The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 33 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Sachs: AI Model Quality Varies Between First-Party APIs and Cloud Providers
“Companies that say they're selling the same model through different vendors, whether it be through first party or Bedrock, Azure, et cetera, we do see different qualities sometimes, and that's not necessarily what's advertised.”
Notion Will Beat Frontier Labs in Collaboration Like Datadog Beat AWS
“Datadog could not exist without cloud storage, right? That it's kind of fundamental that that works. And AWS has like a cloud watch product, but Datadog is an expert on understanding how people want observability on the products they launch. And we're experts …”
Notion Engineers Faced an Identity Crisis Shifting From Coding to AI Delegation
“Every software engineer in Notion this summer went through like this sheer one of our engineering leads at the company called it like, every software engineer is going through the identity crisis that every manager goes through, where all of a sudden they real…”
Over-Simplifying AI Agent Interfaces Abstracts Interpretability and Nerfs Agent Capabilities
“I'd actually say we don't try and make it as easy as possible to use because the more we do that, the more we abstract away that interpretability that Simon's talking about that basically nerfs the model or nerfs the agent from being super capable.”
Frontier AI Product Development Requires Knowing When to Stop Fighting Model Limits
“Is really what will set notion apart for every new capability is we have like two skills that are crucial when it comes to frontier capabilities. One is not letting yourself swim upstream. So like quickly realizing if you're just pressing against model capabil…”
Frontier AI Labs Altered Final Model Snapshots Based on Notion's Enterprise Feedback
“Like they have shipped models that were not the snapshots that we wanted, and they have changed the snapshots that they shipped based on the feedback that we give, because our feedback tends to be more enterprise work focused and not coding agent focused.”
Notion Partnered With Anthropic and OpenAI to Build 30% Pass Rate Evals
“And then what we have, what we call Frontier Headroom evals, where we actively want to be at 30% pass rate. And that's actually been a effort that we took in partnership with Anthropic and OpenAI in the past maybe two or three months, because we actually hit a…”
Model Behavior Engineers Do Not Need Traditional Engineering Backgrounds
“And so, we have, like, very firm conviction And we have had for a number of years now that that is its own career path, and we have always welcomed the misfits, so to speak. So we really firmly believe that you don't need an engineering background to be the be…”
Users Do Not Care About Inference Speed for Asynchronous Custom Agents
“People don't care about speed and custom agents. And so the incentive of like haiku being faster, people don't care when it's asynchronous.”
AI Labs Cluster at Extremes and Ignore Intermediate Intelligence-Latency Tiers
“Everyone's clustered in capability, where everyone's clustered, I mean, Haiku's not that much cheaper. No one's really in the middle. Like, people really tend to cluster around two. Like, this is really capable, and it's really fast, but it's really expensive,…”
Notion Fine-Tunes Its AI Capabilities Entirely Without Relying on User Data
“There are certain aspects of Notion where we do fine tune and do reinforce and fine tuning on our own capabilities. But that's not necessarily trained on user data. You don't need that, that much data in the first place.”
Fine-Tuning Models on Internal Tools Unnecessarily Slows Down Rapid Product Development
“It would actually really slow us down to have a model that was fine tuned on our tools because we'd have to retrain it and cut a new model every time we did that.”
The Majority of Notion Enterprise Search Traffic Now Comes From AI Agents
“We're at a point right now in our business and enterprise or AI enabled plans where the search load and the search traffic, a majority of it's coming from agents, not humans.”
Optimizing Vector Embedding Models Is No Longer the Right RAG Retrieval Focus
“We don't spend a lot of time trying to optimize what vector embedding we use anymore. That was a period of time, but that's just not the right level of optimization, right?”
AI Teams Lose Velocity When Building Cool Tools Without User Journeys
“I actually think the failure of our team is when we focus too much on what are tools that are cool tools. I actually think that's when we make have the least velocity because you still need some sort of focus on a user journey.”
Frictionless AI Prototyping Requires Strong Product Conviction to Avoid Feature Bloat
“I think it's put a big pressure point on us to have really strong product conviction, because if anything can be demoed, You really need a strong filter of making sure that if, you know, you're doing X amount of work, you're making the, you're focusing on one …”
Notion Feature Teams Own Their AI Evaluations Running Nightly in CI
“We maintain the eval framework. Every team owns their own evals and a lot of them we've integrated to opt into CI or we run them nightly and we have a team, a custom agent that triggers to a team to look at the major failures.”
Notion Observes Increased AI API Latency During Standard Business Hours
“We've definitely noticed, particularly for some providers that things are slower during working hours.”
Coding Agents Should Write AI Model Evaluations Like Standard Unit Tests
“It's like the same way that you would have a coding agent write the unit test. You should have a coding agent write the eval.”
Executing Deterministic Code via CLI Avoids Recurring LLM Token Fees From MCP
“And so if we can have our agents properly execute code that calls on CLI deterministically, it's a one time cost, right? Versus constantly having a language model integrate with an MCP over and over and over and paying those like repeated token fees.”
Notion Bypasses Third-Party MCP Search to Maintain Internal Quality Control
“There are a lot of things where we choose not to use MCP because we want to add more high touch to quality. I think search and agentic find is like the largest instance of that, where we have slack and linear and JIRA search and notion that is not using necess…”
Goal-Driven Tool Definitions Allowed Notion to Distribute Agent Tool Ownership
“When we went away from few shots to describing the goal of the tool and like goal driven, basically moving from a DAG to like a true system with feedback, that's when we could distribute tool ownership to the teams much better.”
Claude Sonnet Crashed on Duplicate Tool Names While OpenAI Handled the Error
“Sonic couldn't handle two tools with the same name in OpenAI, GPT, 5.2. It was like, ah, I can figure this out. So that was an interesting one that we learned by accident through a SEV.”
The Majority of Notion AI Tasks Do Not Require Opus-Level Intelligence
“It's not Opus, that's for sure, because a majority of the tasks people are doing aren't Opus level intelligence.”
Agent Search Prioritizes Top-K Retrieval Depth Over Precise Positional Ranking
“Positional ranking matters less, but top care retrieval mode matters more.”
Sachs: Custom Agents Was Notion's Most Successful Launch by Conversions
“It was our most successful launch in terms of free trials and converting people and things like that.”
Notion Fine-Tuned Custom Function-Calling Models Before Standard Tool Calling Existed
“Before function calling came out, we were trying to fine tune with the frontier labs and with fireworks, like a function calling model on notion functions.”
Notion's AI Team Reviews p99 Token-Exhaustive Agent Transcripts Every Friday
“We'll all sit down every Friday and look at the P. 99 of like the most token exhaustive custom agent transcript and just look at why it didn't do well and cut a bunch of tasks.”
Notion Dedicates 50 Engineers to Core AI and 40 to Packaging
“I manage, ah, the team that's what we'll call like core AI capabilities and infrastructure. That's about 50 people. But then we have, I partner teams that do packaging, so how it shows up in the corner chat versus custom agents versus meeting notes, that's ano…”
Notion's Internal Bug-Triaging Custom Agent Completely Changed Company Operations
“One of our biggest use cases is bug triaging. So if someone posts something in Slack and you just have a custom agent that lives there that has its own routing constitution of what team this belongs to. Creates a task in your task database and then posts in th…”
Notion Commits to Supporting MCP Integrations as Long as Users Demand Them
“So we will always support our MCP insofar as other people are using MCPs, right?”