Apr 15, 2026 · 1h 25m · latent-space
Notion’s Sarah Sachs & Simon Last on Custom Agents, Evals, and the Future of Work
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Notion co-founder Simon Last and Head of AI Sarah Sachs join the Latent Space podcast to discuss the engineering architecture, evaluation rigor, and product philosophy behind Notion Custom Agents. They explore how horizontal primitives, autonomous software factories, and robust tool-calling infrastructure are shaping Notion's role as the enterprise system of record.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 16.1% of the talking time here. How this is scored →
The hosts as informed peer 5.3 Guest teaching 5.9 Guest disagreement 1.5 The hosts pushing back 2.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 57:28 Rejecting the accessible UX paradigm
Sarah firmly rejects the premise that AI tools should be simplified for the lowest common denominator, arguing that dumbing down interfaces destroys agent capability.
Hardest push from the hosts ▶ 24:03 Interrogating silent model degradationSwyx presses the guests on whether major frontier labs secretly quantize or throttle model quality during high-traffic enterprise working hours.
Biggest teaching moment ▶ 49:41 Designing systems for model training distributionsSimon breaks down why Notion abandoned its internal XML formats for standard Markdown and SQLite after learning models perform best on representations mirroring pre-training data.
The host holds their own ▶ 1:16:00 Challenging retrieval optimization metricsSwyx challenges Sarah on top-K retrieval nuances, prompting a detailed technical breakdown of loss functions and agentic query distribution patterns.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Latent Space Message: Why We Ask for Subscriptions | 4 | 5 | 1 | 2 | Swyx opens with an ad-free subscriber plea and then presses the guests on why building background agents failed in earlier iterations. Simon and Sarah explain how early context limits and the lack of native tool calling forced multiple redesigns before Sonnet unlocked reliable background execution. | |
| Balancing AGI Ambitions and the Software Factory Concept | 5 | 6 | 2 | 2 | Alessio and Swyx ask how Notion plans product roadmaps when frontier model capabilities shift rapidly. Sarah reframes the problem around sensing which direction the technological current is moving rather than fighting upstream against model limitations. | |
| Horizontal Platform Primitives vs. Vertical SaaS Expertise | 6 | 5 | 2 | 3 | Alessio contrasts Datadog's vertical persona with Notion's horizontal user base, asking how product expertise is encoded at scale. Sarah and Simon explain that horizontal primitives must stay anchored in concrete user journeys rather than building novel tools for their own sake. | |
| Fostering an Egoless Engineering Culture of Code Deletion | 4 | 6 | 1 | 1 | Swyx asks Sarah about her management philosophy on the AI team. Sarah details an egoless engineering culture where engineers are encouraged to throw away old prototypes and rewrite harnesses without tying code ownership to career advancement. | |
| The 'Simon Vortex' and Hackathons as Capability Uplifters | 5 | 4 | 2 | 2 | Swyx asks whether regular hackathons drive Notion's product momentum. Sarah and Simon explain that company-wide hackathons serve to uplift baseline technical literacy, while core skunkworks innovation happens continuously in rapid prototyping cycles. | |
| Organizational Structure and the Design Playground Prototype Repo | 4 | 6 | 2 | 2 | Alessio asks about prototype evaluation bars across engineering and design. Simon and Sarah describe their isolated design playground repository where designers ship functional code rather than static mockups. | |
| Scaling Agent Dev Velocity and Decentralized Team Evals | 5 | 6 | 2 | 2 | Sarah pushes back on Simon's simplified take on prototyping by highlighting the heavy eval and infrastructure burden required before production. She explains that every feature team owns its domain evals while the platform team builds shared harness tooling. | |
| Detecting Model Provider Regressions and Silent Degradation | 6 | 5 | 1 | 3 | Swyx asks whether Notion catches silent quality degradation or hidden quantization during peak hours. Sarah shares that enterprise-focused evaluation suites frequently detect regressions that model providers fail to catch with standard coding benchmarks. | |
| The Rise of Model Behavior Engineers (MBEs) | 5 | 7 | 1 | 2 | Swyx asks about the Model Behavior Engineer role. Sarah outlines how the position evolved from manual spreadsheet audits by linguistics graduates into engineers who direct coding agents and design LLM judges. | |
| The Evolution of Software Engineering and Autonomous Workflows | 6 | 5 | 2 | 3 | Swyx asks if traditional software engineering roles will disappear. Simon and Sarah argue that human engineers are transitioning from syntax implementation to system architecture, verification loops, and delegating to agent fleets. | |
| Demoing Custom Agents: Automating Tenant Lead Ingestion | 6 | 5 | 1 | 1 | Alessio demos a live custom agent that automates coworking applicant ingestion, enrichment, and database tracking. Sarah explains how automating tedious internal processes creates high operational leverage across teams. | |
| Agent Coordination, Manager Agents, and Native Memory | 5 | 6 | 1 | 2 | Alessio and Swyx ask how agents coordinate without creating infinite recursive loops. Simon explains that Notion relies on database primitives and dedicated manager agents rather than complex bespoke messaging architectures. | |
| Native Feature Optimizations vs. Third-Party Integrations | 6 | 6 | 1 | 2 | Swyx asks for Simon's perspective on MCP versus CLI agent interfaces. Simon outlines CLI strengths in progressive disclosure and self-healing while acknowledging MCP's clean sandboxing and permission boundaries. | |
| Deterministic Code Execution and Cost-Effective Pricing Models | 5 | 6 | 2 | 2 | Sarah emphasizes that using LLMs to orchestrate deterministic workflows wastes tokens and drives up user costs. She argues that executing deterministic code sandboxes delivers superior reliability and margin efficiency. | |
| Evolution of Notion's Internal Representation: XML to Markdown and SQLite | 7 | 7 | 1 | 2 | Swyx asks Simon to walk through the technical iterations of Notion's agent architecture. Simon details the journey from brittle custom XML blocks to markdown and SQLite databases that natively align with model pre-training distributions. | |
| Decentralizing Tool Ownership and Progressive Disclosure | 5 | 7 | 2 | 2 | Sarah details how moving away from massive few-shot system prompts to discrete tool definitions allowed Notion to decentralize tool development across independent product squads. | |
| Demystifying System Prompts and Building for Power Users | 5 | 6 | 2 | 2 | Swyx asks whether Notion protects system prompts as intellectual property. Sarah and Simon push back, explaining that making prompts and tools fully transparent enables power users to build sophisticated workflows. | |
| The 'Flippy' Interface: Unifying Settings and Chat Workflows | 5 | 6 | 1 | 1 | Alessio praises the unified editing and execution canvas. Simon and Sarah share the behind-the-scenes decision to delay launch in order to build the 'Flippy' interface where conversational agents modify their own configurations. | |
| Designing Credit-Based Pricing and Usage-Based Economics | 6 | 6 | 1 | 2 | Alessio asks how Notion structures value-based pricing when individual agent tasks vary wildly in business impact. Sarah explains why usage-based credit abstractions protect unit economics without imposing artificial complexity. | |
| The 'Auto' Model Router and Filling the Intelligence-Cost Gap | 5 | 7 | 2 | 2 | Sarah breaks down the 'Auto' model router, explaining how it steers tasks to the most cost-efficient models while highlighting the mid-tier capability vacuum left by frontier model pricing strategies. | |
| Contextual Enterprise Fine-Tuning and Running Overnight Agent Loops | 6 | 5 | 2 | 2 | Swyx asks if Notion plans to train proprietary foundation models. Simon and Sarah clarify that they prioritize enterprise-specific contextual fine-tuning and overnight autonomous coding agent loops over building generic foundation models. | |
| Why Tool Velocity and Outer Loop Robustness Beat Model Training | 5 | 7 | 2 | 1 | Simon argues that engineering attention is better invested in tool interfaces and harness reliability than fine-tuning models on rapidly changing internal APIs. | |
| Adapting Search and Retrieval for Agent-Driven Workloads | 7 | 6 | 1 | 2 | Sarah details how agent-generated search queries differ fundamentally from human queries, requiring a complete redesign of retrieval architectures, top-K ranking, and query expansion models. | |
| Meeting Notes as an Agentic Flywheel and Capture Primitive | 5 | 6 | 2 | 2 | Swyx asks about Notion's meeting notes and wearable devices. Simon and Sarah conclude by positioning meeting notes as a foundational context-capture primitive that feeds Notion's broader collaboration ecosystem. |
Statements from this episode (51)
Sachs: Custom Agents Was Notion's Most Successful Launch by Conversions
“It was our most successful launch in terms of free trials and converting people and things like that.”
Notion Rebuilt Its Agent Framework Four or Five Times Before Launch
“It's probably the fourth or fifth time that we rebuilt that.”
Notion Fine-Tuned Custom Function-Calling Models Before Standard Tool Calling Existed
“Before function calling came out, we were trying to fine tune with the frontier labs and with fireworks, like a function calling model on notion functions.”
Claude Sonnet Was the Breakthrough That Unlocked Notion's Custom Agents
“The big unlock was Probably like Sonic 3.6 or seven early last year. And that's when we started working on our agent, which we shipped last year.”
Simon Last Argues That Coding Agents Are the Kernel of AGI
“I think one thing that's becoming more clear is I think like coding agents are the kernel VGI, sort of everything is a coding agent.”
Last: Coding agents allow software to bootstrap and debug itself
“The exciting thing about that is sort of your agent can sort of bootstrap its own software and capabilities and actually debug and maintain them.”
Frontier AI Product Development Requires Knowing When to Stop Fighting Model Limits
“Is really what will set notion apart for every new capability is we have like two skills that are crucial when it comes to frontier capabilities. One is not letting yourself swim upstream. So like quickly realizing if you're just pressing against model capabil…”
Notion Will Beat Frontier Labs in Collaboration Like Datadog Beat AWS
“Datadog could not exist without cloud storage, right? That it's kind of fundamental that that works. And AWS has like a cloud watch product, but Datadog is an expert on understanding how people want observability on the products they launch. And we're experts …”
AI Teams Lose Velocity When Building Cool Tools Without User Journeys
“I actually think the failure of our team is when we focus too much on what are tools that are cool tools. I actually think that's when we make have the least velocity because you still need some sort of focus on a user journey.”
Notion's AI Team Reviews p99 Token-Exhaustive Agent Transcripts Every Friday
“We'll all sit down every Friday and look at the P. 99 of like the most token exhaustive custom agent transcript and just look at why it didn't do well and cut a bunch of tasks.”
Notion Dedicates 50 Engineers to Core AI and 40 to Packaging
“I manage, ah, the team that's what we'll call like core AI capabilities and infrastructure. That's about 50 people. But then we have, I partner teams that do packaging, so how it shows up in the corner chat versus custom agents versus meeting notes, that's ano…”
Notion's Design Team Builds Live Code Prototypes Instead of Static Mocks
“Our design team made a whole separate GitHub repo called the design playground. And it's basically just to create a bunch of like helper components and for quickly throwing together UIs and it's become like actually quite sophisticated. Like it has like an age…”
Frictionless AI Prototyping Requires Strong Product Conviction to Avoid Feature Bloat
“I think it's put a big pressure point on us to have really strong product conviction, because if anything can be demoed, You really need a strong filter of making sure that if, you know, you're doing X amount of work, you're making the, you're focusing on one …”
Notion Feature Teams Own Their AI Evaluations Running Nightly in CI
“We maintain the eval framework. Every team owns their own evals and a lot of them we've integrated to opt into CI or we run them nightly and we have a team, a custom agent that triggers to a team to look at the major failures.”
Notion Observes Increased AI API Latency During Standard Business Hours
“We've definitely noticed, particularly for some providers that things are slower during working hours.”
Sachs: AI Model Quality Varies Between First-Party APIs and Cloud Providers
“Companies that say they're selling the same model through different vendors, whether it be through first party or Bedrock, Azure, et cetera, we do see different qualities sometimes, and that's not necessarily what's advertised.”
Frontier AI Labs Altered Final Model Snapshots Based on Notion's Enterprise Feedback
“Like they have shipped models that were not the snapshots that we wanted, and they have changed the snapshots that they shipped based on the feedback that we give, because our feedback tends to be more enterprise work focused and not coding agent focused.”
Notion Partnered With Anthropic and OpenAI to Build 30% Pass Rate Evals
“And then what we have, what we call Frontier Headroom evals, where we actively want to be at 30% pass rate. And that's actually been a effort that we took in partnership with Anthropic and OpenAI in the past maybe two or three months, because we actually hit a…”
Model Behavior Engineers Do Not Need Traditional Engineering Backgrounds
“And so, we have, like, very firm conviction And we have had for a number of years now that that is its own career path, and we have always welcomed the misfits, so to speak. So we really firmly believe that you don't need an engineering background to be the be…”
AI Evaluation Harnesses Should Remain Compatible With General CLI Tools
“Yeah, I think it'll be a mistake to like fix it on any particular coding agent. At the end of the day, it's just like CLI tools.”
Coding Agents Should Write AI Model Evaluations Like Standard Unit Tests
“It's like the same way that you would have a coding agent write the unit test. You should have a coding agent write the eval.”
Notion Engineers Faced an Identity Crisis Shifting From Coding to AI Delegation
“Every software engineer in Notion this summer went through like this sheer one of our engineering leads at the company called it like, every software engineer is going through the identity crisis that every manager goes through, where all of a sudden they real…”
Supervising AI Agents Is a Deeply Technical Systems Problem, Unlike Managing Humans
“There's a critical difference
To being a manager, which is that like, it is actually very deeply technical.
The problem of, you know, humans are very like fuzzy and you can't like treat a team of humans like a rigorous system where like, you know, PRs like fle…”
Notion's Internal Bug-Triaging Custom Agent Completely Changed Company Operations
“One of our biggest use cases is bug triaging. So if someone posts something in Slack and you just have a custom agent that lives there that has its own routing constitution of what team this belongs to. Creates a task in your task database and then posts in th…”
Notion Will Soon Release Direct Agent-to-Agent Invocation Settings
“So I think it's actually not released yet. Releasing it like next week is in the settings for an agent, you can give it access to invoke any other agent. So you can have them just talk directly.”
Notion Custom Agents Use Standard Pages and Databases as Their Memory
“Another example of this is we have no built-in memory concept. Memory is, is just pages and databases. And so if you want to give it memory, just give it a page and give it access to that page.”
CLI Environments Let AI Agents Autonomously Debug and Fix Broken Tools
“And then I think the most important thing that's super cool is that there it's also inherently a bootstrapped. So if there's an issue of the agent can debug and fix itself within the same environment that it uses the tool, right?”
MCP Provides a Safer Permission Model Than CLIs by Preventing Token Exfiltration
“MCP inherently has a really strong permission model. Like, all you can do is call the tools. A CLI is a little bit murkier. It's like, can I access the API token? Are you, like, properly sort of, like, re-encrypting the token so it can't, like, exfiltrate it? …”
Notion Commits to Supporting MCP Integrations as Long as Users Demand Them
“So we will always support our MCP insofar as other people are using MCPs, right?”
Executing Deterministic Code via CLI Avoids Recurring LLM Token Fees From MCP
“And so if we can have our agents properly execute code that calls on CLI deterministically, it's a one time cost, right? Versus constantly having a language model integrate with an MCP over and over and over and paying those like repeated token fees.”
Notion Bypasses Third-Party MCP Search to Maintain Internal Quality Control
“There are a lot of things where we choose not to use MCP because we want to add more high touch to quality. I think search and agentic find is like the largest instance of that, where we have slack and linear and JIRA search and notion that is not using necess…”
MCP Lacks a Trigger Protocol, Forcing Notion to Build Custom Trigger Infrastructure
“MCP is a really great way to gain access to tools. Works really well. But you just looked at the trigger UI, for example, there's no trigger protocol. And so those are things we had to build ourselves.”
Agent Environments Must Favor Model Preferences Over Internal System Architecture
“I mean, that was, I would say that was a big learning is just, you know, really be savvy and really careful thinking about what the model wants in terms of, you know, its environment and cater around that and really try so hard not to expose it to any complexi…”
Goal-Driven Tool Definitions Allowed Notion to Distribute Agent Tool Ownership
“When we went away from few shots to describing the goal of the tool and like goal driven, basically moving from a DAG to like a true system with feedback, that's when we could distribute tool ownership to the teams much better.”
Claude Sonnet Crashed on Duplicate Tool Names While OpenAI Handled the Error
“Sonic couldn't handle two tools with the same name in OpenAI, GPT, 5.2. It was like, ah, I can figure this out. So that was an interesting one that we learned by accident through a SEV.”
Notion's Custom Agent Operates With Over 100 Distinct Internal Feature Tools
“So there's now like over a hundred tools just for all, all the crazy notion stuff.”
Over-Simplifying AI Agent Interfaces Abstracts Interpretability and Nerfs Agent Capabilities
“I'd actually say we don't try and make it as easy as possible to use because the more we do that, the more we abstract away that interpretability that Simon's talking about that basically nerfs the model or nerfs the agent from being super capable.”
Self-Healing Agent Capabilities Are Next on Notion's Product Roadmap
“Obviously we should build product of self-healing. That's next on our roadmap.”
Notion Custom Agents Are Deployed With Zero Permissions by Default
“The thing about custom agents that is that by default, it has no permission to do anything. And then you have to explicitly granted all the permissions.”
Notion's Database Autofill Will Soon Become Agentic With Usage-Based Pricing
“The database autofill feature will soon be agentic. That will be associated with usage based pricing”
Users Do Not Care About Inference Speed for Asynchronous Custom Agents
“People don't care about speed and custom agents. And so the incentive of like haiku being faster, people don't care when it's asynchronous.”
The Majority of Notion AI Tasks Do Not Require Opus-Level Intelligence
“It's not Opus, that's for sure, because a majority of the tasks people are doing aren't Opus level intelligence.”
AI Labs Cluster at Extremes and Ignore Intermediate Intelligence-Latency Tiers
“Everyone's clustered in capability, where everyone's clustered, I mean, Haiku's not that much cheaper. No one's really in the middle. Like, people really tend to cluster around two. Like, this is really capable, and it's really fast, but it's really expensive,…”
Notion Fine-Tunes Its AI Capabilities Entirely Without Relying on User Data
“There are certain aspects of Notion where we do fine tune and do reinforce and fine tuning on our own capabilities. But that's not necessarily trained on user data. You don't need that, that much data in the first place.”
Simon Last Runs Autonomous Coding Agents Overnight Before Going to Sleep
“So now every night before I go to bed, I'm like, okay, did I start enough agents to get them done? I get everything done.”
Fine-Tuning Models on Internal Tools Unnecessarily Slows Down Rapid Product Development
“It would actually really slow us down to have a model that was fine tuned on our tools because we'd have to retrain it and cut a new model every time we did that.”
Simon Last: 99% of Agent Failures Are Tool Bugs, Not Model Flaws
“And actually, 99% of the time, it's a bug in one of the tools. Right. And so, just fix the bug.”
The Majority of Notion Enterprise Search Traffic Now Comes From AI Agents
“We're at a point right now in our business and enterprise or AI enabled plans where the search load and the search traffic, a majority of it's coming from agents, not humans.”
Agent Search Prioritizes Top-K Retrieval Depth Over Precise Positional Ranking
“Positional ranking matters less, but top care retrieval mode matters more.”
Optimizing Vector Embedding Models Is No Longer the Right RAG Retrieval Focus
“We don't spend a lot of time trying to optimize what vector embedding we use anymore. That was a period of time, but that's just not the right level of optimization, right?”
Notion's Meeting Notes Automatically Mention Individuals Referenced in Generated Summaries
“One thing that the meeting notes team added recently that was But I've been blowing my mind is they, we have, they made it so it actually, when it makes the summary will actually mention the people that were referenced in it. So I now get notifications wheneve…”