Nov 11, 2024 · 58m · latent-space
Agents @ Work: Dust.tt — with Stanislas Polu
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Former OpenAI reasoning researcher and Dust co-founder Stanislas Polu shares inside perspectives on OpenAI's scaling culture and details the engineering and product architectures needed to deploy reliable enterprise AI agents.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 25.1% of the talking time here. How this is scored →
The hosts as informed peer 5.5 Guest teaching 5.0 Guest disagreement 2.2 The hosts pushing back 2.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 21:59 Dismissing open source cloning risk
Stan directly dismisses Swyx's claim that open-sourcing invites commoditization and cloning, calling it a complete fantasy compared to execution velocity.
Hardest push from the hosts ▶ 57:17 Swyx pushes product-over-platform counterthesisSwyx directly challenges Dust's broad horizontal positioning by arguing founders should build specialized vertical products before ever trying to build platforms.
Biggest teaching moment ▶ 45:08 Explaining structural chunking vs generic ETLStan breaks down why standard connectors like Airbyte fail for LLMs due to structural loss in rich documents like Notion databases, demonstrating the need for bespoke infrastructure.
The host holds their own ▶ 42:39 Swyx breaks down Berkeley Function Calling leaderboardSwyx demonstrates deep domain expertise by citing the newly released Berkeley benchmark data, validating model tiers and specific performance metrics.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Stanislas Polu's Background and Journey into Artificial Intelligence | 4 | 3 | 1 | 1 | Swyx sets the biographical context of Stripe and early OpenAI culture while Stan recounts his journey from Stanford and early robotics to theorem proving. The conversation is collaborative and exploratory without direct friction. | |
| Early OpenAI Days and Formal Mathematics Reasoning Research | 5 | 6 | 1 | 2 | Stan educates the hosts on how formal mathematics systems verify proofs instantaneous via type systems while search tactics require computation. Swyx engages with technical familiarity around complexity limits and competitive math benchmarks. | |
| Compute Governance, Scaling Thesis, and Ilya Sutskever's Vision | 5 | 6 | 1 | 1 | Stan explains the internal dynamics of compute allocation at OpenAI as a management tool and sheds light on Ilya Sutskever's scaling philosophy. The hosts probe into the historical timeline of the scaling laws and compute prioritization. | |
| Leadership at OpenAI and the Historical Anthropic Split | 4 | 4 | 1 | 2 | Swyx prompts Stan on the executive leadership dynamics and the split that formed Anthropic. Stan provides first-hand perspective while noting he was not in the immediate executive weeds, maintaining a measured tone. | |
| Founding Dust, Open-Source Strategy, and the XP1 Browser Extension | 6 | 5 | 4 | 5 | Swyx challenges Stan on the downsides of open-sourcing Dust, asserting that open source exposes them to cloning without business benefit. Stan pushes back firmly, calling the cloning fear a fantasy and defending transparent developer velocity. | |
| Dust's Enterprise Agent Thesis: API Integration vs Browser Automation | 6 | 6 | 5 | 3 | Stan firmly rejects the browser RPA thesis advocated by Adept and David Luan, arguing that internal enterprise automation belongs on native APIs. Swyx explicitly points out the direct contradiction with David Luan's philosophy. | |
| Function Calling Mechanics, Hierarchical Meta-Agents, and Product Usability | 5 | 5 | 2 | 2 | Stan explains the pragmatic value of deterministic single-step agents over brittle autonomous systems, elaborating on hierarchical meta-agents. The hosts contribute concepts around dependency graphs and agent orchestration protocols. | |
| Real-World Agent Evaluation, Feedback Loops, and Model Benchmarks | 7 | 5 | 2 | 3 | Swyx shows deep knowledge of the Berkeley function calling leaderboard, citing rankings of GPT-4 Turbo, 4o, and Salesforce xLAM. Stan explains why raw benchmark evaluations matter less than daily active enterprise adoption in practice. | |
| Engineering Dust's Infrastructure: Custom Connectors, Temporal, and Rust | 7 | 6 | 2 | 3 | Swyx leverages his insider experience with Airbyte and Temporal to drill into data ingestion and workflow engines. Stan explains why generic ETL connectors fail at semantic context chunking, justifying custom Rust connectors. | |
| Billion-Dollar Lean Companies and Horizontal AI Agent Strategy | 6 | 4 | 3 | 5 | Swyx challenges Stan's horizontal company-wide agent thesis by pitching his own 'product over platform' framework. Stan defends the horizontal model, arguing that enterprise workflows have long-tail variations requiring general tooling. |
Statements from this episode (25)
Polu: Transformers failed at code fuzzing because they are too slow
“We're trying to apply transformers to code fuzzing. So code fuzzing, you have kind of an, sorry, an algorithm that goes really fast and tries to mutate the inputs of a library to find bugs. And we try to apply a transformer to that and do reinforcement learnin…”
Polu: Combining LLMs with formal math pairs creativity with proof verification
“Transformers are very creative, but yet they do mistakes. And formal math systems are the ability to verify a proof. And the tactics they can use to solve problems are very mechanical. So you miss the creativity. And so the idea was to try to explore both toge…”
Polu: DeepMind IMO breakthrough relied on scaling RL and autoformalization
“I think the DeepMind team just did a good job of scaling. I think there's nothing too magical in their approach, even if it hasn't been published as a Dan Silver talk from seven days ago, where it goes a little bit into more details. It feels like there's noth…”
Polu: OpenAI's GPT-3 Was Internally Codenamed Project Nest
“Most of the compute was going to a product called Nest, which was basically GPT-free.”
Polu: OpenAI Managed Research Priorities Directly Through Compute Allocation
“In that space, there's a managing tool that is great, which is computer location. Basically, by managing the computer location, you can message the team of where you think the priority should go. And so it was really a question of you were free as a researcher…”
Polu: Ilya Sutskever Spent Surprising Amount of Time Communicating OpenAI Vision
“I think he was really focused on building the vision and communicating the vision within the company, which was extremely It's extremely useful. I was personally surprised that he spent so much time, you know, working on communicating that vision and getting t…”
Polu: OpenAI Believed in Transformer Scaling Pre-Kaplan Paper
“Before that, there really was a strong belief in, in scale. I think it was just the belief that the transformer was a generic enough architecture that you could learn anything, and that this was just a question of scaling.”
Polu: Sam Altman mastered technical ML details within two years at OpenAI
“One thing about Sam Altman, he really impressed me, because when I joined, he had joined not that long ago. And it felt like he was kind of a very high level CEO. And I was mind blown by how deep he was able to go into the subjects within a year or something, …”
Polu: Anthropic split was driven by disagreement over OpenAI's API commercialization
“What I understood of it is that there was a disagreement of the commercialization of that technology. I think the focal point of the disagreement was the fact that we started working on the API and wanted to make those models available through an API. Is that …”
Polu: GPT-4 was ready internally at OpenAI months before September 2022
“I had seen GPT-IV internally at the time. It was September, 20, 22. So it was pre-chat GPT, but GPT-IV was ready since, I mean, I'd been ready for a few months internally.”
Polu: LLM workflow development requires a dozen examples to prevent overfitting
“I had the strong belief from my research time that you cannot create an LLM-based workflow on just one example. Basically, if you just have one example, you overfit. So as you develop your interaction, your orchestration around the LM, you need a dozen example…”
Polu: LLM productization is currently only at the 'Pong' stage
“I think we're at the pong level of LLM productization, and we haven't invented the SIEV-III, we haven't invented Counter-Strike, we haven't invented Cyberpunk”
Polu: Fully autonomous AI models 'get lost' and are not ready
“The AutoGPD approach, obviously, is extremely exciting, but we know that the agentic capability of models are not quite there yet. It just gets lost.”
Polu: The bulk of useful enterprise agent work can use APIs
“The bulk of the useful stuff that you can do within the company can be done through API. The data can be retrieved by API, the actions can be taken through API.”
Polu: Models make mistakes when given high-level instructions and many tools
“If you provide a very high level Kind of an auto GPT-esque level in the instructions and provide 16 different tools to your model. Yes, we're seeing the models in that state making mistakes.”
Polu: Hierarchies of simple agents will unlock Auto-GPT level value
“Once you have those working really well, you can create meta agents that use the agents as actions, and all of a sudden you can kind of have a hierarchy of responsibility that will probably get you almost to the point of the auto GPT value.”
Polu: GPT-4 Turbo performs better than GPT-4o on function calling
“I personally don't have proof, but I know many people, and I'm probably part of them, to think that GPT-IV Turbo is still better than GPT-IV on function calling.”
Polu: Claude Sonnet executes an unpublicized chain-of-thought step during function calling
“They kind of innovated in an interesting way, which was never quite publicized, but it's that they have that kind of chain of thoughts step whenever you use a Clouds model or Sonnet model with function calling. That chain of service step doesn't exist when you…”
Polu: Dust averages 60% to 70% weekly active penetration in enterprise accounts
“The highest penetration we have is 88% daily active users within the entire employee of the company. The kind of average penetration and activation we have in our current enterprise customers is something like more like 60 to 70% weekly active.”
Polu: Airbyte's Notion connector output is not useful for AI models
“And the reality is that if you look at Notion, Airby does the job of taking Notion and putting it in a structured way, but that's a way that is not really usable to actually make it available to models in a useful way. Because you get all the blocks, details, …”
Polu: Post-hyper-growth tech companies may increasingly eliminate traditional SaaS
“So it's interesting that we might see kind of a bad time for SaaS in post-hyper-growth tech companies. So it's still a big market, but it's not that big, because if you're not a tech company, You don't have the capabilities to reduce desk cost. If you're a hig…”
Swyx: ChatGPT rewrote its frontend from Next.js to Remix
“Recently ChatGPT just rewrote from Next.js to Remix.”
Polu: The next generation will see billion-dollar companies with 20 engineers
“All generations of company might be the first billion dollar companies with engineering teams of 20 people. That would be so exciting as well. That would be so great. You know, you don't have the management hurdle. You're just 20 focused people with a lot of a…”
Swyx: The first single-person unicorn will be a content creator
“Semi-hot take is, I actually know what vertical they'll be in. They'll be content creators and podcasters.”
Polu: Vertical AI agents have easier GTM but limited enterprise upside
“Vertical solutions have a good market that is much easier because they're like, oh, I'm going to solve the lawyer stuff. But the potential within the company after that is limited. So there's really a nice tension there. We, we're true believers of the horizon…”