May 29, 2025 · 59m · latent-space
The AI Coding Factory
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Factory AI co-founders Matan Grinberg and Eno Reyes join the Latent Space podcast to discuss their autonomous software engineering platform, detailing how autonomous Droids, first-principles UI design, and advanced scaffolding enable enterprise developers to delegate complex legacy codebases and multi-day migrations.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 3.7 Guest teaching 4.4 Guest disagreement 0.9 The hosts pushing back 1.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 50:08 Eno rejects assertion that observability is solved
When Swyx dismissively points out there are already 80 observability tools, Eno firmly reframes why span-based tracing is inadequate for subjective natural language intent.
Hardest push from the hosts ▶ 53:44 Swyx presses past vague hiring statementsSwyx interrupts generic commentary to challenge the founders on what specific skillset is genuinely rate-limiting their enterprise expansion.
Biggest teaching moment ▶ 27:40 Eno explains post-training RL tool conflictsEno educates the hosts on how frontier models trained on CLI tools develop rigid preferences that actively fight superior custom retrieval systems.
The host holds their own ▶ 45:29 Alessio frames inference limits via Together AI benchmarkAlessio brings deep industry context by citing the Together AI 5,000 tokens-per-second benchmark to rigorously question whether inference latency limits parallel agent fan-out.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Backgrounds in Physics, Hugging Face, and Founding Factory | 4 | 2 | 0 | 1 | Alessio and Swyx engage warmly with the founders, with Alessio drawing parallels to his own experience founding a company out of a hackathon. | |
| Reimagining Software Development: From Collaboration to Delegation | 3 | 5 | 1 | 1 | Eno and Matan articulate the core distinction between IDE-bound collaborative tools and an enterprise delegative model handling legacy codebases. | |
| The Origin of Factory and Autonomous Droids | 3 | 4 | 1 | 2 | Swyx probes into the agent terminology debate, prompting Eno to explain their shift from semi-deterministic workflows to goal-oriented planning agents. | |
| Live Demo: Coding Droid and Delegation Workflow | 2 | 5 | 0 | 0 | The guests walk through a live demo of the Code Droid, explaining their UI design for agent transparency and proactive clarifying questions. | |
| Contextual Intelligence, Rule Ingestion, and Model Upgrades | 4 | 5 | 1 | 2 | Alessio compares Factory's prompt intake to Devin and Cursor rules, while the guests explain dynamic codebase indexing and model shock absorption. | |
| Evaluating Models and Navigating Post-Training Biases | 5 | 6 | 2 | 2 | Swyx and Alessio probe evaluation costs and reinforcement fine-tuning, prompting Eno to reveal how model post-training biases conflict with custom tooling. | |
| The Delegative Paradigm and First-Principles UI Architecture | 4 | 5 | 1 | 3 | Swyx pushes back on delegative test-driven development by noting that changing functionality breaks existing tests, prompting a deeper UI discussion. | |
| Context Retrieval Efficiency and Transparent Usage Pricing | 4 | 4 | 1 | 1 | Swyx and Matan discuss the economic impracticality of dumping entire codebases into large context windows versus precision retrieval. | |
| Measuring Enterprise ROI, Code Churn, and Timelines | 4 | 5 | 2 | 3 | Swyx pushes the guests on executive ROI metrics, prompting Matan to dismiss vanity metrics in favor of massive timeline compression. | |
| Enterprise Legacy Migrations and Forward-Deployed Engineering | 4 | 6 | 1 | 2 | Swyx asks whether Factory relies on forward-deployed engineers, leading Eno to detail how an enterprise legacy migration is decomposed and parallelized. | |
| Model Inference Speeds, Cost Limits, and Parallelization | 5 | 4 | 1 | 1 | Alessio references previous discussions with Together AI on extreme token speeds, exploring whether throughput bottlenecks autonomous execution. | |
| Core Industry Limiting Factors: Models and Observability | 5 | 6 | 2 | 4 | When Swyx claims dev observability is saturated with 80 tools, Eno clarifies that existing span-based tools fail to capture semantic user intent. | |
| Enterprise Go-To-Market Growth and Hybrid Technical Hiring | 3 | 4 | 1 | 2 | Swyx presses past generic hiring claims to identify the exact talent bottleneck in enterprise forward-deployed sales engineering. | |
| Brand Identity, Design Philosophy, and Collaborative Culture | 3 | 3 | 0 | 1 | Alessio and Swyx explore the studio design culture and brand identity created in collaboration with Matan's brother Cal. | |
| AI-Native Team Dynamics and Concluding Remarks | 3 | 2 | 0 | 0 | Swyx synthesizes takeaways around shrinking team sizes and AI-native workflows as the interview wraps up. |
Statements from this episode (26)
Grinberg: LLM coding capability directly boosts performance on all downstream tasks
“Capability in code is really core to performance on any LLM and like loosely, the better any LLM is at code, the better it is at any downstream task, even that's like writing poetry.”
Grinberg: Code is uniquely viable for AI agents due to verifiable ground truth
“Code is one of the very few things, especially at the time that you could actually validate. And so you could have that agentic loop where the LM is generating the output and you're actually, you know, verifying in ground truth, the quality of that output.”
$20/month local IDE pricing limits AI inference quality per task
“The cost when you are local first and your typical consumer is on a free plan or a Like, 20 dollar a month paid plan limits the amount of high quality inference you can do, and the scale or volume of inference you can do per, like, outcome.”
Enterprises demand full AI delegation, not 15% to 20% speed gains
“I think that the product experience of delegation is really, really immature right now. And most enterprises though, see that as the holy grail, not like going 15% or 20% faster.”
Reyes: Inner-loop coding will soon be fully delegated to AI agents
“The outer loop of software development and what a software developer does, planning, talking with other human beings, interacting around what needs to get done, is something that's going to continue to be very human-driven, while the inner loop, the actual exe…”
Grinberg: AI Coding Agents Need Full Workplace Context, Not Just IDEs
“Everyone expects these you know, agentic systems to perform at the level of a human, right? Because that's what they're always going to compare them to. But in a lot of cases, they'll have these agents just in the IDE. And that's like the equivalent of onboard…”
Reyes: Developers Should Not Need to Prompt Engineer AI Agents
“A lot of users we believe should not need to prompt engineer agents, right? If your time is being spent hyper optimizing every line and question that you pass to one of these systems, you're going to have a bad time.”
SWE-bench tasks do not reflect actual enterprise software engineering use cases
“That, and also like just in the enterprise, the use cases are pretty different than those represented in something like Sweebench.”
Claude 3.7 Sonnet post-training heavily biases the model toward CLI tools
“So for example, Sonnet 3.7 clearly has it smells like cloud code, right? Same with codex. It very much impacted the way that those models want to write and edit code such that they seem to have a personality that wants to be in a CLI based tool.”
External scaffolding provides higher leverage for coding agents than fine-tuning models
“But our take in general is that freezing the model at a specific quality level and freezing the model at a specific data set just feels like it's lower leverage than continuing to iterate on all these external systems.”
Grinberg: Human-written code percentage will drop within a few years
“The reality is very clearly in the next few years, the amount of lines of code written by a human will go down. Like the percentage of code written by humans will go down.”
Optimal future AI coding interfaces will not evolve from traditional IDEs
“Our take is that it is very unlikely that the optimal UI or the optimal interaction pattern for this new software development where humans spend much less time writing code. I think it's very unlikely that that optimal interaction pattern will be found by iter…”
Grinberg: AI agents will finally make test-driven development work
“The promise of test driven development is going to finally be delivered with this world of AI agents that are working on software development”
Massive context windows will not eliminate the need for RAG retrieval
“Like if you do have billion token context window model, you throw it all in there. It's still gonna be more expensive. The reason why retrieval is so important for us is because even if there is a model that's going to have these larger context windows, and ce…”
Reyes: High-quality enterprise codebases experience only 3% to 4% code churn
“In very high quality code bases, you'll see three percent, four percent code churn when they're at scale, right? This is like millions of lines of code. In poor code bases or poorly maintained code bases or early stage companies that are just changing a lot at…”
Grinberg: Developer sentiment and timeline compression matter more than engineering metrics
“At the end of the day, no one really cares about the metrics. What people really care about is like developer sentiment. When you're kind of playing that game at the end of the day, if you want to do a metric, talk to developers and ask if they feel more produ…”
Factory AI compressed a four-month enterprise codebase migration to 3.5 days
“There's this one very large public company that we work with and Pulling in just a large migration task from taking four months to taking like three and a half days. That is the best ROI that you don't need to measure this or that.”
Reyes: Enterprise migration bottlenecks lie in planning and bureaucracy, not coding
“So a process that typically gets ultimately bottlenecked, not by like skilled humans writing lines of code, but by bureaucracy and technical complexity and understanding, Now gets condensed into basically how fast can a human being delegate the tasks appropria…”
Grinberg: Software cannot change developer habits through product quality alone
“But if you want to change behavior, you can't just assume that the product is going to be so good that everyone's going to immediately get it because with developers, you know, we need to know who we're selling to and developers have very efficient ways of wor…”
Grinberg: Faster AI Inference Will Boost Tool Adoption, Not Capability Limits
“It probably wouldn't change what's possible, but it would really, really change like ease of adoption for people who maybe aren't as in the weeds on AI tools.”
Factory avoids parallel generation because the quality delta fails cost-benefit analysis
“Originally we had a lot of techniques that would generate a lot of stuff in parallel, and we still know how to do that. And we're very excited to bring that. But right now we don't do it because it's cost prohibitive and the quality Delta Is not enough to just…”
Models post-trained for multi-hour agentic trajectories will be released very soon
“I mean, I'm thinking right off the bat, probably the biggest thing is models that have been post-trained on more general agentic trajectories over very long time spans. That feels like something that is There's an effort for that right now. But what I mean is …”
Amplitude and Statsig are closer to semantic AI observability than LLM tools
“I think Amplitude and Statsig and a lot of the feature flag companies actually are closer to this than the existing tools.”
Grinberg: Developers building for themselves risk missing core design tenets
“A lot of times it's kind of easy to fall victim to, oh, I'm building, like I'm the profile who I'm building for, so I know what's best. And that obviously works a lot of the time, but sometimes there are some like core design tenants that you just might not th…”
Reyes: AI agents can accurately imitate human-defined brand voice and design systems
“Being able to have someone set principles that are then consumable by our own agents, right? Design systems and consistency. I think it's pretty surprising the degree to which even like droids can actually imitate a brand voice and style that Cal created for u…”