Mar 27, 2024 · 49m · latent-space
Why Google failed to make GPT-3 -- with David Luan of Adept
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Adept founder and former OpenAI VP David Luan discusses the organizational history of frontier AI scaling, why Google missed building GPT-3, and Adept's technical vision for multimodal autonomous agents that execute complex enterprise workflows.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 24.7% of the talking time here. How this is scored →
The hosts as informed peer 4.3 Guest teaching 4.5 Guest disagreement 1.3 The hosts pushing back 1.1
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 19:46 Luan rejects 'services as software' full-replacement framing
Luan directly identifies two core flaws in Fanelli's suggested framing of software replacing human workers, firmly arguing that humans prefer supervision over complete displacement.
Hardest push from the hosts ▶ 38:44 Swyx challenges custom fine-tuning as not being true AGISwyx directly challenges Luan's approach after Luan avoids detailing client-specific model customization, arguing that per-customer fine-tuning falls short of general intelligence.
Biggest teaching moment ▶ 9:40 Luan explains Google's internal resource misallocationLuan educates the hosts on the internal mechanics of Google Brain's credit marketplace, showing why Google failed to consolidate compute behind GPT-style scaling despite having superior talent.
The host holds their own ▶ 24:48 Swyx and Fanelli analyze VC pushback on agent hypeSwyx and Fanelli demonstrate insider venture expertise by dissecting why investors avoid diluted 'agent' startups and the realistic capital constraints facing foundation model founders.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| David Luan's Career and AI Research Eras | 4 | 3 | 1 | 0 | Swyx introduces David Luan with precise background facts covering Dextro, Axon, OpenAI, and Google Brain. Luan expands with historical framing on research eras and the organizational shift toward big, coordinated research swings. | |
| Redefining AGI and Behavioral Cloning vs. Reinforcement Learning | 3 | 6 | 2 | 0 | Fanelli prompts Luan regarding the shift from RL in Dota to large language models. Luan reframes standard definitions of AGI away from human replacement and explains LLMs as universal behavioral cloning of human knowledge. | |
| Why Google Failed to Build GPT-3 | 5 | 5 | 1 | 1 | Swyx brings up specific historical details like Alec Radford's early transformer work and Noam Shazeer's trillion-parameter ambition. Luan explains Google's internal failure to produce GPT-3 due to the decentralized brain credit marketplace. | |
| Hardware Optimization with NVIDIA and the Microsoft Pitch | 4 | 4 | 0 | 0 | Swyx references early NVIDIA DGX hardware deliveries and architectural decisions around scaling. Luan shares technical anecdotes about quad sparsity on A100s, writing sparse modeling sections for papers, and pitching Satya Nadella. | |
| Adept's Enterprise Mission and the High-Reliability Agent Standard | 5 | 5 | 2 | 2 | Fanelli challenges Luan's positioning by contrasting Adept's augmentation approach with 'services as software' that fully replace human roles. Luan argues that augmentation creates a superior data flywheel and prevents catastrophic enterprise errors. | |
| The Shift in Public Agent Narratives and Multimodal Architectures | 5 | 5 | 2 | 2 | Swyx and Fanelli probe why Adept is increasing publicity and point out VC skepticism around diluted agent branding. Luan explains why commodity foundation models must be paired with enterprise-focused multimodal pre-training. | |
| The Agent Interaction Layer and Desktop Control | 4 | 5 | 0 | 1 | Fanelli asks how models should interact with software given UI versus API trade-offs. Luan provides a historical analogy to DOS and Windows 3.1, arguing that OS-level computer control is necessary because APIs do not cover arbitrary end-to-end workflows. | |
| Autonomous Vehicles Analogy and Navigating Reliability vs Generality | 6 | 4 | 3 | 3 | Swyx presses Luan on the tension between generality and enterprise reliability, questioning if fine-tuning per client compromises AGI. Luan deflects specific proprietary practices while defending broader data formulation across the Pareto frontier. | |
| Vertical Integration Defensibility and Grounded Agent Benchmarks | 5 | 5 | 2 | 2 | Fanelli relays an audience question regarding GPT-4 Vision commoditization and asks how Adept differentiates against Imbue. Luan explains that pure-play foundation models will get commoditized, highlighting Adept's vertically integrated, customer-grounded evals. | |
| Final Thoughts on Compute, Data, and Industrialization | 2 | 3 | 0 | 0 | Swyx invites concluding thoughts, prompting Luan to deliver a short summary on the industrialization era of AI driven primarily by compute and data efficiency. |
Statements from this episode (16)
Brockman and Sutskever handed Luan their direct reports to do IC work
“I think second or second or third day of my time at OpenAI Greg and Ilya pulled me in a room and were like, hey you know, the you should take over our directs and we'll go mostly do IC work.”
Luan: Product-model co-evolution will drive AI progress over next years
“The number one driver of AI progress over the next couple of years is going to be the deep co-design and co-evolution of like product and users for feedback and actual technology.”
Luan: AGI means performing any task a human can do on a computer
“I think agents are just absolutely the correct long-term direction, right? You just go to find what AGI is, right? You're like, hey, like, Well, first off, actually, I don't love AGI definitions that involve human replacement because I don't think that's actua…”
Luan: LLMs shortcut evolutionary RL by behaviorally cloning all human knowledge
“Like de novo RL is like a pretty terrible way to get there quickly. Why are we rediscovering all the knowledge about the world? Like years ago, I had a debate with a Berkeley professor as to like what will it actually take to build HCI? And his view is basical…”
Luan: AI will converge into a universal byte model across all modalities
“Multimodal models are becoming more of a thing, we're behavioral cloning the visual world, but really what we're just going to have is this like universal byte model, right? Where like tokens of data that have high signal come in, and then all of those pattern…”
Google would have crushed OpenAI by giving Noam Shazeer half its TPUs
“That muscle did not exist during my time at Google. And I think had they had it, what they would have done would be say, hey, Noam Shazir, you're a brilliant guy. You know how to scale these things up? Like, here's half of all of our TPUs. And then I think the…”
Luan: Frontier AI advantages come from unpublished, non-academic scaling knowledge
“There's a collection of really hard-wanted knowledge that you get only by being at the frontiers of scale. And that hard-wanted knowledge, a lot of it's not published. A lot of it is, like, stuff that, like, it's actually not even easily reducible to what look…”
Altman and Luan pitched Microsoft's top leadership before the OpenAI investment
“The last meeting we did with Microsoft, Before Microsoft invested in OpenAI, Sam Altman, myself, and our CFO flew up to Seattle to do the final pitch meeting. And I'd been a founder before, so I always had like a tremendous amount of anxiety about partner meet…”
Luan: Everyone will have an AI teammate at work within years
“I think in a couple years everyone's going to have access to, like, an AI teammate that they can delegate Arbitrary tasks do at work, and then also be able to, you know, use it to the sounding board and like just be way, way, way more productive”
Luan: Augmentation develops core AI capabilities faster than full automation
“I actually think that being an augmentation company Forces you to go develop your core AI capabilities faster than someone who's saying, ah, ok, my job is to deliver you a lights off solution for X.”
Luan: Future AI value will shift from base models to agents
“In a world where foundation models are looking more and more commodity. And if, and I think a huge amount of gain is going to happen from how do you use foundation models as like the, like well learned behavioral cloner to go solve agents.”
Multimodal models will completely supplant text-only large language models
“I actually think like it's really clear today. Multimodal models are the default foundation model, right? It's just going to supplant LLMs. Like why did you just train a giant multimodal model?”
Real-world computer workflows with end-to-end API coverage are close to zero
“If you go itemize out the number of things you want to do on your computer for which every step has an API those numbers of workflows add up pretty close to zero.”
Luan: Agent failures often stem from unreliable actuators, not models
“Everyone under values the importance of really good sensors and actuators. And actually a lot of what's helped us get a lot of reliability is like a really strong focus on like, actually, why does the model not do this thing? And the non-trivial amount of time…”
Pure-play foundation model companies will be commoditized by Llama and big tech
“I think pure play foundation model companies are just gonna be pinched by how good The next couple of llamas are going to be, and the next, like, what next good open source thing, and then seeing the really big players put ridiculous amounts of compute behind …”
Luan: Training foundation models for robotics via behavioral cloning will work
“One, I'm so excited for someone to train a foundation model of robots. Like, it's just, I think it's just gonna work. Like, I will die on this hill. I mean, like, again, this whole time, like, we've been on this podcast, just continually saying, you know, like…”