McGrew: Frontier labs will increasingly rely on model distillation for smaller models
Bob McGrew · Bob McGrew: AI Agents And The Path To AGI · Y Combinator · Jan 31, 2025 · at 16:57
Former OpenAI Chief Research Officer Bob McGrew discusses the rapid performance gains of lightweight AI models with Y Combinator CEO Garry Tan.
0:00 / 0:32exact quote · 32.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think over the last year, the big frontier labs and a lot of other people have figured out the tricks to take big models. And, you know, take a very particular distribution of user input and train a model that is almost as good as the big model, but much, much smaller and much, much faster. And so I think we're going to see this a lot going forward, especially if you look at, you know, the Sonnet versus Haiku, you know, Gemini versus Gemini flash, you know, O-one versus O-one mini, four O-mini every lab has really focused on this.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Bob McGrew
McGrew: LLMs have entered a pure scaling regime toward AGI
“And now that that has been cracked at this point, I think we actually have a very clear path to just focus on scaling. You know, we were, you know, talking about that you know, the zero to one part, that's not about scaling. I think there's a really strong cas…”
McGrew: FDE-led product discovery beats traditional sales-led discovery
“Sales-led product discovery, you're talking to people from the outside. And again, this is important very early on, but it's not as effective as the FDE-led product discovery, where you're solving these problems from the inside.”
McGrew: Building what customers ask for turns startups into consulting firms
“I think one of the other failures, by the way, that's even prior to that, and more, the easier failure to become a consulting firm, it's where you build the product in the field that the customers are asking for, rather than the one that's actually valuable to…”
McGrew: Startups should avoid FDE strategy unless no alternative works
“My first, second, and third pieces of advice to people who are thinking about trying an FDU strategy is like, don't,
Don't do this at home.
If you can avoid it, like it's probably bad for you.
Probably you're going to end up doing services.
And then only if yo…”
McGrew: In five years, AI agents won't be a single category
“Probably in five years, we'll look back, we'll be like, well, AI agents, there wasn't even a thing at all, right?
We were actually doing all these different things.”
McGrew: FDE strategy succeeds by growing contract sizes rather than cutting customization
“In the product market fit strategy, you want to be doing less work for every customer, you want to be driving down costs, you want to keep the contract size the same. In the FTE strategy, you want to drive the contract size up. So you're doing more and more va…”