McGrew: Nobody believed next-token prediction would work before GPT-1
Bob McGrew · Bob McGrew: AI Agents And The Path To AGI · Y Combinator · Jan 31, 2025 · at 4:02
Former OpenAI Chief Research Officer Bob McGrew discusses Alec Radford's early work on GPT-1 and the skepticism around next-token prediction.
0:00 / 0:24exact quote · 24.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And the core idea behind GPT-ONE is that if you have a transformer and you apply this super simple objective of guessing the next token, guessing the next word, that that would be enough signal that you could actually have something that would be able to generate coherent text. And in retrospect, it sounds sort of obvious, right? Like You know, clearly this was going to work, but no one thought this would work at the time.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Bob McGrew
McGrew: LLMs have entered a pure scaling regime toward AGI
“And now that that has been cracked at this point, I think we actually have a very clear path to just focus on scaling. You know, we were, you know, talking about that you know, the zero to one part, that's not about scaling. I think there's a really strong cas…”
McGrew: FDE-led product discovery beats traditional sales-led discovery
“Sales-led product discovery, you're talking to people from the outside. And again, this is important very early on, but it's not as effective as the FDE-led product discovery, where you're solving these problems from the inside.”
McGrew: Building what customers ask for turns startups into consulting firms
“I think one of the other failures, by the way, that's even prior to that, and more, the easier failure to become a consulting firm, it's where you build the product in the field that the customers are asking for, rather than the one that's actually valuable to…”
McGrew: Startups should avoid FDE strategy unless no alternative works
“My first, second, and third pieces of advice to people who are thinking about trying an FDU strategy is like, don't,
Don't do this at home.
If you can avoid it, like it's probably bad for you.
Probably you're going to end up doing services.
And then only if yo…”
McGrew: In five years, AI agents won't be a single category
“Probably in five years, we'll look back, we'll be like, well, AI agents, there wasn't even a thing at all, right?
We were actually doing all these different things.”
McGrew: FDE strategy succeeds by growing contract sizes rather than cutting customization
“In the product market fit strategy, you want to be doing less work for every customer, you want to be driving down costs, you want to keep the contract size the same. In the FTE strategy, you want to drive the contract size up. So you're doing more and more va…”