Depue: RL task specification scales with compute far better than pre-training data
Will Depue · Weekly Recap: The Soham Parekh Drama, Meta Attacks OpenAI, Trump's Mega Bill · Jul 5, 2025 · at 3:05:03
Will Depue discusses the future of frontier AI scaling laws and data curation with TBPN hosts John Coogan and Jordi Hays.
0:00 / 0:24exact quote · 24.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Because like tokens, you kind of hit diminishing returns pretty quick in terms of just like more internet texts or more like human written math solution examples. And it doesn't scale super well, but the thing, the nice thing about the task specification is you can kind of like pour in more compute without necessarily needing more data. It scales much better with compute.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Will Depue
Depue: Yann LeCun had no involvement with Meta's LLaMA models
“I think it's a shame that they didn't necessarily have someone in the llama org who was as visible as him because he was not involved with llama at all. Like fair fundamental AI research is like a whole other group that they mostly write papers and do very aca…”
Depue: Meta may eventually shift to releasing closed flagship AI models
“I w it wouldn't surprise me if they go kind of the Google route where like they are still, they still do open source. They still have the llama brand, but it isn't like their flagship thing where they start doing, whether it's for internal products or it's, th…”
Depue: Meta has not had a homegrown product innovation stick in a while
“It's been a while since they had a product innovation that was homegrown that really stuck.”
Depue: OpenAI holds iPhone-like center of gravity as default AI winner
“I think as long as OpenAI doesn't totally drop the ball on product or research, like, they have the center of gravity for the AI world, like, in the same way that no matter what Android ships, people are not gonna switch from their iPhones, like, It's somethin…”
Depue: Cursor and Claude 3.5 Sonnet made AI coding genuinely useful
“And then it took until cursor for it to really like, I think cursor plus like three, five summit was when it became a thing that was good enough that people were excited about it. And it really ushered in the trend because people were starting to find it more …”