Luan: 2012-2018 AI breakthroughs came from bottom-up curiosity without corporate objectives
David Luan · David Luan: Why Nvidia Will Enter the Model Space & Models Will Enter the Chip Space | E1169 · 20VC with Harry Stebbings · Jun 24, 2024 · at 2:40
David Luan, co-founder and CEO of Adept and former leader at OpenAI and Google Brain, explains how AI research was organized during the early deep learning revolution.
0:00 / 0:43exact quote · 43.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“During that twenty-twelve, twenty-eighteen era, the way people made progress what I mean by bottom-up basic research is you hire the most brilliant scientists, they come to work every day with, like, No near-term objective they're being held accountable to. And they just work together, and they think about, you know like, I wonder what it'd be like if we could solve this, like, open technical problem in AI. Like, how do we go how do we create a model that better understands how to generate images? And they just go work on that of their own curiosity and drive and maybe some interest and glory and fame through papers. And they do that for they do that for, like, six months or so, and then out pops out This, like, research paper that gets posted to archive and goes to a journal that that, like, just solves the problem.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from David Luan
Luan: Vertical integration pressure will merge model builders and chip makers
“That's my expectation. To me, like what's interesting about AI from a business side, right, is like it forces the question of like, what Companies or offerings are going to be bundled or integrated and which ones are going to get unbundled. And I actually thin…”
Luan: Leading AI companies are already lobbying for regulatory capture
“I think I think the move to go pull up the ladder behind them is already beginning. And I think that lawmakers don't really understand this technology at all. And so their default instinct is sort of listened to the most credible source. And usually those cred…”
Luan: AI scaling will not suffer diminishing returns on compute
“The second way of improving model performance is just starting to be tapped now, and that's also going to absorb a boatload of compute. So because of that, I actually am not worried about diminishing returns to the compute over time.”
Luan: Base AI model performance requires doubling compute for consistent gains
“So put another way for just scaling up a base language model you need to double the amount of compute for that language model for it to be predictably consistently smarter.”
Luan: AI model progress is pivoting to synthetic data and reinforcement learning
“Now the critical path for model improvement is, is is shifting over to this to this sort of broader sort of simulation slash synthetic data slash like RL loop sort of path. I think it's just a natural consequence of the fact that it's so expensive to just keep…”