Vinyals: AI will remain hybrid as general models overtaking science is far away
Oriol Vinyals · No Priors Ep. 74 | With Google DeepMind VP of Research Oriol Vinyals · Aug 1, 2024 · at 33:37
Oriol Vinyals, VP of Research at Google DeepMind, discusses task specialization in scientific domains like protein folding, fusion, and weather modeling versus general models.
0:00 / 0:47exact quote · 47.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Probably we can, and we might see more and more bootstrapping from Gemini and these models to then a specific solution that, that the model is going to be throw away, except for, you know, maybe cracking protein folding or, you know, figuring out nuclear fusion or modeling the weather, right? Some of the projects that are currently active In, in the DeepMind portfolio, well, you better choose the ones that will matter because the timeline is such that, well, we'll get there sooner if we specialize on those domains, and then perhaps eventually, like, the general model will overtake, but I mean, that's just probably far away or further away, so when it's worth it, do it, and that's, I think, we're gonna still see this hybrid mode”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Oriol Vinyals
Vinyals predicts pre-training compute will drop to ~50% as RL expands
“So to me, that balance feels correct, like some on pre-training, and here we, we're trying to learn every task. So certainly that's going to be, you know, let's say it can be as high as 50%, not as high as over 90 like today. And then the rest mostly on reinfo…”
Vinyals: Emergent self-correction will make model bootstrapping trivial in specific domains
“At some point, the model might have the capability, let's say, to self-correct. Let's call it self-correction. And as soon as it hits that capability, you can see how, wow, bootstrapping now is trivial. And of course, it's not going to be that blanket across a…”
Vinyals: General reward functions in math will advance AI self-improvement
“Depending how you attack the problem, it's definitely not at that end. And I think even because it's so hard to have a perfect reward when you interleave language, I think by, even by accident, the field will move forward as we get to discover these more gener…”
Vinyals: Achieving monolithic AGI matters less than capability distribution
“The contrarian view would be, I'm not sure it matters that, that we achieved AGI. I think it might not look like Hey, like, it's gonna be exactly like the cognitive task we can do, and then we reach parity. It's gonna be a distribution of things that these mod…”
Vinyals: Context Windows Will Grow 10x for Commodity and SOTA AI in 1-2 Years
“I expect, I mean, order of one, two years, you might see, wow, we went another order of magnitude from both state of the art and, of course, what's considered a commodity. So I'm pretty certain about this.”
Vinyals: AI models show reasoning ability but remain inconsistently brittle
“Reasoning capabilities of the models are there, but I don't think we've perfected sort of making the reasoning very crisp and accurate so that these models would not hallucinate or would not, you know, the model might solve an, you know, Olympia mathematical p…”