Vinyals: Gemini Reasons Over Entire Books Rather Than Compressing to Single Vectors
Oriol Vinyals · No Priors Ep. 74 | With Google DeepMind VP of Research Oriol Vinyals · Aug 1, 2024 · at 14:04
Oriol Vinyals, VP of Research at Google DeepMind, compares traditional vector-based retrieval methods against Gemini's native long-context reasoning capabilities.
0:00 / 0:16exact quote · 16.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The problem with, of course, retrieval based methods is they tend to simplify things to say, hey, like this whole book is just a single vector. Whereas if you just upload a whole book into Gemini and ask questions, it can really reason about every single word, right?”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Oriol Vinyals
Vinyals predicts pre-training compute will drop to ~50% as RL expands
“So to me, that balance feels correct, like some on pre-training, and here we, we're trying to learn every task. So certainly that's going to be, you know, let's say it can be as high as 50%, not as high as over 90 like today. And then the rest mostly on reinfo…”
Vinyals: Emergent self-correction will make model bootstrapping trivial in specific domains
“At some point, the model might have the capability, let's say, to self-correct. Let's call it self-correction. And as soon as it hits that capability, you can see how, wow, bootstrapping now is trivial. And of course, it's not going to be that blanket across a…”
Vinyals: AI will remain hybrid as general models overtaking science is far away
“Probably we can, and we might see more and more bootstrapping from Gemini and these models to then a specific solution that, that the model is going to be throw away, except for, you know, maybe cracking protein folding or, you know, figuring out nuclear fusio…”
Vinyals: General reward functions in math will advance AI self-improvement
“Depending how you attack the problem, it's definitely not at that end. And I think even because it's so hard to have a perfect reward when you interleave language, I think by, even by accident, the field will move forward as we get to discover these more gener…”
Vinyals: Achieving monolithic AGI matters less than capability distribution
“The contrarian view would be, I'm not sure it matters that, that we achieved AGI. I think it might not look like Hey, like, it's gonna be exactly like the cognitive task we can do, and then we reach parity. It's gonna be a distribution of things that these mod…”