Aug 1, 2024 · 46m · no-priors
No Priors Ep. 74 | With Google DeepMind VP of Research Oriol Vinyals
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Google DeepMind Vice President of Research Oriol Vinyals joins hosts Sarah Guo and Elad Gil on No Priors to discuss the frontier of artificial intelligence, examining the transition to unified multimodal models, long-context architectures, the evolution of reasoning via reinforcement learning, and the future trajectory of scientific discovery and AGI.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 15% of the talking time here. How this is scored →
The hosts as informed peer 5.1 Guest teaching 3.3 Guest disagreement 0.5 The hosts pushing back 0.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 38:27 Rejection of the binary AGI milestone concept
Oriol rejects conventional obsession with a singular AGI arrival date, arguing that treating intelligence as a discrete threshold rather than an uneven distribution of capabilities misses the reality of how models operate.
Hardest push from the hosts ▶ 35:08 Challenging math and game domains as dead endsSarah directly confronts the host-lab DeepMind paradigm by asking whether constrained formal systems like mathematics and games represent dead ends rather than true paths toward general reasoning.
Biggest teaching moment ▶ 13:27 Explaining vector compression vs native attention contextOriol clarifies the fundamental architectural trade-off between RAG and long context, pointing out that vector databases reduce an entire book to a single static vector while full-context LLMs preserve word-level relational reasoning.
The host holds their own ▶ 28:47 Introducing Nyquist-Shannon sampling to AI evaluationElad demonstrates sharp theoretical depth by applying the Nyquist-Shannon sampling theorem to conceptualize how an evaluator system must match or exceed the sampling rate of a higher-order intelligence.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Google DeepMind Merger and Gemini’s Role Across Google | 4 | 2 | 0 | 1 | Sarah inquires about the internal reorganization merging Google Brain and DeepMind, as well as Gemini's integration across Google products. Oriol provides a collaborative, thorough overview of the timeline and research-to-product pipelines. | |
| Chatbot Interfaces vs. Search Paradigms | 6 | 2 | 0 | 1 | Elad demonstrates his insider Google background by breaking down historical search query classifications (navigational, commercial, medical) to explore the boundary between search and conversational interfaces. Oriol agrees both will co-evolve rather than one completely subsuming the other. | |
| Long-Context Windows and Multimodal Interfaces | 5 | 3 | 0 | 1 | Oriol reflects on moving from early RNN/LSTM limitations to million-token multimodal context windows in Gemini. Elad probes the expected commercial timeline for very long context adoption in enterprise workflows. | |
| Retrieval Architectures vs. Infinite Context Memory | 6 | 4 | 0 | 1 | Sarah asks about the continued relevance of RAG and hierarchical memory architectures as context windows expand. Oriol explains how embedding-based retrieval flattens rich text into single vectors whereas full context enables word-level cross-attention reasoning. | |
| Defining and Achieving Crisp Reasoning in Language Models | 5 | 5 | 0 | 1 | Sarah presses Oriol on the technical difference between probabilistic reasoning and crisp, deterministic logical reasoning. Oriol explains how token probability distributions inherently leave non-zero error margins, necessitating System 2 architectures to achieve near-perfect reliability. | |
| Compute Allocation: Pre-Training, RL, and Test-Time Search | 6 | 4 | 1 | 1 | Sarah frames compute allocation across pre-training versus inference/test-time search via Sutton's Bitter Lesson. Oriol contrasts AlphaGo's heavy RL compute split with modern LLMs' reliance on pre-training due to imperfect reward signals in natural language. | |
| Scaling Reward Functions, Bootstrapping, and Self-Correction | 7 | 3 | 0 | 1 | Elad brings up Med-PaLM 2 and proposes the Nyquist-Shannon sampling theorem as an analogy for how machines extrapolate and evaluate intelligence greater than their own. Oriol validates the analogy while noting how generative super-resolution models bypass classical sampling limits by learning underlying world structure. | |
| General Models vs. Domain-Specific Scientific AI | 5 | 4 | 1 | 1 | Sarah asks whether foundational generalist models eliminate the need for specialized scientific models. Oriol argues general models provide a 20% baseline capability everywhere, making targeted fine-tuning essential for high-impact problems like fusion and protein folding. | |
| Reward Definition in Formal Mathematics and Language | 6 | 4 | 2 | 2 | Sarah presents the critique that formal constrained domains like math or games are dead ends for AGI research. Oriol counters by highlighting that mathematical reasoning in LLMs naturally interleaves natural language explanations, bridging narrow verifiers with generalized reward modeling. | |
| Contrarian Views on AGI Timelines and Capability Distributions | 4 | 4 | 2 | 1 | When Sarah asks for contrarian takes on AGI timelines, Oriol rejects the binary framing of AGI milestones, arguing instead that capability will manifest as an uneven distribution where models excel at high-level math while making basic mistakes. | |
| Personal Scaling and Parenting in the AI Era | 3 | 2 | 0 | 0 | Sarah asks how Oriol's belief in near-term AGI shifts his day-to-day life and parenting. Oriol shares personal reflections on managing communication overload with LLMs and the challenges of raising young children alongside evolving tech. | |
| Career and Educational Advice for the Future of AI | 4 | 2 | 0 | 0 | Elad asks what educational pathways young students should pursue today. Oriol recommends anchoring on genuine passion first, then identifying how AI transforms that specific field or focusing on under-explored niches. |
Statements from this episode (21)
Vinyals: Gemini formed by merging parallel LLM efforts across Brain and DeepMind
“One was that the Gemini project was formed as a result of having two sort of Parallel efforts on LLMs, mostly led by Google Brain and what we now call Legacy DeepMind. So earlier in the year, there was an effort to merge the two projects, and that's when sort …”
Vinyals: RNNs and LSTMs Never Remembered Beyond a Few Hundred Words
“We come from a world where we had recurrent neural networks and LSTMs that actually had infinite memory, although it was not very capable, right? You, the models in, in practice, they never remember more than a few hundred words or so.”
Vinyals: Context Windows Will Grow 10x for Commodity and SOTA AI in 1-2 Years
“I expect, I mean, order of one, two years, you might see, wow, we went another order of magnitude from both state of the art and, of course, what's considered a commodity. So I'm pretty certain about this.”
Vinyals: Gemini Reasons Over Entire Books Rather Than Compressing to Single Vectors
“The problem with, of course, retrieval based methods is they tend to simplify things to say, hey, like this whole book is just a single vector. Whereas if you just upload a whole book into Gemini and ask questions, it can really reason about every single word,…”
Vinyals: Future AI Research Will Drive Toward Hybrid Long-Context Retrieval Systems
“I think to me, it seems like a feature not a bug that we do have a bit of a hybrid mode perhaps going into the future and research will be driven like this.”
Vinyals: AI models show reasoning ability but remain inconsistently brittle
“Reasoning capabilities of the models are there, but I don't think we've perfected sort of making the reasoning very crisp and accurate so that these models would not hallucinate or would not, you know, the model might solve an, you know, Olympia mathematical p…”
Vinyals: Language models inherently retain non-zero error probabilities
“So then you, of course, are absorbing all the knowledge on the internet and then sharpening those models around being, following instructions, being aligned with humans, but you still have this Probability distribution that will assign non-zero probability to …”
Vinyals: Wrapping LLMs in iterative reasoning programs accelerates accuracy
“To accelerate that, that sort of progress, you want to start sort of really exploring what's the reasoning the model has, and by making it more redundant, more logical, by iterating more on these kind of ideas, you could imagine generating a very small program…”
Vinyals: AlphaGo compute was mostly RL self-play, unlike modern LLM pre-training dominance
“Historically, if you look at AlphaGo, which actually followed quite closely the recipe of you pre-train your model on all human data, you then use RL to make it better, and then you do some search at inference time, the compute there was very skewed for the mi…”
Vinyals predicts pre-training compute will drop to ~50% as RL expands
“So to me, that balance feels correct, like some on pre-training, and here we, we're trying to learn every task. So certainly that's going to be, you know, let's say it can be as high as 50%, not as high as over 90 like today. And then the rest mostly on reinfo…”
Vinyals: LLM bootstrapping works if verification is easier than solution generation
“If checking that something is correct is easier than creating the solution, then we're in business because the language models will be able to evaluate their own samples more accurately than to generate them. And then we have a sort of reinforcement learning l…”
Vinyals: Emergent self-correction will make model bootstrapping trivial in specific domains
“At some point, the model might have the capability, let's say, to self-correct. Let's call it self-correction. And as soon as it hits that capability, you can see how, wow, bootstrapping now is trivial. And of course, it's not going to be that blanket across a…”
Vinyals: General AI models are roughly 20% good at everything
“One way I kind of characterize general models is they are, I mean,
The level of performance is irrelevant, but they're like, 20% good at everything, ok?
So they, that's the level of performance they reach, but it's general, which, I mean, it's powerful, so you…”
Vinyals: AI will remain hybrid as general models overtaking science is far away
“Probably we can, and we might see more and more bootstrapping from Gemini and these models to then a specific solution that, that the model is going to be throw away, except for, you know, maybe cracking protein folding or, you know, figuring out nuclear fusio…”
Vinyals: Domain fine-tuning creates a synergistic loopback into general AI models
“That directionality of taking a generalist model and doing something by fine tuning it, there's going to be a loopback as well, right?
Then you're going to do something amazing, let's say in math, right?
We recently did that.
And then while the data or as a re…”
Vinyals: General reward functions in math will advance AI self-improvement
“Depending how you attack the problem, it's definitely not at that end. And I think even because it's so hard to have a perfect reward when you interleave language, I think by, even by accident, the field will move forward as we get to discover these more gener…”
Vinyals: Shane Legg predicted in 2009 that AGI arrives by 2028
“Shane, I just had a discussion with him about AGI timelines recently. And I mean, in 2009, he predicted it's 2028.”
Vinyals: Achieving monolithic AGI matters less than capability distribution
“The contrarian view would be, I'm not sure it matters that, that we achieved AGI. I think it might not look like Hey, like, it's gonna be exactly like the cognitive task we can do, and then we reach parity. It's gonna be a distribution of things that these mod…”
Vinyals: Reaching industry consensus on when AGI arrives will be impossible
“And honestly, it's gonna be also impossible to get agreement, so it's gonna be quite, Cloud of moments that people might feel it has happened now, but it doesn't matter because the models might be used for amazing things and products and research itself, right…”