Dec 18, 2025 · 55m · mad
”We’re Ahead of Where I Thought We’d Be” — Gemini 3 & the Future of AI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, Google DeepMind Pre-Training Lead Sebastien Bourgeau discusses the technical, architectural, and organizational shifts driving Gemini 3 and frontier AI systems. He explores topics including data-limited regimes, native multimodality, test-time reasoning, vertical hardware integration, and the future of AI-driven research.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 28.5% of the talking time here. How this is scored →
Matt as informed peer 3.3 Guest teaching 4.0 Guest disagreement 0.8 Matt pushing back 1.0
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 35:28 Deflection on proprietary reasoning traces
The guest directly shuts down the host's probe into training on reasoning traces with a firm refusal to comment on specifics.
Hardest push from Matt ▶ 35:32 Host presses after guest deflectionImmediately after the guest refuses to comment on reasoning traces, the host cheekily pushes back, noting that his refusal confirms he asked the right question before reframing.
Biggest teaching moment ▶ 36:16 Correcting premise on finite data vs less dataThe guest explicitly interrupts the host's premise to clarify that shifting to a finite data regime is not the same as learning with less data.
Matt holds his own ▶ 39:19 Connecting Retro paper to Gemini long contextThe host demonstrates deep familiarity with the guest's academic work by contrasting his 2021 Retro paper on retrieval with Gemini 3's context expansion.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Deconstructing the 'Secret' Behind Gemini 3 | 3 | 4 | 1 | 1 | The host cites Oriel Vinyals' tweet about pre-training and post-training simplicity. The guest reframes this by explaining that Gemini 3's leap comes from thousands of combined micro-improvements across a massive team, transitioning the view from single neural nets to system engineering. | |
| Internal Utility and Realistic AI Trajectories | 3 | 4 | 1 | 1 | The host brings up benchmark overfitting fears. The guest counters with internal productivity metrics showing researchers increasingly rely on newer model generations to do their daily engineering. | |
| AI in Research Workflows and Paradigm Continuity Across Labs | 3 | 4 | 1 | 1 | The host references AI 2027 automation scenarios and rival releases like GPT-5.2. The guest breaks down research into infra execution versus high-level hypothesis formation, noting lab specialization branches like DeepMind's vision strength. | |
| Explorative Research and Post-Transformer Paradigms | 4 | 3 | 1 | 2 | The host probes for secret post-Transformer architecture research groups and quotes Demis Hassabis on full-stack integration. The guest acknowledges exploratory research exists but highlights how high failure rates require balancing research risk with Google's infra stack. | |
| Inside the Pre-Training Lead Role | 2 | 2 | 0 | 0 | The host asks about the pre-training lead responsibilities and guest background. The guest details managing 150-200 researchers across data, infra, and model teams. | |
| Joining DeepMind and Shifting to Real-World Data | 1 | 3 | 0 | 0 | The guest recounts joining DeepMind via a Cambridge referral and transitioning from synthetic Atari RL to real-world language datasets. | |
| DeepMind's LLM Milestones: Gopher, Chinchilla, and RETRO | 4 | 5 | 0 | 0 | The host specifically prompts the Retro paper. The guest details early scaling work on Gopher, Chinchilla's revision of OpenAI's compute-optimal scaling laws, and Retro's retrieval mechanism. | |
| Defining 'Research Taste' and Managing Complexity | 4 | 5 | 0 | 1 | The host catches the phrase 'research taste' and asks for a definition. The guest defines it as managing complexity budgets, trading maximum raw performance for system simplicity, and team interoperability. | |
| Balancing Exploration, Execution, and Product Pressure | 3 | 3 | 0 | 1 | The host asks about short-term vs long-term pressure and competing for benchmark wins like IMO. The guest explains how critical paths are de-risked before scale-ups. | |
| DeepMind's Organizational Structure across Pre- and Post-Training | 3 | 4 | 0 | 0 | The host asks about org structure and MoE architecture. The guest explains how Mixture of Experts decouples compute usage from total model parameter size. | |
| Multimodal Computational Costs and Optimization | 4 | 4 | 1 | 2 | The host asks if multimodality inflates token costs and whether pre-training scaling laws are dead. The guest dismisses death-of-scaling narratives as strange, explaining how scale compounds with architectural and data innovations. | |
| Data Mixes, RL Scaling, and Data Shortages | 4 | 4 | 3 | 3 | The host asks directly about training on reasoning traces. The guest explicitly declines to comment on proprietary techniques, leading the host to banter about hitting sensitive topics before shifting to data limits. | |
| Human vs. Machine Data Efficiency | 3 | 5 | 1 | 1 | The host asks if models can learn like children with less data. The guest explicitly corrects the host, clarifying that moving to a finite data regime is conceptually different from training with less data. | |
| Large Context Windows vs. Retrieval-Augmented Generation | 5 | 5 | 1 | 1 | The host connects the guest's earlier Retro paper on retrieval to Gemini 3's massive context windows. The guest explains the long-term vision for end-to-end differentiable retrieval and details the complex pre-training evaluation gap. | |
| AI Model Alignment and Harmful Pre-Training Data | 4 | 4 | 3 | 2 | The host asks if toxic web data should be filtered out during pre-training, then asks about DeepThink internals. The guest declines to give DeepThink details while educating on why models need exposure to bad data to recognize unsafe concepts. | |
| Agentic Workflows, Screen Understanding, and Model Vibes | 3 | 4 | 1 | 1 | The host asks about agentic workflows, Google Anti-Gravity, and 'vibe coding'. The guest highlights screen understanding in pre-training and attributes large model feel to pre-training and RL scaling. | |
| Finite Data Regimes and Inference Cost Optimization | 4 | 5 | 0 | 0 | The host brings up NeurIPS themes like continual learning and asks for career advice for students. The guest advises mastering the complete stack from TPU hardware to model research. | |
| Guidance for Startups Facing Rapidly Advancing Base Models | 3 | 4 | 0 | 1 | The host voices VC and startup founder concerns about rapidly expanding base models. The guest advises founders to extrapolate model capability trajectories rather than build narrow wrappers. |
Statements from this episode (35)
Bourgeau: Gemini 3 progress came from small contributions, not one breakthrough
“In my experience, there's maybe one or two of those things that make a larger difference than other things, but it's really a combination of many, many changes and many, many things from a very large team that actually makes Gemini three so much better than th…”
Bourgeau: AI progress from pre-training improvements is not slowing down
“It's still remarkable how much progress we're able to achieve in this way, and it's not really slowing down.”
Bourgeau: Frontier AI development is about building systems, not just neural networks
“We're not really building a model anymore. I think we're really building a system at this point. People have sometimes this view that we're just training a neural network architecture and that's it. But it's really the entire system around the network as well …”
Bourgeau: Gemini answers computer science benchmark questions taking humans significant time
“They are becoming increasingly difficult, and even for me, who has a background in computer science, some of the questions the model answers, it would take me a significant amount of time to answer.”
DeepMind's Bourgeau: AI progress is ahead of where I expected
“I think, if I'm being honest with myself, I think we're ahead of where I thought we could go.”
Bourgeau: Would not have bet heavily on scaling laws materializing
“I, I'm not sure if I would have bet a lot on, on that actually materializing and being where we are today.”
DeepMind's Bourgeau predicts major AI-driven scientific breakthroughs within years
“I think we will be able to make some large scientific discoveries in the next few years.”
Bourgeau: Agentic workflows will accelerate AI research tasks in the next year
“The first part, I think, especially in the next year with more agentic workflows being enabled more and more, that should be able to really accelerate our work there.”
Bourgeau: Google and DeepMind are actively researching post-Transformer architectures
“I believe so. There's groups doing research on the model architecture side, for sure, within Google and within DeepMind”
Bourgeau: AI scale has blurred the line between research and engineering
“I think over time that boundary has blurred quite a lot because we're working on these very large systems now. Research really looks like engineering and vice versa.”
Bourgeau: DeepMind shifted focus from pure research to research engineering
“And I think that's a mindset that has really evolved over the last few years at DeepMind, especially where maybe there was a bit more of the traditional research mindset before, and now with Gemini, it's really more about research engineering.”
Bourgeau works with 150 to 200 people on Gemini pre-training
“So it's a fairly large team at this point. It's a bit hard to quantify exactly, but maybe a 152 hundred people I work on a day-to-day on the pre-training side between data, model, infrastructure, evals, and so coordinating the work of all of these people into …”
Bourgeau: Early DeepMind research defaulted to synthetic data over real-world data
“And at the time we had to add this from real world data to the name of the project, because people would assume otherwise it would be synthetic environments or synthetic data. And that definitely has shifted completely since then.”
Bourgeau: Trading peak model performance for lower complexity enables faster progress
“Oftentimes we don't necessarily want to use the best performance version of a research idea, but we'd rather trade off some of the performance for a slightly lower complexity version because we think that will allow us to do more and more progress in the futur…”
Bourgeau: In deep learning, negative results often mean unoptimized techniques
“Especially in deep learning, a negative results doesn't mean something doesn't work. It means you haven't made it work yet often.”
Bourgeau: Google leadership's research background shields DeepMind from benchmark pressure
“There's actually very little of that. I think because all of the leadership has a research background that they're very much aware that yes, to some extent you can force and accelerate specific benchmarks and certain goals, but in the end, the progress and the…”
Bourgeau: Google DeepMind separates AI development into dedicated pre- and post-training teams
“At a super high level so we have a pre-training team, a post-training team. On the pre-training side, we have people working on the model, on the data, the infrastructure, evals as well, very important.”
Bourgeau: Gemini 3's architecture hasn't changed much from Gemini 2.5
“At the high level, I don't think the architecture has changed that much compared to the previous one. It's more of what I was saying before, where a few different things come together to gather, give a large, large improvement.”
Bourgeau: Architecture and data innovation currently matter more than scale
“The other parts are architecture and data innovation. These also play a really, really important part in the Performance of pre-training and probably even more so than pure scale these days, but scaling is still an important factor as well.”
Bourgeau: Pre-training scaling lessons apply directly to RL scaling
“On the RL and RL scaling side, I think we're seeing a lot of the same things we're seeing in pre-training or we saw in pre-training. What's interesting here is because we have the experience of pre-training, a lot of the lessons apply, and we can reapply some …”
Bourgeau: AI development is not running out of training data
“The other part of your question are we running out of data? I don't think so, so there's more.”
Bourgeau: AI research is shifting to a data-limited paradigm
“I think what might be happening instead is kind of a shift in paradigm where before we were kind of scaling in the data unlimited regime where, where data would scale as much as you would like. And we're kind of shifting more to a data limited regime, which ac…”
Bourgeau expects significant long-context AI innovations in the next year
“I think there's going to be a lot more innovation on that side in the next year or so to make long context more efficient, but also just to extend the context length of models themselves.”
DeepMind made recent attention discoveries that will shape near-term research
“For us, at least on the attention side, we've made some really interesting discoveries recently that I think will shape a lot of the research we do in the next few months, and I'm personally very excited about that.”
Bourgeau: End-to-end differentiable retrieval and search in training will take years
“I think deep down, I do believe that the long-term answer is to learn this differentiable end-to-end way, which means probably doing pre-training or whatever that looks like in the future, Learn to retrieve as part of the training and learn how to do search as…”
Bourgeau: External AI benchmarks quickly become contaminated via web data
“What we found is that external benchmarks, then you can use them for a little while, but very quickly they become contaminated. So they start to be replicated on different forms Different forms or different parts of the web, and then if we end up training on t…”
Bourgeau: Internal held-out evals are the only way to prevent benchmark self-deception
“The only way you really have to protect against cheating yourself and thinking you're doing better than you are is by actually creating held out evals and not really keeping them held out.”
Bourgeau: AI models must be trained on harmful data to avoid it
“So at a fundamental level, you did, you do need the model to know about those things. So you have to train a bit at least on those so that it knows what those things are and knows to stay away from those, right?”
Bourgeau: Thinking models compute across sequence length to test hypotheses before answering
“Rather than just doing compute in, in the depths or in, in the model side, you also do compute and allow the model to think more on, on the sequence length side of things. So the model actually starts to form hypotheses, test hypotheses, invoke some tools to v…”
Bourgeau: Pre-training perception and screen understanding is critical for agentic AI
“Bringing it back to the topics of pre-training I think that the perception and vision side is very important for this, because now you're asking models to interact with computer screens. So, so being able to do screen understanding really, really well, Is, is …”
Bourgeau: Vibe coding performance stems mostly from RL scaling and post-training
“I think this is, yeah, this is in general for vibe coding specifically, I think that's maybe more of an RL scaling and post training thing where, where you can actually get quite a lot of data and train them all to do that really well.”
Bourgeau: Recent continual learning progress has mostly occurred via post-training search tools
“First, I think a lot of progress has been made on this front since in the last few years. I think this is mostly around post-training, around search, use search tools and then make search calls, then they would have access to that new information.”
Bourgeau: Full-stack hardware and TPU understanding is a superpower for AI research
“So being able to understand how the stack works all the way down from TPUs to research is kind of a superpower, because then you're able to kind of find these gaps in between different layers that other people weren't necessarily able to see, but also to reaso…”
Bourgeau: Retrieval-augmented pre-training could become viable in a few years
“I just think it's not unreasonable to think in the next few years, something like that might actually become viable for a leading model like general.”
Bourgeau: Focus shifts to AI model harnesses and error recovery mechanisms
“And then, so, so what that means is research in terms of how, how you use models and the harness, et cetera, is becoming increasingly important and also how you make models and these harnesses more robust to making errors and recover from such errors.”