Bourgeau: AI models must be trained on harmful data to avoid it
Sebastien Bourgeau · ”We’re Ahead of Where I Thought We’d Be” — Gemini 3 & the Future of AI · Dec 18, 2025 · at 43:14
Sebastien Bourgeau, Google DeepMind pre-training lead, explains why frontier AI models cannot completely omit harmful internet content from pre-training datasets.
0:00 / 0:10exact quote · 10.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So at a fundamental level, you did, you do need the model to know about those things. So you have to train a bit at least on those so that it knows what those things are and knows to stay away from those, right?”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sebastien Bourgeau
Bourgeau: AI progress from pre-training improvements is not slowing down
“It's still remarkable how much progress we're able to achieve in this way, and it's not really slowing down.”
Bourgeau: Google and DeepMind are actively researching post-Transformer architectures
“I believe so. There's groups doing research on the model architecture side, for sure, within Google and within DeepMind”
Bourgeau: Architecture and data innovation currently matter more than scale
“The other parts are architecture and data innovation. These also play a really, really important part in the Performance of pre-training and probably even more so than pure scale these days, but scaling is still an important factor as well.”
Bourgeau: AI development is not running out of training data
“The other part of your question are we running out of data? I don't think so, so there's more.”
Bourgeau: AI research is shifting to a data-limited paradigm
“I think what might be happening instead is kind of a shift in paradigm where before we were kind of scaling in the data unlimited regime where, where data would scale as much as you would like. And we're kind of shifting more to a data limited regime, which ac…”
Bourgeau: End-to-end differentiable retrieval and search in training will take years
“I think deep down, I do believe that the long-term answer is to learn this differentiable end-to-end way, which means probably doing pre-training or whatever that looks like in the future, Learn to retrieve as part of the training and learn how to do search as…”