Sanseviero: MoE models are great for inference but hard to fine-tune
Omar Sanseviero · ⚡️ Google's Open AI Strategy — Omar Sanseviero, Google DeepMind · May 24, 2026 · at 17:28
Google DeepMind's Omar Sanseviero discusses architectural trade-offs between dense models and Mixture of Experts (MoE) models.
0:00 / 0:19exact quote · 19.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“MOEs are challenging to fine tune. I don't know if we've talked about that in the past, but MOEs in general are like an extremely good architecture. They work great for inference. But when people fine tune them, they struggle a bit. Like they are not as easy to fine tune for instruction. Following the standard recipes and hyperparameters that you have may not work out of the box for MOEs.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Omar Sanseviero
Smartphones will run Gemini 3 Pro-level models locally within two years
“I do think We are heading towards a future in one, two years where imagine like you can run a Gemini three pro powerful model directly in your phone, right?”
Sanseviero: AI labs republished model merging techniques previously created on Reddit
“Yeah, like all of the FrankenMoe stuff, like all of the Axolotl library, like all of these tools, and there were papers published by different companies and research labs one or two years later that were rediscovering what was already done by The Reddit or Dis…”
Sanseviero: Local models handle agentic capabilities well, but world knowledge requires scale
“With local models or models that you can run in your own hardware, you can get capabilities, so you can get agent capabilities, function calling, system instructions, like conversational, and that kind of stuff. Knowledge is much trickier, so for knowledge, yo…”
Sanseviero: Most Conversational Model Behavior Changes Can Be Done via Prompting
“Just changing how the model behaves, you can do most, most of that via prompting nowadays, and in terms of capabilities, the models are very good out of the box.”
Sanseviero: Deep architecture research will not be automated within two years
“If you want to do, like, deeper research in the architecture, my hunch is that most likely this will not be, like, automatable, at least in the next one or two years.”
Gemma 4 E2B loads only 2B of 5B parameters into GPU
“So the GEMA for model is a E to B. That means that it effectively has two billion parameters loaded into the GPU. It actually has almost five billion parameters, but those three billion parameters can be in the CPU, they can be in the disk, which means that yo…”