Mostafa Dehghani
Research Scientist, Google DeepMind. On 1 show, 1 appearance. The Shows tab opens the full record on each.
scientistengineerauthor@m__dehghani ↗LinkedIn ↗research.google/people/4169 ↗
Mostafa Dehghani is a machine learning researcher who co-developed foundational architectures including the Vision Transformer (ViT) and Universal Transformers. At Google DeepMind, he focuses on sequence modeling, self-supervised learning, and scaling multimodal models such as Gemini.
1shows
1appearances
27statements
0resolved
0supported
0contradicted
Everything Mostafa Dehghani said on any show that made the record, most notable first. Each card names its show and opens the statement there.
New pre-training techniques will drastically boost base AI model capabilities
“The way that we used to do pre-training, maybe, like, you know, like two, a year ago or two years ago maybe, like, you know, diminishing return is, like, obvious, but I can see how new ideas are bringing, like, you know, fresh, fresh energy into the pre-traini…”
Fully automated AI self-improvement will eliminate human bottlenecks and trigger breakthroughs
“The moment that we had this full automation, I would say we can close the loop of self-improvement and then it becomes the Like, you know, the problems become like, you know, mostly providing compute for these models to actually do what they want to do. And as…”
AI model progress will alternate between pre-training and post-training breakthroughs
“We're going to be having a bit of a swing back and forth between pre-training and post-training.”
Post-training techniques cannot compensate for a weak base AI model
“Pre-training is still the foundation and like, you can never post-train your way out of a week-based model.”
Video data conveys physical world knowledge to AI more efficiently than text
“So because of that, like picking up a lot of knowledge about the word through language is just not really efficient. I don't want to say that it's impossible, but it's not efficient, you know, like to learn about gravity. If you kind of like, you know, have yo…”
Demonstrating that image training lowers text perplexity remains extremely difficult
“So it turned out to be a really, really good model, but it was like really hard to see that. Wow. You know, I train on images and then like Text perplexity goes down. That was hard to see. You know, like the fact that, you know, you train in native model and i…”
Jagged intelligence in AI is a structural flaw, not a patchable bug
“Not easy to pinpoint like specific things, but again, like, you know, this is just like my personal opinion and maybe I have colleagues and like the other people like sharing this with me, but I think we're underestimating how hard like jagged intelligence is …”
RAG will shift from universal use to handling long-tail distribution cases
“Maybe it changes in a way that, you know, like it doesn't need to trigger RAG for like everything, but I'm pretty sure that they're going to be some tail of the distribution that we're going to do RAG still for it.”
Real-world physical grounding will become the bottleneck for AI self-improvement
“As I said, you know, like soon, like the concept of data, like, you know, how to kind of like, you know, enable these models to kind of like, you know, be very good at like self-improvement becomes, how can I ground these models in, in real world? So this is d…”
Every major AI lab uses previous model generations to build new ones
“In almost every lab the new generation of the models are built heavily using the previous generation of the models. I think that's basically like the case again, everywhere.”
Karpathy's AutoResearch is an early example of AI doing sensible research
“That is definitely. And I think that was one of the early examples of like seeing these models actually doing something super sensible on the research side.”
AI research lacks evaluations to measure progress toward true self-improvement loops
“And the fact that we don't have evals that like, or like even defining evals that, that can maybe measure, oh, not how close we are to the point that we can actually get, get a self-improvement loop. It's just like, we don't have that. And it's just making it …”
Safe execution environments are currently the bottleneck for autonomous AI development
“Like in a safe setup, you know, where they can like, because right now we definitely, we don't, we're not confident about, you know, them doing the right things all the time and measuring like how much they can push and how long they can push a task is very di…”
Anchoring self-improvement loops with real external signals prevents AI model collapse
“Model collapse mainly happens when you have a loop that is Completely closed. Right. And if you don't have any outside signal and just the model, for example, talking to itself or operating in a very like a restricted environment there's a good chance that you…”
Building specialist models is currently the fastest path toward generalist AI
“Short term I would say like building a specialist model is like probably the fastest way to learn like what is actually possible. And in, in many cases, these like specialized model are becoming Stepping stone toward a generalist model, which is like super val…”
Becoming a deep expert in a narrow subject will soon lose value
“Becoming absolute expert about a very specific subject, most likely Is not going to be useful in, in, in like the near future.”
Dehghani initially thought Google's Transformer architecture was random and would die
“And I was like, I don't know if I want to go with this team. It's just like, they're doing something random. Like who, like everybody's doing LST. I'm like, why should I go and work with like a group of people who are working on this like random architecture, …”
Looping provides parameter-free FLOPS, whereas MoE architectures provide FLOPS-free parameters
“In mixture of experts, you have flops free parameters. So, so parameters that they're not actually bringing any flops. And in, in like looping, you have parameter free flops where you don't have extra parameters for the extra flops that you're throwing on this…”
Vision Transformers benefited heavily from software infrastructure built for language models
“There was also benefit of doing that simply because the rest of the, that the machine learning field, which was working on, on, on language, they were using this, like architecture. So they were building infra for it, making it faster. And, you know, like the,…”
Interleaved multimodal generation overcomes single-shot limits via step-by-step planning
“But if you have incremental generation, so if you have text and then an image and text and image, you can get your model to generate these details one by one. So you never expect your model to generate an image, a perfect image in the first shot, right? So, so…”
Single AI failures damage user trust more than average performance builds it
“People don't experience average performance of these models. They experience, like, the failures. If you have your model doing a dumb mistake, the damage in the trust that it makes is, like, bigger than, like, you know, the benefit of getting hundred things ri…”
Formal verification is crucial for AI self-improvement, but not a silver bullet
“In my opinion, formal verification is one of the most powerful like keys to enable like self-improvement, but it's not beaky.”
AI data work will shift toward physical world grounding and interactive environments
“At the end of the day, I think, like, the work that we're doing on the data side most likely is going to shift toward building environments or making sure that these models can interact with physical words, and then it becomes more of a problem of, okay, how c…”
AI continual learning research still lacks a proven recipe for productionization
“Like one side is, I think like the research is not like yet to a very, like to a point that you think that, oh, you know, this is the recipe, you know, I just need to kind of like, you know, exploit it and push productionization.”
One line per show, most statements first. The link opens Mostafa's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
 MADLEDGER | Research Scientist, Google DeepMind | 1 | 27 | — | full record on the MAD Podcast → |