Mixture Of Experts
topic on 7 shows · 29 statements across 20 episodes
Cheeky Pint 
Latent Space 
No Priors 
Invest Like the Best 
the MAD Podcast 
Big Technology 
All-In
29 statements about Mixture Of Experts, every show
Frontier Mixture of Experts Models Now Activate Only 1% of Experts
“If you look at a modern MOE model, there are very few models that are more than 10% dense, meaning 10% of the possible number of experts you can activate are activated, and I think the frontier models are closer like one percent.”
All modern AI models requiring multi-GPU parallelization are Mixture-of-Experts
“Effectively, all models today are MOE models that are, you know, at least all models large enough that you would care to parallelize them across multiple GPUs.”
Catanzaro: MoEs have long been the default architecture in frontier AI
“Yeah, I believe MOEs have been the default in Frontier AI for a long time. They're just a really good combination of inference cost and intelligence.”
Catanzaro: Dense models outperform MoE models under strict memory constraints
“You know, they take a lot more memory. If you have a very small amount of memory, a dense model is going to be smarter.”
Edunov: LLM architectures are boring and fundamentally unchanged since 2017
“And honestly, LLM architectures are relatively boring. I don't know, probably alienate half of your audience. But it's like, it's a transforming layer in the end, like paper was published in 2017, and you go to any LLM lab today, you will see very, very simila…”
Sanseviero: Embedding offloading suits edge devices; larger models require MoEs or dense architectures
“This is really optimized and designed for, like, on-device. And when I say on-device, I mean, like, running in a phone, Android, Raspberry Pi, and so on, right? When you go larger, you usually want to come back more You want to have more, like, dense architect…”
Sanseviero: MoE models are great for inference but hard to fine-tune
“MOEs are challenging to fine tune. I don't know if we've talked about that in the past, but MOEs in general are like an extremely good architecture. They work great for inference. But when people fine tune them, they struggle a bit. Like they are not as easy t…”
Looping provides parameter-free FLOPS, whereas MoE architectures provide FLOPS-free parameters
“In mixture of experts, you have flops free parameters. So, so parameters that they're not actually bringing any flops. And in, in like looping, you have parameter free flops where you don't have extra parameters for the extra flops that you're throwing on this…”
Lample: Mistral develops specialized single-capability models before merging them
“The way we kind of do things internally, that we have like one team, focus on one capability, build one model, and then when it's mature enough, we decide to merge this into the Mixture. So, but hey, here's, it was the first time we basically merged all of thi…”
Pope: Mixture of experts maps well to systolic arrays, attention does not
“The mixture of expert layer maps really well, but the attention does not.”
MoE models with 20B active parameters match 72B dense models in memorization
“With the coming of the MOE models, we're starting to see these phenomena, even like models which have a less number of active parameters. So a phenomena that we would observe at a 72 B model in the past is probably now visible at a 20 B active one 20 B MOE als…”
Bakouch: Megatron's local batching hindered MoE expert specialization
“And they find that, for example, in the Megatron item code base this wasn't the case. This was done as the local batch I think. So it basically means that everyone that is using Megatron at the time of the paper I mean, it wasn't even a paper. It was just a bl…”
Swix: Every frontier lab now distills dense models into MoEs
“I think like, I think this is the pattern for every frontier lab now.”
Chen: Nearly all large language models today utilize mixture of experts
“I think pretty much all large language models today use, utilize mixture of experts.”
David Friedberg argues specialized AI model networks will beat monolithic models
“It doesn't make sense to have one large do everything model. This mixture of experts architecture, ultimately you can kind of think about taking a large model, making two copies of it, and then trying to shrink each model down to whatever the necessary, so tha…”
Zhang: Meta Failed at Training MoE Models for Llama Series
“The reason why Lama open-sourced the MOE model, because I think they tried to train our MOE model, but they failed. So that, that's why they didn't open source MOE mode for Lama series.”
He: Token dropping works in MoE pre-training; dropless excels in fine-tuning
“A lot of pre-training experiments show that Token dropping is very efficient, and it doesn't impact performance, but in some of the, like, the downstream fine-tuning, people realize drop-less is better.”
Ethan He: MoE experts do not cleanly specialize into semantic domains
“Unfortunately, people didn't find, like, a significant interpretability inside these experts. Say, one expert focus on math, the other focus on literature. I think the problem is that neural network hidden states are already very entangled. So, when hidden sta…”
He: Upcycling dense models to MoE beats continuing dense training per FLOP
“By training these upcycled models, you can achieve better accuracy than simply training the dense model further for the same number of flops.”
Ethan He: Peak pre-training learning rate works best for MoE upcycling
“We found that the best is to use the original highest peak learning rate from pre-training, which works the best.”
Ethan He: 64 experts is the sweet spot for MoE upcycling
“We found, 64 experts is kind of like the sweet spot. If you increase the number of experts beyond 64, it provides diminishing return.”
Combining MoE and LoRA will eliminate big versus small model trade-offs
“And I do think that's the future so we can get something that is incredibly smart, incredibly huge, but with the latency cost and speed of something, something tiny. So no more big model versus small model paradigm. It's potentially one in the same.”
Scialom: Meta is exploring Mixture of Experts architectures for future models
“So, it's just an hyperparameter we haven't optimized a lot yet, but we have some stuff ongoing, and that's an hyperparameter we will explore in the future.”
Tay: Mixture-of-Experts is fundamentally the right architecture for scaling
“Fundamentally, I just think that MOEs are just, like, the way to go in terms of, like, floppyram ratio, they bring the benefit from the scaling curve, if you do it right, if you, they bring the benefit from the scaling curve, right, and then, Like, that's, lik…”
Tay: Training MoE models from scratch is ideal over sparse upcycling
“I think in the ideal case, you do MOE from scratch.”
Frankle: Training MoE models with FSDP creates severe network bandwidth bottlenecks
“And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP zero three style. Where there's just a lot of parameters getting shuffled back and forth and your ratio of kind of compute to amount of dat…”
Patel: SemiAnalysis reported GPT-4 mixture of experts architecture in January
“Just being clear, I talked about mixture of experts in January, it's just people didn't really notice it.”
Guu: Adapting LLMs to Code Requires Specialized Training Over Pure Retrieval
“So if you just want to get very precise factoid information, like what is my Wi-Fi password, retrieval augmentation is going to be very good. But if, for example, you're trying to adapt a language model to a new enterprise, and they have some kind of a special…”
Guu: Model Surgery Research Enables Unique Modularity Over Retrieval and MoE
“I feel that This is a very exciting area for research because it provides a different kind of modularity from retrieval augmented models or mixture of experts, one that actually allows a kind of generalization that's very interesting.”