Mixture of Experts

“Just being clear, I talked about mixture of experts in January, it's just people didn't really notice it.”

“And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP zero three style. Where there's just a lot of parameters getting shuffled back and forth and your ratio of kind of compute to amount of dat…”

“I think in the ideal case, you do MOE from scratch.”

“Fundamentally, I just think that MOEs are just, like, the way to go in terms of, like, floppyram ratio, they bring the benefit from the scaling curve, if you do it right, if you, they bring the benefit from the scaling curve, right, and then, Like, that's, lik…”

“So, it's just an hyperparameter we haven't optimized a lot yet, but we have some stuff ongoing, and that's an hyperparameter we will explore in the future.”

“A lot of pre-training experiments show that Token dropping is very efficient, and it doesn't impact performance, but in some of the, like, the downstream fine-tuning, people realize drop-less is better.”

“We found that the best is to use the original highest peak learning rate from pre-training, which works the best.”

“We found, 64 experts is kind of like the sweet spot. If you increase the number of experts beyond 64, it provides diminishing return.”

“Unfortunately, people didn't find, like, a significant interpretability inside these experts. Say, one expert focus on math, the other focus on literature. I think the problem is that neural network hidden states are already very entangled. So, when hidden sta…”

“By training these upcycled models, you can achieve better accuracy than simply training the dense model further for the same number of flops.”

“The reason why Lama open-sourced the MOE model, because I think they tried to train our MOE model, but they failed. So that, that's why they didn't open source MOE mode for Lama series.”

“I think like, I think this is the pattern for every frontier lab now.”

“And they find that, for example, in the Megatron item code base this wasn't the case. This was done as the local batch I think. So it basically means that everyone that is using Megatron at the time of the paper I mean, it wasn't even a paper. It was just a bl…”

“With the coming of the MOE models, we're starting to see these phenomena, even like models which have a less number of active parameters. So a phenomena that we would observe at a 72 B model in the past is probably now visible at a 20 B active one 20 B MOE als…”

“The way we kind of do things internally, that we have like one team, focus on one capability, build one model, and then when it's mature enough, we decide to merge this into the Mixture. So, but hey, here's, it was the first time we basically merged all of thi…”

“This is really optimized and designed for, like, on-device. And when I say on-device, I mean, like, running in a phone, Android, Raspberry Pi, and so on, right? When you go larger, you usually want to come back more You want to have more, like, dense architect…”

“MOEs are challenging to fine tune. I don't know if we've talked about that in the past, but MOEs in general are like an extremely good architecture. They work great for inference. But when people fine tune them, they struggle a bit. Like they are not as easy t…”

“And honestly, LLM architectures are relatively boring. I don't know, probably alienate half of your audience. But it's like, it's a transforming layer in the end, like paper was published in 2017, and you go to any LLM lab today, you will see very, very simila…”

“Effectively, all models today are MOE models that are, you know, at least all models large enough that you would care to parallelize them across multiple GPUs.”