Mixture of Experts
also referred to as: moe
4 statements across 3 episodes · 2 bullish · 0 bearish · 3 people on the record · first statement Jul 25, 2024 by Sharon Zhou · across every show →
Everything said about Mixture of Experts, oldest first
Jul 25, 2024 bullish
Combining MoE and LoRA will eliminate big versus small model trade-offs
“And I do think that's the future so we can get something that is incredibly smart, incredibly huge, but with the latency cost and speed of something, something tiny. So no more big model versus small model paradigm. It's potentially one in the same.”
Apr 2, 2026
Looping provides parameter-free FLOPS, whereas MoE architectures provide FLOPS-free parameters
“In mixture of experts, you have flops free parameters. So, so parameters that they're not actually bringing any flops. And in, in like looping, you have parameter free flops where you don't have extra parameters for the extra flops that you're throwing on this…”
Jul 2, 2026 positive