Sparse Mixture of Experts
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Dec 28, 2023 by Arthur Mensch · across every show →
Everything said about Sparse Mixture of Experts, oldest first
Dec 28, 2023
Mensch: Mixture of experts decouples model capacity from inference cost
“A sparse mixture of experts, you take the dense layer and you duplicate it several times. And so that's where you actually increase the number of parameters. So you increase the capacity of the model without increasing the cost. So that's the way of decoupling…”