Switch Transformer
1 statements across 1 episodes · 0 bullish · 1 bearish · 1 people on the record · first statement Oct 29, 2024 by Ethan He · said 5 times in 1 episodes since 2024 · across every show →
Mentions by year
brought up most by Ethan He (5)
tap a year for its mentions
2024 5 mentions in 1 episode
Everything said about Switch Transformer, oldest first
Oct 29, 2024 negative
He: Mixtral's top-k before softmax routing hurts MoE upcycling performance
“We actually found the mix-throughs approach didn't work as well as expected, because the original model, the original switch transformer from Google uses a softmax and topk for a reason. And because of upcycling, if you switch to topk, then softmax, it actuall…”