Switch Transformer
product on 1 show · 1 statements across 1 episodes · said 5 times in 1 episodes since 2024
Mentions by year, every show
tap a year for its mentions
Latent Space 5
2024 5 mentions in 1 episode
every mention on every show, scene by scene, with the transcript →
1 statements about Switch Transformer, every show
He: Mixtral's top-k before softmax routing hurts MoE upcycling performance
“We actually found the mix-throughs approach didn't work as well as expected, because the original model, the original switch transformer from Google uses a softmax and topk for a reason. And because of upcycling, if you switch to topk, then softmax, it actuall…”