ZeRO 3
product on 1 show · 1 statements across 1 episodes · said 1 times in 1 episodes since 2024
Mentions by year, every show
tap a year for its mentions
Latent Space 1
2024 1 mention in 1 episode
every mention on every show, scene by scene, with the transcript →
1 statements about ZeRO 3, every show
Frankle: Training MoE models with FSDP creates severe network bandwidth bottlenecks
“And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP zero three style. Where there's just a lot of parameters getting shuffled back and forth and your ratio of kind of compute to amount of dat…”