ZeRO-3
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Jun 25, 2024 by Jonathan Frankle · said 1 times in 1 episodes since 2024 · across every show →
Mentions by year
tap a year for its mentions
2024 1 mention in 1 episode
Everything said about ZeRO-3, oldest first
Jun 25, 2024
Frankle: Training MoE models with FSDP creates severe network bandwidth bottlenecks
“And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP zero three style. Where there's just a lot of parameters getting shuffled back and forth and your ratio of kind of compute to amount of dat…”