FSDP
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Jun 25, 2024 by Jonathan Frankle · across every show →
Everything said about FSDP, oldest first
Jun 25, 2024
Frankle: Training MoE models with FSDP creates severe network bandwidth bottlenecks
“And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP zero three style. Where there's just a lot of parameters getting shuffled back and forth and your ratio of kind of compute to amount of dat…”