optimizer state
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Jul 28, 2025 by Varun Mohan · across every show →
Everything said about optimizer state, oldest first
Jul 28, 2025 neutral
Mohan: Training optimizer state requires 14x model parameter size in memory
“The optimizer state. So while you're training is 14 times the size of the model. So in this case, if it's like one hundred seventy billion parameters, it's probably, I'm not great at mental math here, but that's probably around 2.5 terabytes to just store the …”