Optimizer State
topic on 1 show · 1 statements across 1 episodes
1 statements about Optimizer State, every show
Mohan: Training optimizer state requires 14x model parameter size in memory
“The optimizer state. So while you're training is 14 times the size of the model. So in this case, if it's like one hundred seventy billion parameters, it's probably, I'm not great at mental math here, but that's probably around 2.5 terabytes to just store the …”