Temperature Zero Inference
topic on 1 show · 1 statements across 1 episodes
1 statements about Temperature Zero Inference, every show
Eugene Yan: GPU floating-point math makes temperature-zero inference non-deterministic
“For GPUs with floating points, and you push it through so many calculations, and so many met miles, the floating points aren't just not gonna be precise. So that's why even if temperature is zero, it's not gonna be the same throughout, ah, for multiple request…”