Multi Token Prediction
topic on 2 shows · 2 statements across 2 episodes
2 statements about Multi Token Prediction, every show
Catanzaro: Multi-token prediction lowers inference costs as model accuracy improves
“With multi-token prediction, the speed that you get is a function of the accuracy of your model. The more accurate your model is, the faster the inference is, the cheaper the inference is, the more accurate it is. That's not usually how it works, but in this c…”