Model Training Efficiency
topic on 1 show · 1 statements across 1 episodes
1 statements about Model Training Efficiency, every show
Scialom: Larger tokenizers allow models to see more text per compute unit
“With a bigger vocabulary, for the same text, you have less tokens, right? And so you can train your model on the same amount of knowledge with fewer steps. So for the same compute, you can see more knowledge if you don't epoch.”