Compute Per Token
topic on 1 show · 1 statements across 1 episodes
1 statements about Compute Per Token, every show
Scialom: Fixed compute per token in transformers forces models to generate extra tokens to think
“We are lacking of flexibility for pre-training architecture, transformers, where We spend the same amount of compute per token. And so, because of that, how can you, like, mitigate this? By generating more tokens, so more thoughts, more compute, because you ha…”