RoPE theta scaling
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement May 31, 2024 by Mark Huang · across every show →
Everything said about RoPE theta scaling, oldest first
May 31, 2024
Huang: Correct RoPE theta scaling causes immediate early perplexity drops
“Specifically when the one trick you should pay attention to is you know that your context length and theta scaling is working right if the early steps in the perplexity go straight down. So like when it wasn't correct, it would oscillate a lot in the beginning…”