rope relative positional encodings
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Sep 5, 2024 by Andrej Karpathy · across every show →
Everything said about rope relative positional encodings, oldest first
Sep 5, 2024 neutral
Karpathy: RoPE is the only major transformer architecture change in five years
“The transformer hasn't changed that much. You know, we've added the rope positional and the rope relative positional encodings. That's like the major change. Everything else doesn't really matter too much. It's like plus three percent on a small few things. Bu…”