attention mechanism
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Sep 19, 2024 by Umar Jamil · across every show →
Everything said about attention mechanism, oldest first
Sep 19, 2024 neutral
Jamil: LLM Attention Allocates Most Weight to Initial Tokens
“So we have seen with the paper called sync attentions that actually the language model allocates a lot of a lot of, because when you do the attention mechanism, you are doing a weighted sum over the tokens, and each token is given a weight, and we see that mos…”