Attention Mechanism
topic on 4 shows · 5 statements across 5 episodes
Acquired 
Latent Space 
Invest Like the Best 
the MAD Podcast
5 statements about Attention Mechanism, every show
Movva: KV cache memory usage is inefficient by 1-2 orders of magnitude
“Specifically, the KV cache is quite uncompressed right now. I think if you look at the entropy in a KV cache, it's nowhere near, it's not earning its keep. Like, we're storing many kilobytes of data in the KV cache per token. And that's probably off by an orde…”
DeepMind made recent attention discoveries that will shape near-term research
“For us, at least on the attention side, we've made some really interesting discoveries recently that I think will shape a lot of the research we do in the next few months, and I'm personally very excited about that.”
Kiela: Attention mechanism, not Transformers, was the real AI breakthrough
“So I would say, and maybe I'm biased because one of my best friends is, is on the original attention paper, but that was the real breakthrough. It's just like figuring out that you have this attention mechanism that actually allows you to yeah, to do a much be…”
Jamil: LLM Attention Allocates Most Weight to Initial Tokens
“So we have seen with the paper called sync attentions that actually the language model allocates a lot of a lot of, because when you do the attention mechanism, you are doing a weighted sum over the tokens, and each token is given a weight, and we see that mos…”
Transformer attention computational cost scales quadratically with input prompt length
“Traditionally, you'd say this is very, very inefficient, and it actually means that the larger your context window, aka token limit, aka prompt length, gets, the more computationally expensive it gets on a quadratic basis. So doubling your input means quadrupl…”