Assertion Supported
Jamil: Writing in the Margins solves the lost-in-the-middle problem
“It improves the ability of any language model to extract relevant information, so solving the lost in the middle problem”
Assertion Supported
Jamil: 'Writing in the Margins' works on any transformer without fine-tuning
“So it can be used with any transformer model without fine-tuning, just by doing it, just by doing this inference differently.”
Assertion Not checkable as stated
Jamil: Chunked prefill is experimental in vLLM, likely used by majors
“This is called the chunked pre-fill, and it's an experimental feature that has been recently introduced in VLLN, but it's probably used in more sophisticated inference engines at major companies.”
Assertion Supported
Jamil: Smaller models gain more improvement from Writing in the Margins
“As you can see, for example smaller models have a better more more improvement.”
Insight
Jamil: Writing in the Margins avoids re-prefilling tokens, halving compute cost
“Again, to the language model to generate the answer, and it would cost you another million, because the model has to reprocess this prefilling again of one million tokens, so it would cost you two million tokens, but with writing in the margins, it would cost …”
Insight
Masking out prior KV cache tokens pushes autoregressive transformers out of distribution
“The token number two in the KVCache is a contextualized version of the token zero, one, and two. So if you tell the model to only look at the last tokens you are creating an autoregressive model that is generating the logits of a P of let's say X, but only loo…”
Assertion Not checkable as stated
Jamil: OpenAI and Cohere Overlap Prefill and Generation to Maximize GPU Utilization
“Token generation is memory bound means that the limitation is only given by how much your KVCache can hold. So the memory can hold in terms of KVCache. While prefilling is compute bound, so to maximize the GPU utilization, whenever you work with OpenAI or Cohe…”
Assertion Supported
Jamil: Ablations Show Supplying Both Context and Margins Outperforms Either Alone
“And we also prove in the ablation studies that actually it's always convenient to send the context plus the margins, never just the margins. So you can see here, this ablation contest compression. So if you only send the margins or only the context, it's alway…”
Assertion Supported
Writing in the Margins requires inference engine modifications but zero fine-tuning
“We are not doing any change to the model architecture, so you don't have to fine-tune anything, you don't have to change anything, like the can you use this stuff with, like, a Lank pane? No, because it requires a modification on how the inference engine is us…”
Insight
Jamil: KV Cache Prefilling Scales Quadratically While Token Generation Is Linear
“Token generation, which means generating one token using whatever is in the kvcash, is linear with respect to whatever is in the side of the kvcash. Prefilling the kvcash is quadratic, and mostly because it's quadratic, it's very expensive, so we are talking a…”
Assertion Supported
Jamil: LLM Attention Allocates Most Weight to Initial Tokens
“So we have seen with the paper called sync attentions that actually the language model allocates a lot of a lot of, because when you do the attention mechanism, you are doing a weighted sum over the tokens, and each token is given a weight, and we see that mos…”
