Jamil: Writing in the Margins solves the lost-in-the-middle problem
Umar Jamil · [Paper Club] Writing in the Margins: Chunked Prefill KV Caching for Long Context Retrieval · Sep 19, 2024 · at 22:54
Umar Jamil discusses the retrieval performance benefits demonstrated in his paper 'Writing in the Margins'.
0:00 / 0:05exact quote · 5.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It improves the ability of any language model to extract relevant information, so solving the lost in the middle problem”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Umar Jamil
Jamil: 'Writing in the Margins' works on any transformer without fine-tuning
“So it can be used with any transformer model without fine-tuning, just by doing it, just by doing this inference differently.”
Jamil: Chunked prefill is experimental in vLLM, likely used by majors
“This is called the chunked pre-fill, and it's an experimental feature that has been recently introduced in VLLN, but it's probably used in more sophisticated inference engines at major companies.”
Jamil: Smaller models gain more improvement from Writing in the Margins
“As you can see, for example smaller models have a better more more improvement.”
Jamil: Writing in the Margins avoids re-prefilling tokens, halving compute cost
“Again, to the language model to generate the answer, and it would cost you another million, because the model has to reprocess this prefilling again of one million tokens, so it would cost you two million tokens, but with writing in the margins, it would cost …”
Masking out prior KV cache tokens pushes autoregressive transformers out of distribution
“The token number two in the KVCache is a contextualized version of the token zero, one, and two. So if you tell the model to only look at the last tokens you are creating an autoregressive model that is generating the logits of a P of let's say X, but only loo…”
Jamil: OpenAI and Cohere Overlap Prefill and Generation to Maximize GPU Utilization
“Token generation is memory bound means that the limitation is only given by how much your KVCache can hold. So the memory can hold in terms of KVCache. While prefilling is compute bound, so to maximize the GPU utilization, whenever you work with OpenAI or Cohe…”