Writing in the Margins

“We are not doing any change to the model architecture, so you don't have to fine-tune anything, you don't have to change anything, like the can you use this stuff with, like, a Lank pane? No, because it requires a modification on how the inference engine is us…”

“Again, to the language model to generate the answer, and it would cost you another million, because the model has to reprocess this prefilling again of one million tokens, so it would cost you two million tokens, but with writing in the margins, it would cost …”

“So it can be used with any transformer model without fine-tuning, just by doing it, just by doing this inference differently.”

“And we also prove in the ablation studies that actually it's always convenient to send the context plus the margins, never just the margins. So you can see here, this ablation contest compression. So if you only send the margins or only the context, it's alway…”

“As you can see, for example smaller models have a better more more improvement.”

“It improves the ability of any language model to extract relevant information, so solving the lost in the middle problem”