Quantization
topic on 2 shows · 6 statements across 5 episodes
6 statements about Quantization, every show
Most inference optimizations like KV caching are lossless, unlike quantization
“Most inference optimizations are lossless. KV caching, for example, you are just recomputing or preventing recomputing the same values. Speculation, of course, If a draft token is wrong, it gets rejected. The main lossy optimization is quantization”
Quantizing more layers can actually improve model fidelity via error cancellation
“It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematical proof where he had like a verifier in is that you can predict which …”
Dettmers: 4-bit precision is the end of quantization
“Four bit precision is the end of quantization.”
Chip Huyen: Quantization works universally well across tasks and models
“Quantizations, which is like very universally very working really well. For a lot of tasks across model.”
Cheah: Over-quantized LLMs rapidly enter repetition loops at long contexts
“When you over-quantize, right, at longer context length, right, it starts going into repetition rapidly.”