quantization

4 statements across 3 episodes · 1 bullish · 0 bearish · 4 people on the record · first statement Nov 3, 2023 by Michael Royzen · across every show →

Everything said about quantization, oldest first

Nov 3, 2023 neutral

Assertion Supported

Royzen: INT8 quantization offers storage optimization without guaranteed inference speedups

“But with int eight, there's not necessarily a Speed increase. It's just the storage optimization.”

Michael Royzen Nov 3, 2023 ▶ 1:05:46 Beating GPT-4 with Open Source Models - with Michael Royzen of Phind

Jul 29, 2024

Insight

Cheah: Over-quantized LLMs rapidly enter repetition loops at long contexts

“When you over-quantize, right, at longer context length, right, it starts going into repetition rapidly.”

Eugene Cheah Jul 29, 2024 ▶ 1:03:20 [LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Aug 3, 2026 positive

Insight

Quantizing more layers can actually improve model fidelity via error cancellation

“It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematical proof where he had like a verifier in is that you can predict which …”

Ali Taha Aug 3, 2026 ▶ 31:54 Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

Aug 3, 2026

Insight

Most inference optimizations like KV caching are lossless, unlike quantization

“Most inference optimizations are lossless. KV caching, for example, you are just recomputing or preventing recomputing the same values. Speculation, of course, If a draft token is wrong, it gets rejected. The main lossy optimization is quantization”

Philip Kiely Aug 3, 2026 ▶ 26:55 Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

quantization

Everything said about quantization, oldest first

Turn any episode into a week of clips.