Autoregressive Models
topic on 2 shows · 9 statements across 7 episodes
9 statements about Autoregressive Models, every show
Wang: Autoregressive Models Type Sequentially, Diffusion Models Iteratively Edit
“What's the difference between autoregressive training versus diffusion language models is that it kind of, you can think of autoregressive training as typing. There is, for example, I like coffee, you have to type I, and then like, and coffee. There's inherent…”
Sanseviero: Text diffusion model quality is still worse than autoregressive models
“I think especially like the model quality is still a bit worse from what you would get from the normal autoregressive model.”
Swix: Reasoning Models Have Better Context Utilization Than Standard LLMs
“I have a theory also that reasoning models have better context utilization because they can loop back. Normal auto-aggressive models, they just kind of go left to right, but reasoning models, in theory, they can loop back and look for things that they needed c…”
Ermon: Diffusion LLMs gain speed by modifying multiple tokens in parallel
“That's kind of like the reason diffusion, diffusion language models are much faster compared to autoregressive models. Is that each neural network evaluation doesn't just give you one token, like in the typical autoregressive world, but it's able to Output, es…”
Ermon: Diffusion LLMs Pareto-dominate autoregressive models on inference efficiency
“On the inference side, what we're seeing is that diffusion models are much more efficient. We're actually able to Pareto dominate autoregressive models. If you think about the typical trade-off between throughput versus latency, which you kind of like cannot, …”
Ermon: Diffusion models naturally self-correct errors during generation unlike autoregressive LLMs
“The fact that you have error correction that is built in. So you think about an autoregressive model. Once you output something, you can never take it back. And so if you want to do, you know, if you want to fix mistakes, maybe you can do a reasoning chain. Ma…”
Masking out prior KV cache tokens pushes autoregressive transformers out of distribution
“The token number two in the KVCache is a contextualized version of the token zero, one, and two. So if you tell the model to only look at the last tokens you are creating an autoregressive model that is generating the logits of a P of let's say X, but only loo…”
Howard: Diffusion should be used to sketch answers before generating tokens
“The idea of, like, there should be a piece of the generative pipeline which is, like, thinking about the answer and coming up with a sketch of what the answer looks like before you start out putting tokens. That's where it kind of feels like diffusion ought to…”