autoregressive models

“The idea of, like, there should be a piece of the generative pipeline which is, like, thinking about the answer and coming up with a sketch of what the answer looks like before you start out putting tokens. That's where it kind of feels like diffusion ought to…”

“The token number two in the KVCache is a contextualized version of the token zero, one, and two. So if you tell the model to only look at the last tokens you are creating an autoregressive model that is generating the logits of a P of let's say X, but only loo…”

“That's kind of like the reason diffusion, diffusion language models are much faster compared to autoregressive models. Is that each neural network evaluation doesn't just give you one token, like in the typical autoregressive world, but it's able to Output, es…”

“On the inference side, what we're seeing is that diffusion models are much more efficient. We're actually able to Pareto dominate autoregressive models. If you think about the typical trade-off between throughput versus latency, which you kind of like cannot, …”

“The fact that you have error correction that is built in. So you think about an autoregressive model. Once you output something, you can never take it back. And so if you want to do, you know, if you want to fix mistakes, maybe you can do a reasoning chain. Ma…”

“I have a theory also that reasoning models have better context utilization because they can loop back. Normal auto-aggressive models, they just kind of go left to right, but reasoning models, in theory, they can loop back and look for things that they needed c…”

“I think especially like the model quality is still a bit worse from what you would get from the normal autoregressive model.”

“What's the difference between autoregressive training versus diffusion language models is that it kind of, you can think of autoregressive training as typing. There is, for example, I like coffee, you have to type I, and then like, and coffee. There's inherent…”