Diffusion Models

“For all the same reasons that those models were really tricky to train for images. Mode collapse was the most famous sort of problem. They didn't work very well for proteins or protein ligand systems. And we sort of had to wait for the right primitive to get c…”

“And here credit to my co-founder, Piotr, who effectively came with that new idea of how you can now create voice models, which are both reliable, high quality, quick, where you would bring a lot of the ideas from transformer models, from diffusion models into …”

“It turns out that the mathematics that we use for diffusion models, but even for reinforcement learning, for Schrodinger bridges, for MCMC sampling, has the same mathematics as this theory, this physical theory of non-equilibrium Systems.”

“So it is a interesting direction to go into these diffusion, diffusion models as alternative to the auto regressive transformers, but it is not I would say the replacement at the state of the art.”

“I think one company will launch a big diffusion model this year.”

“And like, there's enough capital out there to test them, be they like diffusion or SSMs or whatever. And that's going to happen this next year.”

“Yeah, I think you're totally right. It's an anti-pattern. It's pretty tough to pull off.”

“Diffusion models today are, I would say, 80 to 90% of the quality at one-tenth the cost and latency.”

“Left to right reasoning for code doesn't actually really make sense, because in code, we don't, like, we might sometimes write code left to right, but after you write code, you go up and down and figure out, hey, is this variable set? Did I do this? There are …”

“But the thing that people on track is it's going to be LMS and diffusion models. And I think other things with a fabric across them.”

“So, so the diffusion markets are all working. So any area where you bring the marginal cost of creating something, a piece of content to zero is clearly working and creating an image, creating music, you know, creating speech.”

“The challenge is that, yeah, the training objective is quite different because you are training based on denoising as opposed to next token prediction. Diffusion models are not causal and that is also kind of problematic. I mean, it's a big advantage of diffus…”

“Well, you can use architectures. I think that at least the shapes you, that, that can be leveraged. So you don't have to reinvent and necessarily completely different neural network architectures can, a lot of the data can be used. Like I think perhaps there a…”

“Diffusion models. Not surprisingly, they work pretty well at the infilling where you really need to be able to use context to the left and to the right.”

“On the inference side, what we're seeing is that diffusion models are much more efficient. We're actually able to Pareto dominate autoregressive models. If you think about the typical trade-off between throughput versus latency, which you kind of like cannot, …”

“I'm pretty optimistic about a future where diffusion models Can become the dominant solution. I've seen it happen before with GANs a few years ago, so I wouldn't be surprised if that's the case also here.”

“If it happens, it's gonna be driven by efficiency. Like we're all constrained by essentially power. And if you have, I mean, at the end of the day, it's all an inference game, right? Okay. Training is expensive, but then the thing that matters is being able to…”

“We are also working, by the way, on kind of ideas that make the audio generation part diffusion.”

“It's, I think maybe the reason why these models are not great design yet, it's like on the art side you've got diffusion, on the problem solving side you've got LMs, and it's not clear that people have figured out how to, you know, marry techniques together ye…”

“With the regular diffusion models that we all know and love, they have to iterate multiple times generally to generate a good image. Whereas The, these consistency models are designed so they can generate a good image with only one pass through the network.”

“They found in this paper that it's actually inefficient, that you end up with a lot of wasted a way wasted backwards process backwards diffusion steps that you don't need and you can cut out a whole bunch of it just by using a cosine schedule.”

“The locations that you're learning are not basically on the same in the, in this latent space. They're not in the same trajectory As each other, right? So they like and this causes a lot of inefficiency, and that's sort of the whole point to this, the, these …”

“And then this probability flow ODE is sort of a deterministic version that looks at what is the maximum likelihood path If I started at that trajectory, right?”

“So if this Delta T here is very big, you see it goes, like, far from XT to X minus Delta T, then the error that It can have is very big and that can put it on a different trajectory. So you get the wrong trajectory.”

“The idea of, like, there should be a piece of the generative pipeline which is, like, thinking about the answer and coming up with a sketch of what the answer looks like before you start out putting tokens. That's where it kind of feels like diffusion ought to…”

“If you typically use any of the diffusion models out there are two large problems for large enterprises. The first one being it's trained on no one knows which data. So even the companies don't know on which data. So it's like for Austin, it doesn't make sense…”

“As far as I know, all the best text models in the world are still autoregressive models and the best image and video models are diffusion models.”

“When do you want like sort of an LM versus diffusion model solution for something and design is maybe you can define it as like art applied to problem solving. there's many different definitions of design. but I love that one. It's one that I've been thinkin…”

“There's no way that having to remember all these different sort of like magical phrases to summon the right image via diffusion model Is the end state, right? And yeah, maybe there's something where you draw some shapes and you add a prompt, but like, that sti…”

“So for diffusion models, it's really important to have a high batch size, because the gradients gets, like, you can approximate the gradient, which thrives the learning much better if the batch size is higher. And especially for diffusion models, it's like rea…”

“I think there's a couple companies who make, Tens of millions of dollars of revenue from, yeah, from LLMs or diffusion models for porn”

“And so you can actually imagine that in the language world, you're going to have a lot more platforms that people build on in the diffusion model world image, video, et cetera, you're going to have more people kind of grow their own.”

“One outcome might be that we find more success with models that are similar to CodePilot than you do with like a diffusion model, for example, when you're trying to figure out how do you actually create designs using AI systems.”