Large Step Sizes in Discrete ODE Solvers Cause Trajectory Errors
RJ Honicky · [Paper Club] Intro to Diffusion Models and OpenAI sCM: Simple, Stable, Scalable Consistency Models · Nov 2, 2024 · at 25:30
RJ Honicky presents a technical breakdown of OpenAI's Simple, Stable, Scalable Consistency Models (sCM) paper, contrasting discrete ODE sampling errors with continuous-time formulations.
0:00 / 0:13exact quote · 13.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So if this Delta T here is very big, you see it goes, like, far from XT to X minus Delta T, then the error that It can have is very big and that can put it on a different trajectory. So you get the wrong trajectory.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from RJ Honicky
Inconsistent Diffusion Trajectories Drive the Need for Consistency Models
“The locations that you're learning are not basically on the same in the, in this latent space. They're not in the same trajectory
As each other, right? So they like and this causes a lot of inefficiency, and that's sort of the whole point to this, the, these …”
Major Image Models Have Not Yet Adopted Consistency Models
“None of this technology that we're discussing today is in any of really in any of the big models that we know and love with maybe of maybe with the exception of flux.”
Isolating Tangent Function Instability is OpenAI's Core Contribution in sCM
“And this is, I, in my opinion, the meat of the paper. So you have this, part of the, you have this tangent function that I had called out.”
GANs Outperform Diffusion on Benchmarks but Are Abandoned for Mode-Seeking
“The reason why people don't use them is because they're hard to train and they're, they have, they're very, they have mode seeking behavior, meaning it's hard to get any diversity and hard to control them. But for these benchmarks, they do the best.”
Consistency Models Generate High-Quality Images in a Single Pass
“With the regular diffusion models that we all know and love, they have to iterate multiple times generally to generate a good image. Whereas The, these consistency models are designed so they can generate a good image with only one pass through the network.”
Cosine Noise Schedules Eliminate Wasted Backward Steps in Diffusion Models
“They found in this paper that it's actually inefficient, that you end up
with a lot of wasted a way wasted backwards process backwards diffusion steps that you don't need and you can cut out a whole bunch of it just by using a cosine schedule.”