Feb 17, 2024 · 39m · a16z
Text to Video: The Next Leap in AI Generation
Robin Rombach · 13m spoken Andreas Blattman · 10m spoken Anshin Mita · 9m spoken Anjney Midha · 2m spoken
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the a16z podcast, host Anshin Mita interviews Stability AI researchers Robin Rombach and Andreas Blattman on the technical breakthroughs, open-source impact, and physical modeling challenges behind Stable Video Diffusion.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The host as informed peer 3.3 Guest teaching 4.6 Guest disagreement 0.2 The host pushing back 0.0
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 27:09 Gentle reframe on LoRA library scalability
Robin gently counters the host premise that directors will rely on hundreds of LoRAs, explaining that managing such libraries is not a scalable creation approach.
Hardest push from the host ▶ 13:00 Host reframes discussion through physics perspectiveThe host actively reframes the discussion around the guests academic backgrounds in physics and engineering to pivot into 3D world modeling.
Biggest teaching moment ▶ 15:45 Detailed breakdown of video training hardware bottlenecksAndreas educates the host on how CPU decoding speeds unexpectedly became a critical bottleneck during high-resolution multi-GPU video model training.
The host holds their own ▶ 34:00 Host connects guests constrained history to broader AI ecosystemThe host demonstrates deep domain awareness by highlighting how the guests early single-GPU constraints inspired broader university research groups.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| Legal Disclaimers and Compliance Information | 2 | 3 | 0 | 0 | The host opens with high-level introductory questions regarding Stable Diffusion and Stable Video Diffusion. Guest Robin Rombach provides a brief technical baseline explaining diffusion models and university origins. | |
| Diffusion Models vs Autoregressive Models and Distillation | 3 | 6 | 0 | 0 | The host asks open-ended clarifying questions about diffusion versus autoregressive models. Guests educate the host on noise transformation versus sequence generation, as well as single-step distillation. | |
| Evolution of Image Models and Open Source | 3 | 5 | 1 | 0 | The host synthesizes the growth of open-source Lego block developments. Guests share early development details regarding latent diffusion training constraints and classifier-free guidance. | |
| Physics, 3D Understanding, and Video Modeling | 5 | 5 | 0 | 0 | The host demonstrates insight by linking the guests physics and engineering backgrounds to implicit 3D world modeling in video systems. Guests fully agree and expand on world simulation. | |
| Data Pipelines and Training Infrastructure for Video | 3 | 6 | 0 | 0 | The host prompts the guests on hardware and infrastructure differences between image and video models. Guests detailed CPU codec bottlenecks, batch size requirements, and cluster failure risks. | |
| Three-Stage Training and Multi-View 3D Synthesis | 4 | 6 | 0 | 0 | The host raises the problem of 3D structural consistency in video generation. Guest Robin Rombach breaks down the three-stage training process and fine-tuning on multi-view orbit data. | |
| Fine-Grained Control with LoRAs and Future Video Editing | 4 | 5 | 1 | 0 | The host queries whether future video production will rely on managing hundreds of LoRAs. Robin gently reframes this, pointing out that managing extensive LoRA libraries is unscalable compared to direct text and motion prompts. | |
| Community Exploration and Favorite Generated Memes | 2 | 3 | 0 | 0 | A brief lighthearted segment where the host asks about community creations. Guests share favorite examples such as animated memes and classical artwork. | |
| Future Frontiers, Real-Time Inference, and Compute Constraints | 4 | 5 | 0 | 0 | The host asks about hardware wishlists and research bottlenecks. Guests explain how compute constraints drive architectural breakthroughs, which the host summarizes as no constraints, no creativity. | |
| Research Philosophy, Open Source Impact, and Conclusion | 3 | 2 | 0 | 0 | The host asks how a compact team competes with major tech monopolies. Guests express their passion for open-source research, and the host points out OpenAI citing their research in DALL-E 3. |
Statements from this episode (13)
Rombach: Stable Diffusion's core technique was developed during university research
“It's based on a technique that we developed while we were still at the university.”
Rombach: Stability AI distillation research enables single-step diffusion generation
“We ourselves, we have published a distillation work a week ago that actually shows that you can go as low as one sampling step, which is, I would say like a big advantage of these diffusion models.”
Blattman: AI image progress depended on open-sourcing Stable Diffusion
“Open sourcing, like a foundation model as stable diffusion initially, that led to a whole lot of research on these models, which was, yeah, it was in, in retrospect, extremely important to do this. I think otherwise we wouldn't have seen the improvements we sa…”
Blattman: Video AI models must learn physics and 3D geometry
“I think video is, is like an awesome kind of data because you, to solve that task, to solve video generation, a model needs to Learn much about like physical properties of the world of like the physical foundations of the world that there is so much without kn…”
Rombach: Video AI models scaled like LLMs will gain world understanding
“Having something like we are seeing in language modeling, but trained on pixels on videos will probably give like super interesting downstream behavior to not, not only like generating videos, but also understanding of the world.”
Rombach: Stable Video Diffusion took six months to develop
“I would say roughly half a year and like for this model that we just put out, I think the main challenge was that we actually, Yeah, I had to scale the data set and the data loading.”
Blattman: High batch sizes are critical for training diffusion models
“So for diffusion models, it's really important to have a high batch size, because the gradients gets, like, you can approximate the gradient, which thrives the learning much better if the batch size is higher. And especially for diffusion models, it's like rea…”
Rombach: Pre-trained video models learn 3D synthesis faster than image models
“We showed that it's actually like helpful to incorporate like this implicit three D knowledge that Knowledge that is captured in all of the videos into the model, and then the model can learn much quicker than if you start from the pure image model.”
Rombach: Managing hundreds of LoRAs is unscalable for video control
“Maintaining like a library of hundreds of LoRa's is maybe not like the, Most scalable approach.”
Rombach: AI video generation will evolve into real-time interactive experiences
“Because then this will become more like, I don't know, sometimes I think about this as like a video game, right? You type your prompt, and you immediately see what happens given your input view, and I think this might be a super nice user experience, actually.”
Blattman: Stable Video Diffusion acquired 3D reasoning within 2,000 iterations
“We showed that by our three D fine tuning. This was, by the way, this was completely surprising for me seeing that model after a 1002 thousand iterations, like already getting what is like, three D reasoning or like explicit three D reasoning.”
Rombach: Compute constraints drive AI innovation more than scaling hardware
“If you only rely on like more compute it's a bit boring. I think like compute constraints can also Drive innovation, right? So for example, the latent diffusion framework, we developed it at the university because we just, like, we had, like, single GPUs where…”