Rombach: Stable Diffusion's core technique was developed during university research
Robin Rombach · Text to Video: The Next Leap in AI Generation · Feb 17, 2024 · at 3:23
Robin Rombach, AI researcher at Stability AI and co-creator of Stable Diffusion, discusses the academic origins of the core diffusion architecture.
0:00 / 0:04exact quote · 4.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It's based on a technique that we developed while we were still at the university.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Robin Rombach
Rombach: Compute constraints drive AI innovation more than scaling hardware
“If you only rely on like more compute it's a bit boring. I think like compute constraints can also Drive innovation, right? So for example, the latent diffusion framework, we developed it at the university because we just, like, we had, like, single GPUs where…”
Rombach: Video AI models scaled like LLMs will gain world understanding
“Having something like we are seeing in language modeling, but trained on pixels on videos will probably give like super interesting downstream behavior to not, not only like generating videos, but also understanding of the world.”
Rombach: Pre-trained video models learn 3D synthesis faster than image models
“We showed that it's actually like helpful to incorporate like this implicit three D knowledge that Knowledge that is captured in all of the videos into the model, and then the model can learn much quicker than if you start from the pure image model.”
Rombach: AI video generation will evolve into real-time interactive experiences
“Because then this will become more like, I don't know, sometimes I think about this as like a video game, right? You type your prompt, and you immediately see what happens given your input view, and I think this might be a super nice user experience, actually.”