Rombach: Pre-trained video models learn 3D synthesis faster than image models
Robin Rombach · Text to Video: The Next Leap in AI Generation · Feb 17, 2024 · at 23:07
Robin Rombach, co-creator of Stable Diffusion and researcher at Stability AI, discusses the training efficiency of Stable Video Diffusion when adapted for 3D multi-view object synthesis.
0:00 / 0:12exact quote · 12.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We showed that it's actually like helpful to incorporate like this implicit three D knowledge that Knowledge that is captured in all of the videos into the model, and then the model can learn much quicker than if you start from the pure image model.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Robin Rombach
Rombach: Compute constraints drive AI innovation more than scaling hardware
“If you only rely on like more compute it's a bit boring. I think like compute constraints can also Drive innovation, right? So for example, the latent diffusion framework, we developed it at the university because we just, like, we had, like, single GPUs where…”
Rombach: Video AI models scaled like LLMs will gain world understanding
“Having something like we are seeing in language modeling, but trained on pixels on videos will probably give like super interesting downstream behavior to not, not only like generating videos, but also understanding of the world.”
Rombach: AI video generation will evolve into real-time interactive experiences
“Because then this will become more like, I don't know, sometimes I think about this as like a video game, right? You type your prompt, and you immediately see what happens given your input view, and I think this might be a super nice user experience, actually.”
Rombach: DALL-E 3 uses an autoencoder trained on a single GPU
“Dolly three uses a model that, like, the autoencoder that was trained on a single GPU.”