Blattman: Stable Video Diffusion acquired 3D reasoning within 2,000 iterations
Andreas Blattman · Text to Video: The Next Leap in AI Generation · Feb 17, 2024 · at 29:27
Andreas Blattman is an AI researcher at Stability AI. He discusses unexpected spatial reasoning capabilities discovered when fine-tuning Stable Video Diffusion.
0:00 / 0:16exact quote · 16.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We showed that by our three D fine tuning. This was, by the way, this was completely surprising for me seeing that model after a 1002 thousand iterations, like already getting what is like, three D reasoning or like explicit three D reasoning.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Andreas Blattman
Blattman: AI image progress depended on open-sourcing Stable Diffusion
“Open sourcing, like a foundation model as stable diffusion initially, that led to a whole lot of research on these models, which was, yeah, it was in, in retrospect, extremely important to do this. I think otherwise we wouldn't have seen the improvements we sa…”
Blattman: Video AI models must learn physics and 3D geometry
“I think video is, is like an awesome kind of data because you, to solve that task, to solve video generation, a model needs to Learn much about like physical properties of the world of like the physical foundations of the world that there is so much without kn…”
Blattman: High batch sizes are critical for training diffusion models
“So for diffusion models, it's really important to have a high batch size, because the gradients gets, like, you can approximate the gradient, which thrives the learning much better if the batch size is higher. And especially for diffusion models, it's like rea…”