Song: Video Models Unexpectedly Learn 3D Spatial Reasoning From Raw Video
Jiaming Song · Luma's Dream Machine and Reasoning in Video Models · Sep 9, 2024 · at 5:04
Jiaming Song, Chief Scientist at Luma AI, discusses how video foundation models implicitly develop 3D spatial understanding without explicit 3D priors.
0:00 / 0:11exact quote · 11.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It turns out from these type of videos, we are able to show that a video model is able to reason about three D quite well, which in some sense is unexpected before in, in the community.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jiaming Song
Song: Compute Scale Replaces Years of Explicit Graphics and Physics Engineering
“What's really surprising to me is how a large scale of compute is Mostly all you need to capture a lot of the, like, intricate effects that people spend years to develop in, like, graphics and, you know, physics simulation community”
Song: Dream Machine Learned Physical Simulation and Lighting Without 3D Priors
“But what's really surprising about Dream Machine is that we almost did nothing with regards to these NERF datasets or have zero, three D priors into how the model works. But the model just by itself learns to kind of uncover these interesting physical aspects …”
Song: Dream Machine Spontaneously Generates Consistent Cinematic Cuts Without Prompting
“Even though we haven't really asked the model to do anything about it, the model is able to reason about the second shot being a cut of the first shot that happens to have the same things in the first shot. So this is curse definitely some kind of a non-trivia…”
Song: Strict Physics Simulation Cannot Solve Fictional or Artistic Video Generation
“If we were trying to reason about the world via the traditional kind of physics techniques, it is very difficult to kind of reason about things inside a totally fictional world. So I think this is something that more happens like inside our dreams rather than …”
Song: Video AI Models Learn Psychological Causality Beyond Basic Physics
“And this is possibly caused by this eye being very unnaturally looking, and this is some kind of a cause and effect that is, like, even harder to reason, like, strictly in the physics, but more delving into how just human psychology works. So I think the causa…”