Song: Dream Machine Spontaneously Generates Consistent Cinematic Cuts Without Prompting
Jiaming Song · Luma's Dream Machine and Reasoning in Video Models · Sep 9, 2024 · at 21:41
Jiaming Song is Chief Scientist at Luma AI. He discusses emergent camera cut generation and causal reasoning capabilities in Luma's Dream Machine video generation model during an interview with a16z General Partner Anjney Midha.
0:00 / 0:20exact quote · 20.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Even though we haven't really asked the model to do anything about it, the model is able to reason about the second shot being a cut of the first shot that happens to have the same things in the first shot. So this is curse definitely some kind of a non-trivial cause and effect that happens like even in a not, not entirely like physically possible scenario.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jiaming Song
Song: Compute Scale Replaces Years of Explicit Graphics and Physics Engineering
“What's really surprising to me is how a large scale of compute is Mostly all you need to capture a lot of the, like, intricate effects that people spend years to develop in, like, graphics and, you know, physics simulation community”
Song: Dream Machine Learned Physical Simulation and Lighting Without 3D Priors
“But what's really surprising about Dream Machine is that we almost did nothing with regards to these NERF datasets or have zero, three D priors into how the model works. But the model just by itself learns to kind of uncover these interesting physical aspects …”
Song: Strict Physics Simulation Cannot Solve Fictional or Artistic Video Generation
“If we were trying to reason about the world via the traditional kind of physics techniques, it is very difficult to kind of reason about things inside a totally fictional world. So I think this is something that more happens like inside our dreams rather than …”
Song: Video AI Models Learn Psychological Causality Beyond Basic Physics
“And this is possibly caused by this eye being very unnaturally looking, and this is some kind of a cause and effect that is, like, even harder to reason, like, strictly in the physics, but more delving into how just human psychology works. So I think the causa…”
Song: AI Video Models Will Evolve Into 4D Multi-Angle World Simulators
“I don't think it's a much of a stretch to say maybe we can get from like videos to four D. So that's being able to do world simulators meaning, meaning that you might be able to simulate multiple angles at the same time.”
Song: 3D AI Models Face Severe Data Scalability Bottleneck Compared to 2D
“Three D data has this scalability issue, and you If you compare that with like images, everyone can, you know, use their cell phone to take a photo or, you know, take a video. Whereas if you try to do the same with three D is very difficult. You either have to…”