Song: NeRF and Gaussian Splatting Suffer Major Real-World Capture Limitations
Jiaming Song · Luma's Dream Machine and Reasoning in Video Models · Sep 9, 2024 · at 8:00
Jiaming Song, Chief Scientist at Luma AI, explains why popular 3D reconstruction techniques like NeRFs and Gaussian Splatting struggle when deployed to everyday users without specialized capture hardware.
0:00 / 0:31exact quote · 31.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“When you try to develop, like, deploy these techniques in, in, in the wild, there are many issues that comes with this imperfect capture, like, capturing system that comes along. Like, people, when they're trying to capture an object, will oftentimes not capture the object fully, through 60 degrees. There might be motion blur associated with the camera. The object might be moving, so it will It's better to have them, like, perfectly still. So all of these are limitations that we were trying to solve, like, before we even started trying to develop the dream machine.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jiaming Song
Song: Compute Scale Replaces Years of Explicit Graphics and Physics Engineering
“What's really surprising to me is how a large scale of compute is Mostly all you need to capture a lot of the, like, intricate effects that people spend years to develop in, like, graphics and, you know, physics simulation community”
Song: Dream Machine Learned Physical Simulation and Lighting Without 3D Priors
“But what's really surprising about Dream Machine is that we almost did nothing with regards to these NERF datasets or have zero, three D priors into how the model works. But the model just by itself learns to kind of uncover these interesting physical aspects …”
Song: Dream Machine Spontaneously Generates Consistent Cinematic Cuts Without Prompting
“Even though we haven't really asked the model to do anything about it, the model is able to reason about the second shot being a cut of the first shot that happens to have the same things in the first shot. So this is curse definitely some kind of a non-trivia…”
Song: Strict Physics Simulation Cannot Solve Fictional or Artistic Video Generation
“If we were trying to reason about the world via the traditional kind of physics techniques, it is very difficult to kind of reason about things inside a totally fictional world. So I think this is something that more happens like inside our dreams rather than …”
Song: Video AI Models Learn Psychological Causality Beyond Basic Physics
“And this is possibly caused by this eye being very unnaturally looking, and this is some kind of a cause and effect that is, like, even harder to reason, like, strictly in the physics, but more delving into how just human psychology works. So I think the causa…”