Luma AI

“If we were trying to reason about the world via the traditional kind of physics techniques, it is very difficult to kind of reason about things inside a totally fictional world. So I think this is something that more happens like inside our dreams rather than …”

“Even though we haven't really asked the model to do anything about it, the model is able to reason about the second shot being a cut of the first shot that happens to have the same things in the first shot. So this is curse definitely some kind of a non-trivia…”

“It turns out from these type of videos, we are able to show that a video model is able to reason about three D quite well, which in some sense is unexpected before in, in the community.”

“The limitations of images was that it wasn't able to reason about how You know, the camera works in the world because it only has like a relatively independent shots of different objects.”

“So I think that tells me that Dream Machine is definitely able to reason about better than any of the models that we've, you know, worked with before, and this is very much unlike, like, even the models that you try to obtain by fine-tuning on the, you know, l…”

“For example, in this case, what we do is we literally took one of the videos from the last side and we put this video into our three D reconstruction pipeline. And it turns out that it is able to reconstruct a three D scene at this direction quite reasonably w…”

“And this is possibly caused by this eye being very unnaturally looking, and this is some kind of a cause and effect that is, like, even harder to reason, like, strictly in the physics, but more delving into how just human psychology works. So I think the causa…”

“We put the first frame of an image into Dream Machine, and then Dream Machine will give us the output as a video. So as you can see here, like, the three-d consistency of the generated video looks quite amazing”

“Currently. No, we are only at a very, very you know, basic stage at, you know, being able to like generate videos from text and images at this stage. I will say this is more like a, as you said, research preview or version zero of the model.”

“So I think the other interesting thing that we showed a slightly earlier, but also want to kind of reemphasize here is how dream machine is able to reason about the non-physical world as well. Even in cases where it's in entirely a scene of art, it is able to,…”

“But what's really surprising about Dream Machine is that we almost did nothing with regards to these NERF datasets or have zero, three D priors into how the model works. But the model just by itself learns to kind of uncover these interesting physical aspects …”