The Exchanges

Q talk about the progression of AI models becoming more efficient? I know it's like a hot topic right now, but it does seem to me that over the past couple of years, we've definitely seen models become more and more efficient. So what can you tell us about this? We'll just talk about large language models on this front. Um, the efficiency gains that we've seen over time with them.

A I mean, this is, this isn't new. This has been happening. Uh, for the past 1012 years or so, essentially since we first, um, discovered deep learning on our GPUs with AlexNet. Um, if you look at the, the, uh, computational curve, what our GPUs can do, um, in terms of, uh, tensor operations, the AI kind of math that we Need to do. Over the last 10 years, we've had essentially a million x performance increase. And that increase isn't, isn't just from the raw hardware. It's also through, through many layers of the software algorithms. So we're getting these, these benefits, these speed ups continuously at a very rapid rate exponentially by compounding many, um, many layers at, uh, all the different layers at which, uh, this computing happens from the fundamental Hardware, the chips themselves at systems level, networking, system software, algorithms, frameworks, and so on. Um, so what, what we've seen here with, with DeepSeq is, is a great advancement that's on that same curve that we've been on for, for a decade now.

AI assessment note: “Over the last 10 years, we've had essentially a million x performance increase.”

Q So it's interesting. It's not just basically inputting that real world knowledge into LLMs, right? So they can get the question about dropping the paper with the hand. Correct. It is also something that you're working on is building the foundation for robots to go out into our world and operate within it.

A So yes, it's not, it's not inputting it in the same way that we do for these text models. We're not just going to describe, uh, with words, how, what happens when you drop a piece of paper. We're going to give these models, uh, other senses. During the learning process. So they'll, they'll watch, um, watch videos of, of paper dropping. We can also give it, uh, more, more accurate, specific information in the three D realm. Uh, because we can simulate These physical worlds inside a computer today, we have physics simulations of worlds. We can pull ground truth data about, about the position and orientation and, and, uh, state of things inside that three D world and use that as another mode of input into these models. And so what we'll end up with is a, a world foundation model that was trained on many different modes of data, essentially different sense, senses. It can see, it can hear, It can, um, touch and feel and do, do many of the things we can do, or many things other animals or, or even things no, no creature can do, because we can provide it with sensors that don't exist, uh, inside, inside the natural world. And, uh, it can, from that, kind of decipher what are the actual combined rules of, of, of the world, and this, this, um, encoding of the knowledge of how the physical world works can then be the basis for us to build agents inside the real world, to build the brain…

AI assessment note: “encoding of the knowledge of how the physical world works can then be the basis”

Q to be surprised. Maybe not our listeners, but a sizeable portion of the population that would be surprised to hear that Nvidia itself is building these foundation, these world foundational models, releasing weights to help others build on top of them. The perception I think from, uh, some on the outside is, Hey, isn't Nvidia just the company that makes those chips? So what do you say to that, Rev?

A Well, yeah, that's, that's been the perception. It's been the perception since I started InVideo 23 years ago, and it's never been true that we just build chips. Chips are a very, very important part of what we do. Uh, they're the foundation that we build on. But when I joined the company, there were about a thousand people, a thousand employees at the time. The grand majority of them are, were engineers just like today. The majority of our employees are engineers, and the majority of those engineers are software engineers. I myself am a software engineer. I, I, I wouldn't know the first thing about making a chip. And so our form of computing, um, accelerated computing, the form of computing we invented, Is a full stack problem. It's not just a chip. Uh, it's not just a chip that we throw over the fence and leave it to others to figure out how to make use of it. It doesn't work unless we have these layers of software and these layers of software, um, have to have algorithms that, that are harmonized with the architecture of our, of our chips and our systems. Uh, so we, we have to, uh, go in these new markets that we enter, what Jensen calls zero billion dollar industries. We have to actually go invent, uh, these new things kind of top to bottom because they don't exist yet, and nobody else is going to likely to do it. Um, so we build a lot of software and we build, uh, a lot of…

AI assessment note: “it's never been true that we just build chips.”

Q Somewhat real. But I'm, I'm curious, like, do you think that, like, is Hollywood gonna move to a area where it's super real and just simulated? Go ahead.

A Absolutely. I mean, well, um, was it a year or two ago when the last Planet of the Apes came out? I went to go see it with my wife. Now, my wife, Uh, and I have been together since I worked at Disney in the mid nineties, working on visual effects and rendering. We, I had a, a startup company doing rendering and she was a part of that. So she, she has a good eye and she, she's been around computer graphics and rendering for decades now. When we went to go see Planet of the Apes, even though obviously those apes were not real, at one point she turned around and said, That's all CG, right? She couldn't quite believe it. I think what Weta did there is, is amazing. It's indistinguishable from real life, except for the fact that the apes are talking, like other than that, it's indistinguishable. Um, the, the problem with, uh, with that though is to do that level of CG in the traditional way that we've done it requires an incredible amount of artistry and, and skills, uh, that only, only a few studios in the world can do with the teams that they have and the, uh, pipelines they've built, and it's incredibly expensive to produce that. What we're building with AI, with generative AI, and particularly with world foundation models, that once we get to the point where they really understand, ah, the depth of the, the physics that they need to, to produce something like Planet of the Apes, …

AI assessment note: “of course they're gonna use. Those technologies to produce the same images”

Q And so who's going to use it? Is it going to be robotics developers? Is it going to be somebody that's building, let's say LLM based application, but just wants them to be a little smarter? Both?

A It will be all of them. Yes. It's, uh, we, We feel that we're as a, as, as the industry, the world is right at the beginnings of this physical AI revolution, and no one company, no one organization is going to be able to build everything that's, that, that we need. So, so we're building it out there in the open, uh, to encourage others to come build on top of what we've built and come build it with us. And this is gonna be, ah, essentially anybody that has an application that involves the physical world. And so that's definitely robotics companies are part of this and, and robotics in the very general sense that includes self-driving car companies, robo taxi companies, and as well as, uh, companies building robots that are in our factories and warehouses. Anybody that wants to make intelligent robots that have perception and operate autonomously inside the real world, they want this, but, um, it's not, It's not only about robots in the way we think about them as, as these agents that move around. Uh, we have sensors that we're placing in our spaces, in, in, in our cities, in urban environments, inside buildings. Uh, these sensors, uh, need to understand what's happening in that world. Maybe for security reasons, for, for coordinating, um, other robots, changing the climate and, And, uh, energy efficiency of, of our buildings and data centers. So there's, there's many applicatio…

AI assessment note: “It will be all of them. Yes.”

Q DeepMind, who was just on, who commented on this, talk about how these video generators have been surprisingly good at understanding physics, and Jan also, basically in our conversations previously, Effectively saying that it's very difficult for AI to solve these problems. I won't say they've solved it, but everybody's surprised they've gotten to this point. So what is your best understanding of how they've been, though flawed, this good?

A You know, this is the, uh, uh, trillion dollar question, I guess, you know, we've been, We've been betting now for years that if we just throw more compute and more data at, at the problem, that these scaling laws are going to give us a level of intelligence, uh, that's really, really meaningful. That, that will be like step function changes in, in, uh, capabilities. There's no way for us to know for sure. It's very hard to predict that. It feels like we're on an X. We are on an exponential curve with this, but which, um, uh, part of the exponential curve we're on, we, we can't tell. So we don't know how fast that's going to happen. Honestly, uh, I'm, I've been surprised. At how, how well these transformer models have been able to extract the laws of physics at this, to this level by this point in time. Uh, I have. At, at this point, I believe in a few years, we're going to get to a level of a physics understanding with our AIs that are, that's going to unlock, you know, the majority of the applications we need, we need to apply them in, in robotics.

AI assessment note: “if we just throw more compute and more data at, at the problem”