World Models

“That immediately distinguishes us from other so-called world models, whether it's video models, vision models, they all assume during training and inference, it's a fixed resolution.”

“Most of the mode is not going to be technical unless you're a foundation model company or a world model. It's not technical.”

“And if you have perfect alignment between the real world and the world model, I mean, I think you've just sort of solved the universe roughly, right? That's an impossibly difficult problem. But as we make progress towards that it makes training physical AI mod…”

“So I think there'll be a whole industry that's built out where given the data sets that are available, a lot of it is visual data and the right world models that are built around it. You can train these machines and I'm using machine very broadly or robots muc…”

“The model alone, the foundation model alone, is, is not an AGI. No, you need the other guy, which uses all kinds of tricks to exploit The algorithmic information in the world model to come up with better plans.”

“Whenever the agent is, is making plans like that, it's thinking about itself, waking up these internal representations of itself, and then it's self-aware.”

“Unless I'm misunderstanding something, I don't think that they could entirely converge in principle because if an AI is generating exactly the same world that it exists in, The signal to noise ratio is, is non-existent. Like part of what makes us so flexible i…”

“I think that's some of the things the world models are gonna bring us as they begin to watch human behavior.”

“World models, don't get me wrong, are cool and everything, but you and I both know that that is a layer of abstraction that is sometimes not usefully precise enough.”

“A full world model must also be recursive, meaning that the participant in the world model must also be aware that they have a world model. Which is like this whole recursive thing down the line. But yes, and that the world model can be wrong, and that they ne…”

“So word model is like real time, interactive, long horizon videos.”

“So I believe the future is this world model type system, but it's not really well known yet. And there's not really any use cases that are publicly known.”

“These LLMs and even video models, they don't know how to do that. They don't have the ability to understand friction or weight or contact pressure friction, surface texture. Like they're just not able to do these things. And this is the core of what we need. A…”

“I think if you're hoping to do real world deploys and you're purely relying on a world model approach, you probably won't get to something that works before you go bankrupt.”

“There's a fifty-fifty chance there's some things maybe missing that we still need to make breakthroughs in, perhaps their world models”

“I don't think it's going to get replaced. I think it's going to get built on top of these foundation models, just like the way we do with our world models.”

“You only actually have a world model if you can predict, given some action is taken, what is going to change in the world because of that, and in particular that becomes hard over longer time scales, so if you're simply, you know, trying to predict the next vi…”

“I think it's a mixture. I mean, yeah. I mean, there's clearly a visual component of this and it's not that You know, everything can be text, because of course you want to give a visual look, but there's also a massive amount of giving the overall picture of th…”

“Depending on your end goal and purpose, the values should differ. So in the context of games, then the most direct way of measuring is how much time are people actually spending in this world that you create? And if your goal is to say, for example, in the con…”

“But it's a lot easier to change with code, as opposed to a model that is Learned primarily on data of real world and virtual worlds that are, I guess, like for example, junior, like there's actually trained on a lot of real world data and a lot of virtual gami…”

“I'll maybe start with where we see the platform in three years, which is like, okay, the users would tell us what they want to achieve. The end goal could be, hey, I just, I want to make something to teach my kids the value of humility. Or it could be, hey, I …”

“World models are absolutely essential when you want to build agents, because these agents are going to take actions, which is going to change the world. You want to be able to predict these effects.”

“With language models, when you get features, you still have to do auto interpret and things like that to actually get an understanding of what this concept is. But in image and video and world, it's like extremely easy to grok what the concept is because you c…”

“And I think that the way we would see the ultimate architectures of these platforms is Probably for a while, not an all in one thing. We would probably see some hyper efficient thousand person synchronization state engine.”

“In cosmos, we basically, we had all the pieces sitting around. We working on world models, we working on data analysis agent, working on literature agent. And then we're working on, you know, we built a platform for scientific agents. So we had things that can…”

“Your Git repo is like a distillation of all of the work that people put into the PRs, into the commits, and so I think there's a nice analogy between a Git repo and what a world model is.”

“And then, of course, that would be, I think, essential for a GI, because that would allow these systems to plan, long-term plan, in the real world over, perhaps, very long time horizons, which, of course, we as humans can do, you know I'll spend four years get…”

“I don't think about world models that often. I think because world models are just not really well defined in the first place.”

“My, like, take on world models is that, like, we have not yet defined, like, what a world model is.”

“I think one challenge that people have seen is that, like, world models perhaps designed for one specific use case might not generalize to others.”

“What world models do is they actually have to understand the full range of possibilities and outcomes from the current states and based on the action that you take generates the next states, right? So the next frame. And so it is a much more sort of complex pr…”

“Our world models have mouse sensitivity, which is something that, like, gamers absolutely want, right? So you can have these, like, very rapid movements, which you couldn't do in any other world model.”

“However, my understanding is they're currently not interactive, which in my opinion is like the whole point of world models, right? It's environments. They're great environments. And I think from a business perspective, I think they picked a really important p…”

“I think text prediction is just one of the actions that is going to come out of these, you know, these policies and world models. I think speech and text generation will just be one of the actions that, that can be a part of that.”

“I think that there will just be labs coming at this problem from both sides. And everyone ends up in roughly the same place, and the same place will be whatever people think is cool.”

“I think this is a matter of scaling data and bettering model. I don't think there's anything fundamental that separates these two.”

“I think sequence to sequence is actually in world models, I think we are going to see algorithm or architecture beyond sequence to sequence.”

“So I think the very description of LLM Needs to change. In fact, I think now we're talking about world models. I think world models are a more accurate description of what we're talking about.”

“I think you could say that on the surface, Sora, for example, does not look like it's AGI relevant, but I would bet that if we can build really great world models, that'll be much more important to AGI than people think.”

“It's just a matter of like data scale and like the underlying fundamental architectures. But like, I don't think it's going to change that much. We're just like, we're going to put thousand X more data, thousand X more compute, and we'll get like the best phys…”

“I still think, honestly, for my, what I'm excited about for AGI, and which is more embodied agents I really believe this is the fastest path to getting these agents, like, in the real world”

“You might train it on demonstration data of successful data of completing the task, and then evaluate it on to try to actually use it to evaluate actions that are not optimally completing the task, and then the world model will hallucinate a video of completin…”

“Constructing world model might be a little more nuanced. The world is more structured. There might be signals that we need to use to guide it. You can call it in the shape of prior. You can call it supervision in your data, whatever it is.”

“I think it's pretty clear that as these models get bigger, they have a world model, and that world model becomes better with scale. So they are implicitly developing a world model, and I don't think it's something that you need to explicitly model.”

“To create world models, three-D foundation models we require More and more sophisticated data engineering, data acquisition, data processing, and data synthesis. So I am envious of my NLP LLM colleagues that the data is so abundant on the internet, and we don'…”

“I was at that point already talking to various people, not just computer scientists, technologists, but also investors and potentially business partners, and to be honest, most people didn't get it.”

“I actually don't know that I believe that any of the sort of more immediate term commercial applications of these models are interesting. People are like, oh, we can generate games or we'll have like gaming assets or we'll use the data for robotics training or…”

“Is the right way to approach this problem to train better and better video generation systems? And my answer to this is absolutely no. The problem of understanding the world does not go through the solution to the, to generating video at the pixel level.”

“Even if you only have a one percent error in what the model's telling you, that's going to compound over a hundred steps to the point where you'll be in a, you know, you'll kind of get almost a random answer. And so that makes the planning very difficult.”

“So I think that's the next big step actually for agent based systems is to go beyond world models. Can you collect enough data where the agents are also acting in the world and making plans and achieving tasks? And I think for that you will need Ah, ah, not ju…”

“Having deeper models of how, let's see, what are the underlying structures of different domains, how they're related or not related, I think will be an important ingredient for models actually being able to make novel contributions.”

“We've done a ton of work on AI on the platform over the last few years. That was what I would call more narrow specialized models. We can continue to do that, but the bigger step change are gonna come through the more general model, building foundation models,…”

“If you think about predicting the next word, It's, it seems very simple, but you have to really internalize a lot of what is going on in this context. What are the previous words? What's the syntax? What's who's saying them? And all of that information and con…”

“So I think this is not going to be solved in my opinion by you know, more tweaks on the architectures and more training data and things like this. I mean, it may be mitigated, but I think it's not going to be fixed by that. I think it's going to be fixed by sy…”