Physical Intelligence

“He's one of the co-founders of Physical Intelligence, which we think might be the robotics AI lab that brings about the GPT-I moment for all of robotics.”

“Our mission is to build a model that can control any robot, to do any task that is physically capable of, and to do so as such a high level of performance that's going to be useful to people in all walks of life.”

“We think that it's going to be more like a peeling an audience analogy, where you start from a really strong base model that have all sorts of common sense knowledge and already works to some extent on your robot, and you have then a Mixed autonomy system. Ver…”

“You can compare it to the specialist that has been optimized to work well on a particular embodiment. How does it compare? And the interesting result from OpenX is it was 50% better.”

“Let's say if we actually solve robotics, a model that can control any robot to do any task, napkin math maybe contribute 10% to US GDP. Well, that's already a massive number and I think that promise is one of the reasons that warrants the investment into data …”

“And cross embodiment, there is the data collection aspect of, as well, which is to really make sure that your model and your organizations and infrastructure are set up to consume data From many different sources of robots, and that actually allows you to scal…”

“And the argument is that, you know, single robot is simpler to scale. And actually that's not how it plays out in practice. Like how it plays out in practice is even if you have a single robot that you're optimizing for, over time that Platform is going to dri…”

“Today it's possible to perform tasks Zero shot. Zero shot meaning you don't collect any data. And these are the tasks that last year might have required like hundreds and hundreds of hours.”

“No two items of clothing here are the same, and these are also unseen. You know, these are not, like, clothing items that are seen in the training data.”

“And it actually didn't even take us that long to get this result. It was roughly where we set a goal and maybe it was like two weeks afterwards where we got, got a model that was, got a model and a system that was good enough at performing this task.”

“This is packaging real customer real order for customer to be shipped out in a real warehouse. So this is real operations.”

“The interesting thing about the approach is that you're converting it from a very difficult engineering problem into a operation problems of how do I identify the use case, and how do I collect the right data, which is, in some sense, more scalable, because yo…”

“Almost all of the robot evaluation that we run at Pi today, including the really Complicated demo that we have shown making coffee, folding laundry, mobile robots navigating around. The model actually hosted in the cloud.”

“And the, one other interesting thing about our collaboration with Weave and Ultra is one, I've never seen their robot in person. Two is I have very little idea about how their robot actually works. And that's a very intentional choice. I want to stay away from…”

“And when we started the company, we thought that real deployment is going to be a con, it's only going to be in a conversation like five years. Into the life of the company, because the problem is just really hard, and we're two years in, and, you know, this i…”

“You don't need a incredibly expensive robot that is capable of very precise motion today to be able to do this task. And the reason why is this model really reactive? And so they can compensate for some of the inaccuracy in the actual robot movement”

“We open source PI zero and PI zero five. And people also shocked when they asked me, you know, is there any difference between PI zero and PI zero five that you open source versus the model that we use internally PI zero and PI zero five? And the answer was, I…”

“I think this is another area of incredible opportunity of kind of building services for robot company. Like, you know, if you can offer remote tele-op, for example, if you can offer data collections, if you can offer annotation service, because, you know, thes…”

“Evaluation is a really hard problem in robotics because it scales super linearly to model capability. Like, let's say you have a model that can perform a two-minute task. Running evaluation for that is very different from running evaluation for a task that's 2…”

“This only works for simple cases today, and the reason why that's the case is because I think it's pretty fundamental limitation of the model that we have today, which is that they are not at the core model that take action in the world and see the consequence…”

“We have a Claude skill that essentially serving the role of a pre-training on call today. So, you know, we have these pre-training runs that are really large it's very, I think, a difficult exercise to keep them alive, to, you know, for them to continue to chu…”

“And part of the thesis of this company is that we believe that doing it at the full level of generality might actually in the long run be easier than trying to special case very specific narrow application domains.”

“So if we can draw on data from many sources, many applications, many robots, then we can have a model that has a physical understanding, and it'll be much, much easier to put new applications on top of that platform.”

“So we had some demos that we released last April where we showed our robot cleaning kitchens, and like, you know, I think it's kind of cool, but if you watch an individual video out of context, it's just like, okay, it's like picking up plates, like anybody ca…”

“And I think that in the future we'll have A robotic foundation model, which can then be adapted to all sorts of applications, and they might really run the gamut from like, you know, like bulldozers or something, to humanoids, to robotic arms like this thing, …”

“Multimodal language models are really good at pulling in knowledge and trying to articulate that knowledge. They're not very good at, like, grounding that knowledge in physical situations, but they know stuff.”

“So, I think the big challenge, and this is kind of what I'm leaning up to, and what I hope to, ah, that we'll figure out here at Physical Intelligence is how to combine those threads. How to bring in all of that knowledge that you get with generative AI, but a…”

“So the way you get common sense is by essentially using chain of thought. So the robot enters a scene and instead of directly starting to move, it thinks about what it was asked to do. So if it was told to clean up the kitchen, looks at the scene and says, lik…”

“And for example, we had this demo on, ah, making espresso. That system practiced making those espressos many, many times and used that to improve robustness, improve speed, improve throughput.”

“A good learning method can actually like compensate for deficient sensing fairly well.”

“So, I think that the key is not so much to quantify, like, here is exactly the price tag of getting the ultimate robot data set. The key is to get a system that can go into the world that's useful enough That does a wide variety of different things and they ca…”

“What was surprising is that we could also get these systems to perform very dexterous behaviors without really doing anything particularly special for that.”

“Where we could get our models to work on all sorts of other robots, including robots with multi-fingered hands robots with different numbers of degrees of freedom, and obviously we needed to get data, and we needed to fine-tune the model, but the model itself …”

“And I think increasingly what we'll see is a shift where domains where collecting data is straightforward They actually end up falling into the easy bucket over time, even if they are physically intricate. But there will be domains where collecting data is dif…”

“So what that means is that the bottleneck had actually shifted from the lowest level, meaning the robot's ability to physically do the task, to this, like, middle level, where now the system is more bottlenecked by its ability to interpret the scene and select…”

“I think the place where I would see the biggest technical risk is dealing with the breadth of different situations. So, I think if we were talking about a well-defined, but, you know, But slightly chaotic environment like cleaning hotel rooms, or working, ah, …”

“So in my mind, the most important thing to get right is To get the system to be general, and in particular, to get it to be general with respect to how it can be improved, right? So, for example, hand-designed robotic controllers are not very general with resp…”

“And we tried these things, and it actually turned out that we could solve almost all of them. We didn't get there's one we couldn't do, which was turning a dress shirt inside out, because the grippers on this thing wouldn't fit inside the sleeve, so we probabl…”

“And I think that's like, there's something interesting there, because it suggests the power of generality, that when you have this kind of general system, you can really just, like, onboard all these crazy tasks without really doing anything particularly sophi…”

“It turns out to be like pretty straightforward to go in and like find all those pauses and remove them. And you can speed things up further, so you can get a task where a person demonstrates what it means to succeed, and then you can have the robot practice th…”

“I think changing a child's diaper will be really, really hard.”

“We, we're so primed to interact with the physical world, so primed to have physical intelligence that you can use it in everyday speech by saying that company has a lot of momentum, and you can use it when advancing fundamental theoretical physics.”

“Where there's a bootstrap challenge, like getting to a particular level of usefulness so that, ah, robots can be deployed so they can do useful tasks, so they can start collecting data from open world settings at scale, and because that's such a, like a sudden…”

“Well, that's not often the best assumption because you need the right kind of data. Like, Maybe some data is easy, like, it's easy to get, like, videos of people doing something, but that doesn't mean that's the right kind of data, and it might be domain depen…”

“So without, without, like, giving too much away what I can say is that a big focus for us right now is actually better understanding this kind of, like, mid-level reasoning part of the problem. Because we think that we have a pretty good sense for how to acqui…”

“There's a company called Physical Intelligence, known as Pi, that is the leader doing that.”

“I think we're gonna see a real renaissance of, you know, problems that don't just have to be solved with bits. They can also be solved with atoms as well. You know, sometimes to make a real impact on the world, you actually need to move something in reality. A…”

“If you want to truly solve a robotics application, you essentially need to build an entire company around that application. Ah, you need to build a different company for logistics, for wet lab automation, For robots and kitchens, for surgical robots, and so on…”

“And in particular, we're trying to develop a general purpose model that can enable any robot to do any task in any environment.”

“And we think that this sort of generalist model may work better and be easier to use than purpose-built models, just like we've seen in the development of foundation, foundation models for language and other applications.”

“And so I think the lesson here is that scale is necessary for developing these models that can generalize in open world conditions, but they're subordinate to actually solving the problem. So you need scale, but it's not sufficient for the entire problem.”

“I should mention this is large scale by today's robot standards and arguably a minuscule amount of data compared to the sorts of robot data that we should have in the years to come.”

“And to date, I think this is the most impressive thing that I've seen A robot do in the physical world.”

“And this was actually to take some inspiration from the world of language modeling to actually instead of just training a policy on all of our data, can we pre-train on all the data? And then fine tune on a highly, on a curated, consistent, high quality set of…”

“We selected and worked on our curation strategy for curating a higher quality set of demonstration data. We got it from 20 minutes down to 12 minutes for these five items.”

“We took an open source vision language model, a three billion parameter model called polygema. Previously we were using, the previous videos were all with like a hundred to three hundred million parameters that we're iterating on. This model takes as input ima…”

“We're also able to apply that same recipe to robots at other companies. This is a robot that I've actually never seen in person before. They collected data. They sent the data to us. We fine tuned our model on their data. We actually didn't even know exactly h…”

“And in total, we had more than a hundred unique rooms represented in the dataset.”

“And I should point out here that the mobile manipulation data of tidying bedrooms and kitchens only accounted for 2.4% of the overall pre-training mix.”

“And second, it also followed language far better an 80% follow rate rather than a 20% follow rate which suggests that we're able to preserve the kind of pre-training in the vision language model backbone.”