Physical Intelligence

“We selected and worked on our curation strategy for curating a higher quality set of demonstration data. We got it from 20 minutes down to 12 minutes for these five items.”

“We took an open source vision language model, a three billion parameter model called polygema. Previously we were using, the previous videos were all with like a hundred to three hundred million parameters that we're iterating on. This model takes as input ima…”

“And in total, we had more than a hundred unique rooms represented in the dataset.”

“I think that at the end of the day, there's going to be no replacement for real data. And so we're like large amounts of real robot data. It's going to be a necessary component of any like system that's going to work in a generalizable way.”

“And I should point out here that the mobile manipulation data of tidying bedrooms and kitchens only accounted for 2.4% of the overall pre-training mix.”

“We're also able to apply that same recipe to robots at other companies. This is a robot that I've actually never seen in person before. They collected data. They sent the data to us. We fine tuned our model on their data. We actually didn't even know exactly h…”

“And in particular, we're trying to develop a general purpose model that can enable any robot to do any task in any environment.”

“And we find that if we actually increase the amount of homes, the amount of locations that are represented in the data, The performance increases, which is great. And it actually gets to the same level of performance as if we train on data from that target env…”

“We can use language models to relabel and generate hypothetical human prompts for the scenarios that the robots are in.”

“And so I think the lesson here is that scale is necessary for developing these models that can generalize in open world conditions, but they're subordinate to actually solving the problem. So you need scale, but it's not sufficient for the entire problem.”

“And we think that this sort of generalist model may work better and be easier to use than purpose-built models, just like we've seen in the development of foundation, foundation models for language and other applications.”

“And this was actually to take some inspiration from the world of language modeling to actually instead of just training a policy on all of our data, can we pre-train on all the data? And then fine tune on a highly, on a curated, consistent, high quality set of…”

“And second, it also followed language far better an 80% follow rate rather than a 20% follow rate which suggests that we're able to preserve the kind of pre-training in the vision language model backbone.”

“If you want to truly solve a robotics application, you essentially need to build an entire company around that application. Ah, you need to build a different company for logistics, for wet lab automation, For robots and kitchens, for surgical robots, and so on…”

“We ourselves haven't had a lot of challenge with fundraising, and I think that a lot of robotics companies recently have also done a great job and found that there's actually a lot of excitement around this sort of technology, because I think things are actual…”

“And we find that these kind of bars on the right, which are excluding data from static robots in labs and environments and so forth reduces performance significantly. So the performance goes down to less than 60% when you exclude that data when evaluated in no…”

“I should mention this is large scale by today's robot standards and arguably a minuscule amount of data compared to the sorts of robot data that we should have in the years to come.”

“In general, we found that these frontier models generally struggle with visual understanding as it pertains to robotics. Which makes sense because in general, these models aren't kind of really targeting, ah, many physical applications and have very little dat…”

“And to date, I think this is the most impressive thing that I've seen A robot do in the physical world.”

“And cross embodiment, there is the data collection aspect of, as well, which is to really make sure that your model and your organizations and infrastructure are set up to consume data From many different sources of robots, and that actually allows you to scal…”

“The interesting thing about the approach is that you're converting it from a very difficult engineering problem into a operation problems of how do I identify the use case, and how do I collect the right data, which is, in some sense, more scalable, because yo…”

“We open source PI zero and PI zero five. And people also shocked when they asked me, you know, is there any difference between PI zero and PI zero five that you open source versus the model that we use internally PI zero and PI zero five? And the answer was, I…”

“This is packaging real customer real order for customer to be shipped out in a real warehouse. So this is real operations.”

“Today it's possible to perform tasks Zero shot. Zero shot meaning you don't collect any data. And these are the tasks that last year might have required like hundreds and hundreds of hours.”

“We think that it's going to be more like a peeling an audience analogy, where you start from a really strong base model that have all sorts of common sense knowledge and already works to some extent on your robot, and you have then a Mixed autonomy system. Ver…”

“You can compare it to the specialist that has been optimized to work well on a particular embodiment. How does it compare? And the interesting result from OpenX is it was 50% better.”

“No two items of clothing here are the same, and these are also unseen. You know, these are not, like, clothing items that are seen in the training data.”

“And the argument is that, you know, single robot is simpler to scale. And actually that's not how it plays out in practice. Like how it plays out in practice is even if you have a single robot that you're optimizing for, over time that Platform is going to dri…”

“Almost all of the robot evaluation that we run at Pi today, including the really Complicated demo that we have shown making coffee, folding laundry, mobile robots navigating around. The model actually hosted in the cloud.”

“And it actually didn't even take us that long to get this result. It was roughly where we set a goal and maybe it was like two weeks afterwards where we got, got a model that was, got a model and a system that was good enough at performing this task.”

“You don't need a incredibly expensive robot that is capable of very precise motion today to be able to do this task. And the reason why is this model really reactive? And so they can compensate for some of the inaccuracy in the actual robot movement”

“I think this is another area of incredible opportunity of kind of building services for robot company. Like, you know, if you can offer remote tele-op, for example, if you can offer data collections, if you can offer annotation service, because, you know, thes…”

“This only works for simple cases today, and the reason why that's the case is because I think it's pretty fundamental limitation of the model that we have today, which is that they are not at the core model that take action in the world and see the consequence…”

“Let's say if we actually solve robotics, a model that can control any robot to do any task, napkin math maybe contribute 10% to US GDP. Well, that's already a massive number and I think that promise is one of the reasons that warrants the investment into data …”

“Evaluation is a really hard problem in robotics because it scales super linearly to model capability. Like, let's say you have a model that can perform a two-minute task. Running evaluation for that is very different from running evaluation for a task that's 2…”

“Our mission is to build a model that can control any robot, to do any task that is physically capable of, and to do so as such a high level of performance that's going to be useful to people in all walks of life.”

“We have a Claude skill that essentially serving the role of a pre-training on call today. So, you know, we have these pre-training runs that are really large it's very, I think, a difficult exercise to keep them alive, to, you know, for them to continue to chu…”

“And when we started the company, we thought that real deployment is going to be a con, it's only going to be in a conversation like five years. Into the life of the company, because the problem is just really hard, and we're two years in, and, you know, this i…”

“And the, one other interesting thing about our collaboration with Weave and Ultra is one, I've never seen their robot in person. Two is I have very little idea about how their robot actually works. And that's a very intentional choice. I want to stay away from…”

“He's one of the co-founders of Physical Intelligence, which we think might be the robotics AI lab that brings about the GPT-I moment for all of robotics.”