The Exchanges

Q a very unique Technical insight that in the past robotics folks would have kind of gasped and be shocked because robots need to run in real time. A lot of times all of the compute runs in on device, but you guys have done something very different. Can you tell us more about that? So that this works in, in, in real time with large models and, and really well.

A So the context here is that, you know, we talked to many companies that would like to deploy robots and one of the First question we get is what compute units should we get on the robot? You know, it's expensive, it's going to increase the bomb cost, and they're worried that it's going to go out in fashion very quickly because the model changes, the model gets bigger, how do I make sure that the hardware that I'm going to commit to today is going to be viable for, you know, a couple of years? It's a very difficult question. People are often really surprised when I tell them that almost all of the robot evaluation that we run at Pi today, including the really Complicated demo that we have shown making coffee, folding laundry, mobile robots navigating around. The model actually hosted in the cloud. Um, and you know, this is not like a cloud as in a server in the office. It's a real cloud. The model is hosted in a data center somewhere, and within this high frequency control loop that, um, is controlling the robot, the robot is actually querying an API endpoint that hosts the model, sending it images and language command and getting back action that then Executed directly on the robot. And this is surprising because of precisely the reason that you mentioned, you know, how do you actually make it work? This is why it's really important for Pi to couple system, hardware, and model …

AI assessment note: “The model actually hosted in the cloud... you can actually bury the inference time”

Q Quan, because this task is less familiar than laundry folding, do you want to explain what the robot is doing here and what Ultra is, like, doing as a company?

A Ultra is a company that wanna makes it really easy to adapt Robot to, you know, new tasks, um, and right now they're focusing on logistics space, which is really important because, you know, there's lots of labor shortage in logistics, and the task that we focus on together here is, you know, if you order an item from Amazon, you sometimes get this soft pouch that item gets shipped from, and the task here is you have a tray of these items here, and the robot is supposed to pick one of them at the time, and place it inside this pouch. The machine will then close it, And then pick up the pouch and put it, um, on the left here to be ready for shipping. Now, this heart is hard because there are many different types of object that can be in this tray, and the opening here is actually very narrow, so you see this interesting example of the robot kind of nudging the item to go into the pouch, and that's, that's really hard. Like, that requires a very good understanding of the scene and, like, very Precise motion to nudge the object into the pouch. Um, the other thing that's hard about this task is the level of autonomy that's required. Like this is running for an entire day. There is still human intervention. I want to say in, um, this like full day operation. Um, but the level of intervention is actually quite minimal.

AI assessment note: “Ultra is a company that wanna makes it really easy to adapt Robot to, you know, new tasks”

Q Could you help us understand where we are now in terms of, like, what's working and how well it's working? Like, we're not quite at the ChatGBT moment yet. Like, where are we? And I think you brought some videos that you were going to show us to, like, help everybody visualize what the current state of the art actually looks like.

A I think where we are is I think if you have a test where it's okay for the robot to make a mistake, um, and it's possible for you to set up a mixed autonomy system where You have a person that takes over when the robot make a mistake and provide corrections. It is possible to get to a level of performance where it starts to make sense to think about scaling robot deployment. And the example that I specifically want to highlight here is this blog post that we did with Weave and Ultra. And you know, it's great that these are both YC company. I want to provide a little bit of context here first. The context is that Pi Is a primarily research organization. We want to focus on building the best model. Um, but we also want to not be tunnel vision. We want to make sure that the model that we built actually going to be useful and actually perform tasks that people in society cares about. And one of the really good way for us to do so is to partner really closely with company that want to get robot out there today. And the way that these relationships work is that we treat each other like we're on the same team. Very free flow of information, um, and we design a system that try to get the best possible performance for the tasks that this company care about. So, let me talk about we first. What you're seeing in this video is a system that we built together, folding really diverse item of…

AI assessment note: “What you're seeing in this video is a system that we built together”

Q thousand companies like Ultra going after, you know, every like menial job in the economy and like getting a deep understanding of the customer, building a robot that can solve that problem, doing that like mixed human machine deployment until it like can run fully autonomously and Building a company in, in every sector. Is that, is, is that the future that you see people building on top of Pi?

A It's funny that you mentioned Cambrian explosion, because when we wrote this blog post, there was that term that was very kind of like hotly debated. We are, I think, academics at Hurt, and we want to be kind of very major when we communicate. But, you know, myself personally, I believe there's going to be a Cambrian explosion of, um, robotic company across the entire world and across many, many different vertical. Um, just because it's just so much cheaper to build, and it doesn't require, um, you know, someone with 20 years of experience in robotics to start anymore. You know, it requires someone that is really scrappy, that can move really quickly, um, can do the system integration, um, can understand customer what they want, to start the deployment,

AI assessment note: “I believe there's going to be a Cambrian explosion of, um, robotic company”

Q Could we maybe talk a little bit about, um, like the humans behind the robots here? Like, um, how did the company get started? Like who are the, who are your co-founders? How do you all get together and what skills do you each bring to such a complex problem?

A Sometimes the joke I make here is that the human behind the robots are also robots. Not really. Um, yeah, so Pi is a very, I would say, entraditional company. We have a, like, larger than average founding teams, and some of us worked really closely together when we were at the robotic team at Google, and the robotics team at Google was, I think, a really, really great environment for seeing the sign of life and creating the relationships and the community. That allow the robot community and like these advances to flourish. There is Locky, uh, which we met when we, uh, were thinking about starting the company and has just been really instrumental in making sure that we're a good business. And there is Adnan, our hardware lead, um, that came over from Android. And Adnan has a really difficult job because if you want to work on cross embodiment, you know, remember my, uh, joke about how if you want to add two years to your grad school, Bring on one more robots. The, the hardware problem and the operational problem for us is how do we build, improve, and scale a fleet of Heather Joe genius robot. You know, it's just not one robot platform. And because we built the organization from scratch in the beginning to, to, to support that, like, I think we're able to do it, but it's just a really hard, uh, problem. Um, because there's just like No two different robots in the fleet. How do y…

AI assessment note: “There is Locky... And there is Adnan, our hardware lead”

Q Oh, you could. You totally could. What if it's OpenClaw and, um, Obsidian and Markdown files and like, you know, a brain.md with like ontology that's custom to your use case, and what if it's a hundred OpenClaws in the background that you orchestrate?

A I think there's two sides to this. The first is that we already see a little bit of a side of life, where for simple failure modes, um, during evaluation, if you can describe the way that the robot failed in text very precisely and very clearly, Then, you know, you can ask a language model to make very reasonable recommendation about what the next step is. Um, but the, the, the flip side is that this only works for simple cases today, and the reason why that's the case is because I think it's pretty, um, fundamental limitation of the model that we have today, which is that they are not at the core model that take action in the world and see the consequences of its own action, especially action that changes the physical world. Um, and, and so I, I think this kind of very fundamental understanding about how the physical world works is missing from the really large foundation model. Um, and, and I think that that's one of the ingredients that's missing to, to be able to build this automated robot research scientist.

AI assessment note: “this only works for simple cases today, and the reason why that's the case”