Mar 31, 2026 · 1h 15m · invest-like-the-best
World's Top Researcher on AI, LLMs, and Robot Intelligence · Invest Like The Best
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of Invest Like The Best, host Patrick O'Shaughnessy interviews Sergey Levine, co-founder of Physical Intelligence and leading UC Berkeley robotics researcher, on building general-purpose physical foundation models for robotics. Levine details how combining multimodal large language models with deep reinforcement learning enables robots to master diverse tasks across variable environments and physical form factors.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Patrick holds 23.3% of the talking time here. How this is scored →
Patrick as informed peer 3.9 Guest teaching 6.2 Guest disagreement 1.2 Patrick pushing back 1.2
speaking balance: gold is Patrick, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:07:04 Grounded researcher vs hype entrepreneur
Sergey explicitly contrasts his grounded assessment against startup hype, noting that most working industrial robots still run 1980s code.
Hardest push from Patrick ▶ 59:08 Challenging Boston Dynamics' commercial utilityPatrick directly questions whether Boston Dynamics' long history of viral acrobatic demos actually translates to useful customer value.
Biggest teaching moment ▶ 24:28 Explaining Moravec's paradox in manipulationSergey educates the audience and host on why intuitively simple physical tasks for humans represent the hardest machine learning engineering frontiers.
Patrick holds their own ▶ 41:29 Connecting neurobiology to robotic embodimentPatrick introduces his co-founder's intuition about riding a bike, and Sergey backs it with neurological studies on primate tool use.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Patrick as informed peer | Guest teaching | Guest disagreement | Patrick pushing back | Why |
|---|---|---|---|---|---|---|
| Defining Physical Intelligence & Foundation Models | 4 | 6 | 1 | 2 | Patrick opens by asking Sergey to define physical intelligence and probes the trade-offs of building general models versus task-specific narrow robots. Sergey gives a clear foundational explanation drawing analogies to language models. | |
| Generalization Challenges vs. Staged Demonstrations | 3 | 6 | 1 | 1 | Patrick asks about the hardest part of building general models compared to narrow, legible demos. Sergey explains the difference between staged clean demos and true zero-shot generalization in unseen environments. | |
| Humanoid Embodiment vs. Universal Robotic Control | 4 | 6 | 2 | 2 | Patrick asks about humanoid forms versus general control, mentioning the Optimus hand. Sergey counters that humanoids are cool for marketing but physical intelligence should be embodiment-agnostic. | |
| Historical Milestones in End-to-End Robotic Learning | 4 | 7 | 1 | 1 | Patrick asks for historical milestones in robotic learning. Sergey educates on early end-to-end systems like ALVINN in the 1980s and the modern breakthrough of multimodal LLMs providing commonsense priors. | |
| Deep Reinforcement Learning & Multimodal LLMs | 5 | 7 | 1 | 1 | Patrick mentions landmark ML breakthroughs like AlexNet and AlphaGo's Move 37. Sergey explains how physical intelligence seeks to merge generative AI priors with deep RL superhuman policy discovery. | |
| Core Model Architecture: VLAs, Chain-of-Thought, and RL | 3 | 7 | 1 | 1 | Sergey breaks down the technical stack of Vision-Language-Action (VLA) models, explaining how internal chain-of-thought unlocks semantic reasoning before physical execution. | |
| Minimalist Hardware Sensors & Perceptual Learning | 4 | 6 | 2 | 2 | Patrick inquires about sensor configurations and internet-scale robot data collection. Sergey notes that wrist cameras act as visual tactile sensors and references Tesla's flywheel. | |
| Research Surprises: Dexterity & Cross-Embodiment Generalization | 4 | 7 | 1 | 1 | Sergey describes unexpected research breakthroughs in dexterity and Moravec's paradox, detailing how tasks hard for humans are often easy for computers and vice versa. | |
| Defining Common Sense in Robotic Systems | 4 | 7 | 1 | 1 | Patrick asks for a scientific definition of common sense. Sergey defines it as semantic inference from disparate domains grounded into physical action, contrasting it with muscle memory. | |
| Real-World Deployment Risks & The Human Interaction Long-Tails | 3 | 6 | 1 | 1 | Patrick asks about failure modes that could prevent robots from handling home chores by 2050. Sergey discusses human-robot interaction tolerances, safety long-tails, and domestic chaos. | |
| Physical Intelligence's Core Thesis on General System Improvement | 4 | 6 | 2 | 2 | Patrick asks for the simplest mental model of Physical Intelligence's approach versus others. Sergey clarifies that the core thesis is generality of continuous self-improvement rather than specific hardware choices. | |
| Balancing Utility and Cool Demos: The Robot Olympics | 4 | 7 | 1 | 1 | Patrick asks how to balance cool demonstrations with genuine utility, prompting Sergey to describe Benji Holson's Robot Olympics concept of mundane everyday manipulation benchmarks. | |
| Achieving Superhuman Speed & Efficiency in Physical Execution | 4 | 6 | 1 | 1 | Patrick asks how robots can surpass human performance. Sergey illustrates this with cable plugging experiments where autonomous RL removes human pauses and reaction delays. | |
| Embodiment-Agnostic Physical Intelligence | 5 | 6 | 1 | 1 | Patrick quotes co-founder Lachy Groom on physical intelligence feeling like learning to ride a bike. Sergey points to neurobiology studies on tool integration in primate motor cortex. | |
| Major Research Controversies & "The Bitter Lesson" | 5 | 7 | 2 | 1 | Patrick asks about major controversies in robotics research. Sergey steelmans the opposition to Rich Sutton's 'Bitter Lesson' and argues why end-to-end learning ultimately triumphs over hand-coded physics engines. | |
| Human Interaction, Caregiving, and Diaper Changing Challenges | 3 | 6 | 1 | 1 | Sergey explains why human-interactive caregiving and changing diapers will be among the last problems solved, due to subtle contact dynamics and Moravec's paradox. | |
| Scientific Breakthroughs & What Makes Great Researchers | 4 | 6 | 1 | 1 | Patrick asks what differentiates high-impact AI researchers. Sergey explains the delicate intuition of knowing when to persevere on a hard problem versus pivoting to explore new angles. | |
| Personality Diversity Among Leading Scientists | 3 | 5 | 2 | 1 | Patrick asks if elite researchers share distinctive personality traits. Sergey demurs, noting that top researchers range from novelty-seeking explorers to single-minded builders. | |
| Preparing Businesses for Robotic AI Integration | 4 | 6 | 1 | 1 | Patrick asks how business leaders should prepare for robotic AI. Sergey uses coding copilot tools as a realistic analogy for human-robot collaborative labor evolution. | |
| Evaluating Hardware Demos & Boston Dynamics' Contributions | 5 | 6 | 1 | 2 | Patrick pushes on Boston Dynamics having decades of impressive viral demos without commercial enterprise adoption. Sergey defends their role in advancing public imagination and setting ambitious technical targets. | |
| Decreasing Hardware Costs & Evaluating Research Signal | 4 | 6 | 1 | 1 | Sergey discusses dramatic hardware cost deflation from $400k research arms down to low-cost hardware enabled by compliant learning software. | |
| Timeline Uncertainties & The Activation Energy Flywheel | 3 | 6 | 1 | 1 | Patrick asks about the greatest uncertainties in the mission. Sergey points to deployment activation energy and whether teleoperation or autonomous data pipelines will lead the flywheel. | |
| Research Community Pragmatism vs. Entrepreneurial Hype | 4 | 6 | 2 | 1 | Patrick asks Sergey where he sits on the optimism spectrum. Sergey notes he is more optimistic than academic robotics researchers but more cautious and grounded than startup hype founders. | |
| Mentorship Bets & Kindest Career Moments | 3 | 5 | 0 | 0 | Patrick asks the signature closing question about kindness. Sergey reflects on critical inflection points where mentors like Jeff Dean and Pieter Abbeel took bets on him. |
Statements from this episode (49)
Levine: General Robotics Models May Ultimately Be Easier Than Narrow Ones
“And part of the thesis of this company is that we believe that doing it at the full level of generality might actually in the long run be easier than trying to special case very specific narrow application domains.”
Levine: General Language Models Proved Easier Than Narrow NLP Systems
“Again, in much the same way that for language models, it turned out to be Easier in some ways to solve natural language tasks in their full generality than to narrowly target like machine translation or sentiment analysis or whatever.”
Levine: Weakly Labeled Web Data Builds Foundational AI World Understanding
“When you can leverage Weekly labeled data, like data that you like, you know, in the case of language models that you just mine from the web, you actually learn more about the world. So you establish, like, foundation of world understanding, and then on top of…”
Levine: Multi-robot data will enable foundational physical models for rapid deployment
“So if we can draw on data from many sources, many applications, many robots, then we can have a model that has a physical understanding, and it'll be much, much easier to put new applications on top of that platform.”
Levine: Flashy Robot Demos Require Controlled Environments, Not General Intelligence
“Effective robotic learning, effective generalization, isn't actually the optimal way to have, like, a really exciting demo. Like, the way to have a really exciting demo is to pick a really cool task, control everything else in the environment, like, set it up …”
Levine: Physical Intelligence Kitchen Demos Used Zero Prior Training Data
“So we had some demos that we released last April where we showed our robot cleaning kitchens, and like, you know, I think it's kind of cool, but if you watch an individual video out of context, it's just like, okay, it's like picking up plates, like anybody ca…”
Levine: Foundation models will trigger a PC-like explosion in robotics applications
“And I think something like that might happen in the world of robotics, but it can't happen today because if you want to put together some cool new robotics applications, some cool new robotics idea, you kind of have to build this, like, monstrous stack, and yo…”
Levine: Future Robots Won't Just Be Humanoid 'Metal People'
“And I think, you know, we, sometimes we think that, like, robots are going to be, like, one thing. Like, it's just like, you know, there's people, and now we're going to make, like, metal people, and that'll be, like, robots. But I don't think that's how it's …”
Levine: Robotic foundation models will adapt across diverse physical form factors
“And I think that in the future we'll have A robotic foundation model, which can then be adapted to all sorts of applications, and they might really run the gamut from like, you know, like bulldozers or something, to humanoids, to robotic arms like this thing, …”
Levine: Future Surgical Robots Will Not Be Limited to Human Control
“I think that in the long run, this is not by any means a short term thing, but in the long run, I think there's lots of really exciting applications in medicine and surgery where we not only might in the long run not be limited to robots that look like humans,…”
Levine: General purpose robot models require far less data per task
“Being able to train general purpose models that can handle many tasks is essential to this because now you need a lot less data for each new task.”
Levine: Multimodal LLMs hold broad knowledge but lack physical grounding
“Multimodal language models are really good at pulling in knowledge and trying to articulate that knowledge. They're not very good at, like, grounding that knowledge in physical situations, but they know stuff.”
Levine: Deep RL Is Essential for Robots to Exceed Human Performance
“I think that the first deep reinforcement learning systems, which were in the early 20 tens, like those are probably a milestone because deep reinforcement learning gives us a way to go beyond human level performance, which I think will be essential for roboti…”
Levine: Adapting Multimodal LLMs to Robot Control Is a Key Advance
“I do think that the advent of Multimodal LLMs that can be adapted to robotic control to bring in that common sense. I do think that's a really important advance.”
Levine: GenAI Mimics Humans, But Deep RL Discovers Novel Solutions
“The generative AI is impressive because it can reproduce some of the things that humans can do. Like, it can draw pictures that look like people, you know, human pictures write text. Deep RL is impressive for the opposite reason. It does things that humans had…”
Physical Intelligence aims to fuse generative AI prior knowledge with reinforcement learning
“So, I think the big challenge, and this is kind of what I'm leaning up to, and what I hope to, ah, that we'll figure out here at Physical Intelligence is how to combine those threads. How to bring in all of that knowledge that you get with generative AI, but a…”
Levine: Chain-of-thought reasoning allows robots to handle edge cases
“So the way you get common sense is by essentially using chain of thought. So the robot enters a scene and instead of directly starting to move, it thinks about what it was asked to do. So if it was told to clean up the kitchen, looks at the scene and says, lik…”
Levine: Physical Intelligence trained espresso-making robot using repeated RL practice
“And for example, we had this demo on, ah, making espresso. That system practiced making those espressos many, many times and used that to improve robustness, improve speed, improve throughput.”
Levine: Effective AI Learning Methods Compensate for Deficient Hardware Sensing
“A good learning method can actually like compensate for deficient sensing fairly well.”
Levine: Robot Wrist Cameras Function as De Facto Touch Sensors
“The wrist cameras are essentially a touch sensor in disguise because you can see local deformations when you touch something.”
Levine: Scaling physical AI requires real-world data flywheels over fixed datasets
“So, I think that the key is not so much to quantify, like, here is exactly the price tag of getting the ultimate robot data set. The key is to get a system that can go into the world that's useful enough That does a wide variety of different things and they ca…”
Levine: Physical Intelligence achieved robot dexterity without specialized techniques
“What was surprising is that we could also get these systems to perform very dexterous behaviors without really doing anything particularly special for that.”
Levine: Models generalize across robot embodiments without architecture changes
“Where we could get our models to work on all sorts of other robots, including robots with multi-fingered hands robots with different numbers of degrees of freedom, and obviously we needed to get data, and we needed to fine-tune the model, but the model itself …”
Levine: Easy Data Collection Makes Physically Intricate Robotics Tasks Easy
“And I think increasingly what we'll see is a shift where domains where collecting data is straightforward
They actually end up falling into the easy bucket over time, even if they are physically intricate. But there will be domains where collecting data is dif…”
Levine: In Robotics, Common Sense Is the Opposite of Muscle Memory
“For the purpose of robotic learning, ah, we can think of it as applying semantic inferences Using knowledge learned from other domains to a, to the current physical task at hand, right? So you can think of common sense as sort of the opposite of muscle memory.…”
Levine: Robotics Bottleneck Shifted From Execution to Scene Interpretation
“So what that means is that the bottleneck had actually shifted from the lowest level, meaning the robot's ability to physically do the task, to this, like, middle level, where now the system is more bottlenecked by its ability to interpret the scene and select…”
Levine: Handling unexpected home situations is robotics' biggest technical risk
“I think the place where I would see the biggest technical risk is dealing with the breadth of different situations. So, I think if we were talking about a well-defined, but, you know, But slightly chaotic environment like cleaning hotel rooms, or working, ah, …”
Levine: Generality of improvement mechanism is the most critical robotics capability
“So in my mind, the most important thing to get right is To get the system to be general, and in particular, to get it to be general with respect to how it can be improved, right? So, for example, hand-designed robotic controllers are not very general with resp…”
Levine: Acrobatic Humanoid Demos Rely on Simulation, Not Real-World Data
“If you've seen videos of humanoids doing all these acrobatics, right? There's a particular pipeline that makes that work, which is very heavily reliant on simulation, and actually very light on real world data, often almost, often actually zero real world data…”
Levine: Robotic manipulation depends on real-world data over simulation
“There are the approaches that work well for robotic manipulation that often are the opposite. They often use very little simulated data, often use large amounts of real world data, and very large foundation models.”
Levine: General Robots Still Struggle With Turning Shirts Inside Out
“And we tried these things, and it actually turned out that we could solve almost all of them. We didn't get there's one we couldn't do, which was turning a dress shirt inside out, because the grippers on this thing wouldn't fit inside the sleeve, so we probabl…”
Levine: General robot models onboard diverse tasks without task-specific engineering
“And I think that's like, there's something interesting there, because it suggests the power of generality, that when you have this kind of general system, you can really just, like, onboard all these crazy tasks without really doing anything particularly sophi…”
Levine: Robots Surpass Human Speed by Editing Out Cognitive Pauses
“It turns out to be like pretty straightforward to go in and like find all those pauses and remove them. And you can speed things up further, so you can get a task where a person demonstrates what it means to succeed, and then you can have the robot practice th…”
Levine: Traditional control pipelines bottleneck robotic hardware innovation
“I think that in general, in robotics, the ability to innovate on form factors has been very constrained because of the AI challenge, right? So if you have a traditional AI pipeline, like, you know, you're doing some motion planning and stuff like that, It's ha…”
Levine: Studies show the primate brain physiologically treats tools as bodily extensions
“There were studies that were done in monkeys using tools. And you can actually find where in the brain, ah, like which neurons activate for the monkey to figure out where its hand is. It turns out that if it's using a tool, they activate based on location of t…”
Levine: Robotics Community Lacks Universal Acceptance of End-to-End Learning
“I think at this point there's a lot of acceptance that learning is a really important part of robotics, but I don't think there's still universal acceptance that end-to-end learning is the right way to go. Basically, I don't think there's universal acceptance …”
Levine: LLMs Show True Compositional Generalization Through IPA Paragraphs
“But if you ask a good language model, it will write paragraphs in IPA for you. And that is compositional generalization. It means that you have never seen this particular language, this particular alphabet, used to write paragraphs, but you understand paragrap…”
Levine: Changing a Child's Diaper Will Be Exceptionally Hard for Robots
“I think changing a child's diaper will be really, really hard.”
Levine: Physical intelligence underpins human reasoning from daily language to theoretical physics
“We, we're so primed to interact with the physical world, so primed to have physical intelligence that you can use it in everyday speech by saying that company has a lot of momentum, and you can use it when advancing fundamental theoretical physics.”
Levine: Foundation Models Reduce Uncertainty Before Scaling Robot Hardware Manufacturing
“So, yes, making a robot at scale is difficult. Making a robot at scale is even more difficult if you don't know what kind of software's gonna run it afterwards, and you're not even sure whether it's the right kind of robot. So, I think one of the really valuab…”
Levine: Robotics will augment human workers rather than fully replace them
“And I think we'll actually see something like that with robotics, too, that a more realistic template is not like, you know, the humanoid, like, goes in and the people just leave. I think it'll be more like there are some aspects of the job that can be done by…”
Levine: Robotics demos hold value when honestly tied to a mission
“I think that, you know, what I'll say in general terms is that I think it, there is a lot of value in demos that serve to illustrate challenges on the road to something useful and productive. And obviously you can also do a demo without being on the road to so…”
Levine: Robot Arm Costs Plummeted From $400,000 to $3,000 in a Decade
“When I started working in robotics about a decade ago, I worked with a robot called a PR-II, which I believe had a cost of about 400,000 dollars. When I started my lab at UC Berkeley, I used a robot that was in the ballpark of 30,000 dollars. Now, each arm on …”
Levine: Robotics Timelines Are Uncertain Due to Activation Energy Bottlenecks
“Where there's a bootstrap challenge, like getting to a particular level of usefulness so that, ah, robots can be deployed so they can do useful tasks, so they can start collecting data from open world settings at scale, and because that's such a, like a sudden…”
Levine: Easy Video Data Is Often Not Right for Robotics AI
“Well, that's not often the best assumption because you need the right kind of data. Like, Maybe some data is easy, like, it's easy to get, like, videos of people doing something, but that doesn't mean that's the right kind of data, and it might be domain depen…”
Levine: Physical Intelligence Is Focused on Mid-Level Reasoning Representations
“So without, without, like, giving too much away what I can say is that a big focus for us right now is actually better understanding this kind of, like, mid-level reasoning part of the problem. Because we think that we have a pretty good sense for how to acqui…”
Levine: Text Representations in LLMs Are Suboptimal for Embodied AI
“So LLMs make certain kinds of representations very convenient. They make it very convenient to basically turn text into other text. But that's not necessarily the best representation for what an embodied system needs to do. Like sometimes it needs to think abo…”