Chelsea Finn: Observational video alone cannot train robot foundation models
Chelsea Finn · No Priors Ep. 107 | With Physical Intelligence Co-Founder Chelsea Finn · Mar 20, 2025 · at 29:37
Chelsea Finn, Stanford professor and co-founder of Physical Intelligence, discusses why passive human video cannot replace embodied robot experience.
0:00 / 1:00exact quote · 60.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think that data can have a lot of value, but I think that by itself, it won't get you very far and I think that there's actually some really nice analogies you can make where for example, if you watch, like, an Olympic swimmer, swimmer race even if you had their strength just their practice at moving their own muscles to do the, to accomplish what they're accomplishing is, like, essential for being able to do it, or if you're trying to learn how to hit a tennis ball well, You won't be able to learn it by kind of watching the pros. Now, maybe these examples seem a little bit contrived because they're talking about like experts. The reason why I make those analogies is that we humans are experts at motor control, low level motor control already for a variety of things that our robots are not. And I think the robots actually need experience from their own body in order to learn. And so I think that it's really promising to be able to leverage that form of data, especially to expand on the robot's own experience, but it's really going to be essential to like actually Have the data from the robot itself, too.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Chelsea Finn
Finn: Humanoids are overrated because collecting teleoperation data is too difficult
“On the other hand, I think that they're a little overrated, and one way it kind of to practically look at it is, I think that we're generally fairly bottlenecked on data right now, and some people argue that with humanoids, you can maybe collect data more easi…”
Finn: Robot training data transfers across different physical hardware embodiments
“We've seen a lot of evidence that you could actually transfer a lot of rich information across these different embodiments and allows you to use data. And also if you iterate on your robot platform, you don't have to throw all your data away.”
Finn: Biggest risk in generalist robotics is technical failure, not competition
“The last thing that I'll mention is that I think the biggest risk with this bet is that it won't work. Like, I'm not really worried about competitors. I'm more worried that no one will solve the problem.”
Finn: Wrist-mounted RGB cameras capture much of what tactile sensors provide
“And we found that actually that mounting RGB cameras to the wrists ends up being very, very helpful and probably giving you a lot of the same information that tactile sensors can give you.”
Finn: Pre-trained VLMs let robots perform tasks with unseen internet concepts
“We had a research result
a couple years ago where we showed that if you leverage vision language models, then you could actually get the robot to do tasks that require concepts that were never in the robot's training data, but were in the internet.”
Finn: Embodied AI and motor control are underrated compared to language models
“I feel like actually people underestimate how much intelligence goes into motor control. Many, many years of evolution is what led to us being able to use our hands the way that we do. And there are many animals that they can't do it even though they had so mu…”