Jul 22, 2025 · 44m · y-combinator
Chelsea Finn: Building Robots That Can Do Anything · Y Combinator
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Y Combinator AI Startup School talk, Chelsea Finn presents Physical Intelligence's breakthrough foundation model architecture, π₀, which combines broad multi-robot pre-training with curated fine-tuning to achieve general-purpose dexterity and zero-shot generalization across unseen real-world environments.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The partners as informed peer 0.3 Guest teaching 0.6 Guest disagreement 0.3 The partners pushing back 0.3
speaking balance: gold is the partners, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 33:00 Pushing back on fundraising difficulty premise
Chelsea directly challenges an audience member's skepticism about convincing investors to fund chore robots by emphasizing broad physical intelligence applications and thriving investor appetite.
Hardest push from the partners ▶ 33:00 Audience question on investor skepticismAn audience member asks how Physical Intelligence manages fundraising given the perceived difficulty of convincing investors to back domestic chore automation.
Biggest teaching moment ▶ 41:40 Clarifying the synthetic data analogy in roboticsChelsea educates the audience by clarifying that the true robotics equivalent of LLM synthetic data is autonomous reinforcement learning and self-attempted data rather than passive physics simulation.
The partners hold their own ▶ 37:45 Audience question comparing parametric memory to retrievalAn audience member Frederick demonstrates domain knowledge by asking whether small models with external knowledge retrieval databases could outperform large unified parameter models in robotics.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The partners as informed peer | Guest teaching | Guest disagreement | The partners pushing back | Why |
|---|---|---|---|---|---|---|
| General-Purpose Models vs. Purpose-Built Robotics | 0 | 0 | 0 | 0 | This segment is entirely a solo lecture by Chelsea Finn introducing Physical Intelligence and foundation models in robotics. With no host dialogue or interaction, all host-side and adversarial scores are baseline zero. | |
| Data Episode Demonstration and Presentation Outline | 0 | 0 | 0 | 0 | Chelsea presents initial experimental attempts and failure modes of folding laundry with imitation learning. This is an uninterrupted presentation monologue. | |
| The Post-Training Breakthrough: Pre-Training and Curated Fine-Tuning | 0 | 0 | 0 | 0 | Chelsea explains the breakthrough discovery of pre-training across broad data followed by curated post-training fine-tuning. The format remains a solo presentation. | |
| Introducing π₀: Pre-Trained VLMs and Flow Matching Diffusion | 0 | 0 | 0 | 0 | The presentation details integrating PaliGemma VLMs with flow matching diffusion action heads for continuous control. Solo technical lecture segment. | |
| Quantitative Evaluation and Multi-Task/Robot Transfer | 0 | 0 | 0 | 0 | Chelsea shares quantitative ablation benchmarks and zero-shot cross-robot/cross-task transfer results. Continued solo technical talk. | |
| Takeaways from Part 1 and Environmental Generalization Limit | 0 | 0 | 0 | 0 | Chelsea discusses mobile manipulation data across over a hundred real-world rooms and gradient-stopping methods to retain language grounding. Solo lecture format. | |
| Y Combinator Interstitial & Quantitative Evaluation of Unseen Generalization | 0 | 0 | 0 | 0 | Contains a short YC application voiceover interstitial followed by Chelsea reviewing quantitative evaluation curves across novel Airbnb test environments. | |
| Open-Ended Interaction with Hierarchical VLA Models (Hi Robot) | 0 | 0 | 0 | 0 | Chelsea details hierarchical VLA architectures and LLM-generated synthetic prompt annotations for handling open-ended requests and human interjections. | |
| Benchmarking Frontier Models and Closing Remarks | 1 | 2 | 1 | 1 | Chelsea wraps up the talk and takes audience questions regarding post-training data quality and fundraising viability. Chelsea politely pushes back against the skepticism around commercial robotics demand. | |
| Audience Q&A: World Models, Infrastructure, and Architecture | 1 | 2 | 1 | 1 | Audience members ask technical questions about world model integration, realtime robot infrastructure, and RAG versus model parameters. Chelsea educates the audience on world model hallucinations and retrieval limitations. | |
| Audience Q&A: Synthetic Data, Academia vs. Industry, and Tokenization | 1 | 2 | 1 | 1 | Chelsea answers final audience questions on synthetic data, academia versus industry compute dynamics, and action tokenization. She reframes synthetic data in robotics as being analogous to RL self-improvement rather than pure simulation. |
Statements from this episode (29)
Finn: Solving robotics applications traditionally requires building separate companies from scratch
“If you want to truly solve a robotics application, you essentially need to build an entire company around that application. Ah, you need to build a different company for logistics, for wet lab automation, For robots and kitchens, for surgical robots, and so on…”
Finn: Physical Intelligence is building a general-purpose model for all robots
“And in particular, we're trying to develop a general purpose model that can enable any robot to do any task in any environment.”
Finn: Generalist robotics models may outperform purpose-built models
“And we think that this sort of generalist model may work better and be easier to use than purpose-built models, just like we've seen in the development of foundation, foundation models for language and other applications.”
Finn: Modern AI coding assistants build on general data, not just code
“For example, if you want to build a coding assistant, you don't nowadays develop something specifically for coding, but you develop and you build on models that were trained on large amounts of data, not just on code.”
Finn: Industrial automation data lacks behavioral diversity for general robotics
“So for example, we might look at data from industrial automation and you get tons and tons of data of robots doing tasks over and over again like this, but the sort of data isn't going to allow robots to go into disaster zones or to make a sandwich or to bag g…”
Finn: Human video data is constrained by robot-human embodiment gaps
“Alternatively, maybe we look at data from YouTube, which has also a massive data source and many videos of humans doing tasks that can be useful for training robots. But at the same time, we don't learn how to write by watching other people write, and we don't…”
Finn: Scale is necessary but not sufficient for open-world robotics models
“And so I think the lesson here is that scale is necessary for developing these models that can generalize in open world conditions, but they're subordinate to actually solving the problem. So you need scale, but it's not sufficient for the entire problem.”
Finn: Current Robot Datasets Are Minuscule Compared to Future Scale
“I should mention this is large scale by today's robot standards and arguably a minuscule amount of data compared to the sorts of robot data that we should have in the years to come.”
Folding laundry is the most impressive physical robot feat Finn has seen
“And to date, I think this is the most impressive thing that I've seen A robot do in the physical world.”
Finn: Pre-training and curated fine-tuning unlocked robotic laundry folding
“And this was actually to take some inspiration from the world of language modeling to actually instead of just training a policy on all of our data, can we pre-train on all the data? And then fine tune on a highly, on a curated, consistent, high quality set of…”
Finn: Data curation reduced five-item folding time to 12 minutes
“We selected and worked on our curation strategy for curating a higher quality set of demonstration data. We got it from 20 minutes down to 12 minutes for these five items.”
Finn: Physical Intelligence uses PaliGemma 3B with flow matching diffusion head
“We took an open source vision language model, a three billion parameter model called polygema. Previously we were using, the previous videos were all with like a hundred to three hundred million parameters that we're iterating on. This model takes as input ima…”
Finn: Physical Intelligence adapted its model to an unseen third-party robot
“We're also able to apply that same recipe to robots at other companies. This is a robot that I've actually never seen in person before. They collected data. They sent the data to us. We fine tuned our model on their data. We actually didn't even know exactly h…”
Physical Intelligence collected robot manipulation data across over 100 unique rooms
“And in total, we had more than a hundred unique rooms represented in the dataset.”
Finn: Bedroom and kitchen tidying data was only 2.4% of pre-training mix
“And I should point out here that the mobile manipulation data of tidying bedrooms and kitchens only accounted for 2.4% of the overall pre-training mix.”
Finn: Architectural fix boosted robot language following rate from 20% to 80%
“And second, it also followed language far better an 80% follow rate rather than a 20% follow rate which suggests that we're able to preserve the kind of pre-training in the vision language model backbone.”
Finn: Full Pre-Training Mixture Boosts Robot Performance Over 20% in Novel Homes
“And we find that these kind of bars on the right, which are excluding data from static robots in labs and environments and so forth reduces performance significantly. So the performance goes down to less than 60% when you exclude that data when evaluated in no…”
Finn: Diverse Home Training Matches Custom Target-Environment Performance
“And we find that if we actually increase the amount of homes, the amount of locations that are represented in the data, The performance increases, which is great. And it actually gets to the same level of performance as if we train on data from that target env…”
Finn: Real-world human-robot interaction data is hard to scale
“It's going to be challenging to collect a large number of human robot interactions with the real robot in the loop. And this is also going to be fairly hard to scale.”
Finn: LLMs can generate synthetic prompts to relabel robot data
“We can use language models to relabel and generate hypothetical human prompts for the scenarios that the robots are in.”
Finn: Frontier models struggle with visual understanding for robotics
“In general, we found that these frontier models generally struggle with visual understanding as it pertains to robotics. Which makes sense because in general, these models aren't kind of really targeting, ah, many physical applications and have very little dat…”
Finn: Reinforcement Learning Outperforms Pure Imitation Learning in Robotics
“I think that reinforcement learning can play a very large role in it actually, in post-training. I think that online data from the robots which reinforcement learning allows you to use, Can allow robots to have a much higher success rate and also be faster tha…”
Finn: Physical Intelligence Has Not Struggled to Raise Capital
“We ourselves haven't had a lot of challenge with fundraising, and I think that a lot of robotics companies recently have also done a great job and found that there's actually a lot of excitement around this sort of technology, because I think things are actual…”
Finn: World models hallucinate success when evaluating suboptimal actions
“You might train it on demonstration data of successful data of completing the task, and then evaluate it on to try to actually use it to evaluate actions that are not optimally completing the task, and then the world model will hallucinate a video of completin…”
Finn: Retrieval systems struggle because models frequently ignore retrieved content
“So in my experience working on like retrieval based systems is that it actually is a little bit tricky to first figure out what should be offloaded versus actually done by the model. And second sometimes the model will ignore the retrieved content and try to g…”
Finn: Robot-side infrastructure is an underworked opportunity for builders
“There's some open source code for that sort of thing, but there's a lot of opportunities to make robot infrastructure better. And not a lot of people I think are working on that aspect of the problem.”
Chelsea Finn: Real robot data cannot be replaced by synthetic data
“I think that at the end of the day, there's going to be no replacement for real data. And so we're like large amounts of real robot data. It's going to be a necessary component of any like system that's going to work in a generalizable way.”
Chelsea Finn: Synthetic data's robotic analog is RL, not simulation
“I think that the analog of synthetic data in language models is actually not necessarily simulation in robotics, but closer to something like reinforcement learning.”
Chelsea Finn: Abundant resources can lead to wasteful compute usage
“Sometimes when you have a lot of resources, you don't actually think as carefully and as critically about what runs are going to be doing and so forth, and you end up being sometimes more wasteful of compute than if you were kind of more compute constrained.”