Finn: Architectural fix boosted robot language following rate from 20% to 80%
Chelsea Finn · Chelsea Finn: Building Robots That Can Do Anything · Y Combinator · Jul 22, 2025 · at 21:08
Chelsea Finn explains how modifying the π₀ architecture to stop diffusion gradients from deteriorating the VLM backbone improved language instruction adherence.
0:00 / 0:11exact quote · 11.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And second, it also followed language far better an 80% follow rate rather than a 20% follow rate which suggests that we're able to preserve the kind of pre-training in the vision language model backbone.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Chelsea Finn
Finn: Generalist robotics models may outperform purpose-built models
“And we think that this sort of generalist model may work better and be easier to use than purpose-built models, just like we've seen in the development of foundation, foundation models for language and other applications.”
Finn: Scale is necessary but not sufficient for open-world robotics models
“And so I think the lesson here is that scale is necessary for developing these models that can generalize in open world conditions, but they're subordinate to actually solving the problem. So you need scale, but it's not sufficient for the entire problem.”
Finn: Diverse Home Training Matches Custom Target-Environment Performance
“And we find that if we actually increase the amount of homes, the amount of locations that are represented in the data, The performance increases, which is great. And it actually gets to the same level of performance as if we train on data from that target env…”
Finn: Frontier models struggle with visual understanding for robotics
“In general, we found that these frontier models generally struggle with visual understanding as it pertains to robotics. Which makes sense because in general, these models aren't kind of really targeting, ah, many physical applications and have very little dat…”
Chelsea Finn: Real robot data cannot be replaced by synthetic data
“I think that at the end of the day, there's going to be no replacement for real data. And so we're like large amounts of real robot data. It's going to be a necessary component of any like system that's going to work in a generalizable way.”