Dean: 1,000x energy penalty for moving data forces machine learning batching
Jeff Dean · Jeff Dean: The 1% Rule for Building in AI · Y Combinator · Jul 30, 2026 · at 12:52
Google Chief Scientist Jeff Dean discusses AI hardware bottlenecks and memory bandwidth constraints with YC Managing Partner Diana Hu at Startup School.
0:00 / 0:38exact quote · 39.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I mean, I think the example you raised of a thousand X difference in bringing moving data versus actually computing on it in, in terms of energy is, is a pretty significant one. And it shapes a lot of aspects of what we do in machine learning. Because if you didn't have that thousand X difference, then, you know, you wouldn't have to do batching, but you have to do batching of You know, many examples, or maybe many tokens at once, in order to amortize that data movement, so that you can, ah, you know, not pay a thousand x slowdown, but pay a thousand x divided by batch size, ah, energy cost.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jeff Dean
Jeff Dean: AI models have reached junior engineer capability level
“The models have been getting a lot better at sort of agent-based, longer-running coding tasks, and it seems pretty clear that they are now actually pretty capable, and depending on exactly your definition of junior engineer, it seems pretty spot on, I would sa…”
Dean: Specialized inference hardware will surpass general GPUs and TPUs
“I think, ah, you're gonna see more and more, ah high performance and low energy inference hardware systems, because I think everyone is now realizing that inference is the key to making, you know, these agent-based systems be available to more and more people,…”
Jeff Dean: AI agents can run autonomously for days or weeks
“Probably one thing is people don't quite realize how possible it is to have, you know, agent based systems that can run not just for an hour or two hours on a problem you care about, but for some problem domains and with highly capable models underlying them, …”
Dean: AI startups should target tasks with 1% model success, not 20%
“If they're completely failing, that's probably a good sign. If they're kind of able to do some of it, but not very well, That's maybe not a great sign because that's probably a sign that the capability is starting to be present in those models and with more tr…”
Jeff Dean: AI models will not have good taste in problem selection
“I think models are not necessarily going to be that good at it. So you're going to have people steering a lot of AI assisted computation in order to accomplish great things and more quickly. But that essence of what it is you want your models to do is the key …”
Jeff Dean: There is no impediment to fully automating ML model research
“There's no, ah, you know, real impediment to making that be a much more automated loop, where the model itself decides it's going to explore, or maybe with a nudge from some people, ah, at the various highest level, like, oh, why don't you try some new ideas a…”