Zaharia: Scaling down from bulk ingestion to serving is easier than scaling up
Matei Zaharia · The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin · Jun 24, 2026 · at 55:55
Databricks co-founder Matei Zaharia explains why starting with large-scale MapReduce/Spark data processing gave Databricks a strategic architectural advantage over Snowflake.
0:00 / 0:19exact quote · 20.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It turned out that, you know, it's easier to go from that bod thing that's really good at the scale and ingesting and super low cost and create versions in it that have the speed and features of the, you know, super easy to use, like smaller data for business users thing.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Matei Zaharia
Zaharia: Open agent hosting layers will win over proprietary alternatives
“Another way to think about it is like, imagine, you know we, our thing wasn't open. We had some kind of agent hosting thing, but it's not open. And then there is an open one. If you're, which one's gonna win in the long run? So like here, because there is this…”
Matei Zaharia: AI agents need stateful contextual security policies, not binary permissions
“A lot of coding agents today have very basic things like you can tell me which tool patterns I'll allow or disallow or whatever. It's like yes or no, but that puts you in a very tough spot. So just as an example, like, should my agent be able to read, you know…”
Zaharia: AI agents are useless without a collaboration and history layer
“Plus the agent is like completely useless if you can't share sessions with someone and have history and have search and all this like layer on top of it for collaboration.”
Zaharia: Databricks abandons frontier AI models to focus on agent systems
“Even though we did launch open source model DBRX, and, you know, we went up to, like, sort of above the LAMA-R III scale, we decided that we really want to focus on, there'll be so many people releasing models, and instead of doing the general model where, lik…”
Zaharia: Coding agents and custom agents share identical underlying technical problems
“They're like, why are you doing coding agents and custom agents in the same thing, but I said it's basically the same problems.”
Zaharia: Protocol design remains essential for multi-party interoperability despite fast coding
“For this type of interoperability where multiple parties that are moving at different speeds are building stuff and you still want some layer on top to coordinate you do want to design it and build it. So it reminds me of that, like agents talking to each othe…”