Apr 25, 2023 · 40m · no-priors
No Priors Ep. 11 | With Matei Zaharia, CTO of Databricks
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Databricks co-founder and CTO Matei Zaharia joins No Priors to discuss open-source AI development, the limitations of brute-force parameter scaling, and the system architectures needed to ground enterprise language models in private data.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 12.6% of the talking time here. How this is scored →
The hosts as informed peer 4.7 Guest teaching 4.3 Guest disagreement 0.7 The hosts pushing back 0.4
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 32:40 Dismissing pure neural network scaling as inefficient
Matei pushes back against model purists who celebrate trillion-parameter networks for memorizing trivia, arguing from an engineer's perspective that it is computationally wasteful compared to building a database lookup.
Hardest push from the hosts ▶ 31:13 Sarah questions the scaling ceiling hypothesisSarah directly challenges Matei's assumption about diminishing returns, pointing out that many prominent AI researchers do not foresee an asymptote and pressing him to explain where the bottleneck originates.
Biggest teaching moment ▶ 8:00 Deconstructing the standard GPT-3 scaling assumptionsMatei educates the hosts on the history of LLM paradigms, explaining how researchers mistakenly assumed conversational capability required massive parameter counts before instruction-tuning proved otherwise.
The host holds their own ▶ 35:08 Sarah references Cicero and multi-system planningSarah demonstrates strong technical domain grasp by referencing Noam Brown's research on Cicero to reinforce the distinction between monolithic neural scaling and structured planning architectures.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Origins, Cloud Evolution, and Scale of Databricks | 3 | 3 | 0 | 0 | Sarah Guo asks foundational questions regarding the founding of Databricks from UC Berkeley research and its growth to over a billion in ARR. Matei explains the confluence of big data, cloud migration, and machine learning in a purely informative, non-adversarial manner. | |
| Stanford Academic Research and Knowledge-Intensive Systems | 4 | 4 | 0 | 0 | Sarah and Elad ask about Matei's dual role at Stanford and the genesis of Dolly. Matei outlines his work on knowledge-intensive systems and open-sourcing instruction-following models inspired by Stanford's Alpaca. | |
| Instruction Tuning vs. Brute-Force Scaling | 5 | 5 | 1 | 0 | Elad demonstrates domain expertise by framing the industry's focus on scaling compute vs instruction following. Matei educates on why smaller 6B models with curated instruction tuning challenge standard beliefs previously established by the GPT-3 scaling paradigm. | |
| Surprising Model Capabilities and the Dolly Naming Origin | 4 | 4 | 0 | 0 | Elad asks about emergent capabilities and the naming origin of Dolly. Matei describes how smaller models surprised researchers by succeeding at creative fluency while struggling more with factual precision. | |
| Grounding Language Models with Reliable Enterprise Data | 4 | 5 | 1 | 0 | Elad inquires about future Databricks directions. Matei explains why grounding LLMs with vetted external enterprise data sources is vastly superior to relying on raw parameter memorization to reduce hallucinations. | |
| The Commoditization of AI Models and Scaling Limitations | 5 | 5 | 2 | 1 | Sarah pushes on whether massive model scaling actually matters for near-term enterprise deployments. Matei offers a contrarian engineering perspective, arguing that core models are commoditizing quickly and autoregressive token generation faces inherent reasoning limits. | |
| Enterprise AI Tooling, Real-World Use Cases, and Defensibility | 6 | 4 | 0 | 0 | Sarah and Elad actively discuss operational tooling, customer support use cases, and embeddings. Matei explains practical enterprise architectures, emphasizing that custom datasets and feedback loops provide defensible startup moats. | |
| From PhD Researcher to CTO: Lessons on Leadership and Systems | 7 | 5 | 2 | 3 | Sarah offers thoughtful pushback by pointing out that many top researchers see no asymptote to scaling, citing Noam Brown's research on planning models. Matei holds his ground, arguing from systems engineering principles that parameter memorization exhibits diminishing returns compared to database retrieval. | |
| The Evolution of Unstructured Data and AI Software Engineering | 4 | 4 | 0 | 0 | Sarah asks about the broader trajectory of AI software engineering. Matei closes with an insightful historical parallel to 1990s web application development, predicting every software engineer will eventually become an ML and data engineer. |
Statements from this episode (16)
Zaharia: Databricks reached 6,000 employees and crossed $1B in ARR
“The company has about 6000 employees now, and last year we said that we cost a billion dollars in AR and we're continuing to go.”
Zaharia: Nearly 1,000 Databricks customers used LLMs before ChatGPT
“I think we had like almost a thousand customers that were using these in, in some form”
Zaharia: Databricks created Dolly by cloning Stanford's Alpaca approach
“Dolly is partly based on this great result from some other faculty members at Stanford called Alpaca, where they tested a way to, you know, basically they use the model to generate a bunch of realistic conversations, and then they use this to train another mod…”
Zaharia: 6B parameter models can achieve instruction following with 50x less data
“We just had a larger data set of, you know, human-like conversations, and we had this you know, very kind of modest size open source model that's only six billion parameters, only trained on less than one terabyte of text. So like, 50 times less data than GPD …”
Zaharia: Small models excel at creative generation but struggle with factual recall
“It's surprisingly good at just freeform, like kind of fluent text generation. So you can tell it to like create a story or create a tweet or create a scientific paper abstract, and it does a pretty good job at that. And before that, whenever I talked to my, yo…”
Zaharia: Grounding LLMs with external vetted data fixes stale knowledge and hallucinations
“The two big problems with it are number one, like the knowledge is not up to date. You know, it's only, it only knows stuff it was strained on. And number two, a lot of the things it says are inaccurate and it's confident, but like wrong in various ways. And I…”
Zaharia: Reducing hallucinations may be easier with small models than big ones
“It may actually be easier with small models than with big ones to reduce hallucination from them, but it, you know, I think it's still an open question”
Zaharia: Core LLM technology is commoditizing rapidly and becoming much cheaper
“The thing I can say for sure, especially, and Dolly and like other, you know, results like this really highlighted is it does seem that the core tech is getting commoditized very quickly. So just, if you just want to run, you know, something like today's chat …”
Zaharia: ChatGPT-level AI models will soon run locally on smartphones
“At least to get something with today's capabilities, I think it'll be you know, it'll be very affordable and you might just be able to run it locally on, you know, your phone or something.”
Zaharia: Linear token generation is inadequate for complex reasoning and planning
“This kind of token by token generation we're doing now is not an amazing format for reasoning because you have to like linearly, like do one, say one thing at a time. So it's not really good for like making plans or comparing versions. I think to get a really …”
Zaharia: Data-rich enterprises may become model and data vendors
“And I think, I actually think even a lot of the enterprises that like have a lot of the data and various domains might turn more into data or model vendors of some form in the future you know, as they use this to like build something that no one else can.”
Zaharia: Long-term startup moats require unique datasets or user feedback
“I would say, you know, for people thinking about startups and so on, like you want your startup to have you know, a long-term defensible mode, ideally something that goes over time also. So anything around a unique dataset, for example, or unique, like feedbac…”
Zaharia: Academic founders must unlearn short-term prototyping for maintainable engineering
“A lot of research, at least in computer science, the kind of stuff that I've worked on, a lot of research is basically, is mostly prototyping. It's like, can we showcase an idea, but it's not really software engineering of like, we'll build a thing that can be…”
Zaharia: Model quality experiences diminishing returns from parameter scaling
“And also there's usually, there are usually diminishing returns from scale in, in terms of quality of models in general. And you can also kind of see it in other areas, like in computer vision, for example, we don't have, you know, trillion parameter models.”
Zaharia: Trillion-parameter models are computationally inefficient for knowledge retrieval
“I think actually, I think from a computation perspective, it's very inefficient to have like a trillion parameters and have to actually load them all and add and multiply by them. Each time you make an inference, because they're just encoding knowledge, most o…”
Matei Zaharia: Every software engineer will become an ML and data engineer
“And I think over time, like I increasingly think that basically, especially because of the capabilities of these AI models, every software engineer will need to become an ML engineer and a data engineer also. As they build their application and we'll, we'll fi…”