Zaharia: Trillion-parameter models are computationally inefficient for knowledge retrieval
Matei Zaharia · No Priors Ep. 11 | With Matei Zaharia, CTO of Databricks · Apr 25, 2023 · at 32:43
Matei Zaharia, CTO of Databricks, argues that database retrieval and search indexes are architecturally superior to scaling model weights for factual recall.
0:00 / 0:14exact quote · 14.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think actually, I think from a computation perspective, it's very inefficient to have like a trillion parameters and have to actually load them all and add and multiply by them. Each time you make an inference, because they're just encoding knowledge, most of which you don't need for that inference.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Matei Zaharia
Zaharia: Model quality experiences diminishing returns from parameter scaling
“And also there's usually, there are usually diminishing returns from scale in, in terms of quality of models in general. And you can also kind of see it in other areas, like in computer vision, for example, we don't have, you know, trillion parameter models.”
Zaharia: 6B parameter models can achieve instruction following with 50x less data
“We just had a larger data set of, you know, human-like conversations, and we had this you know, very kind of modest size open source model that's only six billion parameters, only trained on less than one terabyte of text. So like, 50 times less data than GPD …”
Zaharia: Small models excel at creative generation but struggle with factual recall
“It's surprisingly good at just freeform, like kind of fluent text generation. So you can tell it to like create a story or create a tweet or create a scientific paper abstract, and it does a pretty good job at that. And before that, whenever I talked to my, yo…”
Zaharia: Core LLM technology is commoditizing rapidly and becoming much cheaper
“The thing I can say for sure, especially, and Dolly and like other, you know, results like this really highlighted is it does seem that the core tech is getting commoditized very quickly. So just, if you just want to run, you know, something like today's chat …”
Zaharia: Linear token generation is inadequate for complex reasoning and planning
“This kind of token by token generation we're doing now is not an amazing format for reasoning because you have to like linearly, like do one, say one thing at a time. So it's not really good for like making plans or comparing versions. I think to get a really …”
Matei Zaharia: Every software engineer will become an ML and data engineer
“And I think over time, like I increasingly think that basically, especially because of the capabilities of these AI models, every software engineer will need to become an ML engineer and a data engineer also. As they build their application and we'll, we'll fi…”