Zaharia: Core LLM technology is commoditizing rapidly and becoming much cheaper
Matei Zaharia · No Priors Ep. 11 | With Matei Zaharia, CTO of Databricks · Apr 25, 2023 · at 16:44
Matei Zaharia, co-founder and CTO of Databricks, discusses the trajectory of foundation models and infrastructure efficiency with Sarah Guo.
0:00 / 0:34exact quote · 34.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The thing I can say for sure, especially, and Dolly and like other, you know, results like this really highlighted is it does seem that the core tech is getting commoditized very quickly. So just, if you just want to run, you know, something like today's chat GPD it will be a lot cheaper because all these hardware manufacturers are building devices that are specialized and much cheaper. And another thing that's making it Less expensive is we're figuring out ways to get a smaller model with less data, fewer parameters and stuff to get similar performance.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Matei Zaharia
Zaharia: Model quality experiences diminishing returns from parameter scaling
“And also there's usually, there are usually diminishing returns from scale in, in terms of quality of models in general. And you can also kind of see it in other areas, like in computer vision, for example, we don't have, you know, trillion parameter models.”
Zaharia: 6B parameter models can achieve instruction following with 50x less data
“We just had a larger data set of, you know, human-like conversations, and we had this you know, very kind of modest size open source model that's only six billion parameters, only trained on less than one terabyte of text. So like, 50 times less data than GPD …”
Zaharia: Small models excel at creative generation but struggle with factual recall
“It's surprisingly good at just freeform, like kind of fluent text generation. So you can tell it to like create a story or create a tweet or create a scientific paper abstract, and it does a pretty good job at that. And before that, whenever I talked to my, yo…”
Zaharia: Linear token generation is inadequate for complex reasoning and planning
“This kind of token by token generation we're doing now is not an amazing format for reasoning because you have to like linearly, like do one, say one thing at a time. So it's not really good for like making plans or comparing versions. I think to get a really …”
Zaharia: Trillion-parameter models are computationally inefficient for knowledge retrieval
“I think actually, I think from a computation perspective, it's very inefficient to have like a trillion parameters and have to actually load them all and add and multiply by them. Each time you make an inference, because they're just encoding knowledge, most o…”
Matei Zaharia: Every software engineer will become an ML and data engineer
“And I think over time, like I increasingly think that basically, especially because of the capabilities of these AI models, every software engineer will need to become an ML engineer and a data engineer also. As they build their application and we'll, we'll fi…”