Zaharia: Small models excel at creative generation but struggle with factual recall
Matei Zaharia · No Priors Ep. 11 | With Matei Zaharia, CTO of Databricks · Apr 25, 2023 · at 10:38
Matei Zaharia, CTO and co-founder of Databricks, discusses unexpected findings from building their open-source language model, Dolly.
0:00 / 0:46exact quote · 46.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It's surprisingly good at just freeform, like kind of fluent text generation. So you can tell it to like create a story or create a tweet or create a scientific paper abstract, and it does a pretty good job at that. And before that, whenever I talked to my, you know, NLP, like researcher friends, they thought that that creativity was the thing that required a lot of parameters from something like GPT-C. Like they actually told me all the knowledge intensive stuff, like remembering facts, tell me the capital of like France and whatever. That's not surprising that a small model with a few parameters can do it, but the creativity, that's like really hard. So this one is actually pretty good at the creativity and generation. It's less good at remembering lots of facts, which kind of makes sense given the parameters.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Matei Zaharia
Zaharia: Model quality experiences diminishing returns from parameter scaling
“And also there's usually, there are usually diminishing returns from scale in, in terms of quality of models in general. And you can also kind of see it in other areas, like in computer vision, for example, we don't have, you know, trillion parameter models.”
Zaharia: 6B parameter models can achieve instruction following with 50x less data
“We just had a larger data set of, you know, human-like conversations, and we had this you know, very kind of modest size open source model that's only six billion parameters, only trained on less than one terabyte of text. So like, 50 times less data than GPD …”
Zaharia: Core LLM technology is commoditizing rapidly and becoming much cheaper
“The thing I can say for sure, especially, and Dolly and like other, you know, results like this really highlighted is it does seem that the core tech is getting commoditized very quickly. So just, if you just want to run, you know, something like today's chat …”
Zaharia: Linear token generation is inadequate for complex reasoning and planning
“This kind of token by token generation we're doing now is not an amazing format for reasoning because you have to like linearly, like do one, say one thing at a time. So it's not really good for like making plans or comparing versions. I think to get a really …”
Zaharia: Trillion-parameter models are computationally inefficient for knowledge retrieval
“I think actually, I think from a computation perspective, it's very inefficient to have like a trillion parameters and have to actually load them all and add and multiply by them. Each time you make an inference, because they're just encoding knowledge, most o…”
Matei Zaharia: Every software engineer will become an ML and data engineer
“And I think over time, like I increasingly think that basically, especially because of the capabilities of these AI models, every software engineer will need to become an ML engineer and a data engineer also. As they build their application and we'll, we'll fi…”