Zaharia: Grounding LLMs with external vetted data fixes stale knowledge and hallucinations
Matei Zaharia · No Priors Ep. 11 | With Matei Zaharia, CTO of Databricks · Apr 25, 2023 · at 14:21
Matei Zaharia, co-founder and CTO of Databricks, explains retrieval-augmented generation architectures for grounding LLMs in enterprise data.
0:00 / 0:31exact quote · 31.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The two big problems with it are number one, like the knowledge is not up to date. You know, it's only, it only knows stuff it was strained on. And number two, a lot of the things it says are inaccurate and it's confident, but like wrong in various ways. And I think you can tackle both of these by combining some kind of language model with you know, a system that, that, you know, pulls out like vetted data, either from documents like a search engine or from you know, APIs and tables and stuff like that inside your company.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Matei Zaharia
Zaharia: Model quality experiences diminishing returns from parameter scaling
“And also there's usually, there are usually diminishing returns from scale in, in terms of quality of models in general. And you can also kind of see it in other areas, like in computer vision, for example, we don't have, you know, trillion parameter models.”
Zaharia: 6B parameter models can achieve instruction following with 50x less data
“We just had a larger data set of, you know, human-like conversations, and we had this you know, very kind of modest size open source model that's only six billion parameters, only trained on less than one terabyte of text. So like, 50 times less data than GPD …”
Zaharia: Small models excel at creative generation but struggle with factual recall
“It's surprisingly good at just freeform, like kind of fluent text generation. So you can tell it to like create a story or create a tweet or create a scientific paper abstract, and it does a pretty good job at that. And before that, whenever I talked to my, yo…”
Zaharia: Core LLM technology is commoditizing rapidly and becoming much cheaper
“The thing I can say for sure, especially, and Dolly and like other, you know, results like this really highlighted is it does seem that the core tech is getting commoditized very quickly. So just, if you just want to run, you know, something like today's chat …”
Zaharia: Linear token generation is inadequate for complex reasoning and planning
“This kind of token by token generation we're doing now is not an amazing format for reasoning because you have to like linearly, like do one, say one thing at a time. So it's not really good for like making plans or comparing versions. I think to get a really …”
Zaharia: Trillion-parameter models are computationally inefficient for knowledge retrieval
“I think actually, I think from a computation perspective, it's very inefficient to have like a trillion parameters and have to actually load them all and add and multiply by them. Each time you make an inference, because they're just encoding knowledge, most o…”