Liang: LLMs are not just memorizing because novel concept fusion requires invention
Dr. Percy Liang · No Priors Ep. 7 | With Stanford Professor Dr. Percy Liang · Apr 25, 2023 · at 21:53
Stanford Professor Dr. Percy Liang discusses whether large language models exhibit genuine creativity or merely regurgitate memorized text from their training data.
0:00 / 0:20exact quote · 20.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“You know, people say that sometimes all language models just memorize because they're so big and train on clearly a lot of texts, but these examples, I think really indicate that there's no way That these language models are just memorizing because this text just doesn't exist and you have to have some creative juice and invent something new.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dr. Percy Liang
Liang: The AI industry is retreating from its open culture
“And what we're seeing now is sort of a retreat of that open culture where models are now being only accessible via APIs. We don't really know all the secret sauce that's going behind them, and there's sort of limited access.”
Liang: Big tech scale forced AI academia to focus on understanding models
“And now today I think it's the dynamic is, is quite different because it's no longer academia's job isn't just to get things to work because you can do that in other ways. There's a lot of resources going into big tech companies where there's if you have data …”
Liang: AI benchmarks should target superhuman reliability over human mimicry
“I think we're getting to a point where along many axes, it's a superhuman or should be superhuman. And I think we should maybe define more of an objective measure of like what we actually want. We want something that's very reliable, is grounded. You know, I o…”
Liang: Next-token prediction forces language models to build world models
“If you think about predicting the next word, It's, it seems very simple, but you have to really internalize a lot of what is going on in this context. What are the previous words? What's the syntax? What's who's saying them? And all of that information and con…”
Liang: Foundation Models Cause the Concept of an AI Task to Dissolve
“And this was a paradigm shift, in my opinion, because it changed the way that we conceptualize Machine learning and NLP systems from these bespoke systems where you're, it's trained to do question answering, to train to do this, to just a general substrate whe…”
Liang: Future AI models may be too powerful for unrestricted release
“I think these models are extremely powerful and maybe the models right now I think are, well, if they were out and open, it would be maybe okay, but in the future these models could be extremely good and having them, you know, anyone, anything goes. Might we m…”