Guu: BERT Demonstrated Vast Unprogrammed World Knowledge Purely From Pre-Training
Kelvin Guu · No Priors Ep. 15 | With Kelvin Guu, Staff Research Scientist, Google Brain · May 4, 2023 · at 1:23
Google Brain staff research scientist Kelvin Guu discusses early findings during the development of BERT that inspired research into retrieval-augmented generation.
0:00 / 0:14exact quote · 14.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think one of the things that became very apparent early on when playing with BERT was unlike all the prior generations of models, it had a large amount of world knowledge that we didn't deliberately encode into it. It wasn't in, you know, the fine tuning data. It was all within pre-training.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Kelvin Guu
Guu: Without Attribution, Distinguishing LLM Generalization From Memorization Is Impossible
“Without being able to track that down, we'll never know if these models are generalizing or just kind of cleverly patching together what they know.”
Guu: Fundamental AI Advances May Make Agent Workflow Engineering Obsolete
“But it's quite possible that fundamental advances could come along and this kind of workflow hacking or workflow engineering could go the same way as feature engineering or specialized architectures.”
Guu: FLAN Demonstrated Zero-Shot Generalization After Multi-Task Training on 100 Tasks
“What we were able to show was that if you train on a hundred tasks, if we then show the model a new 101 task, it will be able to adapt to that without having seen that particular task before.”
Guu: AI Agents Relying Purely on Explicit Reasoning May Fail to Scale
“That's something that seems to be missing from these autonomous agents right now. Everything is explicit reasoning, and at some point that might not scale or that might become brittle.”
Guu: Tool-Using AI Models Still Struggle to Fulfill Original Retrieval Promises
“And since then we've encountered many interesting challenges on top of that idea that I would say even in, in the sort of systems you see today, these tool using models that issue Google searches or provide citations, they still face some challenges in terms o…”
Guu: Retrieval Models Best Serve Enterprise Needs Through Modularity and Privacy
“So it's actually been a case where I think more and more the benefits of a retrieval augmented model are for modularity. For personal information that you might not want in the main model, for adapting to, say, like an enterprise customer that has special info…”