LLM logic problems are best solved by having models implicitly execute code
Jack Krawczyk · The Bard blueprint | Creating value, shipping fast, and advancing AI | Jack Krawczyk (Google) · Nov 30, 2023 · at 45:48
Jack Krawczyk, Product Director for Google Bard, describes the engineering pattern used to handle arithmetic and logic in LLMs.
0:00 / 0:09exact quote · 9.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The way that you start solving math and some logic-oriented problems is under the hood implicitly ask the model to write and execute code.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jack Krawczyk
Google frames Bard as an experiment because LLMs aren't fully baked products
“It's part of why we talked about Bard as an experiment at this point. We don't think that this technology is yet ready to be a fully baked product.”
Information retrieval is not the majority use case for Google Bard
“Some people, and it turns out not the majority use case, but some people are going to use it to try to find information.”
Persistent memory for LLMs will take much longer to solve than expected
“And so I am beyond Fascinated and intrigued by the research that's taking place around memory and being able to reference things from previous conversations, but I think it's going to take way longer than we expect.”
LLM hallucination is a core generative feature, not a bug
“And so finding the way of harnessing that as a feature rather than a bug, I think is going to be challenging, especially when one of the perceived shortcomings of your product is actually going to be, is like the core of what makes it function. I don't know th…”
Google Bard refuses pediatric medical dosage questions to avoid hallucination risk
“How much Tylenol should I give to my kid when they have a fever? Very well-intentioned question, but you want to make sure that the answer that you provide is as tuned toward not hallucinating as possible. And so for early versions, and even to now, we think t…”