Katzenberg: Commercial LLMs do not provide extraction confidence scores
Howard Katzenberg · How Glean AI Slashes Vendor Costs: CEO Howard Katzenberg on AI Accounting · Nov 29, 2023 · at 9:55
Howard Katzenberg, CEO of Glean AI, explains the technical obstacles when extracting financial invoice data using LLMs.
0:00 / 0:05exact quote · 5.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“One of the drawbacks of using LLMs is like, we're not receiving confidence scores. On the extractions.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Howard Katzenberg
LLM migration eliminated annotation teams and increased Glean AI's gross margins
“The accuracy rates are better. Our cost to compute has gone down and like, it is the data scientist. We don't need a team to do annotations anymore for the models. And it's just like, it's like our gross margins. It's done like two step functions over the cour…”
Katzenberg: 99 out of 100 vendor bills were approved automatically at Better
“And when I actually analyzed what our approval rate was at Better, 99 out of a hundred bills were getting approved.”
Katzenberg: LLMs are very poor at performing calculations
“We're going to do the calculations because the LLMs are very poor at that.”
Katzenberg: Glean AI business grew over 3x in 2023
“Our business is up like three X, over three X so far this year.”
Katzenberg: External AI model pricing will fall over 12-18 months
“I think their pricing will continue to improve. So we can't just look at like where their pricing is today and assume it's going to be static over the next 12 months or 18 months. Cause like as they compete, their compute costs will come down. Like they'll, I …”