Eugene Yan: LLMs are fairly inaccurate on complex document processing tasks
Eugene Yan · [Paper Club] DocETL: Agentic Query Rewriting + Eval for Complex Document Processing w Shreya Shankar · Nov 29, 2024 · at 0:46
Eugene Yan introduces DocETL during the Latent Space Paper Club, outlining the core problem the research aims to solve.
0:00 / 0:08exact quote · 8.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“But the problem is, is that for fairly complex tasks and data, LLM outputs for what we wanted to do is fairly inaccurate.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Eugene Yan
Eugene Yan: LLM ranking should use pairwise comparisons for stability
“I'm actually strongly convinced that it should be pairwise and we can debate that and see how it works. And I also think that... I think it's just more reliable and stable that way.”
Eugene Yan: Verifying LLM outputs is often harder than generating them
“And Shreya also has an interesting point, that it's much easier to verify the output and generate it. I actually observe the opposite. Or maybe it depends on the task. Like, for classification tasks, yes, it's easy. For, like, factuality, or comprehensiveness,…”
Eugene Yan: LLM pipeline validation still requires seed human-labeled data
“I'm of a slightly different take. I feel like we do need some set of seed human labeled data.”
Eugene Yan: LLM-as-a-Judge works reliably when reduced to binary classification
“I think when we simplify it to binary classification metrics, I think it can work. And I think a lot of things can be simplified, like Shreya mentioned, I think a lot of things can be simplified to binary classification metrics. And I've seen evidence of it wo…”
Yan: Using LLMs as evaluators is the only way to scale
“I know that we have to use an LLM as an evaluator. There's no way around it. If we want to scale, I think that's the only way.”
Yan: Pairwise evaluation fails for objective metrics like factuality
“The reason why pairwise preferences cannot work is that if you give two things that are both factual or if you give two things that are both non-factual, you would say that one is better than the other, but it still doesn't meet the bar of being factual enough…”