Nov 29, 2024 · 55m · latent-space
[Paper Club] DocETL: Agentic Query Rewriting + Eval for Complex Document Processing w Shreya Shankar
Eugene Yan · 28m spoken Shreya Shankar · 10m spoken Vibhu (Veebu) · 5m spoken Eugene Cheah · 2m spoken Shawn Wang · 1m spoken
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Paper Club session, host Eugene Yan and author Shreya Shankar break down DocETL, an agentic framework that formalizes complex document processing through database-style operators, query rewriting, and automated plan optimization. The discussion covers operator design, LLM-as-a-judge evaluation pipelines, and how DocETL enables comprehensive dataset analytics beyond standard RAG systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 6.8 Guest teaching 3.4 Guest disagreement 1.4 The hosts pushing back 2.1
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 12:24 Pairwise ranking disagreement
Swix and Eugene directly spar over ranking architectures, with Swix doubting the necessity of pairwise comparisons and Eugene asserting strong conviction that pairwise is required.
Hardest push from the hosts ▶ 48:36 Preempting criticism of weak academic evaluationsShreya pushes back on conventional academic benchmark expectations, preempting Eugene's notes by candidly admitting the evaluation lacks ground truth because real-world end users never have labeled data.
Biggest teaching moment ▶ 47:13 Reframing DocETL against RAG misconceptionsShreya systematically educates Vibhu on the architectural scope of DocETL, correcting the idea that it is merely a pre-processing pipeline for RAG and defining it as an analytical semantic layer.
The host holds their own ▶ 43:36 Citing literature on LLM judge ensembling superiorityEugene demonstrates deep domain expertise by citing the Cohere PoLL paper to explain how majority-vote ensembles of smaller models achieve higher human correlation than single large models.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Overview of Core and Auxiliary DocETL Operators | 7 | 0 | 0 | 0 | Eugene walks through DocETL's core operators (Map, Reduce, Resolve, Filter, Split, Gather) and connects them to production data engineering challenges like entity deduplication. | |
| Discussion on Ranking, Filtering, and Operator Categorization | 7 | 3 | 3 | 3 | Swix questions the absence of a rank operator, prompting a technical exchange between Eugene, Swix, and Shreya over pairwise ranking versus embedding similarity. | |
| Rewrite Directives: Data Decomposition and Hierarchical Aggregation | 7 | 3 | 1 | 2 | Eugene explains rewrite directives and shares empirical findings where removing chunking improved metrics, while Shreya reinforces why automated optimization is necessary due to task-dependent variability. | |
| Iterative Refinement, Gleaning, and Projection Synthesis | 7 | 4 | 2 | 3 | Vibhu probes ambiguous paper phrasing regarding data generation versus validation models, and Eugene challenges the premise that LLM verification is always easier than generation. | |
| The Optimization Engine and Pipeline Rewrite Evaluation | 7 | 3 | 1 | 2 | Eugene details the candidate rewrite evaluation pipeline and cost dynamics, while Shreya clarifies practical constraints around synthetic operators lacking ground-truth labels. | |
| Evaluation Reliability and LLM-as-a-Judge Ensembling | 8 | 2 | 1 | 2 | Eugene and Shreya explain why LLM-as-a-judge works for binary precision-recall tasks, with Eugene citing empirical literature on multi-model ensembling beating GPT-4 standalone evaluation. | |
| Differentiating DocETL from Traditional RAG Systems | 4 | 7 | 1 | 2 | Vibhu asks whether DocETL is a document chunker for RAG, prompting Shreya to clearly explain that DocETL is an analytical semantic layer for whole-dataset processing rather than point-lookup retrieval. | |
| Ground Truth Challenges, Evaluation Trade-offs, and Paper Takeaways | 7 | 5 | 2 | 3 | Shreya candidly preempts Eugene's critique of the paper's heuristic evaluation metrics, explaining the reality of messy enterprise data without ground truth, while Eugene emphasizes precision risks. |
Statements from this episode (17)
Eugene Yan: LLMs are fairly inaccurate on complex document processing tasks
“But the problem is, is that for fairly complex tasks and data, LLM outputs for what we wanted to do is fairly inaccurate.”
Eugene Yan: DocETL applies database concepts to unstructured document processing
“Essentially, what this is trying to do is it's trying to take database, database concepts and pandas data frame concepts and try to apply them to shapeless documents.”
Eugene Yan: Loose deduplication thresholds create destructive pipeline mega-clusters
“One several things I've seen in production is that when you do resolve, resolving and deduplication, if your deduplication is too loose, everything deduplicates the single huge mega cluster. And that's when you get that's when you have severity issues”
Swix: Filtering, sorting, and ranking belong together at one API layer
“I had always had this strong view that filtering ranking Sorting, all this stuff is, is kind of like the same layer in the API stack and should be, you know, kind of done together. And this is my number one problem with AI news right now, which is that all fil…”
Eugene Yan: LLM ranking should use pairwise comparisons for stability
“I'm actually strongly convinced that it should be pairwise and we can debate that and see how it works. And I also think that... I think it's just more reliable and stable that way.”
Eugene Yan: Removing document chunking boosted pipeline downstream metrics by 50%
“I was asked to help with a pipeline, and I was able to improve downstream metrics significantly by 20 to 50% by removing chunking.”
Eugene Yan predicts LLM data pipelines will become reliable within two years
“It's gonna be a bit lossy, it's gonna be a bit stochastic, but I think we will figure it out in the next one to two years to get it to a more reliable state.”
Shreya Shankar: Chunking efficacy is task-specific, requiring automated pipeline optimization
“Sometimes it's beneficial to chunk and sometimes you should not chunk. And we have observed this in a number of workloads and the insight that we've gained is that we will never know what it's all task specific and data specific. And we are so further convince…”
Eugene Yan: Verifying LLM outputs is often harder than generating them
“And Shreya also has an interesting point, that it's much easier to verify the output and generate it. I actually observe the opposite. Or maybe it depends on the task. Like, for classification tasks, yes, it's easy. For, like, factuality, or comprehensiveness,…”
Vibhu: Synthetic data research shows LLMs verify better than they generate
“A lot of the synthetic datagen papers, like orca-three, wizard-lm, they show that models are better at verifying output than generating output.”
Eugene Yan: LLM pipeline validation still requires seed human-labeled data
“I'm of a slightly different take. I feel like we do need some set of seed human labeled data.”
Eugene Yan: DocETL pipeline optimizer costs roughly $100 and 30 minutes
“Running the optimizer, right, they used the optimizer. I don't know how many plans the optimizer generated, but it cost approximately a hundred dollars and less than half an hour. Just go get lunch and you come back and you get your optimized pipeline. And the…”
Eugene Yan: LLM-as-a-Judge works reliably when reduced to binary classification
“I think when we simplify it to binary classification metrics, I think it can work. And I think a lot of things can be simplified, like Shreya mentioned, I think a lot of things can be simplified to binary classification metrics. And I've seen evidence of it wo…”
Eugene Yan: Ensembled LLMs outperform standalone GPT-4 for evaluation tasks
“So in this paper here by Kohir, what they did was they have a reference model, and this reference model is GPT-IV. And then essentially what they did was the ensemble command R, Haiku, and GPT-IV. And I can't remember what the I think the ensemble was just maj…”
Shreya Shankar: GPT-4o Mini reduces DocETL optimization cost by 90%
“The reason it was a hundred dollars, if I ran the optimizer with GPT-Foro mini as the LLMs, it would be 10 dollars. But we use GPT four. Oh, just because I think we did this at a time where many hadn't come out yet.”
Shreya Shankar: DocETL builds semantic unstructured layers, not point-lookup RAG systems
“This is very different from traditional rag or Q&A or document processing for a chatbot. Like, the kinds of queries that people are, people want to use .etl for can be expressed as etl style sweep and harvest, kind of, I want to look at my entire dataset. I wo…”