The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 19 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Eugene Yan: LLM ranking should use pairwise comparisons for stability
“I'm actually strongly convinced that it should be pairwise and we can debate that and see how it works. And I also think that... I think it's just more reliable and stable that way.”
Eugene Yan: Verifying LLM outputs is often harder than generating them
“And Shreya also has an interesting point, that it's much easier to verify the output and generate it. I actually observe the opposite. Or maybe it depends on the task. Like, for classification tasks, yes, it's easy. For, like, factuality, or comprehensiveness,…”
Eugene Yan: LLM pipeline validation still requires seed human-labeled data
“I'm of a slightly different take. I feel like we do need some set of seed human labeled data.”
Eugene Yan: LLM-as-a-Judge works reliably when reduced to binary classification
“I think when we simplify it to binary classification metrics, I think it can work. And I think a lot of things can be simplified, like Shreya mentioned, I think a lot of things can be simplified to binary classification metrics. And I've seen evidence of it wo…”
Yan: Using LLMs as evaluators is the only way to scale
“I know that we have to use an LLM as an evaluator. There's no way around it. If we want to scale, I think that's the only way.”
Yan: Pairwise evaluation fails for objective metrics like factuality
“The reason why pairwise preferences cannot work is that if you give two things that are both factual or if you give two things that are both non-factual, you would say that one is better than the other, but it still doesn't meet the bar of being factual enough…”
Meta used stepwise reward models and Monte Carlo Tree Search for Llama 3.1
“They actually went the extra step to, no pun intended, to actually train stepwise reward models. That's kind of crazy, no? I mean, they wanted each step in the chain of thought to be so good that they actually took the extra effort to train step, to train step…”
Eugene Yan: GPU floating-point math makes temperature-zero inference non-deterministic
“For GPUs with floating points, and you push it through so many calculations, and so many met miles, the floating points aren't just not gonna be precise. So that's why even if temperature is zero, it's not gonna be the same throughout, ah, for multiple request…”
Eugene Yan: LLMs are fairly inaccurate on complex document processing tasks
“But the problem is, is that for fairly complex tasks and data, LLM outputs for what we wanted to do is fairly inaccurate.”
Eugene Yan: Removing document chunking boosted pipeline downstream metrics by 50%
“I was asked to help with a pipeline, and I was able to improve downstream metrics significantly by 20 to 50% by removing chunking.”
Eugene Yan predicts LLM data pipelines will become reliable within two years
“It's gonna be a bit lossy, it's gonna be a bit stochastic, but I think we will figure it out in the next one to two years to get it to a more reliable state.”
Eugene Yan: Ensembled LLMs outperform standalone GPT-4 for evaluation tasks
“So in this paper here by Kohir, what they did was they have a reference model, and this reference model is GPT-IV. And then essentially what they did was the ensemble command R, Haiku, and GPT-IV. And I can't remember what the I think the ensemble was just maj…”
Yan: LLM evaluation criteria cannot be determined without inspecting real outputs
“What they're saying is that it is impossible to completely determine good evaluation criteria without actually looking at LLM outputs. So essentially the point is you have to look at the data before you overcome evaluation criteria.”
Yan: Annotators must update past grades when evaluation criteria drift
“If you find that your criteria has drifted, instead of trying to maintain the same criteria, And aligning to the previous grades. Instead, what we should do is we should revisit those previous grades and fix it because it's an iterative process.”
Yan: Optimizing LLM evaluation prompts requires 100 to 400 labeled examples
“I actually think the right number should be maybe a hundred to 400 if you want to be optimizing based on this.”
Eugene Yan: DocETL applies database concepts to unstructured document processing
“Essentially, what this is trying to do is it's trying to take database, database concepts and pandas data frame concepts and try to apply them to shapeless documents.”
Eugene Yan: Loose deduplication thresholds create destructive pipeline mega-clusters
“One several things I've seen in production is that when you do resolve, resolving and deduplication, if your deduplication is too loose, everything deduplicates the single huge mega cluster. And that's when you get that's when you have severity issues”
Eugene Yan: DocETL pipeline optimizer costs roughly $100 and 30 minutes
“Running the optimizer, right, they used the optimizer. I don't know how many plans the optimizer generated, but it cost approximately a hundred dollars and less than half an hour. Just go get lunch and you come back and you get your optimized pipeline. And the…”
Yan: EvalGen requires fewer assertions than SPADE for comparable evaluation
“Evalgen only needed three assertions, be quote assertions, or LLM prompts. So essentially, it needed less than spade, which needed five to get comparable results. Now, when we look at the product pipeline we see that evalgen only requires four assertions, whic…”