The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 3 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Corbitt: Prompt optimization methods like JEPA failed OpenPipe's agent benchmarks
“It didn't work on the problems we tried it on. It just didn't. It got like a minor boost over the sort of like more naive prompt we had and was just like, it was like, okay, Just kind of like our naive prompt with our model gets maybe like 50% on this benchmar…”
Liu: DSPy excels at micro-tasks but fails at end-to-end business workflows
“I think something like DSPy can work because there are like very short term metrics to measure success, right? It is like, did you find the PII or like, did you write the multi-hop question the correct way? But in these like workflows that I've been managing, …”
Schulhoff: Automated Prompt Optimization Fails on Open Generation Without Ground Truth
“One limitation, I guess, is that you really need ground truth labels, so it's harder, if not impossible currently, to optimize open generation tasks, so like Writing, writing newsletters, I suppose. It's harder to automatically optimize those”