The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 12 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Clark: AI testing should use many weak estimators to detect behavioral differences
“Instead of trying to come up with a small number of strong estimators for performance, where we want to be able to conclusively say A is better than B, Instead, what we want is a large number of potentially weak estimators to be able to determine whether or no…”
Clark: Untuned deep learning models perform worse than tuned simple algorithms
“An untuned, sophisticated system will underperform a tuned simple system.”
Clark: System trust, not performance, limits enterprise AI value
“The thing that's holding back people getting value from these AI systems is not performance. It's not about squeezing out that last half a percent from some eval function or some performance metric. It's about being able to confidently trust these systems.”
Clark: High-level LLM evaluations mask undesired AI system behaviors
“We're seeing people do the exact same thing again today with LLMs, where they're focusing on these high-level metrics, these end outputs, these performance evals, and that ends up masking all of these potentially undesired behaviors within the system itself.”
Clark: Evaluating end-to-end AI performance hides upstream system failures
“And what I think a lot of firms are running into right now is if you're only looking at that last step, if you're only looking at the system's performance as a whole, it can be very difficult to understand when, where, and why behaviors are shifting within thi…”
Clark: Machine learning is normalized tech; AI is cutting-edge novelty
“Like machine learning is the stuff that's now become easy and then AI is all the fun new stuff. And then as soon as it stops becoming the cutting edge, Then it just becomes, oh, that's just machine learning.”
Clark: AI adoption faces misaligned incentives between providers and enterprise users
“So one big complication is That sometimes the incentives are misaligned. So open AI obviously wants to create the best general purpose foundational models, but an individual business may want a model that solves a very specific problem a very specific way very…”
Clark: An AI confidence gap leaves enterprise generative AI in prototypes
“We talked to a lot of firms that are terrified to cross this AI confidence gap from I've developed something that works good in, in, in theory. How do I actually scale it up in practice? And A lot of times we'll talk to individuals who say, every single time I…”
Clark: Expanding RAG datasets with historical data degrades search quality
“And so RAG has obviously become very prevalent in a wide variety of industries and people use it for a lot of different things. We've spoken with different firms that they were like, okay, well, I'm just going to continue to add more and more data to the corpu…”
Clark: Manual hyperparameter tuning fails as machine learning pipelines expand
“Yeah, the complexity grows exponentially. And so some of the standard techniques that people do, like trying to solve this tuning problem in their head or via brute force, just completely fall flat.”
Clark: Machine learning optimization intuition does not transfer across different problems
“Well, yeah, the intuition for how to configure these systems does not transfer, which is why you need to retune, re-optimize, and reconfigure these systems to make sure they're maximizing that business value.”
Clark: Data availability and engineering form the base of AI's needs hierarchy
“The data problem is the first, like, layer in Maslow's hierarchy of AI. Like, you need to actually have the data. Then you need to be able to understand the business context of what you're aiming for and Do a lot of the data engineering to make sure that you c…”