The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 20 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Clark: AI testing should use many weak estimators to detect behavioral differences
“Instead of trying to come up with a small number of strong estimators for performance, where we want to be able to conclusively say A is better than B, Instead, what we want is a large number of potentially weak estimators to be able to determine whether or no…”
Clark: Untuned deep learning models perform worse than tuned simple algorithms
“An untuned, sophisticated system will underperform a tuned simple system.”
Clark: System trust, not performance, limits enterprise AI value
“The thing that's holding back people getting value from these AI systems is not performance. It's not about squeezing out that last half a percent from some eval function or some performance metric. It's about being able to confidently trust these systems.”
Clark: High-level LLM evaluations mask undesired AI system behaviors
“We're seeing people do the exact same thing again today with LLMs, where they're focusing on these high-level metrics, these end outputs, these performance evals, and that ends up masking all of these potentially undesired behaviors within the system itself.”
Clark: Evaluating end-to-end AI performance hides upstream system failures
“And what I think a lot of firms are running into right now is if you're only looking at that last step, if you're only looking at the system's performance as a whole, it can be very difficult to understand when, where, and why behaviors are shifting within thi…”
Clark: Production generative AI adoption will drive dedicated AI ops teams
“I think as we see the rise of these gen AI platforms, we're going to see the rise of more AI ops, the people who have to make sure the system's working and understand when it isn't and then fix it.”
Clark: AI will continue to require supervised learning alongside unsupervised paradigms
“I think there's going to be need for all of it, to be honest. When it comes down to solving a very specific business problem like fraud detection, you don't want the algorithm to learn on its own. Just let a lot of fraud through as you slowly come up with an i…”
Clark: Machine learning is normalized tech; AI is cutting-edge novelty
“Like machine learning is the stuff that's now become easy and then AI is all the fun new stuff. And then as soon as it stops becoming the cutting edge, Then it just becomes, oh, that's just machine learning.”
Clark: Generative AI platform leaders are traditional machine learning veterans
“A lot of the people who are now in charge of building Gen AI platforms or productionizing these massive use cases are the same people who built those original machine learning systems.”
Clark: Generative shadow AI exposes intellectual property to external SaaS vendors
“And it was a somewhat localized problem because like you're doing data science on your laptop versus now I'm just shipping off secret IP to some SaaS company or something like that.”
Clark: AI adoption faces misaligned incentives between providers and enterprise users
“So one big complication is That sometimes the incentives are misaligned. So open AI obviously wants to create the best general purpose foundational models, but an individual business may want a model that solves a very specific problem a very specific way very…”
Clark: An AI confidence gap leaves enterprise generative AI in prototypes
“We talked to a lot of firms that are terrified to cross this AI confidence gap from I've developed something that works good in, in, in theory. How do I actually scale it up in practice? And A lot of times we'll talk to individuals who say, every single time I…”
Clark: Expanding RAG datasets with historical data degrades search quality
“And so RAG has obviously become very prevalent in a wide variety of industries and people use it for a lot of different things. We've spoken with different firms that they were like, okay, well, I'm just going to continue to add more and more data to the corpu…”
Clark: Enterprises avoid high-value AI use cases due to unwieldy risk
“We see some firms attacking the low hanging fruit internal chatbots to like ask questions about HR because they're afraid to take that leap to develop the difficult problem because it's so unwieldy and there is so much risk associated with it. A lot of the mos…”
Clark: Manual hyperparameter tuning fails as machine learning pipelines expand
“Yeah, the complexity grows exponentially. And so some of the standard techniques that people do, like trying to solve this tuning problem in their head or via brute force, just completely fall flat.”
Clark: Google pays $1 million for talent with deep learning intuition
“This is why Google will pay like a million dollars for someone with 10 years of deep learning experiences is that intuition that's built up.”
Clark: Machine learning optimization intuition does not transfer across different problems
“Well, yeah, the intuition for how to configure these systems does not transfer, which is why you need to retune, re-optimize, and reconfigure these systems to make sure they're maximizing that business value.”
Clark: Data availability and engineering form the base of AI's needs hierarchy
“The data problem is the first, like, layer in Maslow's hierarchy of AI. Like, you need to actually have the data. Then you need to be able to understand the business context of what you're aiming for and Do a lot of the data engineering to make sure that you c…”
Clark: One developer today matches a researcher team from a decade ago
“A single person can do now what would have taken a team of researchers a decade ago.”
Clark: Enterprises are moving from generative AI prototypes to centralized platforms
“One thing that we've seen that's really interesting over the last year, year and a half is people have started to shift from kind of science project prototype land where they have a bunch of individual teams trying to roll their own stack and trying to like bu…”