May 23, 2025 · 45m · a16z
Building AI Systems You Can Trust
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the a16z podcast, Scott Clark, Co-founder and CEO of Distributional, discusses the paradigm shift in artificial intelligence from chasing raw model performance to establishing system trust, reliability, and enterprise governance. He outlines strategies for testing non-deterministic models, managing AI technical debt, and building robust platform engineering to safely scale production Gen AI.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The host as informed peer 3.4 Guest teaching 3.8 Guest disagreement 1.1 The host pushing back 1.1
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 22:22 Pushback on platform abstraction benefits
Scott gently counters Matt's skepticism about platform gateways, arguing that uniform abstractions solve essential governance and model-switching hurdles for developers.
Hardest push from the host ▶ 22:22 Challenging platform developer appealMatt directly challenges Scott's narrative that developers want centralized platform layers, pointing out that abstraction between developers and LLMs can feel like unnecessary friction.
Biggest teaching moment ▶ 32:02 Mathematical rationale for weak estimatorsScott educates Matt on the core mathematical distinction between traditional LLM evals using strong estimators and distributional testing using many weak, high-entropy estimators to detect subtle behavior shifts.
The host holds their own ▶ 39:41 Conway's Law in prompt engineeringMatt demonstrates sharp domain knowledge by connecting Conway's Law to prompt engineering, observing how corporate system prompts reflect internal organizational structure and bureaucracy.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| Defining Machine Learning vs. Artificial Intelligence | 3 | 4 | 1 | 1 | Matt asks foundational questions about the distinction between traditional ML and modern generative AI. Scott reframes the core challenge in AI from pure accuracy optimization to system trust and behavior verification based on his experience building SigOpt and leading AI at Intel. | |
| Building Production AI Systems: Then and Now | 2 | 5 | 1 | 1 | Matt prompts Scott on product owner worries in production AI. Scott educates the host on the three structural complexities of modern GenAI—non-determinism, non-stationarity, and component chaining—showing how upstream shifts propagate chaos downstream. | |
| Establishing Enterprise Trust and Defining Model Behavior | 3 | 4 | 1 | 1 | Matt synthesizes Scott's point using analogies like job interview behavior and AI bedside manner. Scott explains why enterprise trust requires testing non-performance characteristics such as toxicity, reading level, tone, and reasoning steps. | |
| Centralization, Platform Engineering, and Mitigating Shadow AI | 4 | 3 | 2 | 2 | Matt pushes back on whether developers actually want standardized platform gateways. Scott explains how enterprise engineering teams use uniform gateways to manage governance, shadow AI risks, and multi-model access. | |
| Scaling AI in Production and the AI Confidence Gap | 3 | 4 | 1 | 1 | Matt asks Scott for real-world stories of failed deployments. Scott explains the AI confidence gap where prototypes stall because teams fear how non-deterministic models and expanding RAG corpora behave under full production scale. | |
| Statistical Testing Approaches and Unsupervised Change Detection | 4 | 5 | 1 | 1 | Matt frames the technical discussion with a metaphor of putting AI systems in glass tanks with sensors. Scott delivers a detailed mathematical explanation of why distributional testing uses high-entropy weak estimators rather than standard evaluation metrics. | |
| Managing AI Tech Debt, Prompt Hygiene, and Cost Trade-Offs | 5 | 2 | 1 | 1 | Matt demonstrates high technical familiarity by applying Conway's Law to prompt engineering, noting how system prompts mirror corporate bureaucracy. Scott agrees and expands on how behavioral test suites prevent build failures during prompt refactoring. | |
| AI Labs, Enterprise Dynamics, and AI Ops | 3 | 3 | 1 | 1 | Matt highlights operational realities by asking who gets paged late at night when a production model fails. Scott outlines how enterprise platform layers connect frontier research labs with operational needs, predicting the rise of dedicated AI Ops teams. |
Statements from this episode (14)
Clark: Machine learning is normalized tech; AI is cutting-edge novelty
“Like machine learning is the stuff that's now become easy and then AI is all the fun new stuff. And then as soon as it stops becoming the cutting edge, Then it just becomes, oh, that's just machine learning.”
Clark: System trust, not performance, limits enterprise AI value
“The thing that's holding back people getting value from these AI systems is not performance. It's not about squeezing out that last half a percent from some eval function or some performance metric. It's about being able to confidently trust these systems.”
Clark: High-level LLM evaluations mask undesired AI system behaviors
“We're seeing people do the exact same thing again today with LLMs, where they're focusing on these high-level metrics, these end outputs, these performance evals, and that ends up masking all of these potentially undesired behaviors within the system itself.”
Clark: Generative AI platform leaders are traditional machine learning veterans
“A lot of the people who are now in charge of building Gen AI platforms or productionizing these massive use cases are the same people who built those original machine learning systems.”
Clark: Evaluating end-to-end AI performance hides upstream system failures
“And what I think a lot of firms are running into right now is if you're only looking at that last step, if you're only looking at the system's performance as a whole, it can be very difficult to understand when, where, and why behaviors are shifting within thi…”
Clark: Enterprises are moving from generative AI prototypes to centralized platforms
“One thing that we've seen that's really interesting over the last year, year and a half is people have started to shift from kind of science project prototype land where they have a bunch of individual teams trying to roll their own stack and trying to like bu…”
Clark: Generative shadow AI exposes intellectual property to external SaaS vendors
“And it was a somewhat localized problem because like you're doing data science on your laptop versus now I'm just shipping off secret IP to some SaaS company or something like that.”
Clark: AI adoption faces misaligned incentives between providers and enterprise users
“So one big complication is That sometimes the incentives are misaligned. So open AI obviously wants to create the best general purpose foundational models, but an individual business may want a model that solves a very specific problem a very specific way very…”
Clark: An AI confidence gap leaves enterprise generative AI in prototypes
“We talked to a lot of firms that are terrified to cross this AI confidence gap from I've developed something that works good in, in, in theory. How do I actually scale it up in practice? And A lot of times we'll talk to individuals who say, every single time I…”
Clark: Expanding RAG datasets with historical data degrades search quality
“And so RAG has obviously become very prevalent in a wide variety of industries and people use it for a lot of different things. We've spoken with different firms that they were like, okay, well, I'm just going to continue to add more and more data to the corpu…”
Clark: AI testing should use many weak estimators to detect behavioral differences
“Instead of trying to come up with a small number of strong estimators for performance, where we want to be able to conclusively say A is better than B, Instead, what we want is a large number of potentially weak estimators to be able to determine whether or no…”
Clark: Enterprises avoid high-value AI use cases due to unwieldy risk
“We see some firms attacking the low hanging fruit internal chatbots to like ask questions about HR because they're afraid to take that leap to develop the difficult problem because it's so unwieldy and there is so much risk associated with it. A lot of the mos…”
Bornstein: System prompts mirror company organizational structures like Conway's Law
“My biggest takeaway from this is the system prompt often reflects the organization it came from. Like it's some new form of Conway's law. You know, Conway's law is this thing that you kind of ship your org structure as a software company. Like I think as an AI…”