Jan 16, 2025 · 1h 12m · mad
What You MUST Know About AI Engineering | Chip Huyen, Author of “AI Engineering”
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, host Matt Turck speaks with computer scientist and author Chip Huyen about her book AI Engineering. They cover the shift from traditional ML to foundation models, prompt engineering disciplines, evaluation challenges, RAG architectures, and autonomous agent planning.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 15.8% of the talking time here. How this is scored →
Matt as informed peer 2.9 Guest teaching 5.4 Guest disagreement 0.3 Matt pushing back 0.6
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:00:50 Chip teases host on trap question and VC incentives
When asked to define AI agents, Chip lightheartedly turns the question back on Matt, calling it a trap question and poking fun at his VC role.
Hardest push from Matt ▶ 40:04 Matt re-routes from theoretical metrics to practical engineering needsMatt refuses to stay on theoretical LLM concepts like entropy and perplexity, explicitly redirecting Chip to provide practical evaluation metrics for software engineers building real applications.
Biggest teaching moment ▶ 38:00 Chip explains Claude Shannon's theoretical entropy boundChip delivers a deep theoretical breakdown of Claude Shannon's 1950s experiments, explaining irreducible loss and context-length dynamics in language predictability.
Matt holds his own ▶ 55:28 Matt articulates the system prompt versus user prompt distinctionMatt steps in with sharp expertise to clearly distill the difference between background system prompts and front-end user prompts, earning a compliment from Chip on his teaching ability.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Overview of the Book AI Engineering | 1 | 1 | 0 | 0 | Matt opens the episode with a warm welcome and highlights key praise for Chip's new book AI Engineering. Chip responds with pleasantries and mutual admiration. | |
| Defining AI Engineering vs Traditional ML and MLOps | 3 | 5 | 1 | 1 | Chip defines AI engineering in contrast to traditional MLOps, detailing silent failures and the shift to product-driven development. Matt checks his understanding by clarifying if the development lifecycle is reversed. | |
| Comparing AI Engineers and Machine Learning Engineers | 4 | 5 | 1 | 2 | Chip outlines hybrid architectures combining classical ML classifiers with generative models. Matt pushes back gently against the media narrative that generative AI completely replaces traditional ML. | |
| The Core Components of the Generative AI Stack | 1 | 5 | 0 | 0 | Matt asks Chip to break down the generative AI technology stack. Chip systematically outlines the application development layer, model fine-tuning layer, and underlying infrastructure layer. | |
| Why Language Models Scale Exceptionally Well | 2 | 6 | 0 | 0 | Chip explains the historical mystery of scaling text models compared to computer vision and the simplicity of next-token prediction. Matt interjects briefly to confirm technical terminology around autoregressive models. | |
| Defining Supervised, Unsupervised, and Self-Supervised Learning | 2 | 6 | 0 | 1 | Chip defines self-supervised learning relative to supervised and unsupervised approaches. Matt asks a philosophical question about why larger models require proportionally larger datasets to achieve potential. | |
| Optimizing Smaller Models and Mixture of Experts | 3 | 6 | 0 | 0 | Matt brings up Mixture of Experts as an approach for small model optimization. Chip details parameter sharing across routing heads as well as practical techniques like quantization and distillation. | |
| Pre-Training vs Post-Training Phases | 3 | 6 | 1 | 1 | Matt asks Chip to explain post-training dynamics, noting its crucial role. Chip critiques industry terminology and reveals why post-training alignment represents proprietary lab secrets. | |
| Understanding Sampling and Token Selection Strategies | 2 | 6 | 0 | 0 | Matt introduces the topic of sampling strategies. Chip explains temperature, token probability selection, and how sampling provides a cost-effective way to tune application behavior. | |
| Evaluation as the Central Bottleneck in AI Adoption | 3 | 5 | 0 | 0 | Matt cites Chip's book assertion that evaluation is the central bottleneck in enterprise AI adoption. Chip elaborates on ROI, evaluation-driven development, and why deterministic tasks like coding are easier to measure. | |
| Entropy, Perplexity, and Theoretical Limits of Language Modeling | 2 | 7 | 0 | 0 | Matt asks about theoretical evaluation metrics like entropy and perplexity. Chip educates the host on Claude Shannon's historical work, irreducible loss limits, and pre-training performance plateaus. | |
| Practical Metrics for Evaluating Real-World AI Applications | 4 | 5 | 0 | 2 | Matt steers the conversation from theoretical language metrics toward pragmatic evaluation methods for practical AI engineers. Chip uses Text-to-SQL as an example to illustrate task-based metrics. | |
| AI as a Judge: Benefits, Pitfalls, and Non-Stationarity | 3 | 6 | 0 | 1 | Matt introduces the AI-as-a-judge paradigm. Chip highlights the pitfalls of non-stationarity, hidden prompt drift, and lack of standardized metrics across downstream engineering teams. | |
| Understanding and Approaching Systematic Prompt Engineering | 3 | 5 | 0 | 0 | Matt references Chip's argument that prompt engineering's apparent simplicity deceives practitioners. Chip emphasizes treating prompts as systematic experiments with disciplined tracking. | |
| In-Context Learning and Few-Shot Prompting Concepts | 3 | 6 | 0 | 0 | Chip breaks down in-context learning, zero-shot, and few-shot prompting concepts. Matt steps in to emphasize why adapting models via context without retraining is a major industry shift. | |
| Defensive Prompt Engineering and System Prompts | 5 | 4 | 0 | 0 | Matt asks about defensive prompt engineering. When Chip mentions system prompts, Matt concisely steps in to articulate the clear structural boundary between user prompts and system prompts. | |
| Retrieval-Augmented Generation vs. Long Context Windows | 4 | 6 | 1 | 1 | Matt brings up Chip's book thesis that long context windows will not eliminate RAG. Chip explains context processing inefficiency, lost-in-the-middle degradation, and memory limits. | |
| Defining AI Agents and Multi-Step Planning Challenges | 4 | 7 | 2 | 2 | Matt asks Chip to define AI agents. Chip humorously calls it a trap question and turns the tables by probing Matt's VC investments before grounding her definition in classic 1990s AI textbooks and explaining planning bottlenecks. |
Statements from this episode (23)
Chip Huyen: AI becomes harder to evaluate as intelligence increases
“As a more intelligent AI becomes like the harder it is to evaluate it.”
Chip Huyen: AI engineering requires developers to have stronger product sense
“It requires engineers to have a much, much better product sense.”
Chip Huyen: Most generative AI systems combine traditional ML with generative AI
“It's a vast majority of GF-AI systems have seen, like, you have, like, traditional or, like, analytical ML components with GF-AI.”
Chip Huyen: Many developers build strong AI applications without traditional ML backgrounds
“I definitely see a lot of people building very good applications without traditional ML background.”
Chip Huyen: Machine translation is largely solved for major languages
“Now it was like pretty much like people are saying that machine translation is like pretty much sold for like major languages.”
Chip Huyen: Lack of labeled data requirements makes language modeling uniquely scalable
“You don't need to curate, like, labels, like, reference data, so that, that you can use a train models that make language modeling, like, so, so much easier to scale than other types of tasks.”
Chip Huyen: Smaller Llama 3 models likely outperform largest first-gen Llama models
“The smaller model in Lama three Families probably perform better than the bigger the model in the first Lama generations.”
Chip Huyen: Quantization works universally well across tasks and models
“Quantizations, which is like very universally very working really well. For a lot of tasks across model.”
Chip Huyen notes major AI labs keep post-training research proprietary
“And unfortunately, a lot of labs that are doing it are not quite, like, publishing papers about it.”
Chip Huyen argues post-training is what differentiates frontier AI models
“So, so I do think that post-training is what makes this, like, really big lab models are, like, different.”
Chip Huyen says sampling improves AI performance without model retraining
“It's a cheap way to improve some model, like, so, so, so, so application's performance without to retreat, you're going to have to retreat in the model.”
Code generation leads generative AI adoption because it is easily evaluated
“For JDIF AI, like, one of the most common JDIF AI use cases today is coding. Okay, so there are many reasons why coding is popular, and I said, like, one of the reasons is that it's very, it's a lot easier to evaluate coding than, like, other, because, like, h…”
Evaluation is the single biggest bottleneck holding back enterprise AI adoption
“So, so I do think that evaluation is the biggest bottleneck for AI adoptions, because unless, like, if we can, like, if we can, like, develop a more reliable way to evaluate the application, that application is not going to get adopted. Like, or maybe, maybe i…”
Chip Huyen: Language models will never achieve perfect next-token prediction
“I don't think we would ever reach the point that we can predict the next token, like perfectly, because there's always some, like some variations in the way we speak, right?”
Chip Huyen: AI evaluation metrics must be derived backward from business use cases
“For applications it's really, really important to understand the use cases well, so they can design like the set of metrics and then you can walk backward from that and map it to like the model metrics.”
Chip Huyen: Most AI engineering teams currently use AI as a judge
“Nowadays you talk to like teams, I think like most teams have like some variations of AI as a judge going on.”
Chip Huyen says prompt engineering should treat each prompt as an experiment
“I do think that it can be very systematic. You know, you should need to make it very systematic. So if you consider like each prompt is experiment, it should be watching like versions of prompt. It should be able to systematically track your progress with diff…”
Defensive prompt engineering grows critical as AI executes complex tasks
“So I do think that's a topic is getting increasingly important, especially as AI is being like first it is being used for more like high stack tasks, right? And a more complex task. And the second is, is like, it's now AI has increasing access to more tools an…”
Open-source AI developers lack usage visibility to ensure model safety
“Open source model developers try their best to make the models, like, safe as well, right? But they also have less visibilities into how the open source models are being used. So which gives them like less information for them to like make the model safe.”
Chip Huyen predicts users will always expand data to fill available context
“I think we always expand our usage to fit in whatever context length. That's going to be available.”
Chip Huyen warns million-token context capacity does not imply efficient processing
“Just because a model, I think the second reason may be actually more important at least for now is that just because a model can fit in a million con token context doesn't mean that it can process that million token efficiently.”
Chip Huyen argues human-generated plans are poor training data for AI agents
“When we ask humans to generate like what they consider the best plan for an actions, for a task, it's actually like not quite the best plan for AI, because what is what is easy or efficient for humans is not the same as easy and efficient for AI, right?”