Huyen: Evaluate Marginal Gains and Switching Costs Before Adopting New AI Tech
Chip Huyen · Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix) · Oct 23, 2025 · at 5:57
AI engineer and author Chip Huyen explains how software teams should evaluate emerging AI frameworks and protocols rather than chasing untested tools.
0:00 / 0:51exact quote · 51.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And I think it's a specific question you should ask them is like, first, Like if how much of the improvements could you get, like from like optimal solutions versus non-optimal solutions, right? And sometimes they were like, actually it's not much, right? And I was like, okay, if it's not much improvement, why do you want to spend so much time debating something that doesn't makes a much difference to your performance? And another question they asked is like, if you adopted a new technology, like how hard it would be to switch That out to another. And sometimes it was like, oh, I think it would be like a lot of work switching it out. And I was just like let's say here's a new technology. It hasn't been tested by a lot of people. And if you adopt it would be like stuck with it forever. Like, do you actually want to adopt it? And maybe you want to think twice about like over commit to like a new technology that hasn't been better tested.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Chip Huyen
Huyen: Internet data is maxed out, making post-training the key AI differentiator
“At some point, we are actually, like, have kind of maxed out on, like, internet data, right? And then people, like, text data, people max out. I think a lot of people are doing, like, with other data, like audios and videos, and, like, everyone's trying to thi…”
Huyen: AI data labeling startups face severe customer concentration risk
“It's very lopsided, right? Because like, is there only like a very small numbers of frontier labs, right? And they want a lot of data. And there's like a massive amount of like startups or companies providing data. So like, you can see these companies, like th…”
Huyen: Engineering managers prefer headcount while VPs prefer AI coding agent subscriptions
“Would you rather have access could you rather have, give everyone on the team, like very expensive. Coding agent subscriptions, or you get an extra head count, right? Let's say it's like maybe like and almost everyone could say the managers could say head coun…”
Huyen: A randomized engineering trial found top performers gained most from Cursor
“He was like, okay, here's more like currently like best performing, average performing, and lowest performing. And then there's a randomized trial. So like they give like half of each group, like access to like cursor. And then he was noticed like over time, I…”
Chip Huyen: Sampling strategy is very underrated for boosting model performance
“Sampling strategy, I think is something extremely important. It can have you boost the performance in a huge way and very, very underrated.”
Huyen: Data preparation drives bigger RAG gains than database choice
“Data preparations for Rack is extremely important. And I would say that's like in the, a lot of the companies that I have seen, that's like the biggest performance in their Rack solutions coming from like better data, data preparations, not agonizing over like…”