Huyen: Internet data is maxed out, making post-training the key AI differentiator
Chip Huyen · Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix) · Oct 23, 2025 · at 14:58
AI engineer Chip Huyen discusses pre-training data constraints and why AI labs increasingly concentrate resources on post-training.
0:00 / 0:22exact quote · 22.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“At some point, we are actually, like, have kind of maxed out on, like, internet data, right? And then people, like, text data, people max out. I think a lot of people are doing, like, with other data, like audios and videos, and, like, everyone's trying to think of, like, what is the new source of data. But we're, like, post-training, but, like, middle class of, like, it's just more of, like, everyone kind of have very similar pre-training data. It's, like, post-training is where, They make a big difference nowadays.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Chip Huyen
Huyen: AI data labeling startups face severe customer concentration risk
“It's very lopsided, right? Because like, is there only like a very small numbers of frontier labs, right? And they want a lot of data. And there's like a massive amount of like startups or companies providing data. So like, you can see these companies, like th…”
Huyen: Engineering managers prefer headcount while VPs prefer AI coding agent subscriptions
“Would you rather have access could you rather have, give everyone on the team, like very expensive. Coding agent subscriptions, or you get an extra head count, right? Let's say it's like maybe like and almost everyone could say the managers could say head coun…”
Huyen: A randomized engineering trial found top performers gained most from Cursor
“He was like, okay, here's more like currently like best performing, average performing, and lowest performing. And then there's a randomized trial. So like they give like half of each group, like access to like cursor. And then he was noticed like over time, I…”
Huyen: Evaluate Marginal Gains and Switching Costs Before Adopting New AI Tech
“And I think it's a specific question you should ask them is like, first, Like if how much of the improvements could you get, like from like optimal solutions versus non-optimal solutions, right? And sometimes they were like, actually it's not much, right? And …”
Chip Huyen: Sampling strategy is very underrated for boosting model performance
“Sampling strategy, I think is something extremely important. It can have you boost the performance in a huge way and very, very underrated.”
Huyen: Data preparation drives bigger RAG gains than database choice
“Data preparations for Rack is extremely important. And I would say that's like in the, a lot of the companies that I have seen, that's like the biggest performance in their Rack solutions coming from like better data, data preparations, not agonizing over like…”