Insight
Chip Huyen argues post-training is what differentiates frontier AI models
“So, so I do think that post-training is what makes this, like, really big lab models are, like, different.”
Insight
Evaluation is the single biggest bottleneck holding back enterprise AI adoption
“So, so I do think that evaluation is the biggest bottleneck for AI adoptions, because unless, like, if we can, like, if we can, like, develop a more reliable way to evaluate the application, that application is not going to get adopted. Like, or maybe, maybe i…”
Assertion Not checkable as stated
Chip Huyen warns million-token context capacity does not imply efficient processing
“Just because a model, I think the second reason may be actually more important at least for now is that just because a model can fit in a million con token context doesn't mean that it can process that million token efficiently.”
Insight
Chip Huyen argues human-generated plans are poor training data for AI agents
“When we ask humans to generate like what they consider the best plan for an actions, for a task, it's actually like not quite the best plan for AI, because what is what is easy or efficient for humans is not the same as easy and efficient for AI, right?”
Insight
Huyen: Legacy systems prevent US companies from matching Chinese online learning
“So a lot of American, American internet companies are like a lot older than the average, like the new, like Chinese internet company. So it means it's like American internet companies have legacy systems that you from like, 20 years ago. And just have to build…”
Insight
Chip Huyen: AI becomes harder to evaluate as intelligence increases
“As a more intelligent AI becomes like the harder it is to evaluate it.”
Assertion Not checkable as stated
Chip Huyen: Machine translation is largely solved for major languages
“Now it was like pretty much like people are saying that machine translation is like pretty much sold for like major languages.”
Insight
Chip Huyen: Lack of labeled data requirements makes language modeling uniquely scalable
“You don't need to curate, like, labels, like, reference data, so that, that you can use a train models that make language modeling, like, so, so much easier to scale than other types of tasks.”
Prediction Not checkable as stated
Chip Huyen predicts users will always expand data to fill available context
“I think we always expand our usage to fit in whatever context length. That's going to be available.”
Insight
Chip Huyen: Machine learning projects should start with problems, not models
“So so the main idea is you go backward from the problems. So I think a lot of approach machine is saying it's like, you start with the solutions and it tries to have like five problems when machine can I be applied. So, and it's like, so tend to be like, oh, H…”
Insight
Chip Huyen: MLOps startups ignore real-time learning for low-hanging fruit
“We don't have tools for it yet. And I see very, very, very little tools focusing on it because most people are like targeting on focusing on low hanging fruit.”
Assertion Not checkable as stated
Huyen: Alibaba and ByteDance lead US firms in online ML scale
“When I was looking into online learning and I realized it's like on the examples I felt were by Chinese companies. And it could be I think I've heard some American companies doing that, but they are doing a much smaller scale, like a lot less complex models th…”
Insight
Chip Huyen: AI engineering requires developers to have stronger product sense
“It requires engineers to have a much, much better product sense.”
Assertion Not checkable as stated
Chip Huyen: Most generative AI systems combine traditional ML with generative AI
“It's a vast majority of GF-AI systems have seen, like, you have, like, traditional or, like, analytical ML components with GF-AI.”
Assertion Not checkable as stated
Chip Huyen: Many developers build strong AI applications without traditional ML backgrounds
“I definitely see a lot of people building very good applications without traditional ML background.”
Assertion Supported
Chip Huyen: Smaller Llama 3 models likely outperform largest first-gen Llama models
“The smaller model in Lama three Families probably perform better than the bigger the model in the first Lama generations.”
Assertion Not checkable as stated
Chip Huyen notes major AI labs keep post-training research proprietary
“And unfortunately, a lot of labs that are doing it are not quite, like, publishing papers about it.”
Insight
Code generation leads generative AI adoption because it is easily evaluated
“For JDIF AI, like, one of the most common JDIF AI use cases today is coding. Okay, so there are many reasons why coding is popular, and I said, like, one of the reasons is that it's very, it's a lot easier to evaluate coding than, like, other, because, like, h…”
Prediction Not checkable as stated
Chip Huyen: Language models will never achieve perfect next-token prediction
“I don't think we would ever reach the point that we can predict the next token, like perfectly, because there's always some, like some variations in the way we speak, right?”
Insight
Chip Huyen: AI evaluation metrics must be derived backward from business use cases
“For applications it's really, really important to understand the use cases well, so they can design like the set of metrics and then you can walk backward from that and map it to like the model metrics.”
Assertion Not checkable as stated
Chip Huyen: Most AI engineering teams currently use AI as a judge
“Nowadays you talk to like teams, I think like most teams have like some variations of AI as a judge going on.”
Insight
Chip Huyen says prompt engineering should treat each prompt as an experiment
“I do think that it can be very systematic. You know, you should need to make it very systematic. So if you consider like each prompt is experiment, it should be watching like versions of prompt. It should be able to systematically track your progress with diff…”
Insight
Open-source AI developers lack usage visibility to ensure model safety
“Open source model developers try their best to make the models, like, safe as well, right? But they also have less visibilities into how the open source models are being used. So which gives them like less information for them to like make the model safe.”
Prediction Not checkable as stated
Chip Huyen: Coding examples using current AI frameworks quickly become outdated
“All the frameworks today change so fast. So I feel like any coding example using any of them is going to go, like, outdated, like, pretty quickly.”
