The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 31 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Chip Huyen argues post-training is what differentiates frontier AI models
“So, so I do think that post-training is what makes this, like, really big lab models are, like, different.”
Evaluation is the single biggest bottleneck holding back enterprise AI adoption
“So, so I do think that evaluation is the biggest bottleneck for AI adoptions, because unless, like, if we can, like, if we can, like, develop a more reliable way to evaluate the application, that application is not going to get adopted. Like, or maybe, maybe i…”
Chip Huyen warns million-token context capacity does not imply efficient processing
“Just because a model, I think the second reason may be actually more important at least for now is that just because a model can fit in a million con token context doesn't mean that it can process that million token efficiently.”
Chip Huyen argues human-generated plans are poor training data for AI agents
“When we ask humans to generate like what they consider the best plan for an actions, for a task, it's actually like not quite the best plan for AI, because what is what is easy or efficient for humans is not the same as easy and efficient for AI, right?”
Huyen: Legacy systems prevent US companies from matching Chinese online learning
“So a lot of American, American internet companies are like a lot older than the average, like the new, like Chinese internet company. So it means it's like American internet companies have legacy systems that you from like, 20 years ago. And just have to build…”
Chip Huyen: AI becomes harder to evaluate as intelligence increases
“As a more intelligent AI becomes like the harder it is to evaluate it.”
Chip Huyen: Machine translation is largely solved for major languages
“Now it was like pretty much like people are saying that machine translation is like pretty much sold for like major languages.”
Chip Huyen: Lack of labeled data requirements makes language modeling uniquely scalable
“You don't need to curate, like, labels, like, reference data, so that, that you can use a train models that make language modeling, like, so, so much easier to scale than other types of tasks.”
Chip Huyen predicts users will always expand data to fill available context
“I think we always expand our usage to fit in whatever context length. That's going to be available.”
Chip Huyen: Machine learning projects should start with problems, not models
“So so the main idea is you go backward from the problems. So I think a lot of approach machine is saying it's like, you start with the solutions and it tries to have like five problems when machine can I be applied. So, and it's like, so tend to be like, oh, H…”
Chip Huyen: MLOps startups ignore real-time learning for low-hanging fruit
“We don't have tools for it yet. And I see very, very, very little tools focusing on it because most people are like targeting on focusing on low hanging fruit.”
Huyen: Alibaba and ByteDance lead US firms in online ML scale
“When I was looking into online learning and I realized it's like on the examples I felt were by Chinese companies. And it could be I think I've heard some American companies doing that, but they are doing a much smaller scale, like a lot less complex models th…”
Chip Huyen: AI engineering requires developers to have stronger product sense
“It requires engineers to have a much, much better product sense.”
Chip Huyen: Most generative AI systems combine traditional ML with generative AI
“It's a vast majority of GF-AI systems have seen, like, you have, like, traditional or, like, analytical ML components with GF-AI.”
Chip Huyen: Many developers build strong AI applications without traditional ML backgrounds
“I definitely see a lot of people building very good applications without traditional ML background.”
Chip Huyen: Smaller Llama 3 models likely outperform largest first-gen Llama models
“The smaller model in Lama three Families probably perform better than the bigger the model in the first Lama generations.”
Chip Huyen notes major AI labs keep post-training research proprietary
“And unfortunately, a lot of labs that are doing it are not quite, like, publishing papers about it.”
Code generation leads generative AI adoption because it is easily evaluated
“For JDIF AI, like, one of the most common JDIF AI use cases today is coding. Okay, so there are many reasons why coding is popular, and I said, like, one of the reasons is that it's very, it's a lot easier to evaluate coding than, like, other, because, like, h…”
Chip Huyen: Language models will never achieve perfect next-token prediction
“I don't think we would ever reach the point that we can predict the next token, like perfectly, because there's always some, like some variations in the way we speak, right?”
Chip Huyen: AI evaluation metrics must be derived backward from business use cases
“For applications it's really, really important to understand the use cases well, so they can design like the set of metrics and then you can walk backward from that and map it to like the model metrics.”
Chip Huyen: Most AI engineering teams currently use AI as a judge
“Nowadays you talk to like teams, I think like most teams have like some variations of AI as a judge going on.”
Chip Huyen says prompt engineering should treat each prompt as an experiment
“I do think that it can be very systematic. You know, you should need to make it very systematic. So if you consider like each prompt is experiment, it should be watching like versions of prompt. It should be able to systematically track your progress with diff…”
Open-source AI developers lack usage visibility to ensure model safety
“Open source model developers try their best to make the models, like, safe as well, right? But they also have less visibilities into how the open source models are being used. So which gives them like less information for them to like make the model safe.”
Chip Huyen: Coding examples using current AI frameworks quickly become outdated
“All the frameworks today change so fast. So I feel like any coding example using any of them is going to go, like, outdated, like, pretty quickly.”
Chip Huyen highlights customer support as a massive untapped online ML market
“Another use case, I think it's like heavily underexplored and I don't see a lot of companies doing it except for a few, like I think except for very few, few big companies. So it's a customer service support.”
Chip Huyen: Quantization works universally well across tasks and models
“Quantizations, which is like very universally very working really well. For a lot of tasks across model.”
Chip Huyen says sampling improves AI performance without model retraining
“It's a cheap way to improve some model, like, so, so, so, so application's performance without to retreat, you're going to have to retreat in the model.”
Defensive prompt engineering grows critical as AI executes complex tasks
“So I do think that's a topic is getting increasingly important, especially as AI is being like first it is being used for more like high stack tasks, right? And a more complex task. And the second is, is like, it's now AI has increasing access to more tools an…”
Huyen: Snorkel AI is one of the better machine learning startups
“Snuggle is, is a great tool. And I think like it's one of the, like one of the better ML startups out there.”