The Wisdom Wall
15 quotable lessons, heuristics and mental models. Every one is playable at the moment it was said. No fortune cookies allowed.
“So, so I do think that post-training is what makes this, like, really big lab models are, like, different.”
“Um, so, so I do think that evaluation is the biggest bottleneck for AI adoptions, because unless, like, if we can, like, if we can, like, develop a more reliable way to evaluate the application, that application is not going to get adopted. Like, or, or maybe, maybe it can. Maybe we need some billionaires to just,…”
“when we ask humans to generate like what they consider the best plan for an actions, for a task, it's actually like not quite the best plan for AI, because what is, uh, what is easy or efficient for humans is not the same as easy and efficient for AI, right?”
“So a lot of American, American internet companies are like a lot older than the average, like the new, like Chinese internet company. So it means it's like American internet companies have legacy systems that you from like, 20 years ago. And just have to build the system on top of it, whereas either Chinese can be a…”
“as a more intelligent AI becomes like the harder it is to evaluate it.”
“you don't need to curate, like, labels, like, reference data, so that, that you can use a train models that make language modeling, like, so, so much easier to scale than other types of tasks.”
“So, uh, so the main idea is you go backward from the, the problems. So I think a lot of approach machine is saying it's like, you start with the solutions and it tries to have like five problems when machine can I be applied. So, and it's like, so tend to be like, oh, Hey, this is fancy model coming out like bird or…”
“we don't have tools for it yet. And I see very, very, very little tools focusing on it because most people are like targeting on focusing on low hanging fruit.”
“it requires engineers to have a much, much better product sense.”
“for JDIF AI, like, one of the most common JDIF AI use cases today is coding. Okay, so there are many reasons why coding is popular, and I said, like, one of the reasons is that it's very, it's a lot easier to evaluate coding than, like, other, because, like, here's JDIF code, right? You can evaluate, like, does it…”
“for applications, um, it's really, really important to understand the use cases well, so they can design like the set of metrics and then you can walk backward from that and map it to like, uh, the model metrics.”
“I do think that it can be very, uh, systematic. You know, you should need to make it very systematic. So if you consider like each prompt is experiment, it should be watching like versions of prompt. It should be able to systematically track your progress with different prompts.”
“open source model developers try their best to make, to make the models, like, safe as well, right? But, but they also have less visibilities into how the open source models are being used. So which gives them like less, uh, information for them to like make the model safe.”
“it's a cheap way to improve some model, like, so, so, so, so application's performance without to retreat, you're going to have to retreat in the model.”
“So I do think that's a topic is getting increasingly important, especially as AI is being like first, um, it is being used for more like, um, high stack tasks, right? And a more complex task. And the second, um, is, is like, it's now AI has increasing access to more tools and it can make changes.”