The Wisdom Wall
16 quotable lessons, heuristics and mental models. Every one is playable at the moment it was said. No fortune cookies allowed.
“The issue really is the fact that these observability companies isn't actually doing observability for the system, it's just doing the LLM thing. Like I still end up using like Datadog, right? Or like, you know, Sentry to do, like, latency. And so I just have those systems handle it, and then the, like, prompt in,…”
“so much of it is changing that if you give control of these systems away too early, you end up ultimately wanting them back. Like many companies I know that I reach out or ones were like, oh, we're going off of the frameworks because now that we know what the business outcomes we're trying to optimize for, these…”
“instead of doing like a react type reasoning loop, I think my belief is that we should be using like workflows, right? If we do this, then we always have a request and a complete workflow. We can fine tune a model that has a better workflow. Whereas it's hard to think about, like, how do you fine tune a better react…”
“Um, we, we, because, I mean, at this point, we would have, like, you know, three million pieces of inventory, over, like, a billion interactions between users and clothes. Um, any kind of fine-taining would, would definitely outperform, uh, like, some off-the-shove model.”
“In terms of whether or not, like, JSON mode is better, I usually think it's, it's almost worse unless you want to spend less money on, like, the prompt tokens that the function call represents. Um, primarily because with JSON mode you don't actually specify the schema.”
“if you're running into issues where you have, like, 20 or 50 or 60 function calls, I think you're much better having those specifications saved in a vector database, and then have them be retrieved. So if there are 30 tools, like, you should basically be, like, ranking them, and then using the top K to, uh, do…”
“embeddings really is kind of like the lowest hanging fruit, and using something like Instructor can really help produce a data structure, and then you can just use your computer science to reason about this data structure.”
“I think something like DSPy can work because there are like very short term, uh, metrics to measure success, right? It is like, did you find the PII or like, did you write the multi-hop question the correct way? But in these like workflows that I've been managing, a lot of it is like, are we minimizing, like minimizing…”
“Narrative matters a lot to human beings. And I think the recommendation system, that's really hard to capture. Like, it's easy to sell, it's easy to use AI to sell, like, a 20 dollar shirt, but it's really hard for AI to sell, like, a 500 dollar shirt.”
“Function calling lets you define the schema separate from the data and the instructions. And what this meant was, uh, you can kind of have a lot more complex schemas and just map them in Pydantic, and then you can just keep those very separate.”
“People want to build their own frameworks, but people don't want to build, like, JSON parsing.”
“in terms of an extraction workflow, I definitely think it's probably more helpful to have everything be a single schema. Just because you can sort of specify relationships between these entities, right, that you can't do in parallel function calling, you can have a single chain of thought before you generate a list of…”
“one of the things we found out about these language models is that not only can you define nodes, it's really good at figuring out what are nodes and what are edges.”
“I think the higher agency person is more focused on like process, uh, metrics versus, um, outcome metrics. Right? Like, from pottery, like, one thing I learned was, if you want to be good at pottery, you shouldn't count, like, the number of cups or bowls you make. You should just weigh the amount of clay you use.”
“So, agency lets you sort of capture the volume of experiments, and, like, experience lets you figure out, like, oh, that other half, it's not worth doing.”
“Like what you should write down is like, here are the conditions. This is the inputs and the outputs we tried the experiment on. And then one thing that's really valuable is basically writing down under what conditions would I revisit these experiments?”