error analysis

“What we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn't have the context needed to understand whether something might be, you know, bad product smell or, you know, not”

“Just write down the first thing that you see that's wrong. The most upstream error. Don't worry about all the errors. Just capture the most, the first thing that you see that's wrong and stop and move on.”

“You want to usually ground yourself in your actual errors. You don't want to skip this step. And so the reason I'm kind of spending so much time on this is like, this is where people get lost. They go straight into evals. Like, let me just write some tests. An…”