The Wisdom Wall
4 quotable lessons, heuristics and mental models. Every one is playable at the moment it was said. No fortune cookies allowed.
“my hunch is some of the, um, Uh, enterprise deployments that don't actually work out likely don't have the scaffolding or infrastructure for these agents to interact with as well. Um, a lot of the, like, really successful deployments that we've made, a lot of what our FDs end up doing with some of these customers is to…”
“And evals also, oftentimes, need to come up bottom up. Right? Because all of these things are kind of in people's heads, in the actual operator's heads. Like, it's actually very hard to have a top-down mandate of, like, you got, like, this is how the evals should look. A lot of it needs the bottom-up adoption.”
“as a product builder, there's a latency, there's a real latency trade-off that you have to deal with where, you know, your user might not be happy waiting 10 minutes for, like, the best answer in the world. It might be more okay with the substandard answer and, like, no wait at all.”
“Reinforcement fine tuning introduces like RL or reinforcement learning to this loop. Way more complex, way more finicky, but an order of magnitude more powerful.”