The Exchanges

Q Yeah. What was the model evaluation at the time? So I'm sure part of, like, the We Need Plan and Act is, like, maybe the models are not able to do it end to end. When you started working on that, what were the model limitations, what were the best models, and then how has that evolved over time?

A Yeah, when I first started working on Klein, this was, I think, 10 days after Cloud through Five Sonic came out. I was reading Anthropics Model Card Addendum, and there was this section about agentic coding and how it was so much better at this step-by-step Accomplishing tasks. And they talked about running this internal test where they let the model run in this loop where it could call tools. Um, and it was obvious to me that, okay, they have some version and they have some application internally. That's really different from how the, you know, the other things at the time were things like copilot and cursor and Adr. They didn't do this for like step by step reasoning and accomplishing tasks. They were more suited for the Q and A and, and one shot prompting paradigm. Uh, at the time, I think it was, uh, June, 24. Anthropic was doing a build with cloud hackathon. So I thought, okay, this is a really cool new capability that none of the models have really been capable of doing before. And, uh, I think being able to create something from the ground up and take advantage of kind of like the nuances of how much the models improved in that point in time. So for example, cloud through five was also really good at this test called needle in a haystack, where if it has A lot of context in its context window. For example, you know, 90% of its 200 K context window is filled up. It's real…

AI assessment note: “when I first started working on Klein, this was, I think, 10 days after Cloud through Five Sonic”

Q And was this the first shape of the product or did you get to the plan act iteratively? And maybe was this the first idea of the company itself or were you exploring other stuff?

A It was a lot of Especially in the early days of the client, it was a lot of experimenting and talking to our users and seeing what kind of workflows came up that they found that were useful for them and translating them into the product. So plan and act was really a byproduct of just talking to people in our Discord, just asking them what would be useful to them, what kind of prompt shortcuts we could add into the UI. I mean, that's really all plan and act mode is. It's, is essentially a shortcut for the user to save them the trouble of having to Type out, you know, I want you to ask me questions and put together a plan, um, the way that you might have to and, you know, some of the other tools you'd have to, like, be explicit about. I want you to come up with a plan before, you know, acting on it or editing files. Incorporating that into the UI just saves the user the trouble of having to type that out themselves.

AI assessment note: “plan and act was really a byproduct of just talking to people in our Discord”

Q I'm happy to move on. I would say like the last thing that's kind of curious is like, if Anthropic hasn't, hadn't come along and made MCP, what would have happened? What's the alternative history, right? Like, would you have come with MCP?

A So we saw some of, uh, our competitors who have been kind of working on their own version of Plug and play tools into these agents. They kind of had to natively create these tools and integrations themselves directly into their product. And so I think anybody in the space would have had to just do the laborious work of having to recreate these tools and integrations for. So I think Anthropic just saved us all a lot of trouble and tapped into the power of open source and community driven development and allowed, you know, individual contributors to make an MCP for anything people could think of. I'm gonna really take advantage of people's imagination in a way that I think is, like, necessary right now for us to really tap into full potential of, of this sort of thing, so.

AI assessment note: “anybody in the space would have had to just do the laborious work”

Q Yeah. Uh, double clicking on the AST mention. That's very verbose. When do you use that?

A Right now it's a tool. The way that it works is when Klein, once, when Klein is doing sort of the agentic exploration of trying to pull in relevant context, and it wants to sort of get an idea of what's going on in a certain directory, for example, there's a tool that lets it pull in all the sort of language from a directory. So it could be the names of classes, the names of functions, And that gives it some idea of, okay, here's what's going on in this, in this folder. And if it's, if it seems relevant to whatever the task is trying to accomplish is, then it sort of like zooms in and starts to actually read those entire files into context. So it's, it's essentially a way to help it kind of figure out how to navigate through large code bases.

AI assessment note: “when Klein is doing sort of the agentic exploration of trying to pull in relevant context”

Q Any thought on CloudMD versus AgentsMD versus AgentMD? I built an open source tool called Agents-Nine-to-seven, like the XKCD that just copy-pastes it across all the different file names, so all of them have access to it. Do you think there should be a single file? Like, there's also like the IDE rules versus the agent rules. There's kind of like a lot of issues.

A I actually think it's fine that each of these different tools have their own specific instructions, because I find myself using A cursor rules and a client rules separately. When I want client, the agent, I want him to work, you know, a certain way that's different than how I might want, you know, cursor to interact with my code base. So I think each tool is specific to the kind of work that I do and I have different instructions for how I want these things to operate. So I think I've seen like a lot of people complain about it and I get that it could make code bases look a little bit ugly, but for me, it's been like incredibly helpful for them to be separated.

AI assessment note: “I actually think it's fine that each of these different tools have their own specific instructions”

Q over time. I mean, if we sat down, I mean, this discussion 12 months ago, complex was, like, much easier than today for the models. Do you feel like that's evolving quickly enough that, like, you know, in 18 months, it's like, you should probably just do follow GenTech for, like, 75% of work, 80% of work, or do you feel like it's not moving as quickly as you thought?

A I think, I think what was complex a couple years ago is totally different to what is complex today. Now, I think what we need to be more intentional about are, The architectural decisions we make really early on and how the model kind of builds on top of that. If you have kind of a clear direction of where things are headed and what you want, you kind of have a good idea to about how you might want to like lay the foundation for the code base that you're producing. And I think what we might have considered complex a few years ago, algorithmic, you know, challenges, that's pretty trivial for models today and stuff that we don't really necessarily have to think too much about anymore. We kind of give it, you know, A certain expectation or unit test about what we want, and it kind of goes off and puts together the, you know, the perfect solution. So I think there's a lot more thought that has to go into tasteful architectural decisions that really comes down to you having experience with what works and what doesn't work, having a clear idea for the direction of where you want to take the project and sort of your vision for the code base. Those are all decisions that I think is, is, is hard to rely on a model for because of its limited context and It's, you know, it's inability to kind of see your vision for things and really have a good understanding of, of, uh, what you're trying…

AI assessment note: “algorithmic, you know, challenges, that's pretty trivial for models today”