The Exchanges

Q Let's run through maybe outside of, I think the, this is the prompt to code kind of thing. And then I think what people like about clock code is, uh, the plan mode, kind of like the thinking mode, there's kind of like all these different ways to use it. What does this look like here?

A Yeah. So we're still exploring exactly how that's gonna look. Uh, so plan mode is one of the most requested features. We're probably gonna ship it this week, but we're not implementing it as An explicit plan mode. We're implementing the concept of modes. So you'll be able to hit tab and cycle through different modes. We'll have a built in like plan mode and implement mode, but you'll be able to extend this with your own types of modes. And all a mode is a combination of a custom system prompt, optionally a different model and optionally a different set of tools. So you can make a mode where you're like, I want, I like talking to Gemini when I'm doing like larger code based things. So you can have a good Gemini mode. That you can tab into. It'll switch to using Gemini. It'll take advantage of the larger context. It'll maybe only pass it read only tools. Um, and then once you have a plan there, you can, you can tab back and get to implement mode, which might be done what's on it and actually do the implementation.

AI assessment note: “We're implementing the concept of modes. So you'll be able to hit tab”

Q Amazing. And then like, so, you know, I think there's, there's a lot of questions about the core agent harness agent loop. Um, yeah, let's just like talk about the technical lesson on the journey that you went through. Where did you start and where have you ended up?

A Yeah. So I, I obviously didn't know about any of how any of this stuff worked until we're like, maybe like about two months into development. So roughly two months ago, I knew that all an agent was, was a loop running somewhere. And in that process of doing this, I've learned some of the details about that. Like I said, we don't try to innovate on how that works. We look at cloud code. We dump all their system prompts. We dump all their tool descriptions. We dump all their tool schemas. We re-implement the tools. And when you're using an anthropic model, we basically have the exact same implementation. Some of the details of how to figure, like some of the prompt caching stuff was tricky to understand how that worked. But at the end of the day, like it's just a loop where you send the prompt with a bunch of tool descriptions. It tells you what tools to call. You call it tools and the results back. And there's really not too much magic there.

AI assessment note: “didn't know about any of how any of this stuff worked until we're like”

Q I mean, I think either is like up there, but also like, this is curious, like since you build in public, Probably this is public, but like, what's an example of a thing that, you know, locally temporarily improved things, but actually you regretted it two weeks later?

A Yeah, so I did this thing. So the edit tool, the way the tools work on most of these tools now is you ask the LM for the file you want to change the old string to look for and the new string to replace. And now there's some percentage of the time where it nails it. You search the old string, you find it, you replace it. Sometimes it kind of messes up what the old string looks like, and you search for and it's not there. But it's usually A lot of times it's a very simple mistake. Like it used tab spacing when you use spaces. Um, so you can have a series of fallback strategies and the client team published all of their fallback strategies, which I then ported to open code. And then I was like, that was easy. I just had open code port it. Let me go crazy. I went to Gemini's CLI and I was like, let me take all their strategies too. And Gemini ones were not good. It was, uh, initially it looked good, but then I noticed it was getting caught in these crazy loops and it was like editing my files in these totally messed up ways. And I should have known because you can tell when you're looking through the Gemini code base, it's like really rushed. So I ended up dropping those strategies, but there was at least like two or three weeks where That was definitely a regression. And I only knew because I was using the tool.

AI assessment note: “there was at least like two or three weeks where That was definitely a regression.”

Q Uh, I mean, there's, there's a, there's a lot, like, do you, um, do you run evals, um, you know, in a, in a way that Aether does?

A Yeah, so we looked at all the different benchmarks we could find, and it's almost comical how none of them look like my day-to-day work. Most of these evals are like, You know, solve this maze. And I'm just like, I'm never solving a maze. Like it's never anything that I've ever asked an LM to do. I'm usually asking you to do something like, Hey, I'm implementing this feature. The backend's pretty easy. I just need you to go like implement a new data model in the database, create a migration, expose an API. And I'll work on the front end while you do that. Almost no benchmarks have anything that resemble anything close to that. So one of the people on our team, Frank, he's working right now on developing our own set of benchmarks. We don't really, I don't really care too much about being like, Hey, open code's like the best on his benchmark. I more care about, I made a change to how our editor tool works. Did I make things worse? Did I make things better? Um, this is another funny thing you see right now because nobody really has a great way to do this. So every new feature is positioned as we made things better. But I've seen myself make changes where I make it better in one dimension. And it's just like horrible. I realized it's actually horrible. And like the dimensions that matter two weeks down the road. So I just feel lost trying to improve this stuff. So at first we're tr…

AI assessment note: “Frank, he's working right now on developing our own set of benchmarks.”