The Exchanges

Q Do you feel like that will just be built by each company on their own, or do you think there's like a same default that you guys are going to build, um, for that that is going to be effective for most, most code bases and test structures?

A This is where scale helps, and we have a lot of scale, so, you know, increasingly we're able to see in this framework for this, you know, Go, the standard Go unit test package, that's easy. Uh, vTest in JavaScript, that's easy. And Once you start getting the more of the long tail, then, you know, it might have to just fall back to a really good model. But I think that we could probably, um, make something that's optimized for some of these more popular unit testing frameworks. And it's a combination of deterministic stuff and non-deterministic stuff. Because right now in my VS code, I can hit Apple T if I'm positioned in a test file inside of one of those test blocks, and it's only going to run that one. So, you know, even that is a benefit.

AI assessment note: “I think that we could probably, um, make something that's optimized”

Q Are there any numbers that you share about developers, like, you know, AMP usage overall?

A It's growing really fast. It's growing more than 50% month over month, a lot faster in, you know, some weeks. And really, what we have seen, too, is there's a huge change in who's using it. So we have teams with like two or three people that are on annual run rates of like hundreds of thousands of dollars. So that's it. We also made a decision to not try to go to every single dev in an enterprise, which we had done with Cody. We pick off the people that want to move as fast as we want to move, that want to stay at the model product frontier like us. So it's all about just being able to move really fast. And I think that the way that agents work today, most of them are used in your editor or CLI interactively. You have one agent at most running with you at all times. That's going to be blown up with async agents when they're running 24, seven concurrently in the background, then you can have 10 or a hundred times as many, and that's going to dominate inference. That's going to dominate the output you get. So it's really, you know, amp is growing really fast, but it's, it's more about how do we get to be the first ones with that, like 10 to a hundred X improvement. And everything is about how can we move fast and learn along the way? It just so happens that we are positive gross margins along the way.

AI assessment note: “It's growing more than 50% month over month, a lot faster in, you know”

Q internally from a team perspective? Because, you know, obviously you have a lot of customers already on the Sourcegraph product. There's kind of like this tension of, you know, going founder mode and kind of burning the bridge on maybe some of the old use cases versus having a smaller team and a dictator for a new product. How does that look like from like a building the company perspective?

A When you have a really popular, successful product that's highly profitable, that funds a lot of this craziness. And we're able to do this also with the customer trust. So there's a lot of things on AMP that we do, like no consistent pricing, no user model choice, no checking off all the boxes that security and compliance and legal want that, you know, takes nine months. We're able to get away without doing that stuff because we have that customer trust. So, you know, that That has been a big thing. It requires you to totally change how you think about an existing business. It's not a way to sell through that same channel to those same users. It's a way to use that trust and that revenue to fund crazy stuff that you got to do. But it's something that we deal with all the time and we've got really smart devs and yet it is hard for people to throw away everything that they have learned about how to build software development. And so in some cases, it's been really refreshing to have people that Have only ever been at tiny, like one person companies.

AI assessment note: “it is hard for people to throw away everything that they have learned”

Q actual Docker file and whatnot, it's helpful for them to explain what they have access to. What do you think are like other things that you don't expect the model to like have in the model that you want to still expose to it? So we can assume it's gonna test. We can assume it's gonna have some definition of its environment. Are there other things that come to mind?

A I think test is a big one and there's many different kinds of tests. So we had sub agents in AMP. You know, among the, the first that come out with this conception of sub agents, which is a separate context window, separate cute, more curated set of tools. And I think there's a lot of potential to take a tool like test. And right now you invoke it by the bash tool and you have some complex invocation too often. It'll run all of your tests, which is noisy and it takes a long time. If you're in your editor and you've got something nice set up, you can hit like a hot key and then it'll only run the tests that you need, you know, at your cursor. So giving the LLM a tool like that seems to have a lot of potential. And then that could even potentially be a smaller model, a fine tuned model for that task. It could be multiple based on what, uh, projects or, or stack you're using. And that, uh, could eliminate a lot of the confusion, even with a good agents that MD guidance about how to run tests, I still see with AMP. And I think, you know, we've, we've tried to make this really good. It only gets it right maybe 90, 95% of the time. Sometimes it'll run the wrong testing or it won't escape it correctly, and I think we can eliminate that with a sub-agent. So there's so much more potential to go deep in areas like that, and then for every language, it's a little bit different, so handle …

AI assessment note: “I think test is a big one and there's many different kinds of tests.”