The Exchanges

Q Tell me a bit more about that. You guys have obviously deployed with a lot of the biggest companies in the world. What are some of the more impressive results you guys have seen in the wild?

A Yeah. So the earliest results that we, where we, where we saw and we were like, oh, there's, there's something here where they started with, um, basically like modernization programs. So people that had large legacy existing things, they needed to transform. And if you sort of did the math to scope out how long it would take, maybe it would be like a two year project. And, you know, relatively quickly by like late 20, 24, we were measuring You know, somewhere between a six to 12 x productivity gain for those types of projects. Meaning that, you know, one hour of human time spent managing Devin was worth like six to 12 hours of that human time doing the work themselves. Um, and so that was a big, that was like a big early result, and we started doing lots of, lots of engagements where our customers would use, would use Devin to just refactor, migrate, modernize these large systems. Now the interesting trend that we're seeing is, Um, a shift from really, uh, reactive to proactive engineering work. So, you know, if you think of the early days of the internet, most of the packets that were sent on the internet, um, it was like a human clicking a button or, or visiting a link or initiating some requests. And then at some point it, it totally flipped and now most of the packets are initiated by machines talking to other machines. And I think we're now seeing the sort of the flippenin…

AI assessment note: “we were measuring You know, somewhere between a six to 12 x productivity gain”

Q and making it talk to each other, ontology, the processes, and we had all sorts of, Different frameworks over time. There are conceptual frameworks we'd use when you go in. Do you guys have, like, your own conceptual frameworks for business value, and do you have something called ontologies? Like, like, not to, not to get the secret sauce, but other things like this you could, you could tell us.

A We, we really look at it from the perspective of the software development life cycle. So, we go inside an organization, um, they are, they have a way of doing things, right? Of, of developing software, starting from planning and deciding what they even want to write, to understanding all of their existing code and process, to then maybe scoping it out, and And maybe writing some code, testing it, fixing it when it's wrong, iterating on it, deploying it in production, monitoring it. You know, there's a pretty standardized software development lifecycle at this point. And what's happening is agents are just eating more and more of the cycle. And it kind of started with the writing of the code. And now we're like well past that, right? And so in fact, one of the more recent products we, we shipped is called Devon Review. It's because we observed that there's this totally new bottleneck in the software development, uh, in the software development lifecycle that wasn't the case previously, which is there's this abundance of code being written by AI now. How can humans even keep up with it all to understand what's going on? Again, we work with, you know, a lot of like regulated, large, complex organizations that are, they're running mission critical systems, and you can't just sort of vibe code, you know, your way and like YOLO merge, uh.

AI assessment note: “We, we really look at it from the perspective of the software development life cycle.”

Q I mean, could, could the product people themselves create some things now, or how does that work?

A I mean, we see it every, there, there's like a joke where it's sort of, you know, the, the, the engineer and the designer and the product manager all look at each other and say, I don't need you guys anymore. Uh, because they're all, they're all just doing it all themselves, right? It's like every person is empowered to, to do the other aspects of the product development life cycle. And, and I think it's really rewarding people who are, you know, actually personally highly agentic and thinking about, okay, what's the impact I can go have on Um, and, and be sort of like self-reliant in that way. And I know, you know, with Devon, a lot of product managers, one of the very first things they, they started using Devon for was actually to not bother the engineers with questions, with silly questions. You know, how often have, you know, you're a new employee at a company, you don't understand what's going on somewhere, and you're a little nervous to say, hey, can you explain this to me? You know, everyone, Devon is very non-judgmental. You know, you ask a question, you get the answer.

AI assessment note: “every person is empowered to, to do the other aspects of the product development”

Q This is probably harder for someone who's my age, I'm 43, than someone who's like 18 and still learning, or no?

A Yes, because some people say, oh, you know, it's gonna be really hard for junior engineers now because, you know, the entry level, the entry level tasks are being done automatically by AI, but I think a lot of what we see internally, it's kind of the opposite in some way, where if you're coming in with no preconceptions about how things are supposed to be done, or how things are supposed to work, Then you can just go all in, just really embracing this completely new way of working. Um, but I think the AI technical depth piece is, um, it's actually not just in the sort of modern generative AI era. You know, when I was at, uh, autopilot, uh, I was a machine learning scientist working on the sort of the vision neural network, and Elon had this phrase that he really drilled into us, which is, you know, everyone is chief engineer. You know, everyone on the autopilot team has to understand how the full stack worked. And this is actually extra important in AI because what happens is, The abstraction boundaries between different teams start to break down. You know, the sort of classical way that the self-driving system worked, which you had a, you know, a perception team, you had a planning team, you had a controls team, and they had these, like, thin interface boundaries between them. But the nice thing about AI is you can optimize systems end to end. So if you want to actually optimi…

AI assessment note: “Yes, because... if you're coming in with no preconceptions about how things are supposed”

Q a year in IT. It's ironic because you're right. Sometimes government in the past, especially when innovation was really expensive, they, they pushed some new things that otherwise wouldn't have happened. Uh, today, most of that a hundred billion dollars is spent by on special interests that don't seem to be using the money well. So it's, it's a giant mess. What are the types of projects you're working on?

A Totally. I mean, the, the incentives are obviously super screwed up for a bunch of reasons that your listeners are probably familiar to. One of the less One of the less known ones that I think we actually might be able to just sidestep is the government is a really unique buyer of software for a bunch of reasons, but one of them is that a lot of times they want to own the IP of the software they're using. And this is a, this has a really big implications for most SaaS businesses. You know, if you make scheduling software and your business is a SaaS business, you don't want the government to own your IP. You want them to have a license to it, to use it for scheduling. And actually that desire Is literally incompatible with how a lot of government contracting has worked and happened historically. So you end up in this situation where the government says, oh, you know, this scheduling provider is a real example. This scheduling provider that has great SAS that can do scheduling. I can't use it cause I wouldn't own the IP. So I have to go work with the systems integrator and completely custom build my own. Right. Insane. Now, now what we could, could we lobby and go try to convince the government to change their policies? Yes. But actually I think easier for us to just sidestep the problem and say, look, Devon can just write this thing for you, you know. Just build it really fast. …

AI assessment note: “easier for us to just sidestep the problem and say, look, Devon can just write this thing for you”

Q And cognition for government, this is a fast growing business for you then right now?

A I would say government is one of those things that it, it really takes some time to kind of build and, and be compliant and work in the way that people want to work, and then once you're there, Uh, once you're there, it's, it's much easier to be helpful. And so we've, over the past year, we've really done a lot of the legwork to, you know, how do you get your FedRAMP certification? How do you, um, understand the needs of these agencies, which are actually pretty different in a lot of ways. Again, just in the, just in the civilian sector, these, they're operating under completely different trade-offs, right? A lot of the software they use is actually by statute, not allowed for them to write themselves. Talk about regulatory capture and, you know, there are literally laws saying you government agency are not allowed to maintain your own website. You have to bid this out to, uh, to, to a contractor. And, and so it kind of sounds insane, but you know, that's the way the system works. And I guess my experience working on AI with technology is that a lot of times it's actually easier to solve like a frontier science or engineering problem than it is to sort of change the molasses of the existing world, right? Uh, when we were working on self-driving at Tesla, a lot of folks would ask us, hey, like, why don't you make the cars talk to each other? Wouldn't that be way easier for self-…

AI assessment note: “government is one of those things that it, it really takes some time”