The Exchanges

Q The obstacles to that right now are some new model advancements. Is it just building out some other core technology? Is it just the UI? Like what, what is keeping that from happening right now?

A Yeah, I think the first thing is the model, uh, the full O three model that's not available yet, that OpenAI showed, um, uh, as part of the ship miss, uh, uh, right before the holidays. We're going to see, you know, improved reasoning, and I think it's as the models get better in reasoning, uh, we're going to get closer to a hundred percent of this VBench, which is that benchmark out of 12 repos, um, uh, open source Python repos, um, a team in Princeton identified, uh, 2200 or so, uh, issue pull request pairs. Effectively, all the models and agents are measured against. And so that's number one, you know, the, the model and the agent combination. I think the second piece is just, uh, figuring out what's the right user interface flow. Um, if you think about the workflow of a developer, right, you, you have an issue that somebody else filed for you, you know, user or product manager or something that you filed yourself. Now, how do you know whether you should assign co-pilot to this, um, the agent to it, um, or, or whether you need to refine the issue to be more specific, right? It's crucial that the agent is predictable, that you know that this is a task that the agent can solve. If not, then you need to steer it. So steerability is the next thing you need to either, you know, extend the, The definition, um, uh, or the agent needs to come back to you and, and ask you additional …

AI assessment note: “I think the first thing is the model... I think the second piece is”

Q next, you know, level of Minecraft in 20, 25 is like use AI aggressively, learn to use it. But like, does that concern from this segment of, I'm sure you've heard it. Does that Concern like resonate with you at all? Like, can you foster the requisite depth of understanding of engineering at an abstract level when we're not writing the code, or is it like a silly, silly concern?

A I wouldn't call it silly, um, because obviously, you know, there, there's some truth, truth to that, right? It's easy, uh, you know, to cheat at a programming exercise or advent of code and, and those kinds of things. Um, as these AI models get better, these competitions of who's the best hacker or coder Are going to have to move to a whole different level, um, where you assume that the developer is using AI to, to solve the challenges because otherwise, um, uh, it's going to be way too easy. If you think, you know, the next generation of developers, you know, maybe not 20, 25, but 2035, like, look, you know, and you mentioned, you know, me growing up in East Germany and then the wall fell and they bought a Commodore 64, but they had no internet. And so I bought books and magazines and, That was it, right? Like there, there was no forum I could go to and ask questions. Um, I could go to, I went to computer club, uh, every, every Wednesday or so until nobody there had anything to say anymore that I didn't know already. Right. Um, if you, if you take that and compared to today, um, the kids of today and those that want to learn coding have an infinite amount of knowledge available to them. Um, and you know what? Also an infinite amount of patience, um, because, you know, copilot doesn't run out of patience. Parents do. I am one. And so it's incredibly, um, democratizing, uh, to h…

AI assessment note: “I wouldn't call it silly, um, because obviously, you know, there, there's some truth”

Q curious how you think about the actual flow in terms of very different users in some sense, although with some overlapping either responsibilities or goals. And what, what are the set of tools that they interact with? And is it, is it a singular tool? Is it many? Is it one company? Is it many? Where does it launch out of? Like, how do you think about all that stuff?

A One of our strongest belief at GitHub is developer choice. Um, and you know, imagine a GitHub, um, as a platform where you had only JavaScript libraries available or only, you know, React, um, uh, available to you. And we would tell you that's the only open source library you need to build an application, right? Like there would be a set of users, uh, using React, uh, using GitHub because they love React and the rest would go somewhere else because some other platform would offer them all these other open source components. Right. In AI, I think we're going to see the same thing. We're going to see a stack or universe of, um, um, companies that offer different parts of the software development life cycle, and developers pick the one that, um, you know, they like the most, that they have experience with, and you know, that convicted are the future. You know, a lot of that is, um, part of a belief system, you know, programming languages in, in many ways are, uh, uh, very similar. Um, and then if you look at, uh, the discussion between developers, you, you have get the feeling they're very different. Uh, to each other, right? Like the end of the day, they're all, all compiling, uh, down to, uh, uh, an instruction set that runs on, on your Apple M four chip or, or your, your Intel CPU or AMD or Nvidia or whatever, right? So I think we are going to have a stack of different tools an…

AI assessment note: “We're going to see a stack or universe of, um, um, companies”

Q How far away do you think, um, these agents are from being sort of the median programmer equivalent? And then how much longer do you think it takes to get to sort of superhuman?

A You know, I thought about this this morning, right? Like what, um, if regardless of what agent, uh, you're thinking of a travel agent or a coding agent, or maybe it's an agent that designs your house, the fundamental challenge is actually the same as, as you as have as a human, uh, developer, right? Like you have this big idea in your head and you can sketch it on a whiteboard, but then you want to start coding and you have to take this big idea and break it down into small chunks of work. I think that's the part where we're far away from, from agents actually being good enough to take a very rough idea and break it down to small pieces without you as, as, um, as developer or as architect, or, or even when planning your travel, constantly getting questions back of what decisions you want to make, you know, what database, what cloud, like imagine you give the agent a task saying, you know, build GitHub or build a mobile app or something like it will just be not specific enough, right? So that's the systems thinking that I think the medium developer will not Uh, be replaced by agent. And the flip side of that is a lot of, you know, what developers do is just picking up issues and fixing bugs and finding where to fix the bugs, adding a feature that comes from a, from a customer. And then you have to navigate the code base and figure out what files you have to modify. And I think t…

AI assessment note: “I think that's the part where we're far away from”

Q this transition, how software development is done, right? Like, so for example, you know, you feel like we're somewhere between crossing the tipping point of the majority of code being generated this year to maybe Like all of the code in like some cases or some, some tasks. How does that change like testing or, or, you know, the way we should look at technical debt or any of that?

A To be clear, I don't think all of the code is written by AI. I think the way, uh, this will work is that we have two layers. We have the machine language layer, you know, which is Python or Ruby or Rust, right? Those are effectively abstractions of the, of the chipset, um, and the machine instruction set. Um, and that's the last layer that's deterministic, right? Like programming language inherently when I, it does exactly what I wanted to do. And then human language is inherently non-deterministic, right? We can all, the three of us can say the same sentence and, and mean a different thing. And so while we will use human language to describe a lot of the, you know, features and, and behaviors that we're going to build, we will still have the programming language layer below that, and that we are going back and forth, um, as engineers to, to figure out Uh, is the code that was written by AI actually the correct one? Is it the one that, you know, aligns with my cost profile, if you will, as an, as an example, right? Like at the end of the day, we're still all running businesses that have to have, uh, positive profit margins. I think we're going to, as engineers have both of these layers, um, and, and we're heading into a world of more human language and, and, and less, and less programming language. Um, but at the same time, you know, we are in a world where lots of financial se…

AI assessment note: “figure out Uh, is the code that was written by AI actually the correct”

Q like, didn't exist as an idea. A year ago, you know, we now have a market full of folks experimenting with these products. Um, how, how do you think about winning the, like, GitHub is obviously a very dominant force overall, as is Copilot, but how do you think about, you know, winning the developer over and what they care about in that, um, you know, changing and competitive market?

A The way we think about winning is that, you know, we care deeply about developers, um, and that's always been, you know, the heart of GitHub is that we put developers first and that we are developers that are building products for developers. You know, we have the saying at GitHub is that we're building GitHub, uh, with GitHub, uh, on GitHub using GitHub, right? And so everything that, that we do in the company, including, you know, the, our legal terms and our HR policies and, um, Or product management, sales, um, sales enablement, all these functions are in GitHub issues and GitHub discussions and GitHub repos. So I think that's number one, that we deeply care about our own product and we're using it for everything day in, day out. You know, the first thing I do in the morning is open the GitHub app on, on my mobile phone and, and then Slack, um, as a lot of our, you know, operations company chat runs through Slack. Number two is, you know, you mentioned competition. I mean, it's obviously like they've never seen anything like that in the developer space. It's, it's the most exciting time I think for developer tools. And, you know, I've been a developer for, Uh, over 30 years. It's amazing to see, um, uh, the innovation, you know, um, the, the news, uh, that is coming out every day. And I think that energy, you know, that is in the market and innovation driven both on the ope…

AI assessment note: “The way we think about winning is that, you know, we care deeply about developers”