The Exchanges

Q You mentioned that initially GitHub Copilot ran on Codex, which was the early, um, coding model from, from OpenAI, uh, in, I think at the end of two of last year, 2024, you, uh, introduced GitHub models, uh, which, uh, feels like a library of different underlying models you can use. Walk us through that. How does that work? And what can you do as a developer?

A In 24, we did two things and GitHub models was, um, early August and that gives, um, developers access to catalog of models. Integrated into the GitHub platform. And in fact, last month, uh, in May, 25, uh, we brought, uh, these models into the repository. And so you can, you know, integrate these models into your repository to build AI scenarios into your own applications. And, um, that's really cool because you don't have to go to another place, you know, and sign up for a new account. And then you have your model stuff here and your code on the other side. Um, GitHub ultimately was always About, uh, millions of developers collaborating on, on a project together. And so for lots of years, we bring what we call the primitives, um, that developers need, um, into the repository, you know, issues, wikis, pull requests, actions, and now models. And then late, um, I think it was late October, uh, we announced multi-model choice for Copilot, moving from just having one model provider, OpenAI, to having multiple model providers, Uh, for Copilot Chat and for now Copilot Agent Mode. And so we added back then Anthropic Claude, 3.5 Sonnet, and now it's Anthropic Claude Sonnet four and Opus four. And we added Google Gemini back then 1.5. Now we are 2.5 pro. And what really, you know, it is about choice. Um, we fundamentally at GitHub believe that we need to offer developers choice, right?…

AI assessment note: “integrate these models into your repository to build AI scenarios into your own applications”

Q And the, the agent works with prompts, right? It's vibe coding where you describe what it is that you want it to do and it does it. Is that, is that right?

A It's prompts when you use it within the IDE. Um, although even there you can start with, um, brainstorming Uh, a cycle first. Uh, and so that, for example, that works really great, uh, with, with Claude, uh, Sonnet or Claude Opus for, as you can first ask it, you know, how would I build this and what are the, what's the system design for, for this feature, for example, and then have it first write a markdown file with bullets doing the engineering together with the model. And then you take the first task of that and feed that into the agent mode to write the code. For the coding agent, because it sits, you know, on GitHub platform, the starting point is an issue, and the issue can be, you know, the description that comes from a product manager or from a user of your open source project, but it's also all the comments in the issue, you know, attached images, um, uh, file references or, uh, web pages, and of course the coding agent and agent mode both can use MCP servers and tools, and so you can then You know, further connect into additional context. Uh, but yeah, fundamentally it's a prompt. It's just that the prompt when it's, it's no longer just one input field, you know, with the three lines of code, it can be, you know, a long description, uh, a specification, uh, from a product manager, just as they would fight for, for the human developer. And again, the product manager t…

AI assessment note: “but yeah, fundamentally it's a prompt.”

Q And for the enterprise, do you allow people to fine tune their model based on their data? How does that part work?

A We do not today. Um, we, um, have pursued that path in the past. Um, so I think 20, 23, 24, a number of companies in our space saw fine tuning as the next, the next opportunity. The challenge from my perspective on fine tuning is that, uh, A, as there's another model every other day, uh, you're, you're ultimately always going to be behind with your fine tune model. And we are putting you then in a position where you're having the choice between the model that you fine tuned Uh, that is based on an older version of, you know, of the base model, or you can pick the new model, but that isn't fine-tuned yet. Um, B, I think most, uh, customers' code bases, um, especially if you look into, you know, the individual project, you know, most companies that are at a certain scale have not just a single programming language and a unified stack. That's the dream every CIO or CTO is talking about. Like, I want a standardized stack for all my developers, and Then you look into a 30 person startup, and of course, even they don't have that because as soon as they go from web development to iPhone development to Android development, they already have three stacks. And so then if you look at the individual repository or set of repositories, that code base isn't actually big enough to have a meaningfully fine tuned model. And lastly, I think this is where fine tuning kind of like got left behind b…

AI assessment note: “We do not today.”

Q Yeah. Let's double click on that for a few minutes. The launch of that agent mode, uh, was, uh, a major announcement, major new step in the history of, of GitHub that just happened at that build 2025, a few weeks ago. What does it do? Do and what tasks in particular would you suggest people should direct it to?

A So the coding agent, the way it works is that you can just give it a task, coding task, um, documentation, generating test cases, or simple things like find all the bugs in my code base. The agent then, you know, goes off in the cloud and spins up a virtual machine and checks out the repository, installs other tools, and Solves that task for you. And the magic here is that in the meantime, you can keep working on your part of the code base on a different task, a different issue on your local machine. And so effectively the coding agent is like a new member of your team that can take on certain tasks. And you can obviously, you know, not only assign one task to one coding agent, you can assign 10 tasks to 10 versions of, of that coding agent, and they can all run in parallel. And when they're done with their work, they submit a pull request exactly like one of your human team members would do. And then they alert you and say, Hey, this pull request is ready for review. And then you go in and you review the code and you can comment on it and the co-pilot will pick up those comments and, and keep iterating. And so if you don't like the code or, you know, I tested it yesterday with one of my hobby projects and I realized the README is, Uh, completely outdated because, you know, I didn't spend time on writing a README for myself and just told the coding agent, look at the code base,…

AI assessment note: “give it a task, coding task, um, documentation, generating test cases”

Q And when you mention 30% or 40% success, is that across the board or from your experience using the product and early feedback from users, is the agent particularly good at certain tasks versus other tasks? If I'm a GitHub user and I want to play with product, what should I do first?

A The 30% I was referencing the SWE benchmark that was developed Uh, by a number of researchers and originally started as in Python only, um, and it's 2000 issue pull request pairs out of a dozen Python open source repositories, but was recently expanded to other languages. And so in Python, I think the big best benchmarks is 60 to 70%, depending what model, uh, agent combination you take. Uh, but if you look at multilingual, we are in the 20 to 30% range for these benchmarks. So that gives you an idea of real life Issues in open source projects and the corresponding pull requests. How, how, how good is the agent on solving that existing issue compared to the original solution? For our coding agent, the way to approach is for developers to just go and try it out. Because worst case scenario here is that you get a pull request that is so far off from what you would build yourself that you close the pull request without merging it. But then that gave you a learning cycle between that The description that you gave it and the code generated. And then, uh, you know, there's multiple ways you can approach that, um, if you don't want to throw it away. One is, uh, to provide custom instructions in a file within the repository to give the agent more context of what you expect it to do. So it's kind of like a, you know, how, how to, uh, that you provide to the agent in the same way that if…

AI assessment note: “the way to approach is for developers to just go and try it out.”

Q I understand where, uh, your, your plugin into VS Code, and then the, another part of Microsoft needs to make sure that VS Code remains very versatile, uh, broadly applicable to lots of different Use case and scenarios. Do you feel constrained by that in terms of how much you can do that's truly AI native because you need to operate in that framework where there are other driving factors?

A Not at all. And the VS Code team, while they don't report into me as the Gap CEO, they are part of the developer division that we are also part of. And together with the developer division, we are part of an organization within Microsoft called Core AI. Uh, that has the whole AI stack in it as well. And so, you know, under Jay Parik, who's the leader for Core AI, uh, we're forming one unit, uh, that competes in, in this age of AI with, with all these companies across the categories that I lined out. And as I mentioned, we're also partnering with them. So a different, a different part of the Core AI team might have, you know, uh, one of our GitHub competitors as, as their customers and, uh, and have partnership meetings with them and negotiate Deals and, and, you know, Azure spent and those kind of things. Coming back to VS Code, um, at Microsoft Build in May, we announced that we are open sourcing, uh, the Copilot within VS Code, which also means, uh, we're moving it from an extension, uh, which is how Copilot originally started into a core part of VS Code. And so it, it becomes open source as part of the VS Code open source project or the code, uh, .OSS project. And as such, the VS Code team, and that's been true for a while, is actually the one building the client side of GitHub Copilot. And so for, I think, the last two years now, the VS Code team owns GitHub Copilot client …

AI assessment note: “Not at all. And the VS Code team, while they don't report into me”

Q world where anyone can spin up applications quickly? Uh, do we all build our own applications ultimately? And then another big, meaty question, uh, which, you know, I'm not suggesting we go into it in great detail, but what does that mean in terms of the future of being a coder? Uh, what, what are the skills I need to develop or not care about in the, in the future?

A The future of, of coders, uh, of software developers, software engineers is probably the best way of framing it is bright from my perspective, and software developers will learn Very fast, if they haven't already, to adopt AI within their workflows, not only co-pilot, uh, and not only in the IDE or on GitHub, but also on the command line, when processing files, when organizing files, you know, all the things that we also, when, when operating clusters, all these things that we also do, um, and we have always, you know, as, as software developers, automated parts of our workflow and moved higher up the abstraction ladder. You know, when I started coding on a Commodore 64, there, there, Wasn't a debugger. Everything was, you know, trial and error. You would print stuff onto the console to see, uh, what, what the variable, uh, content is. And then came, you know, Turbo Pascal and, uh, and Visual Studio and so on. And our tools got, got more complex. And as such, we, we moved up and, and get more done in the same amount of time. And I think for developers, the same old thing will happen again. It's in fact already happening. It happened with the cloud. It happened, you know, with, Um, uh, containers like Docker and Kubernetes. Um, today most developers in the web don't really care about the hardware anymore. Certainly not about the chip itself, other than, you know, in, in models w…

AI assessment note: “software developers will learn Very fast, if they haven't already, to adopt AI”

Q on those, uh, two points, which are super interesting. The GitHub helping Microsoft. To which extent does GitHub fit into the Azure broader picture? The hyperscaler business model is, you know, generally described as building a bunch of applications that drive demand for the underlying compute. Is that one of the key objectives? And what is the intersection between GitHub in general, GitHub Copilot in particular, and underlying Azure business?

A The Azure business within Microsoft came out of division called Server and Tools. Um, today that division is called Cloud and AI, and the Server and Tools team also had the developer division, but this core product, uh, back, back in 2018 being Visual Studio, you know, in itself, um, uh, the IDE, you know, the old school IDE in itself, a billion dollar business, um, and had a bunch of new stuff like Visual Studio Code. It had a platform for DevOps Back then called Visual Studio Team Services, nowadays called Azure DevOps. And, um, so that tools division, you know, within the cloud, within Azure, uh, had, had the job of, um, providing the ecosystem, the tools, the platform for developers to build on top of Azure, to build on top of Windows, you know, to build on top of other Microsoft platforms and GitHub snapped into that developer division and bringing basically the open source platform, the home of all developers. Uh, into that same ecosystem, you know, from a business perspective, that meant the GitHub revenue became part of the Azure KPI that is reported, you know, in, in earnings calls, um, uh, every quarter, um, from Microsoft. And, uh, you know, we announced last July, um, uh, so almost a year ago that GitHub had passed two billion in, in annual revenue run rate ARR. And, um, you can go back in time, 2017, a year before the deal, then GitHub leadership team had, uh, anno…

AI assessment note: “from a business perspective, that meant the GitHub revenue became part of the Azure KPI”

Q developers that already wants pure AI. I was curious what the bear case is for them. Like the, the reason why they're going to be, you know, a roadkill from the power of Microsoft and perhaps Google. So that's distribution. That's developer mindshare. Uh, you know, you're a very competitive guy and Microsoft is a very competitive organization. Like what, why is the, how are you going to crush them?

A I'm not thinking about crushing them because I think that's not, you know, how we are seeing, uh, the space. Um, like I, I like for developers to pick the tool they love most, and I, I like to build the ones that most of them are picking. And so I think realizing that in developer tools and in automated technology, There's always going to be multiple players. Um, and there's always, and some of them and some technologies stay around forever. Kobo on mainframes is one such example that comes up way more often than you would think for me as a good CEO, because many companies still deal with that Kobo on mainframe. Now, is that, you know, the state of the art and, and something that college kids get, get excited about? Of course not. But we should realize that once some software is deployed, it sticks around. It's incredibly Sticky and it sticks around forever and migrating things and migrating even a repository, you know, from subversion into, into Git and then into, onto GitHub that are often, uh, very complicated, uh, and expensive projects that engineering leadership only wants to fund if they can see the ROI. I think the bear case for, for any company in our space is missing out on the next big thing. Uh, and there's always going to be a next big thing in, in software demo, whether it's, you know, front end frameworks, whether that's program languages, uh, Whether it's IDEs, …

AI assessment note: “the bear case for, for any company in our space is missing out on the next big thing”

Q You know, again, in an effort to make this interesting to both a developer coder audience, but a broader audience as well, how would you describe GitHub Copile in two minutes, and how does that interface with VS Code, and what is VS Code, and so if I'm a non-developer, what does that all mean?

A The way I'd like to start describing what GitHub Copile does is describe what a developer's job actually is, which is Writing code. And so most developers in the morning, they go somewhere where they have their backlog, you know, the tasks that they define themselves or often that comes from their manager or from their product team. And they pick up a task and then they go into an editor, which is, you know, ultimately a program where I can edit files with plain text. And, um, the key skill of a developer is to take this description that is in human language, because that's what all the specs All the issues, all the bug reports are in, right? Like they're in English, German, French, um, whatever language the stakeholder used. The key job of the developers to translate that language into, into code, um, and to take human language and transfer it into programming language. And so they type code and as any writer, whether it's a coder or whether it's a journalist, um, or a VC writing a blog post, it's really hard to do that because you're Getting stuck when you have an empty file, you're getting stuck because you don't remember specifically how to call a method or how to do, you know, a certain algorithm, uh, connecting, you know, to some backend system and so on. And so what you did before Copilot would command tap or alt tap on your computer into a different application, often t…

AI assessment note: “The way I'd like to start describing what GitHub Copile does is describe”

Q uh, Early to this market, which when you think about the company that tends to be the other way around, right? Especially for people like me in the venture world is always like, you know, the startup moves faster than the big company. That was not the case here. Uh, what was the, uh, secret and the kind of management tricks and techniques that enable you guys to do that?

A Tying it back to the acquisition, you know, one story I can give you is that, um, beyond all the other reasons I gave you that we are part of the strategy document, uh, that we had to write, uh, Uh, to send it to Satya and then for Satya to go to the Microsoft board and automatically get approved for the deal. One additional thing was in fact AI and saying, we believe there's a future that where we can train AI models on the graph is I think what we called it of all the source code that is stored on, on GitHub, but also on the relationships between the developers, you know, how they, how they work together. And, and so in 2018, obviously we didn't know anything about, uh, co-pilot, um, Uh, but AI was, was certainly a reason, uh, to, to do the deal. Um, comes 20, 20, and, uh, in the meantime, Microsoft had invested into open AI. I think that was in, in mid 2019. And GPT three was on the horizon and, um, we got early access because of the partnership between Microsoft and open AI. And, um, we played, we played for the model. Um, just like, you know, today you play with a new model whenever a new one comes out, which feels like it's every other day. And we ask it, uh, you know, to, to write methods, uh, like prime number detection, sorting algorithms, those, those coding exercises available. In fact, you know, through that process, we looked at our own coding exercises that we use…

AI assessment note: “we got early access because of the partnership between Microsoft and open AI.”

Q as a side matter, you've diversified away from one supplier. That tension also between, uh, Anthropic, uh, potentially and Cursor, where Cursor has started to diversify away from them. So how do you view this evolving? Ultimately, everybody competes, but also Cooperates or is there a moment when uh,, I don't know, you guys maybe need to develop your own model specifically for coding? How do you view it evolving?

A It's the DNA of Microsoft um, to be competing and partnering uh, with, with many companies in the industry, right The,? The most classic example comes to mind is that Microsoft for the longest time Has, um, sold software, um, or provided software nowadays for the Mac, um, back, going back all the way to the Apple II. Uh, I think most of us remember Bill Gates appearing in, in a Steve Jobs keynote, big on screen behind him, uh, to announce, uh, that, um, Microsoft is investing into Apple, uh, and, uh, bringing, um, Internet Explorer, uh, and Office, uh, uh, To, to macOS. And then of course, you know, after Asatia became the CEO, um, the decision was made to bring the office suite onto the iPad. And nowadays we take it for granted that, uh, office teams outlook, um, all the modern Microsoft applications, uh, run on a Mac and run on iPhone and Android. And, um, as such, I think, you know, and I learned that when I joined Microsoft, uh, 10 and a half years ago, it was weird as a startup founder to come into the company and realizing that What I considered competitors for the product I owned at Microsoft are often also partners. Um, uh, I, I can't name any names, but, you know, many of these AI, um, code generation companies that compete with GitHub Copilot are running their inference on Azure AI Foundry. And as such, uh, you know, they're paying Microsoft, uh, for, for the lower pa…

AI assessment note: “It is part of our DNA to both partner and compete.”