The Exchanges

Q One, one question for you before we zoom out from the, from some of the technical stuff. Does the offering of small models, like the seven or eight by seven B size that are quite capable, I think surprised a lot of people, um, from, from Mistral, like do small models that show higher level reasoning change your point of view at all or how you guys approach this?

A Uh, we're very bullish on small models, so we, we've actually integrated Mixtrol into Kodi. You can use Mixtrol, uh, as one of the models in Kodi chat, uh, as of, uh, last week, and it's just amazing to see the progress on that side. Uh, I mean, there's a lot to like about small models. They're cheaper and faster, and if you can make them, uh, approach the quality of the larger models for your specific use case, then, you know, there's, it's a no-brainer, uh, to use them. Um, I think we also like them in the context of completions. The primary model that Cody uses for inline completions right now is, uh, StarCoder, uh, seven billion. Um, and with the benefit of context, uh, that actually matches, uh, the performance of, uh, you know, larger proprietary models. And we're just scratching the tip, uh, of what's possible there with context fetching right now. So I think, uh, we're very bullish on, uh, Pushing, pushing that boundary up even further. And again, with a smaller model, inference goes much faster. It's also much, much cheaper, which means we can provide a faster, cheaper, uh, product to our users. What's not to like there? Um, I think there is a question, uh, with the smaller models, uh, specifically in the context of, uh, RAG, uh, because I think there's been some research that shows that the kind of like in context learning ability of large language models is, is a lit…

AI assessment note: “we're very bullish on small models, so we, we've actually integrated Mixtrol into Kodi.”

Q Do you take into account the quality of code in this pipeline in, in some way? Because, you know, you're working on customer code bases. Like if there's anything like the code bases I've interacted with, like there's, you know, there's a variance of quality, but that's the real world. So like, what do you mean by, you know, high quality code here?

A I mean, we kind of implicitly do right now because we, uh, built into Cody is, is this notion of like, you know, which code is it referencing? It's going to reference the code. In your code base, uh, first, and that's probably the, the most relevant code if you're trying to work on day-to-day tasks in a private code base. We're probably going to release a feature soon. This is something that our customers have requested. Um, basically the ability to point Cody at areas of the code base that are better models of what good, uh, looks like. We, we've talked with a lot of enterprise customers where when we say like, hey, you know, Cody has the context of your code base and we'll go and do a bunch of code searches. When it's generating code for you, uh, their initial reaction is like, uh, can I tell it to ignore, uh, large parts of the code? Because there's certain parts of the code where like, yeah, those are antipaths. We're trying to like deprecate that or migrate away from that pattern. And we're like, yeah, absolutely. That's actually like a very easy thing to do at the, the, the search layer. And, and the nice part of, of this too, is, um, when you're doing rag, you can, you can be very explicit about, The, the type of information, the type of data you're fetching into the context window. You basically like can give someone like a lever that they could turn on or off or like a…

AI assessment note: “we kind of implicitly do right now because we built into Cody”

Q surprising to me, and I think, like, March of this year was, uh, like, we just don't need to learn to code anymore, right? And I'm like, like, how could you say that? It's like, you know, like, they don't even teach a garbage collection anymore, like, grumpy old man. Um, like, where's the CS fundamentals? Like, what do you think people need to know? Like, what would be valued?

A So my take on this and, you know, here's the advice I would give to myself or, you know, a younger sibling or my child, you know, if they were, uh, you know, at that age where they're trying to determine what skills they should invest in. I think coding is still going to be incredibly valuable skill moving forward. Um, I think in the limit, the things that are going to be valuable that are going to differentiate, uh, humans operating in collaboration with AI, if you think about, like, layers through which software delivers value, you know, at the very top, you have kind of like the product level concerns, the user level concerns, like, how do I design the appropriate user experience? How do I make this piece of software meet the business objectives, uh, that I'm trying to achieve? And then you have at the very bottom, the very low level, okay, like what data structures, what algorithms, what sort of, uh, specific things underneath the hood are happening that are going to roll up to the high level goals that I want to achieve. And then you have like a lot of stuff in the middle, uh, that is really just mapping the low level capabilities, uh, that you're implementing to the high level goals that you're trying to achieve. And I think what AI will do is it will compress the middle, um, because in the middle is really just a lot of, like, abstractions, uh, and, um, middleware and ot…

AI assessment note: “I think coding is still going to be incredibly valuable skill moving forward.”

Q Actually, I'm very interested. Do you do both, let's say, other traditional information retrieval approaches like ranking along with AST transversal? Or like, is there, is there information missing from the graph context that's also useful either for your humans using search or for the models using search?

A Yeah, there's a ton of data sources. Let's start with the search side, which the search problem is really like, hey, the user asked a question. Now find me all the pieces of code or pieces of documentation that could be relevant to answering that question. We really view that as a generalized search problem. Um, it has a lot of, like, parallels to end user search, uh, with the difference being, you know, for human search, it's really important to get the quote unquote right result in the top three. Otherwise people will ignore it. Whereas with, uh, language models, you actually have a little bit more flexibility because, you know, you have a context window of these days at least, you know, 2000 tokens, some cases much longer, right? And then in terms of how you do that fetching, um, The, the overall architecture is very similar to how you would design a search engine. So you have a two-layered architecture. At the, the bottom layer are your, kind of, like, underlying retrievers. Um, so the base case here would be just keyword search. Um, or, you know, the fancy way of saying that nowadays is, uh, sparse vector, uh, search. Uh, if you use the, kind of, like, one hot encoding where ones correspond to the presence of certain dictionary words, um, Anyways, that's just keyword search. It actually works reasonably well. I think if you talk to a lot of RAG practitioners, uh, you'll fi…

AI assessment note: “The, the overall architecture is very similar to how you would design a search engine.”

Q We can't At this time of year, not make predictions. So one is just, you have thought about software development and how to change it for literally a decade now, probably longer since you had to like think about it to start the company. Um, what does it look like five years from now?

A That is a great question. Where my mind goes is, um, Well, I guess to answer where software development will go in the next five years, um, maybe it's, it's kind of informative to look how at how it's evolved over the past. There's a seminal work called the mythical man month, uh, that was written in the seventies about software development that today, oddly enough, despite all the technological changes, uh, still rings very true. And the, the core thesis of that book is that software development is this Strange beast of, of knowledge work, uh, that's very difficult to, to measure. The common mistake that people make again and again is to treat it, uh, as some sort of like factory style work where, uh, you know, commits or lines of code, uh, are kind of commodities. And, and the goal is just to try to like ship as many of those widgets out, uh, as possible. Uh, whereas, you know, anyone who's spent, you know, a month inside, uh, A software development or working as an actual software creator knows that there's such a high variance in, in terms of the impact that, uh, a line of code can make. You know, you have some features that eat up many lines of code that have very little business impact. And, uh, there's also kind of like one line changes that, uh, can, uh, be game changers for, for the product, uh, that you're building. And so when I look Look forward at how software deve…

AI assessment note: “I like to place it in the context of solving a lot of the challenges”

Q um, The Stanford AI Research Lab way back when. Yeah. Like, that wasn't the starting point for Sourcegraph. It was more like, oh, we need, like, super grep, right? Like, we just need a version of search that works in real environments and is useful for getting to flow. When, when in the story of Sourcegraph did you start thinking about how advancements in AI could, could change the product?

A My first love in terms of computer science was, it was actually AI and machine learning. That's what I concentrated in. Uh, when I was a student at Stanford, I worked in the Stanford AI lab, uh, with Daphne Koller, she's my advisor, uh, mostly doing computer vision stuff in those days. And it was very different in those days. We're now living through the neural net revolution, you know, we're well into it. Uh, it's just like neural nets, uh, everywhere. And in those days, it's still kind of like the dark ages of neural nets, where it was after the first initial Uh, successes they had in like the late eighties and nineties doing OCR with them. Um, but then after that, the use cases sort of petered out. And by the time that I was doing it, uh, the conventional wisdom, the thing that they told us in, you know, machine learning one-on-one was like, you know, neural, that's where this thing that we tried, you know, a decade or so ago, but it didn't really pan out. So these days we're, we're mostly focused on, uh, graphical models and statistical, uh, learning techniques, you know, really trying to be explicit about modeling the, uh, Probability distribution of what we're trying to represent.

AI assessment note: “My first love in terms of computer science was, it was actually AI”