The Exchanges

Q Um, what's driving that? What are you finding?

A So, We have been rolling people off the waitlist that we sort of amassed when we announced when I last saw you. Um, and it's been really interesting because as I may have mentioned on the podcast, like we had to be very opinionated about the data mix and the data set that we put together for like sort of the V zero of Gini. Um, again, like to your point, JavaScript, JavaScript, JavaScript, Python, right? There's a lot of JavaScript and it's various forms in there. Um, But it turns out that when we've shipped it, um, to the, to the very early alpha users we rolled it out to, um, for example, we had some guys using it, um, with a C-sharp code base, and C-sharp currently represents, I think, about three percent of the overall data mix, um, and they weren't getting the levels of performance that they saw when they tried it with the Python code base, and it was obviously, like, not great for them to have a bad experience, but it was nice to be able to correlate it with the, the actual, like, Objective data mix that we saw. So we did, um, what we've been doing is like little top up fine tunes where we take like the General Genie model and do an incremental fine tune on top with just a bit more data for a given, you know, vertical language. Um, and we've been seeing improvements coming from that. So again, this is one of the great things, um, about sort of baptism by fire and letting …

AI assessment note: “we had some guys using it, um, with a C-sharp code base”

Q Yeah. I thought maybe we could just kind of dive into each of those actions. Code retrieval, one of the core problems you had at Indexer that You've worked on, uh, even as, as built. What makes it hard? What approach you thought would work? Didn't work? Anything like that?

A It's funny. I had a similar conversation to this when I was chatting to the guys from OpenAI yesterday. The thing is that searching for code specifically semantically, at least to start with, I mean, like keyword search and stuff like that is a, is a sole problem. It's been around for ages, but at least being able to, the phrase we always used back in the day was searching for what code does rather than what code is. Like searching for functionality is really hard, really hard. The way that we approached that problem was that obviously like a very basic and easy approach is right. Let's just embed the code base. We'll chunk it up in some arbitrary way, maybe using an AST, maybe using number of lines, maybe using whatever, like some overlapping, just chunk it up and embed it. And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the cosine similarity and get the top of K right. That doesn't work. And I wish it did work. Don't get me wrong. It doesn't work well at all because fundamentally, if you think about like semantically how code looks is very different to how English looks. And there's like not a huge amount of, of signal that's carried between the two. So what we ended up, the first approach we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take …

AI assessment note: “Let's just embed the code base... That doesn't work.”

Q Yeah. I thought maybe we could just kind of dive into each of those actions. Code retrieval, one of the core problems you had at Indexer that You've worked on, uh, even as, as built. What makes it hard? What approach you thought would work? Didn't work? Anything like that?

A It's funny. I had a similar conversation to this when I was chatting to the guys from OpenAI yesterday. The thing is that searching for code specifically semantically, at least to start with, I mean, like keyword search and stuff like that is a, is a sole problem. It's been around for ages, but at least being able to, the phrase we always used back in the day was searching for what code does rather than what code is. Like searching for functionality is really hard, really hard. The way that we approached that problem was that obviously like a very basic and easy approach is right. Let's just embed the code base. We'll chunk it up in some arbitrary way, maybe using an AST, maybe using number of lines, maybe using whatever, like some overlapping, just chunk it up and embed it. And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the cosine similarity and get the top of K right. That doesn't work. And I wish it did work. Don't get me wrong. It doesn't work well at all because fundamentally, if you think about like semantically how code looks is very different to how English looks. And there's like not a huge amount of, of signal that's carried between the two. So what we ended up, the first approach we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take …

AI assessment note: “searching for functionality is really hard... Let's just embed the code base... That doesn't work”

Q Uh, so you just spoke at OpenAI Dev Day. What was the experience like? Did they reach out to you? Um, you seem to have a very close relationship.

A Yeah, so off the back of, off the back of the work that we've done, that we spoke about last time we saw each other, um, I think that OpenAI definitely felt that the work we've been doing around fine-tuning was worth sharing. Um, I would obviously tend to agree, but today, um, today I spoke about some of the techniques that we learned. Obviously, it was like a non-linear path Arriving to where we've arrived and the techniques that we built to build Genie. Um, so I definitely, I think I shared, um, a few, a few extra pieces about some of the techniques and how it really works under the hood, how you generate a data set to show the model, how to do what we show the model. Um, and that was mainly what I spoke about today. I mean, yeah, they reached out and I was, I was super excited at the opportunity, obviously, like it's not every day that you get to come and do this, um, especially in San Francisco. So

AI assessment note: “they reached out and I was, I was super excited at the opportunity”

Q How do you set that up? It's like, how do you make a model actually understand how to run a code base, which is different than writing code for the code base?

A The model itself is not in charge of like setting up the code base and running it. So genie sits on top of GitHub. And if you have CI running GitHub, you have GitHub Actions and stuff like that, then Genie essentially makes a call out to that, runs your CI, sees the outputs, and then, like, moves on. Making a model itself set up a repo wasn't scoped in what we wanted Genie to be able to do, because for the most part, like, at least most enterprises have some sort of CI pipeline running, and like, a lot of, if you're doing some, even like, a lot of hobbyists software development has some sort of, like, basic CI running as well. And that was, like, the lowest hanging through approach that we took. So when, when Genie ships, like, the way it will run its own code is it will basically run your CI, and it will, like, take the, um, I'm not in charge of writing this, the rest of the team is, but I think it's the CheX API on GitHub allows you to, like, grab that information and throw it in the context window.

AI assessment note: “Genie essentially makes a call out to that, runs your CI, sees the outputs”

Q Do you see, how good is it at just like generalizing, you know, if you're running Rust or C++ or whatever else, it's quite different?

A It's pretty good at generalizing. Um, obviously though, I think there's 15 languages in that Technical Report, I think, that we've, that we've covered. The ones that we picked in the highest mix were the ones that selfishly we internally use the most, and also that are, I'd argue, some of the most popular ones. When we have more resource as a company and more time and, you know, once all the craziness that has just happened sort of dies down a bit, we are going to, you know, work on that mix. I'd love to see everything ideally be represented in a similar level as it is. If you, if you took GitHub as a data set, If you took like, how are the languages broken down in terms of popularity, that would be my ideal data mix to start. It's just that it's not cheap doing all this. Um, yeah, trying to have an equal amount of, of Ruby and, and Rust and, and all these different things is just at the, at our current state is, is, is not really what we're looking for.

AI assessment note: “It's pretty good at generalizing. Um, obviously though, I think there's 15 languages”

Q Call to action. Uh, I assume you're hiring. I assume you're also looking for customers. What's the ideal customer, ideal employee?

A On the customer side, honestly, people who are just willing to try something new, like the genie UX is, is different to a conventional IDE. Give it a chance like that. We, we really do believe in this whole idea of like developers work is going to be abstracted, you know, levels higher than just the code. We still let you touch the code. We still want you to dive into the code if you need to, but Fundamentally, we think that if you're trying to offload the coding to a model, the model should do the coding and you should be in charge of guiding the model. So people who are willing to give something new a chance, size of company, and honestly, well, preferably the languages that are the most represented in our, in our training days. So like, anyway, if you're like doing TypeScript, JavaScript, Python, Java, that sort of thing. And in terms of size of company, like, so long as you're willing to try it, um, and there aren't any massive like infosex Things that get in the way, like it doesn't really matter. Like code base size can be arbitrary for us. We can deal with any code base size and essentially any language, but your mileage may vary. But for the most part, like anyone who's willing to give it a try is the ideal customer. And on the employee, honestly, we just want people who, um, we're going to be hiring both on like what we call like the traditional tech side. So like buil…

AI assessment note: “On the customer side, honestly, people who are just willing to try something new”

Q Call to action. Uh, I assume you're hiring. I assume you're also looking for customers. What's the ideal customer, ideal employee?

A On the customer side, honestly, people who are just willing to try something new, like the genie UX is, is different to a conventional IDE. Give it a chance like that. We, we really do believe in this whole idea of like developers work is going to be abstracted, you know, levels higher than just the code. We still let you touch the code. We still want you to dive into the code if you need to, but Fundamentally, we think that if you're trying to offload the coding to a model, the model should do the coding and you should be in charge of guiding the model. So people who are willing to give something new a chance, size of company, and honestly, well, preferably the languages that are the most represented in our, in our training days. So like, anyway, if you're like doing TypeScript, JavaScript, Python, Java, that sort of thing. And in terms of size of company, like, so long as you're willing to try it, um, and there aren't any massive like infosex Things that get in the way, like it doesn't really matter. Like code base size can be arbitrary for us. We can deal with any code base size and essentially any language, but your mileage may vary. But for the most part, like anyone who's willing to give it a try is the ideal customer. And on the employee, honestly, we just want people who, um, we're going to be hiring both on like what we call like the traditional tech side. So like buil…

AI assessment note: “On the customer side, honestly, people who are just willing to try something new”

Q How does that differ when you work with customers that have private workflows? Like do you think, is there usually a big delta between what you get in open source and maybe public data versus like.

A Yeah, yeah, yeah. When you scrape enough of it, most of open source is updating readmes and docs. It's hilarious. Like we had to filter out so much of that stuff because when we first did the, uh, 3.5, 16 K model, like the amount of readme updating that went in, we did like no data cleaning, no real, like. We just sort of threw it in and saw what happened. And it was just like, it was really good at updating readme's, really good at writing some comments, really good at, um, complaining in Git reviews, in PR reviews rather. And it would, again, like we didn't clean the data, so you'd like give it some feedback and it would just like reply and like, it would just be quite insubordinate when it was getting back to you. Like, no, I don't think you're right. And it would just sort of argue with you. So the process of, of doing all that was super interesting. Because we realized from the beginning, okay, there's a huge amount of work that needs to go into like cleaning this, getting it aligned with what we want the model to do to be able to get the model to be useful in some way.

AI assessment note: “most of open source is updating readmes and docs. It's hilarious.”

Q just putting code in it. There's kind of like a lot more than code that goes into software engineering. How do you think about the data mixture, you know, and like, uh, there's this kind of known truth that code makes models better when you put in the pre-training data. But since we put so much in the pre-training data, what else do you add when you turn into Genian?

A Yeah, I think, well, I think that sort of boils down fundamentally to the difference between a model writing code and a model doing software engineering, because the, the, the software engineering sort of discipline goes wider, because if you look at something like a PR, that is obviously a artifact of some thought and some work that has happened and has eventually been squashed into, you know, some diffs, right? What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most part, unless there are some comments, which, you know, anyone who's worked in a company realizes PR reviews can be a bit dodgy at times, but you see that you lose so much information at the end. And that's perfectly fine because PRs aren't designed to be something that perfectly preserves everything that happened. But what we realized was if you want something that's A software engineer, and very crudely, we started with like something that can do PRs for you, essentially. You need to be able to figure out why those things happened. Otherwise, you're just going to rely, essentially, you just have a code writing model. You have something that's good at human eval, but, but not very good at SweetBenge, essentially. Tha…

AI assessment note: “extract what happened in the past, like as forensically as possible”

Q Are you using the VS code LSP or?

A No, that's not, we're not like, we're not doing this in VS We're just using the language servers running, but we really wanted to try to mimic the way we do it as best as possible. And we did that during the self play process when we were generating the data set. So although we did all that work originally, and although like Genie still has access to these tools, so it can do keyword searches and it can do, you know, basic semantic searches and it can use the graph. It uses them through this process and, and figures out, okay, I've learned from data how to To find stuff in code bases. And I think in our technical report, I can't remember the exact number, but I think it was around 65 or 66% retrieval accuracy overall measured on. We know what lines we need for these tasks to find for the task to actually be able to be completed. And we found about 66% of all those lines, which is one of the biggest areas of free performance that we can get hold of. Because when we were building Genie truthfully, like a lot more focus went on Assuming you found the right information, you've been able to reproduce the issue. Assuming that's true, how do you then go about solving it? And the bulk of the work we did was on the solving. But when you go higher up the funnel, obviously like the funnel looks like, have you found everything you need for the task? Are you able to reproduce the problem th…

AI assessment note: “No, that's not, we're not like, we're not doing this in VS”

Q How does that differ when you work with customers that have private workflows? Like do you think, is there usually a big delta between what you get in open source and maybe public data versus like.

A Yeah, yeah, yeah. When you scrape enough of it, most of open source is updating readmes and docs. It's hilarious. Like we had to filter out so much of that stuff because when we first did the, uh, 3.5, 16 K model, like the amount of readme updating that went in, we did like no data cleaning, no real, like. We just sort of threw it in and saw what happened. And it was just like, it was really good at updating readme's, really good at writing some comments, really good at, um, complaining in Git reviews, in PR reviews rather. And it would, again, like we didn't clean the data, so you'd like give it some feedback and it would just like reply and like, it would just be quite insubordinate when it was getting back to you. Like, no, I don't think you're right. And it would just sort of argue with you. So the process of, of doing all that was super interesting. Because we realized from the beginning, okay, there's a huge amount of work that needs to go into like cleaning this, getting it aligned with what we want the model to do to be able to get the model to be useful in some way.

AI assessment note: “most of open source is updating readmes and docs”