Cosine

“I think it's fairly obvious that with the foundational models, like you can ask them to think by step by step and ask them to plan and stuff, but that isn't enough because if you look at how those models score on these benchmarks and then they're not They're n…”

“For the moment, as a closed source company, like fighting for an edge, we've decided not to publish that information for that exact reason. I don't want someone basically taking my tragedies and then taking a model that's suing them in GA and just distilling i…”

“And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the cosine similarity and get the top of K right. That doesn't work. And I wish it did work. Don't get me wrong. It doesn't work w…”

“So what we ended up, the first approach we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take in English code queries and then produce a hypothetical Code snippet that might look like the answer, embed that,…”

“What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most par…”

“At least with what we've seen, the browser is helpful, but it's not as helpful as like writing the correct files. If that makes sense. Like, it is still helpful, but obviously there are more fundamental things you have to get right before you get to like, oh y…”

“Eventually we were able to get on the experimental access program and we got access to four turbo fine tuning. As soon as we did that, because in the entire run up to that, we'd built the data pipeline. We already had all that set up. So we're like, right, we …”

“We got 219 out of 500, which is 43.8%, which is To my knowledge, at least right now, state of the art also”

“We train Genie to write diffs and, you know, essentially patches, right? Because it's more token efficient”

“When models like that come out, obviously the signal in my data, when I regenerate it goes up. And then I can then train that model that's already better at reasoning with improved reasoning data. And just like, I can keep bootstrapping and keep leapfrogging e…”

“And I think in our technical report, I can't remember the exact number, but I think it was around 65 or 66% retrieval accuracy overall measured on. We know what lines we need for these tasks to find for the task to actually be able to be completed. And we foun…”

“The certainty of code writing is so much more certain than every other aspect of Genie's loop. So whatever's going on under the hood, the model is really comfortable with writing code. There is no doubt, and it's like in, in the token probabilities.”

“We have a version of Genie that is fine-tuned on our code base. So we basically, it's the base Genie, and then we run the same data pipeline that we run on, like, all the stuff that we did to generate the main data set on our repo.”

“Fundamentally to build a product like this, you need to get as much information in front of the model as possible and make sure that everything ever writes in output can be Traced back to something in the context window, so it's not hallucinating it.”

“The patches are still public. So like the, dare I say, traditional Sweebench submission, you can go to a GitHub repo and see it and run them for yourself and verify that the numbers come out correctly.”

“And that was in sort of two parts. We synthetically generated runtime errors where we would Intentionally mess with the AST to make stuff not work or index out of bounds or refer to a variable that doesn't exist or errors that the foundational models just make…”

“So we're going to make everything editable, including the code it writes. Like you can, if it makes a small error in a patch, you can just change it yourself and let it continue and it will be fine. So yeah, like those things are super important. We'll be doin…”

“We eventually built the world's most flimsy system, which was back in the day, like 4000 token context windows, like chaining prompts, trying to keep as much context from one to the other, all these different things where essentially you'd put in an app idea i…”

“Actually we use models that are larger than what's publicly available, something publicly available yet, but when this goes out, it will be, but we have larger law adapters available to us, just because the amount of data that we're pumping through it”

“The way that we Decided we want to try to extract what happened in the past, like as forensically as possible, has been and is currently like one of the main things that we focus all our time on. Because doing that as we're getting as much signal out as possib…”

“The genie has both of those tools available to it as well. So yeah, yeah. So we have a tool where you can like put in URLs and it will just read the URLs and you can, it also uses perplexities API under the hood as well to be able to actually ask questions if …”

“Mostly we've actually seen, I'd say people becoming more amenable to the idea over time, actually rather than more skeptical, because I think they can see the upside.”

“The model itself is not in charge of like setting up the code base and running it. So genie sits on top of GitHub. And if you have CI running GitHub, you have GitHub Actions and stuff like that, then Genie essentially makes a call out to that, runs your CI, se…”

“As soon as you can fine-tune Gemini 1.5, then you best be sure that Genie will have, will work, will run on Gemini 1.5, and, like, we'll probably get very good performance out of that.”

“Well, there are going to be some things that you need to build this when the tech does catch up. So retrieval being one of the most important things, like the model is going to have to be able to like pull code out for code base somehow. So we were like, well,…”

“Everyone is using the same base models here, or the same model APIs period. I don't know if anyone except for cosine was even training, but like the diff here is all in prompting.”

“So we had Cosign on they had a, like a 50 something on full, on Sweebench full, which is the hardest one. And they were rejected because they didn't want to submit their traces.”

“And then we also on this podcast, we interviewed Cosign that actually fine tune four O to, on three bench, sorry to achieve data on three bench. And that score was actually higher than O one when it came out.”