The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 28 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 1 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Cosine refuses to publish SWE-bench trajectories to prevent competitor model distillation
“For the moment, as a closed source company, like fighting for an edge, we've decided not to publish that information for that exact reason. I don't want someone basically taking my tragedies and then taking a model that's suing them in GA and just distilling i…”
Extracting problem-solving history most determines Cosine Genie's SWE-bench performance
“The way that we Decided we want to try to extract what happened in the past, like as forensically as possible, has been and is currently like one of the main things that we focus all our time on. Because doing that as we're getting as much signal out as possib…”
Wang: Cosine scored 50+ on SWE-bench Full but was rejected over traces
“So we had Cosign on they had a, like a 50 something on full, on Sweebench full, which is the hardest one. And they were rejected because they didn't want to submit their traces.”
Step-by-step prompting on foundational models cannot achieve SOTA SWE-bench performance
“I think it's fairly obvious that with the foundational models, like you can ask them to think by step by step and ask them to plan and stuff, but that isn't enough because if you look at how those models score on these benchmarks and then they're not They're n…”
Training on pull request diffs yields code generators, not software engineers
“What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most par…”
File modification is more fundamental for coding agents than browser access
“At least with what we've seen, the browser is helpful, but it's not as helpful as like writing the correct files. If that makes sense. Like, it is still helpful, but obviously there are more fundamental things you have to get right before you get to like, oh y…”
Cosine's Genie achieved a state-of-the-art 43.8% on SWE-bench Verified
“We got 219 out of 500, which is 43.8%, which is To my knowledge, at least right now, state of the art also”
Cosine's fine-tuned GPT-4o scored higher on SWE-bench than OpenAI's original o1
“And then we also on this podcast, we interviewed Cosign that actually fine tune four O to, on three bench, sorry to achieve data on three bench. And that score was actually higher than O one when it came out.”
Jesse Hu: SWE-bench competitors use identical base model APIs, differentiating via prompting
“Everyone is using the same base models here, or the same model APIs period. I don't know if anyone except for cosine was even training, but like the diff here is all in prompting.”
Standard semantic embedding similarity fails on codebase retrieval tasks
“And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the cosine similarity and get the top of K right. That doesn't work. And I wish it did work. Don't get me wrong. It doesn't work w…”
Generating hypothetical code from queries significantly improves codebase retrieval accuracy
“So what we ended up, the first approach we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take in English code queries and then produce a hypothetical Code snippet that might look like the answer, embed that,…”
Access to GPT-4 Turbo experimental fine-tuning enabled Cosine Genie's creation
“Eventually we were able to get on the experimental access program and we got access to four turbo fine tuning. As soon as we did that, because in the entire run up to that, we'd built the data pipeline. We already had all that set up. So we're like, right, we …”
Upgraded frontier models will automatically improve Cosine's synthetic data flywheel
“When models like that come out, obviously the signal in my data, when I regenerate it goes up. And then I can then train that model that's already better at reasoning with improved reasoning data. And just like, I can keep bootstrapping and keep leapfrogging e…”
Cosine's Genie achieved roughly 66% codebase retrieval accuracy across benchmark tasks
“And I think in our technical report, I can't remember the exact number, but I think it was around 65 or 66% retrieval accuracy overall measured on. We know what lines we need for these tasks to find for the task to actually be able to be completed. And we foun…”
Token log probabilities show models are most certain when writing code
“The certainty of code writing is so much more certain than every other aspect of Genie's loop. So whatever's going on under the hood, the model is really comfortable with writing code. There is no doubt, and it's like in, in the token probabilities.”
Autonomous coding agents require grounding all generated code in context windows
“Fundamentally to build a product like this, you need to get as much information in front of the model as possible and make sure that everything ever writes in output can be
Traced back to something in the context window, so it's not hallucinating it.”
Cosine injects synthetic AST errors into training data to teach error recovery
“And that was in sort of two parts. We synthetically generated runtime errors where we would Intentionally mess with the AST to make stuff not work or index out of bounds or refer to a variable that doesn't exist or errors that the foundational models just make…”
Pullen: Enterprise customers are growing more willing to share proprietary codebases
“Mostly we've actually seen, I'd say people becoming more amenable to the idea over time, actually rather than more skeptical, because I think they can see the upside.”
Cosine's Genie hooks into existing GitHub CI rather than building environments
“The model itself is not in charge of like setting up the code base and running it. So genie sits on top of GitHub. And if you have CI running GitHub, you have GitHub Actions and stuff like that, then Genie essentially makes a call out to that, runs your CI, se…”
Cosine trains Genie to output diffs rather than full file rewrites
“We train Genie to write diffs and, you know, essentially patches, right? Because it's more token efficient”
Cosine built a version of Genie fine-tuned on its own codebase
“We have a version of Genie that is fine-tuned on our code base. So we basically, it's the base Genie, and then we run the same data pipeline that we run on, like, all the stuff that we did to generate the main data set on our repo.”
Cosine's SWE-bench model patch outputs are publicly verifiable on GitHub
“The patches are still public. So like the, dare I say, traditional Sweebench submission, you can go to a GitHub repo and see it and run them for yourself and verify that the numbers come out correctly.”
Cosine will make all Genie plans and generated code patches editable
“So we're going to make everything editable, including the code it writes. Like you can, if it makes a small error in a patch, you can just change it yourself and let it continue and it will be fine. So yeah, like those things are super important. We'll be doin…”
Cosine receives larger OpenAI LoRA adapters than public tiers due to volume
“Actually we use models that are larger than what's publicly available, something publicly available yet, but when this goes out, it will be, but we have larger law adapters available to us, just because the amount of data that we're pumping through it”
Cosine's Genie uses the Perplexity API and URL reading tools
“The genie has both of those tools available to it as well. So yeah, yeah. So we have a tool where you can like put in URLs and it will just read the URLs and you can, it also uses perplexities API under the hood as well to be able to actually ask questions if …”
Cosine will fine-tune and run Genie on Gemini 1.5 once supported
“As soon as you can fine-tune Gemini 1.5, then you best be sure that Genie will have, will work, will run on Gemini 1.5, and, like, we'll probably get very good performance out of that.”
Cosine built code retrieval tooling before LLMs could support autonomous agents
“Well, there are going to be some things that you need to build this when the tech does catch up. So retrieval being one of the most important things, like the model is going to have to be able to like pull code out for code base somehow. So we were like, well,…”
Cosine's early prototype generated entire apps from scratch using OpenAI Codex
“We eventually built the world's most flimsy system, which was back in the day, like 4000 token context windows, like chaining prompts, trying to keep as much context from one to the other, all these different things where essentially you'd put in an app idea i…”