The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 34 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 1 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Pullen: CoScene's Genie outperforms OpenAI o1 out of the box on SWE-bench
“So it was obviously great to see, like, we still are better than O-one out of the box. You know, even with an older model, and I'm sure that that, that Delta will continue to grow once we're able to train O-one and once we've done more work on our dataset usin…”
Pullen: SWE-bench is a poor proxy for real-world AI coding competence
“I know Sweebench is, like, the most commonly talked about thing, and honestly, it's a very, it's an amazing project, but one of the things we've learned the most from actually shipping this product to users is, it's a pretty bad proxy at telling us how compete…”
Extracting problem-solving history most determines Cosine Genie's SWE-bench performance
“The way that we Decided we want to try to extract what happened in the past, like as forensically as possible, has been and is currently like one of the main things that we focus all our time on. Because doing that as we're getting as much signal out as possib…”
Cosine refuses to publish SWE-bench trajectories to prevent competitor model distillation
“For the moment, as a closed source company, like fighting for an edge, we've decided not to publish that information for that exact reason. I don't want someone basically taking my tragedies and then taking a model that's suing them in GA and just distilling i…”
Pullen: You cannot build a successful startup solely on the YC advantage
“You can't build a startup on the YC advantage. It's obviously nice and it makes you feel warm and fuzzy inside, but like at the end of the day, it's not that that's going to make you win.”
Pullen: SWE-bench is currently the best metric for software engineering agents
“It was actually a very useful tool in building Genie because beforehand it was like, yes, vibe check this thing and see if it's useful. And then all of a sudden you have an actual measure to see like, couldn't it do software engineering? Not the best measure, …”
Training on pull request diffs yields code generators, not software engineers
“What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most par…”
File modification is more fundamental for coding agents than browser access
“At least with what we've seen, the browser is helpful, but it's not as helpful as like writing the correct files. If that makes sense. Like, it is still helpful, but obviously there are more fundamental things you have to get right before you get to like, oh y…”
Step-by-step prompting on foundational models cannot achieve SOTA SWE-bench performance
“I think it's fairly obvious that with the foundational models, like you can ask them to think by step by step and ask them to plan and stuff, but that isn't enough because if you look at how those models score on these benchmarks and then they're not They're n…”
Genie's SWE-bench success rate drops to roughly 50% past 60k tokens
“Performance of Jeannie over the length of the context window degrades fairly linearly. So actually, I actually broke it down by probability of solving a SWE bench issue. Given the number of tokens of the context window at 60 K, it's basically .5. So if you go …”
Cosine's Genie achieved a state-of-the-art 43.8% on SWE-bench Verified
“We got 219 out of 500, which is 43.8%, which is To my knowledge, at least right now, state of the art also”
Pullen: Replacing Genie's reasoning traces with o1 traces improves performance
“Even now we've started Replacing some of the reasoning traces in our Genie model with reasoning traces generated by O-one, or at least in tandem with O-one, and we've already started seeing improvements in performance from that point.”
Autonomous coding agents require grounding all generated code in context windows
“Fundamentally to build a product like this, you need to get as much information in front of the model as possible and make sure that everything ever writes in output can be
Traced back to something in the context window, so it's not hallucinating it.”
Access to GPT-4 Turbo experimental fine-tuning enabled Cosine Genie's creation
“Eventually we were able to get on the experimental access program and we got access to four turbo fine tuning. As soon as we did that, because in the entire run up to that, we'd built the data pipeline. We already had all that set up. So we're like, right, we …”
Pullen: Most scraped open-source data consists of README and documentation updates
“When you scrape enough of it, most of open source is updating readmes and docs.”
Pullen: Enterprise customers are growing more willing to share proprietary codebases
“Mostly we've actually seen, I'd say people becoming more amenable to the idea over time, actually rather than more skeptical, because I think they can see the upside.”
Standard semantic embedding similarity fails on codebase retrieval tasks
“And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the cosine similarity and get the top of K right. That doesn't work. And I wish it did work. Don't get me wrong. It doesn't work w…”
Generating hypothetical code from queries significantly improves codebase retrieval accuracy
“So what we ended up, the first approach we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take in English code queries and then produce a hypothetical Code snippet that might look like the answer, embed that,…”
Cosine's Genie achieved roughly 66% codebase retrieval accuracy across benchmark tasks
“And I think in our technical report, I can't remember the exact number, but I think it was around 65 or 66% retrieval accuracy overall measured on. We know what lines we need for these tasks to find for the task to actually be able to be completed. And we foun…”
Anthropic model fine-tuning will be offered through AWS Bedrock
“They are partnered with AWS, and it's going to be in bedrock. As far as I know, I think that's true.”
Upgraded frontier models will automatically improve Cosine's synthetic data flywheel
“When models like that come out, obviously the signal in my data, when I regenerate it goes up. And then I can then train that model that's already better at reasoning with improved reasoning data. And just like, I can keep bootstrapping and keep leapfrogging e…”
Token log probabilities show models are most certain when writing code
“The certainty of code writing is so much more certain than every other aspect of Genie's loop. So whatever's going on under the hood, the model is really comfortable with writing code. There is no doubt, and it's like in, in the token probabilities.”
Cosine's Genie hooks into existing GitHub CI rather than building environments
“The model itself is not in charge of like setting up the code base and running it. So genie sits on top of GitHub. And if you have CI running GitHub, you have GitHub Actions and stuff like that, then Genie essentially makes a call out to that, runs your CI, se…”
Cosine injects synthetic AST errors into training data to teach error recovery
“And that was in sort of two parts. We synthetically generated runtime errors where we would Intentionally mess with the AST to make stuff not work or index out of bounds or refer to a variable that doesn't exist or errors that the foundational models just make…”
Pass-fail metrics like SWE-bench are too sparse for evaluating AI agents
“Sweet Bench is a pass or fail, right? Like you either have solved the problem or you haven't. That is quite sparse. Like it doesn't give you a huge amount of information because your model could have got a lot of it right. Like looking through when you do a ma…”
Cosine built code retrieval tooling before LLMs could support autonomous agents
“Well, there are going to be some things that you need to build this when the tech does catch up. So retrieval being one of the most important things, like the model is going to have to be able to like pull code out for code base somehow. So we were like, well,…”
Cosine's Genie uses the Perplexity API and URL reading tools
“The genie has both of those tools available to it as well. So yeah, yeah. So we have a tool where you can like put in URLs and it will just read the URLs and you can, it also uses perplexities API under the hood as well to be able to actually ask questions if …”
Cosine will fine-tune and run Genie on Gemini 1.5 once supported
“As soon as you can fine-tune Gemini 1.5, then you best be sure that Genie will have, will work, will run on Gemini 1.5, and, like, we'll probably get very good performance out of that.”
Cosine will make all Genie plans and generated code patches editable
“So we're going to make everything editable, including the code it writes. Like you can, if it makes a small error in a patch, you can just change it yourself and let it continue and it will be fine. So yeah, like those things are super important. We'll be doin…”
Cosine trains Genie to output diffs rather than full file rewrites
“We train Genie to write diffs and, you know, essentially patches, right? Because it's more token efficient”
Cosine receives larger OpenAI LoRA adapters than public tiers due to volume
“Actually we use models that are larger than what's publicly available, something publicly available yet, but when this goes out, it will be, but we have larger law adapters available to us, just because the amount of data that we're pumping through it”
Cosine's SWE-bench model patch outputs are publicly verifiable on GitHub
“The patches are still public. So like the, dare I say, traditional Sweebench submission, you can go to a GitHub repo and see it and run them for yourself and verify that the numbers come out correctly.”
Cosine built a version of Genie fine-tuned on its own codebase
“We have a version of Genie that is fine-tuned on our code base. So we basically, it's the base Genie, and then we run the same data pipeline that we run on, like, all the stuff that we did to generate the main data set on our repo.”
Cosine's early prototype generated entire apps from scratch using OpenAI Codex
“We eventually built the world's most flimsy system, which was back in the day, like 4000 token context windows, like chaining prompts, trying to keep as much context from one to the other, all these different things where essentially you'd put in an app idea i…”