Aug 22, 2024 · 1h 1m · latent-space
Is finetuning GPT4o worth it?
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of Latent Space, Cosine CEO Ali Pullen joins Alessio Fanelli and Swix to break down the technical architecture, synthetic training pipelines, and large-scale OpenAI fine-tuning behind Genie, their state-of-the-art autonomous AI software engineer. The discussion explores Cosine's founding journey, empirical discoveries in context window degradation, and benchmark leadership on SWE-bench.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 16.7% of the talking time here. How this is scored →
The hosts as informed peer 5.5 Guest teaching 5.3 Guest disagreement 1.5 The hosts pushing back 0.5
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 22:27 Rejecting the wrapper framing
Ali firmly rejects Swyx's assertion that he deemed browser tools unimportant, clarifying that he was attacking superficial wrapper architectures rather than the utility of browsing.
Hardest push from the hosts ▶ 22:07 Challenging Genie's dismissal of agent toolingSwyx directly challenges Ali by quoting his claim that competitors are mere wrappers and questioning why Genie downplays tools like browsers.
Biggest teaching moment ▶ 36:15 Context window degradation thresholdAli delivers proprietary empirical data showing that SWE-bench problem resolution degrades linearly, falling to a 50% failure rate past 60k tokens.
The host holds their own ▶ 47:51 Synthesizing Llama 3 back-translation methodsSwyx demonstrates deep domain awareness by connecting Ali's synthetic data methodology to the Llama 3 paper's back-translation findings.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Ali Pullen's Early Background and the Fancy Acquisition | 4 | 3 | 1 | 0 | Swyx introduces Ali and summarizes his resume from Exeter to Fancy/GoPuff. Ali recounts his transition into startups and the acquisition in a narrative, collaborative tone. | |
| Discovering GPT-3 and Early App Generation Prototypes | 5 | 3 | 1 | 0 | Ali and the hosts bond over early GPT-3 playground completions and early Codex prototypes. The dynamic is conversational and nostalgic with shared domain context. | |
| The Y Combinator Experience and the Genesis of Build | 5 | 3 | 1 | 1 | Ali details the brutal YC interview and the early pivot toward codebase retrieval. Swyx teases him about the poor performance and unpronounceable name of 'Build'. | |
| Expanding Context Windows and the Thesis for Fine-Tuning | 5 | 5 | 2 | 1 | Ali explains why 128k context windows unlocked viable software engineering agents via fine-tuning. Swyx observes how OpenAI's fast-tracking of fine-tuning diminishes the YC Slack channel moat. | |
| Partnering with OpenAI and Adopting SWE-bench | 5 | 6 | 1 | 0 | Ali details getting early experimental access to GPT-4 Turbo fine-tuning and diving headfirst into SWE-bench after Devin launched. The hosts listen attentively as Ali breaks down early training setbacks. | |
| Distinguishing Code Generation from True Software Engineering | 6 | 7 | 2 | 0 | Alessio asks how Genie differentiates code generation from software engineering. Ali explains that PR diffs are lossy artifacts, educating the hosts on reconstructing the developer's decision trajectory. | |
| Data Cleansing and Customer Appetite for Enterprise Code Sharing | 6 | 5 | 1 | 0 | Alessio queries customer willingness to share proprietary codebases for model training. Ali explains enterprise appetite for productivity outweighs security resistance in practice. | |
| Genie's Core Architecture vs. Generic Agent Tooling | 6 | 5 | 4 | 3 | Swyx challenges Ali on claiming browser and code interpreter tools are unimportant compared to Genie's core loop. Ali corrects the premise, asserting he derided generic wrapper architecture, not the tools themselves. | |
| Advancing Codebase Retrieval through Self-Play and Language Servers | 6 | 7 | 2 | 1 | Ali explains why standard semantic embeddings fail for code functionality and how training Genie on LSP traversal and self-play boosted retrieval accuracy to 66%. Swyx follows with technical suggestions. | |
| Foundation Model Agnosticism vs. Custom Architectures | 6 | 5 | 2 | 1 | Swyx brings up Magic.dev's LTM approach with long contexts. Ali defends remaining model-agnostic, arguing that fine-tuning data can easily transfer across foundation models like Gemini or Anthropic. | |
| Context Window Degradation and Token Log Probabilities | 6 | 8 | 1 | 0 | Ali shares proprietary insights shared with OpenAI showing that task success drops to 50% past 60k tokens and breaks down token log probabilities. Alessio asks about context scaling factors. | |
| Multi-Language Data Distribution and Model Generalization | 6 | 5 | 1 | 0 | Alessio examines multi-language distribution and enterprise test execution. Ali clarifies that Genie offloads test runs to existing GitHub Actions and CI pipelines rather than managing local repo environments. | |
| Deep Collaboration with OpenAI on LoRA and Adapter Scaling | 5 | 7 | 1 | 0 | Ali describes deep technical collaboration with OpenAI, discussing LoRA adapter capacity, learning rate schedules, and scaling laws when pushing billions of tokens through fine-tuning APIs. | |
| Generating Synthetic Data and Modeling Iterative Error Correction | 7 | 6 | 1 | 0 | Ali explains synthetically corrupting ASTs to teach models iterative error correction. Swyx connects this methodology to Llama 3's back-translation findings in code synthesis. | |
| SWE-bench Trajectory Controversies and IP Protection | 6 | 6 | 3 | 1 | Swyx asks why Genie's score isn't on the official SWE-bench leaderboard. Ali explains his refusal to submit full model trajectories, prioritizing IP protection against distillation over academic listing. | |
| SWE-bench Verified Results and Granular Evaluation Metrics | 5 | 6 | 1 | 0 | Ali discusses achieving 43.8% on SWE-bench Verified and outlines plans to move beyond binary pass/fail evals toward step-level divergence metrics. | |
| Future Horizons: Scaling Data and Repo-Specific Fine-Tuning | 5 | 4 | 1 | 0 | Swyx and Ali conclude by discussing fine-tuning models on specific customer repositories, Cursor's UX excellence, hiring needs, and Ali's viral launch video bloopers. |
Statements from this episode (33)
Cosine's early prototype generated entire apps from scratch using OpenAI Codex
“We eventually built the world's most flimsy system, which was back in the day, like 4000 token context windows, like chaining prompts, trying to keep as much context from one to the other, all these different things where essentially you'd put in an app idea i…”
Cosine built code retrieval tooling before LLMs could support autonomous agents
“Well, there are going to be some things that you need to build this when the tech does catch up. So retrieval being one of the most important things, like the model is going to have to be able to like pull code out for code base somehow. So we were like, well,…”
Autonomous coding agents require grounding all generated code in context windows
“Fundamentally to build a product like this, you need to get as much information in front of the model as possible and make sure that everything ever writes in output can be
Traced back to something in the context window, so it's not hallucinating it.”
Y Combinator companies had a direct Slack channel to OpenAI
“Which is a little bit, you know, for a year or so, YC companies had like a direct Slack channel to open AI.”
Pullen: You cannot build a successful startup solely on the YC advantage
“You can't build a startup on the YC advantage. It's obviously nice and it makes you feel warm and fuzzy inside, but like at the end of the day, it's not that that's going to make you win.”
Access to GPT-4 Turbo experimental fine-tuning enabled Cosine Genie's creation
“Eventually we were able to get on the experimental access program and we got access to four turbo fine tuning. As soon as we did that, because in the entire run up to that, we'd built the data pipeline. We already had all that set up. So we're like, right, we …”
Pullen: SWE-bench is currently the best metric for software engineering agents
“It was actually a very useful tool in building Genie because beforehand it was like, yes, vibe check this thing and see if it's useful. And then all of a sudden you have an actual measure to see like, couldn't it do software engineering? Not the best measure, …”
Training on pull request diffs yields code generators, not software engineers
“What the, very crudely, what the pre-trained models are reading is they're reading those final diffs and they're Emulating that and then being able to output it. Right. But of course it's a super lossy thing, a PR. You have no idea why or how, for the most par…”
Extracting problem-solving history most determines Cosine Genie's SWE-bench performance
“The way that we Decided we want to try to extract what happened in the past, like as forensically as possible, has been and is currently like one of the main things that we focus all our time on. Because doing that as we're getting as much signal out as possib…”
Pullen: Most scraped open-source data consists of README and documentation updates
“When you scrape enough of it, most of open source is updating readmes and docs.”
Pullen: Enterprise customers are growing more willing to share proprietary codebases
“Mostly we've actually seen, I'd say people becoming more amenable to the idea over time, actually rather than more skeptical, because I think they can see the upside.”
File modification is more fundamental for coding agents than browser access
“At least with what we've seen, the browser is helpful, but it's not as helpful as like writing the correct files. If that makes sense. Like, it is still helpful, but obviously there are more fundamental things you have to get right before you get to like, oh y…”
Cosine's Genie uses the Perplexity API and URL reading tools
“The genie has both of those tools available to it as well. So yeah, yeah. So we have a tool where you can like put in URLs and it will just read the URLs and you can, it also uses perplexities API under the hood as well to be able to actually ask questions if …”
Standard semantic embedding similarity fails on codebase retrieval tasks
“And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the cosine similarity and get the top of K right. That doesn't work. And I wish it did work. Don't get me wrong. It doesn't work w…”
Generating hypothetical code from queries significantly improves codebase retrieval accuracy
“So what we ended up, the first approach we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take in English code queries and then produce a hypothetical Code snippet that might look like the answer, embed that,…”
Cosine's Genie achieved roughly 66% codebase retrieval accuracy across benchmark tasks
“And I think in our technical report, I can't remember the exact number, but I think it was around 65 or 66% retrieval accuracy overall measured on. We know what lines we need for these tasks to find for the task to actually be able to be completed. And we foun…”
Cosine will fine-tune and run Genie on Gemini 1.5 once supported
“As soon as you can fine-tune Gemini 1.5, then you best be sure that Genie will have, will work, will run on Gemini 1.5, and, like, we'll probably get very good performance out of that.”
Anthropic model fine-tuning will be offered through AWS Bedrock
“They are partnered with AWS, and it's going to be in bedrock. As far as I know, I think that's true.”
Step-by-step prompting on foundational models cannot achieve SOTA SWE-bench performance
“I think it's fairly obvious that with the foundational models, like you can ask them to think by step by step and ask them to plan and stuff, but that isn't enough because if you look at how those models score on these benchmarks and then they're not They're n…”
Upgraded frontier models will automatically improve Cosine's synthetic data flywheel
“When models like that come out, obviously the signal in my data, when I regenerate it goes up. And then I can then train that model that's already better at reasoning with improved reasoning data. And just like, I can keep bootstrapping and keep leapfrogging e…”
Cosine will make all Genie plans and generated code patches editable
“So we're going to make everything editable, including the code it writes. Like you can, if it makes a small error in a patch, you can just change it yourself and let it continue and it will be fine. So yeah, like those things are super important. We'll be doin…”
Token log probabilities show models are most certain when writing code
“The certainty of code writing is so much more certain than every other aspect of Genie's loop. So whatever's going on under the hood, the model is really comfortable with writing code. There is no doubt, and it's like in, in the token probabilities.”
Cosine trains Genie to output diffs rather than full file rewrites
“We train Genie to write diffs and, you know, essentially patches, right? Because it's more token efficient”
Genie's SWE-bench success rate drops to roughly 50% past 60k tokens
“Performance of Jeannie over the length of the context window degrades fairly linearly. So actually, I actually broke it down by probability of solving a SWE bench issue. Given the number of tokens of the context window at 60 K, it's basically .5. So if you go …”
Cosine's Genie hooks into existing GitHub CI rather than building environments
“The model itself is not in charge of like setting up the code base and running it. So genie sits on top of GitHub. And if you have CI running GitHub, you have GitHub Actions and stuff like that, then Genie essentially makes a call out to that, runs your CI, se…”
Cosine receives larger OpenAI LoRA adapters than public tiers due to volume
“Actually we use models that are larger than what's publicly available, something publicly available yet, but when this goes out, it will be, but we have larger law adapters available to us, just because the amount of data that we're pumping through it”
Cosine injects synthetic AST errors into training data to teach error recovery
“And that was in sort of two parts. We synthetically generated runtime errors where we would Intentionally mess with the AST to make stuff not work or index out of bounds or refer to a variable that doesn't exist or errors that the foundational models just make…”
Wang: Natural language-to-code translation is ripe for synthetic data generation
“I think that translation between natural language, English versus code and back and forth, I think is actually actually a really ripe source of synthetic data and Lama three specifically called out that, that they trained on that.”
Cosine refuses to publish SWE-bench trajectories to prevent competitor model distillation
“For the moment, as a closed source company, like fighting for an edge, we've decided not to publish that information for that exact reason. I don't want someone basically taking my tragedies and then taking a model that's suing them in GA and just distilling i…”
Cosine's SWE-bench model patch outputs are publicly verifiable on GitHub
“The patches are still public. So like the, dare I say, traditional Sweebench submission, you can go to a GitHub repo and see it and run them for yourself and verify that the numbers come out correctly.”
Cosine's Genie achieved a state-of-the-art 43.8% on SWE-bench Verified
“We got 219 out of 500, which is 43.8%, which is To my knowledge, at least right now, state of the art also”
Pass-fail metrics like SWE-bench are too sparse for evaluating AI agents
“Sweet Bench is a pass or fail, right? Like you either have solved the problem or you haven't. That is quite sparse. Like it doesn't give you a huge amount of information because your model could have got a lot of it right. Like looking through when you do a ma…”