Cosine will fine-tune and run Genie on Gemini 1.5 once supported
Alistair Pullen · Is finetuning GPT4o worth it? · Aug 22, 2024 · at 30:41
Ali Pullen is the CEO of Cosine, creator of the Genie AI software engineer. He explains Cosine's model-agnostic strategy compared to building custom foundation models.
0:00 / 0:08exact quote · 8.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“As soon as you can fine-tune Gemini 1.5, then you best be sure that Genie will have, will work, will run on Gemini 1.5, and, like, we'll probably get very good performance out of that.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Alistair Pullen
Pullen: CoScene's Genie outperforms OpenAI o1 out of the box on SWE-bench
“So it was obviously great to see, like, we still are better than O-one out of the box. You know, even with an older model, and I'm sure that that, that Delta will continue to grow once we're able to train O-one and once we've done more work on our dataset usin…”
Pullen: SWE-bench is a poor proxy for real-world AI coding competence
“I know Sweebench is, like, the most commonly talked about thing, and honestly, it's a very, it's an amazing project, but one of the things we've learned the most from actually shipping this product to users is, it's a pretty bad proxy at telling us how compete…”
Extracting problem-solving history most determines Cosine Genie's SWE-bench performance
“The way that we Decided we want to try to extract what happened in the past, like as forensically as possible, has been and is currently like one of the main things that we focus all our time on. Because doing that as we're getting as much signal out as possib…”
Cosine refuses to publish SWE-bench trajectories to prevent competitor model distillation
“For the moment, as a closed source company, like fighting for an edge, we've decided not to publish that information for that exact reason. I don't want someone basically taking my tragedies and then taking a model that's suing them in GA and just distilling i…”
Pullen: You cannot build a successful startup solely on the YC advantage
“You can't build a startup on the YC advantage. It's obviously nice and it makes you feel warm and fuzzy inside, but like at the end of the day, it's not that that's going to make you win.”
Pullen: SWE-bench is currently the best metric for software engineering agents
“It was actually a very useful tool in building Genie because beforehand it was like, yes, vibe check this thing and see if it's useful. And then all of a sudden you have an actual measure to see like, couldn't it do software engineering? Not the best measure, …”