Liu: Cody matches GitHub Copilot completion acceptance rates using open-source StarCoder
Beyang Liu · The "Normsky" architecture for AI coding agents — with Beyang Liu + Steve Yegge of SourceGraph · Dec 17, 2023 · at 15:22
Beyang Liu (CTO of Sourcegraph) explains that Cody achieves competitive inline completion acceptance rates against market leader GitHub Copilot using an open-source model paired with context fetching.
0:00 / 0:33exact quote · 33.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Like today, Cody uses StarCoder for inline completions, and with the benefit of the context that we provide, we actually show, like, comparable completion acceptance rate metrics. It's kind of like the standard metric that folks use to evaluate inline completion quality. It's like, if I show you a completion, what's the chance that you actually accept the completion versus you reject it? And so we're at par with Copilot, which is at the head of the industry right now. And we've been able to do that with the StarCoder model, which is open source, and the benefit of the context fetching stuff that we provide.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Beyang Liu
Liu: Pure transformer models are insufficient to support autonomous AI agents
“We're actually a little bit, I think, more bearish than the average, you know, AI hypefluencer out there on the feasibility of agents with purely kind of like transformer-based models.”
Liu: Reliable AI coding workflows require search-based algorithmic backbones
“The way that we get to this, like, more reliable, multi-step workflows that can do things beyond, you know, generate unit test is, is, it's really gonna be, like, a search-based approach, where, where you use an LLM as, kind of, like, an advisor or a proposal …”
Liu: Open-source AI models are currently state-of-the-art for code completion
“Yeah, I mean, for completions, open source is, is state of the art right now.”
Liu: Reliable single-step generation is a strict prerequisite for true AI agents
“If you want to get to the point where you can actually be truly agentic or like multi-step automated a necessary part of that is like the single step has to be robust and reliable.”
Liu: AI coding assistants must pull context beyond Git repositories to succeed
“And I don't think the AI developer will be any different. It will need to pull context from all these different sources.”
Liu: Post-hype reality check will expose the limits of AI search techniques
“And I don't know, I think in the next year or two, maybe as like the, as we get past like the peak AI hype, we'll start to see the gap emerge or become more obvious to more people about like how, how, how many of like the newfangled techniques actually work in…”