Liu: Synthetic data and task-specific fine-tuning provide alpha for code automation
Beyang Liu · The "Normsky" architecture for AI coding agents — with Beyang Liu + Steve Yegge of SourceGraph · Dec 17, 2023 · at 1:11:39
Beyang Liu is the co-founder and CTO of Sourcegraph, discussing data bottlenecks and model training improvements for AI coding agents.
0:00 / 0:33exact quote · 33.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I feel like most models today, they still use, like, combination of, like, the stack and the pile as, like their training corpus but you can only stretch that so far. At some point, we need more data and I don't know. I think there's still more alpha in, like, synthetic data. Like, we have a couple efforts where, like, we think fine-tuning some models on specific coding tasks will yield alpha, will yield more, kind of, like, reliable code generation of the sort where it's like reliable enough that we can fully automate it at least like the one hop thing.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Beyang Liu
Liu: Cody matches GitHub Copilot completion acceptance rates using open-source StarCoder
“Like today, Cody uses StarCoder for inline completions, and with the benefit of the context that we provide, we actually show, like, comparable completion acceptance rate metrics. It's kind of like the standard metric that folks use to evaluate inline completi…”
Liu: Pure transformer models are insufficient to support autonomous AI agents
“We're actually a little bit, I think, more bearish than the average, you know, AI hypefluencer out there on the feasibility of agents with purely kind of like transformer-based models.”
Liu: Reliable AI coding workflows require search-based algorithmic backbones
“The way that we get to this, like, more reliable, multi-step workflows that can do things beyond, you know, generate unit test is, is, it's really gonna be, like, a search-based approach, where, where you use an LLM as, kind of, like, an advisor or a proposal …”
Liu: Open-source AI models are currently state-of-the-art for code completion
“Yeah, I mean, for completions, open source is, is state of the art right now.”