StarCoder

“Like today, Cody uses StarCoder for inline completions, and with the benefit of the context that we provide, we actually show, like, comparable completion acceptance rate metrics. It's kind of like the standard metric that folks use to evaluate inline completi…”

“Stars are not a good predictor of whether data is useful for models or not. Like, I think that's, like, the most popular repos are not necessarily higher quality, at least with respect to do they improve a model's coding capabilities.”

“After just 10,000 steps of training, which this training one took about two hours, this orange curve, you see that it fully matches the original loss.”

“So yeah, but we don't pay a quadratic cost. If you were looking at the star coder baseline, it would get even more, more expensive way more quickly.”

“In the end, this converges to, I believe, about 35% accuracy on human eval, whereas the star coder baseline was about 30%.”