Disclosure
GitHub CPO: Next-Gen Copilot Aims to Turn Non-Programmers Into Creators
“What I'm really excited about the next generation of Copaldo is transforming into this product that Your grandmother, your child, or you yourself, even if you're not a programmer, can actually interact and create something overall.”
Opinion
GitHub CPO: 95% Offline Evaluation Scores Still Yield Bad AI Products
“Even if you get an offline 95, by the way, usually that means the product, when it gets to market, it's a really bad product, at least in AI world, in my opinion.”
Opinion
GitHub CPO: Every Existing Public AI Benchmark Can Be Gamed
“I don't like benchmarks out there by the way, because you could game every single one of them. In my opinion, but what I do like about benchmarking is that it gives you a view into a set of scenarios that then you could then figure out, are you getting better …”
Disclosure
GitHub CPO: Copilot Development Is Decentralized Across All Product Teams
“What I would say is we try to not centralize it. There shouldn't be just one co-pilot team. There's a co-pilot intelligence platform team, and that's true, but I would say the repos team is a co-pilot team. The PR team is a co-pilot team. The issues team is a …”
Prediction Not checkable as stated
GitHub CPO: Platform Will Become Completely AI-Native in Five Years
“What I would say is it looks more AI native. One of the beauties of what we're living through right now is that these LLMs can actually understand natural language in a way that none of the other tools before could. So I think if we really lean into that and g…”
Insight
GitHub CPO: Product Predictability Is Impossible With Unpredictable LLMs
“In AI, you cannot have predictability. LLMs are not predictable even on that.”
Insight
GitHub CPO: AI Experimentation Is Useless Without Over-Investing in Offline Evals
“At GitHub we invest in this thing we call coffee, which is compiler offline evaluation, and that's one area that I have found to be a place that you have to over invest in AI if you will, if you need to experiment, because you could experiment all day in AI an…”
Insight
GitHub CPO: AI Products Require AI Engineers Alongside Applied Scientists
“The right thing is not to say you need a bunch of applied science. I think you also need to grow the company to be AI engineers plus applied science. And that's what at the end might create the right product overall. Instead of just investing in the deep tech.”
Assertion Not checkable as stated
GitHub CPO: Copilot Engineered for 100-150ms Global Latency
“We worked out really hard in making the latency be a hundred milliseconds, a 150 milliseconds, no matter where you were in the world.”
