GitHub CPO: 95% Offline Evaluation Scores Still Yield Bad AI Products
Mario Rodriguez · Adding AI to SaaS: Inside the AI Product Strategies of Figma, Cloudflare, GitHub and Ramp · Jan 10, 2025 · at 26:57
Mario Rodriguez, Chief Product Officer at GitHub, highlights why offline AI evaluation scores alone do not ensure a successful user product experience.
0:00 / 0:08exact quote · 8.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Even if you get an offline 95, by the way, usually that means the product, when it gets to market, it's a really bad product, at least in AI world, in my opinion.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mario Rodriguez
GitHub CPO: Next-Gen Copilot Aims to Turn Non-Programmers Into Creators
“What I'm really excited about the next generation of Copaldo is transforming into this product that Your grandmother, your child, or you yourself, even if you're not a programmer, can actually interact and create something overall.”
GitHub CPO: Every Existing Public AI Benchmark Can Be Gamed
“I don't like benchmarks out there by the way, because you could game every single one of them. In my opinion, but what I do like about benchmarking is that it gives you a view into a set of scenarios that then you could then figure out, are you getting better …”
GitHub CPO: Copilot Development Is Decentralized Across All Product Teams
“What I would say is we try to not centralize it. There shouldn't be just one co-pilot team. There's a co-pilot intelligence platform team, and that's true, but I would say the repos team is a co-pilot team. The PR team is a co-pilot team. The issues team is a …”
GitHub CPO: Platform Will Become Completely AI-Native in Five Years
“What I would say is it looks more AI native. One of the beauties of what we're living through right now is that these LLMs can actually understand natural language in a way that none of the other tools before could. So I think if we really lean into that and g…”
GitHub CPO: Product Predictability Is Impossible With Unpredictable LLMs
“In AI, you cannot have predictability. LLMs are not predictable even on that.”
GitHub CPO: AI Experimentation Is Useless Without Over-Investing in Offline Evals
“At GitHub we invest in this thing we call coffee, which is compiler offline evaluation, and that's one area that I have found to be a place that you have to over invest in AI if you will, if you need to experiment, because you could experiment all day in AI an…”