Ryan Chi: Direct recursive self-improvement testing is too slow and expensive
Ryan Chi · Inside the Race to Measure Frontier Intelligence · Sep 9, 2026 · at 11:08
Ryan Chi, co-founder of Vals AI, explains why proxy evaluations are necessary for measuring recursive self-improvement in frontier models.
0:00 / 0:09exact quote · 9.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In an ideal world, what you want to do is actually take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ryan Chi
Chi: Meta Llama 4 Underperformed on Private Benchmarks Despite Public Scores
“One of the early indications of that you saw was when Meta released Lama four that was a bit of a disaster, and interestingly, what we saw is that on our held out private benchmarks, the model is actually underperforming, but on all of the major public benchma…”
Chi: AI data vendors create gimmick benchmarks to sell data
“And actually a lot of that industry has now Built these gimmick style benchmarks as a mechanism to sell their data. And so that, that's become kind of their go-to-market as well.”
Chi: Bundling model evaluation with data consulting produces pay-to-win benchmarks
“If you look at auditing as an industry, you end up with issues like Enron, where if you have the same group who's responsible for doing the audit, as well as also consulting and supporting the company, you have a mixed incentive structure, and then it just bec…”
OpenRouter functions predominantly as a gateway rather than an automated router
“Open route is a bit of a misnomer in that most of their usage comes from being a model gateway. And so it's actually up to their users to decide which models they want to use when.”