SWE Bench Pro
product on 3 shows · 1 statements across 1 episodes · said 14 times in 5 episodes since 2025
Latent Space 12 
the Startup Ideas Podcast 1 
TBPN 1
Mentions by year, every show
tap a year for its mentions
Latent Space 12the Startup Ideas Podcast 1TBPN 1
2026 9 mentions in 3 episodes 3 per episode
every mention on every show, scene by scene, with the transcript →
1 statements about SWE Bench Pro, every show
Watkins: SWE-bench Verified is saturated, contaminated, and should be retired
“SweetBenchVerified has been one of the Northstar coding benchmarks that the field has looked at to measure coding progress. But recently we've seen that progress has kind of stalled and this, we realized that this is because the eval is effectively saturated a…”