AI industry has completed only 10% of necessary agent evaluation progress
Sherwin Wu · DevDay 2025: Apps SDK, Agent Kit, MCP, Codex and why Prompting is More Important than Ever · Oct 7, 2025 · at 17:44
OpenAI platform lead Sherwin Wu speaks at DevDay 2025 about the development challenges of evaluating long-horizon, multimodal agent trajectories.
0:00 / 0:06exact quote · 6.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I actually think agent evals is still a work in progress. So I think we've, like, made maybe 10% of the progress that we need here.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sherwin Wu
Wu: OpenAI was first to launch stateful responses API
“Obviously we were the first one to launch responses API, but like a couple of other people have kind of adopted, I think Grok has it in their API. I think I saw LMSYS just did something”
OpenAI integrates with OpenRouter for multi-provider evals
“We have a really cool setup with Open Router, where we're working with them, and then you can bring your Open Router setup. And then with that, you can actually, you know, you write your evals using our data sets tool, or use our data set tool to create a bunc…”
Prompt engineering has grown more entrenched despite predictions of its demise
“I feel like two years ago, people were like, oh, at some point, prop, like, prompting's gonna be dead. Like, you know, and it's like, you know... And if anything, it is, like, become more and more entrenched. And I think that, you know, there's this interestin…”
Apple Siri routes requests based on the user's ChatGPT subscription tier
“If you sign into your ChatGPT account the Siri integration will actually use your subscription status to decide what type of model to use when it passes things over to ChatGPT. And so if you're you know just a free user you get, you know, the free model. But i…”
Wu: Cheaper inference does not cut developer spending due to surging demand
“What we realized is as we make it cheaper, you know, the demand for that goes up even more, and you end up, you know, still spending quite a bit”
OpenAI Codex successfully one-shots entire features 30% to 40% of the time
“What a lot of the interns would do is just, like, full YOLO mode, like, trust it to, like, write the whole feature. And it, like, it doesn't work. It, like, doesn't work sometimes. But, like, I don't know, like, 30, 40% of the time it just, like, one-shots it.”