OpenAI will cover inference costs for developers who share custom evaluation data
Michelle Pokrass · GPT 4.1: The New OpenAI Workhorse · Apr 15, 2025 · at 42:04
OpenAI researcher Michelle Pokrass discusses ways developers can collaborate with OpenAI and optimize costs by contributing evaluation datasets to their evals platform.
0:00 / 0:06exact quote · 6.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“You can upload an eval and opt in such that we'll pay for the inference, inference costs if we can also use the eval.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Michelle Pokrass
Pokrass predicts developers will abandon RAG vector stores for direct long-context
“So we do expect a lot of developers to start, you know, uploading their full context more directly to the model. So for smaller tasks, you maybe don't need The whole vector store.”
Pokrass: AI model gains now driven by post-training, not larger pre-trains
“We find that actually a significant amount of the gains come from new post-training techniques. So I think in the past the narrative is that you need to pre-train these larger and larger models to get better performance, and we're finding that we're able to sq…”
Pokrass: Prototype with GPT-4.1, then downscale for latency or upscale for reasoning
“I think the answer is always going to be the fastest model that accomplishes your task, right? So maybe you start prompting 4.1 as a starting point if it does your task super well, Then maybe you could drop down a 4.1 mini and save latency, or even nano. Where…”
Pokras: Vision fine-tuning is the most underrated release for bespoke OCR
“Vision fine-tuning is so underrated. For the past, like, two months, whenever I talk to founders, they tell me this is the thing they need most. A lot of people are doing, like, OCR on, on very bespoke formats, like government documents, and vision fine-tuning…”
Pokrass: Every successful company eventually outgrows Postgres for NoSQL
“At some point, every company gets the scale, every successful company gets the scale where Postgres is not cutting it. And then you migrate to some sort of NoSQL database.”
Function calling benchmarks like BFCL are largely saturated
“I find that a lot of these evals are mostly saturated, like for BFCL. All the models are near, near the top. Already. And kind of the errors are more, I would say like just differences in default behaviors. I think most of the models on the leaderboard can kin…”