Prediction Not checkable as stated
Pokrass predicts developers will abandon RAG vector stores for direct long-context
“So we do expect a lot of developers to start, you know, uploading their full context more directly to the model. So for smaller tasks, you maybe don't need The whole vector store.”
Insight
Pokrass: AI model gains now driven by post-training, not larger pre-trains
“We find that actually a significant amount of the gains come from new post-training techniques. So I think in the past the narrative is that you need to pre-train these larger and larger models to get better performance, and we're finding that we're able to sq…”
Insight
Pokrass: Prototype with GPT-4.1, then downscale for latency or upscale for reasoning
“I think the answer is always going to be the fastest model that accomplishes your task, right? So maybe you start prompting 4.1 as a starting point if it does your task super well, Then maybe you could drop down a 4.1 mini and save latency, or even nano. Where…”
Opinion
Pokras: Vision fine-tuning is the most underrated release for bespoke OCR
“Vision fine-tuning is so underrated. For the past, like, two months, whenever I talk to founders, they tell me this is the thing they need most. A lot of people are doing, like, OCR on, on very bespoke formats, like government documents, and vision fine-tuning…”
Insight
Pokrass: Every successful company eventually outgrows Postgres for NoSQL
“At some point, every company gets the scale, every successful company gets the scale where Postgres is not cutting it. And then you migrate to some sort of NoSQL database.”
Insight
Function calling benchmarks like BFCL are largely saturated
“I find that a lot of these evals are mostly saturated, like for BFCL. All the models are near, near the top. Already. And kind of the errors are more, I would say like just differences in default behaviors. I think most of the models on the leaderboard can kin…”
Insight
Multi-step agentic apps fail at 95% reliability due to compounded errors
“Like if something is 95% reliable, but you're chaining together a bunch of calls, if you magnify that error rate, it makes your like application not work. So that's a really exciting thing here from going from like 95% to a hundred percent. I'm very biased wor…”
Opinion
Pokrass: LLMs are far more rational 'econs' than humans
“So I think way more than all of us, they are e-cons.”
Assertion Supported
OpenAI launches GPT-4.1 model lineup featuring 1M-token context window
“Yeah, I'll just say we released three new models today, GPT-Fort.one, GPT-Fort.one mini, and GPT-Fort.one data, and the real focus on these were just making the models that were great for developers so we improved instruction following, coding, and shipped our…”
Assertion Open · timeframe Apr 2028
GPT-4.1 Nano and Mini are new pre-trains; base 4.1 is mid-train
“Nano is obviously a new pre-train. We also have a new pre-train for Mini, and then, ah, the larger version is, ah, a new mid-train.”
Insight
Pokrass warns against close collaboration between AI evaluators and model developers
“Honestly, I think it's best when eval authors and model developers don't collab too much because you want things you know, as objective as possible, not trying to game any evals.”
Insight
Open-source AI benchmarks omit critical tasks because they are hard to grade
“And these are useful instructions, but we find that many of the really interesting instructions are actually challenging to grade. And so the open source evals often don't have them.”
Assertion Supported
GPT-4.1 reduces extraneous edit rate to 2%, down from GPT-4o's 9%
“And we found that from four O, which got nine percent, which is pretty crazy, nine percent of the time making an extraneous edit is a lot. 4.1 is at two percent, so it's a pretty big improvement.”
Insight
Pokrass: Use XML for structuring LLM inputs and JSON for parsing outputs
“I do think XML is very helpful for structuring prompts, whereas for parsing outputs maybe the story is a bit different. Like sometimes it's really useful to get outputs in JSON, so you can plug them directly into your application. But I do think the models wor…”
Insight
Pokrass: Pair reasoning models for planning with smaller models for execution
“I do think reasoning models for planning and using kind of more targeted models to execute is definitely a good architecture.”
Insight
GPT-4.1 excels at exploring repositories, while reasoning models dominate targeted file changes
“Basically, where GPT, 4.1, can it kind of explore, go through a repo? It's been trained to do that particularly well. Whereas you know, to just get some code and produce a change, a reasoning model might do better because it can kind of reason over the entire …”
Assertion Not checkable as stated
OpenAI researcher uses GPT-4.1 for 49 of 50 commits on massive PR
“I was actually just talking to one of the researchers on the team who worked on something over the weekend. And he said that this model, GBT, 4.1 was able to like get 49 out of 50 of his commits on this massive PR done.”
Assertion Not checkable as stated
GPT-4.1's multimodal vision improvements stem from pre-training, not post-training
“We talked about like coding instruction following long context, a lot of gains coming from post training, but in particular multimodal, like basically everything you're seeing, the gains are there from pre-training.”
Opinion
Pokrass: Developers are sleeping on preference fine-tuning for model style steering
“One thing I will say is that I think people have slept on the preference fine tuning offering or the, I think that's what we call the product. So SFT is, people know it pretty well. It's the original fine tuning we had, whereas this preference fine tuning is s…”
Assertion Supported
OpenAI increases prompt caching discount from 50% to 75% on GPT-4.1
“We've increased our prompt caching discount from 50% to 75% on these models.”
Disclosure
Pokras: OpenAI will ship raw audio in Chat Completions API
“We're actually going to be shipping audio capabilities in chat completions. So this is like the lowest level capability. So you supply in audio and you can get back raw audio and it works at the request response layer.”
Opinion
Pokras: OpenAI Assistants API requires too many initial API requests
“Some of the things that are good in the assistance API is hosted tools. People really like posted tools and especially RAG. And then some things that are, you know, less intuitive is just how many API requests you need to get going with the Assistant's API.”
Insight
Constrained decoding alone degrades output quality without model training
“And so it's not enough to just kind of constrain the model. I think of that as the engineering side, whereas basically you mask the available tokens that are produced every time to only fit the schema. And so you can do this engineering thing and you can force…”
Assertion Not checkable as stated
OpenAI built its constrained decoding engine from scratch
“Yeah, we didn't use any kind of Other stuff. We kind of built, you know, our solution from scratch to meet our specific needs.”
