The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 58 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 2 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Pokrass predicts developers will abandon RAG vector stores for direct long-context
“So we do expect a lot of developers to start, you know, uploading their full context more directly to the model. So for smaller tasks, you maybe don't need The whole vector store.”
Pokrass: AI model gains now driven by post-training, not larger pre-trains
“We find that actually a significant amount of the gains come from new post-training techniques. So I think in the past the narrative is that you need to pre-train these larger and larger models to get better performance, and we're finding that we're able to sq…”
Pokrass: Prototype with GPT-4.1, then downscale for latency or upscale for reasoning
“I think the answer is always going to be the fastest model that accomplishes your task, right? So maybe you start prompting 4.1 as a starting point if it does your task super well, Then maybe you could drop down a 4.1 mini and save latency, or even nano. Where…”
Pokras: Vision fine-tuning is the most underrated release for bespoke OCR
“Vision fine-tuning is so underrated. For the past, like, two months, whenever I talk to founders, they tell me this is the thing they need most. A lot of people are doing, like, OCR on, on very bespoke formats, like government documents, and vision fine-tuning…”
Pokrass: Every successful company eventually outgrows Postgres for NoSQL
“At some point, every company gets the scale, every successful company gets the scale where Postgres is not cutting it. And then you migrate to some sort of NoSQL database.”
Function calling benchmarks like BFCL are largely saturated
“I find that a lot of these evals are mostly saturated, like for BFCL. All the models are near, near the top. Already. And kind of the errors are more, I would say like just differences in default behaviors. I think most of the models on the leaderboard can kin…”
Multi-step agentic apps fail at 95% reliability due to compounded errors
“Like if something is 95% reliable, but you're chaining together a bunch of calls, if you magnify that error rate, it makes your like application not work. So that's a really exciting thing here from going from like 95% to a hundred percent. I'm very biased wor…”
Pokrass: LLMs are far more rational 'econs' than humans
“So I think way more than all of us, they are e-cons.”
OpenAI launches GPT-4.1 model lineup featuring 1M-token context window
“Yeah, I'll just say we released three new models today, GPT-Fort.one, GPT-Fort.one mini, and GPT-Fort.one data, and the real focus on these were just making the models that were great for developers so we improved instruction following, coding, and shipped our…”
GPT-4.1 Nano and Mini are new pre-trains; base 4.1 is mid-train
“Nano is obviously a new pre-train. We also have a new pre-train for Mini, and then, ah, the larger version is, ah, a new mid-train.”
Pokrass warns against close collaboration between AI evaluators and model developers
“Honestly, I think it's best when eval authors and model developers don't collab too much because you want things you know, as objective as possible, not trying to game any evals.”
Open-source AI benchmarks omit critical tasks because they are hard to grade
“And these are useful instructions, but we find that many of the really interesting instructions are actually challenging to grade. And so the open source evals often don't have them.”
GPT-4.1 reduces extraneous edit rate to 2%, down from GPT-4o's 9%
“And we found that from four O, which got nine percent, which is pretty crazy, nine percent of the time making an extraneous edit is a lot. 4.1 is at two percent, so it's a pretty big improvement.”
Pokrass: Use XML for structuring LLM inputs and JSON for parsing outputs
“I do think XML is very helpful for structuring prompts, whereas for parsing outputs maybe the story is a bit different. Like sometimes it's really useful to get outputs in JSON, so you can plug them directly into your application. But I do think the models wor…”
Pokrass: Pair reasoning models for planning with smaller models for execution
“I do think reasoning models for planning and using kind of more targeted models to execute is definitely a good architecture.”
GPT-4.1 excels at exploring repositories, while reasoning models dominate targeted file changes
“Basically, where GPT, 4.1, can it kind of explore, go through a repo? It's been trained to do that particularly well. Whereas you know, to just get some code and produce a change, a reasoning model might do better because it can kind of reason over the entire …”
OpenAI researcher uses GPT-4.1 for 49 of 50 commits on massive PR
“I was actually just talking to one of the researchers on the team who worked on something over the weekend. And he said that this model, GBT, 4.1 was able to like get 49 out of 50 of his commits on this massive PR done.”
GPT-4.1's multimodal vision improvements stem from pre-training, not post-training
“We talked about like coding instruction following long context, a lot of gains coming from post training, but in particular multimodal, like basically everything you're seeing, the gains are there from pre-training.”
Pokrass: Developers are sleeping on preference fine-tuning for model style steering
“One thing I will say is that I think people have slept on the preference fine tuning offering or the, I think that's what we call the product. So SFT is, people know it pretty well. It's the original fine tuning we had, whereas this preference fine tuning is s…”
OpenAI increases prompt caching discount from 50% to 75% on GPT-4.1
“We've increased our prompt caching discount from 50% to 75% on these models.”
Pokras: OpenAI will ship raw audio in Chat Completions API
“We're actually going to be shipping audio capabilities in chat completions. So this is like the lowest level capability. So you supply in audio and you can get back raw audio and it works at the request response layer.”
Pokras: OpenAI Assistants API requires too many initial API requests
“Some of the things that are good in the assistance API is hosted tools. People really like posted tools and especially RAG. And then some things that are, you know, less intuitive is just how many API requests you need to get going with the Assistant's API.”
Constrained decoding alone degrades output quality without model training
“And so it's not enough to just kind of constrain the model. I think of that as the engineering side, whereas basically you mask the available tokens that are produced every time to only fit the schema. And so you can do this engineering thing and you can force…”
OpenAI built its constrained decoding engine from scratch
“Yeah, we didn't use any kind of Other stuff. We kind of built, you know, our solution from scratch to meet our specific needs.”
Standard HTTP status codes are limiting for AI model failures
“I think the HTTP spec is a little bit limiting for AI in a lot of ways. Like there are things that are in between your fault and my fault. There's kind of like the model's fault and there's no, you know, error code for that.”
OpenAI rejected pre-registering schema IDs to avoid developer complexity
“The alternative design space that we explored It's like pre-registering your schema, so like a totally different endpoint, and then passing in like a schema ID. But we thought, you know, that was a lot of overhead, and like another endpoint to maintain, and ju…”
Structured response format is limited to GPT-4o and GPT-4o mini
“Actually, the new response format is only available on two models. It's Foro Mini and the new Foro. So the old Foro doesn't have the new response format. However, for function calling, we were able to enable it for all models that support function calling, and…”
OpenAI Structured Outputs enforces schemas in one shot without retries
“We are not retrying, you know, we're doing it in one shot and this is how you save on latency and cost.”
Fine-tuning requires only 100 to 1,000 high-quality examples
“It's actually a lot easier to get started than a lot of people expect. I think they might need Tens of thousands of examples, but even a hundred really high quality ones or a thousand is enough to get going.”
OpenAI's API is its broadest vehicle for distributing AGI
“So I believe that the API is kind of our broadest vehicle for distributing AGI. You know, we're building some first party products, but they'll never reach every niche in the world and kind of every corner in community.”
OpenAI's seed parameter is best-effort and not fully deterministic
“Yeah, the seed parameter is not fully deterministic, and it's kind of a best effort thing. So you'll notice there's more determinism in the first few tokens. That's kind of the current implementation.”
OpenAI aims to become an AI development platform, not just LLM-as-a-service
“We want to do more in this space and not just be an LLM as a service, but kind of AI development platform as a service.”
Whisper v2 outperforms v3 at certain tasks, delaying its API rollout
“And so whisper V two is better at some things than whisper V three. And so it didn't seem that worthwhile to ship whisper V three compared to like the other things in our priorities. I think we still will at some point, but yeah, it's just, you know, there's a…”
Standard request-response APIs will not work for speech-to-speech models
“I think just the regular request response probably isn't going to be the right solution.”
OpenAI hires applied engineers without prior AI experience
“We've hired people with all kinds of backgrounds, people who have PhD in an ML or folks who have just done engineering like me, and we're really hiring for a lot of teams. We're hiring across the applied org, which is where I sit for engineering, and for a lot…”
OpenAI stealth-tested GPT-4.1 models on OpenRouter before official release
“Yeah yeah, we really wanted to get as much developer feedback as possible on this model to make sure it worked well in the real world, and so we tested it kind of through Open Router and it was super cool to see people latch on to the names and get the theorie…”
OpenAI has no current plans to add GPT-4.1 to Realtime API
“I don't think we don't have any current plans to release 4.1 in the real time API, but you know, things, things may change.”
GPT-4.1 powers OpenAI API while enhanced memory remains ChatGPT-exclusive
“So, 4.1 is powering the API, whereas the enhanced memory is, is ChatGPT only.”
OpenAI uses its own models to categorize anonymized developer prompts
“Well, I will say we do use our own products internally where we can, and so we're not manually by hand reading every prompt. After they're like anonymized, we scrub them with any identifying data, then we use our models to take passes to categorize them.”
GPT-4.1 significantly improves chain-of-thought planning over previous non-reasoning models
“We have found that 4.1 is a lot better at doing planning and thinking through its steps in COT when prompted than our previous non-reasoning models.”
GPT-4.1 Mini significantly outperforms 4o Mini, nearing original GPT-4o performance
“4.1 mini is actually quite significantly better than four o mini and not that far away from the old four o.”
OpenAI currently restricts reinforcement fine-tuning exclusively to its reasoning models
“No, that's reinforcement fine tuning is only for reasoning models.”
OpenAI is working to inject GPT-4.5's humor and nuance into future models
“We're working on incorporating kind of those improvements into the models more generally. People loved about 4.5 is like the humor, the green text, the nuance. So we've heard that feedback and I know, yeah, there's lots of folks working on that and trying to b…”
OpenAI will cover inference costs for developers who share custom evaluation data
“You can upload an eval and opt in such that we'll pay for the inference, inference costs if we can also use the eval.”
OpenAI's API team had just five engineers before ChatGPT launched
“I would say the applied team was maybe like 30 or 40 people, and yeah, probably closer to 30, and there was maybe like five-ish total working on the API at most.”
OpenAI's Applied team now exceeds its entire pre-ChatGPT headcount
“Applied now is bigger than the company when I joined.”
ChatGPT's launch caused Postgres bottlenecks by reusing API developer accounts
“Surprisingly there were a lot of Postgres issues when ChatGPT came out because the accounts for like ChatGPT were tied to the accounts in the API. And so you're basically creating a developer account to log into ChatGPT at the time, because it's just what we h…”