Sep 17, 2024 · 1h 12m · latent-space
Building AGI with OpenAI's Structured Outputs API
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
OpenAI Tech Lead Michelle Pokrass sits down with Alessio Fanelli and Shawn Wang to provide an in-depth technical look at the engineering, constrained decoding mechanics, and developer best practices behind Structured Outputs. She also shares lessons from scaling AI platform infrastructure, expanding multimodal APIs, and building resilient agentic systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 36.7% of the talking time here. How this is scored →
The hosts as informed peer 5.9 Guest teaching 4.9 Guest disagreement 2.1 The hosts pushing back 3.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 13:20 Clear rejection of equating function calling with structured outputs
Michelle firmly rejects Swyx's suggestion that function calling and structured outputs are synonymous, clearly demarcating programmatic action taking from structured user-facing generation.
Hardest push from the hosts ▶ 19:11 Swyx challenging the refusal field design instead of HTTP error codesSwyx pushes back against returning model refusals in the response body payload rather than conventional HTTP error codes, arguing that developers already have established error-handling pipelines.
Biggest teaching moment ▶ 11:27 Explaining why token masking alone causes infinite whitespace loopsMichelle explains that engineering-level constrained decoding without model post-training leads to degenerative whitespace looping, educating the hosts on the necessity of co-training models for structured outputs.
The host holds their own ▶ 37:42 Swyx providing concrete production cost reduction dataSwyx demonstrates deep domain expertise by citing exact production figures from his application, revealing that ripping out third-party retry wrappers saved 55% on API costs.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Michelle Pokrass's Background and Scaling Engineering Infrastructure | 6 | 2 | 1 | 2 | The hosts engage with Michelle's early career at Waterloo, Coinbase, and Clubhouse, sharing technical domain knowledge about derivative trading, Postgres scaling bottlenecks, and DynamoDB architecture. The tone is highly collegial and conversational. | |
| Joining OpenAI and Scaling Early API Infrastructure | 5 | 3 | 1 | 2 | Swyx and Alessio ask about joining OpenAI pre-ChatGPT and early infrastructure scaling challenges. Michelle shares operational insights about authorization systems and GPU allocation during the ChatGPT launch surge. | |
| The Evolution and Architecture of Structured Outputs | 6 | 6 | 2 | 2 | Michelle explains the technical difference between JSON mode token masking and training models specifically for constrained decoding to prevent whitespace loops. Alessio compares this to pre-fills and instructor patterns. | |
| Distinguishing Function Calling from Structured Outputs | 5 | 7 | 4 | 3 | Swyx asks whether function calling and structured outputs should be considered synonymous. Michelle firmly clarifies the architectural and semantic difference between programmatic tool calling and structured conversational responses. | |
| Constrained Decoding Grammars and API Safety Refusals | 6 | 6 | 3 | 5 | Swyx and Michelle debate the design choice of returning safety refusals inside the JSON response body rather than throwing standard HTTP 4xx or 5xx status codes. Michelle explains why HTTP semantics break down for model-side non-compliance. | |
| ChatML Protocol and Eliminating Schema Errors | 5 | 6 | 2 | 2 | Michelle educates the hosts on how structured outputs eliminated ChatML recipient hallucination errors under high temperatures by enforcing valid tool selection constraints. | |
| Benchmarking Structured Outputs and Parallel Function Calling | 6 | 6 | 3 | 4 | Swyx questions benchmark saturation and why parallel function calling is not yet supported with structured outputs. Michelle explains the latency overhead of re-indexing combined schemas on dynamic function lists. | |
| Latency Optimization, Caching, and Agentic Reliability | 6 | 5 | 2 | 3 | Alessio and Michelle discuss the foundation of autonomous agents, analyzing how moving from 95% to 100% schema reliability prevents error magnification across multi-step chains. | |
| JSON Schema Dialects, Dynamic UIs, and Prompting Best Practices | 8 | 4 | 2 | 3 | Swyx shares concrete engineering benchmarks from his production system AI News, reporting a 55% API cost reduction by switching from Instructor retries to native Structured Outputs. Michelle details OpenAI's JSON schema dialect decisions. | |
| Model Selection Strategy, Fine-Tuning GA, and Version Pinning | 6 | 5 | 3 | 4 | The conversation covers fine-tuning GA announcements and version pinning. Swyx questions why developers would ever use unpinned rolling ChatGPT models over deterministic API endpoints. | |
| OpenAI's Platform Vision and DevDay Outage War Stories | 5 | 4 | 1 | 2 | Michelle shares behind-the-scenes war stories from DevDay 2023, including fixing live API outages two hours before Sam Altman's keynote, and explains upgrades to the Assistants file search architecture. | |
| Platform Capabilities: Logit Bias, Rate Limits, and Batch Processing | 6 | 5 | 2 | 4 | The hosts explore logit bias, rate-limit tiering, and the Batch API. Swyx pushes back on the Batch API's rigid 24-hour SLA, arguing developers need a 2-4 hour turnaround option. | |
| Multimodal APIs: Vision, Whisper, Advanced Voice, and Enterprise Security | 7 | 5 | 3 | 4 | Alessio presses Michelle on why Whisper API lacks speaker diarization despite community demand. Michelle explains Whisper v2 vs v3 performance tradeoffs and suggests prompt hacks for spelling corrections. | |
| Engineering Culture, Decision Science, and OpenAI Hiring | 5 | 4 | 1 | 2 | The discussion closes on behavioral economics, Richard Thaler's Misbehaving, cognitive biases in engineering, and the specific low-ego traits sought in OpenAI hiring. |
Statements from this episode (32)
Pokrass: Every successful company eventually outgrows Postgres for NoSQL
“At some point, every company gets the scale, every successful company gets the scale where Postgres is not cutting it. And then you migrate to some sort of NoSQL database.”
OpenAI's API team had just five engineers before ChatGPT launched
“I would say the applied team was maybe like 30 or 40 people, and yeah, probably closer to 30, and there was maybe like five-ish total working on the API at most.”
OpenAI's Applied team now exceeds its entire pre-ChatGPT headcount
“Applied now is bigger than the company when I joined.”
ChatGPT's launch caused Postgres bottlenecks by reusing API developer accounts
“Surprisingly there were a lot of Postgres issues when ChatGPT came out because the accounts for like ChatGPT were tied to the accounts in the API. And so you're basically creating a developer account to log into ChatGPT at the time, because it's just what we h…”
Constrained decoding alone degrades output quality without model training
“And so it's not enough to just kind of constrain the model. I think of that as the engineering side, whereas basically you mask the available tokens that are produced every time to only fit the schema. And so you can do this engineering thing and you can force…”
OpenAI combined model training with constrained decoding for Structured Outputs
“We trained a model which is significantly better than our past models at following formats. And we did the end work to serve like this constrained decoding concept at scale.”
OpenAI's function calling originated from early Code Interpreter prototypes
“The history here is we started with function calling and function calling, you know, came from the idea of like, let's give the model access to tools and let's see what it does. And we basically had these internal prototypes of what code interpreter is now. An…”
OpenAI built Structured Outputs because developers were hacking function calling
“A lot of people were hacking function calling to get the response format they needed, and so this is why we shipped kind of this new response format. So you can get exactly what you want, and you get kind of more of the models verbosity.”
OpenAI built its constrained decoding engine from scratch
“Yeah, we didn't use any kind of Other stuff. We kind of built, you know, our solution from scratch to meet our specific needs.”
Standard HTTP status codes are limiting for AI model failures
“I think the HTTP spec is a little bit limiting for AI in a lot of ways. Like there are things that are in between your fault and my fault. There's kind of like the model's fault and there's no, you know, error code for that.”
Structured Outputs prevent models from hallucinating invalid tools
“The model before was able to hallucinate a tool, but now it's it can't when you're using structured outputs.”
Function calling benchmarks like BFCL are largely saturated
“I find that a lot of these evals are mostly saturated, like for BFCL. All the models are near, near the top. Already. And kind of the errors are more, I would say like just differences in default behaviors. I think most of the models on the leaderboard can kin…”
Structured Outputs does not yet support parallel function calling
“All our models support it, or all of our newer models support it, but we don't support it with structured outputs right now.”
OpenAI rejected pre-registering schema IDs to avoid developer complexity
“The alternative design space that we explored It's like pre-registering your schema, so like a totally different endpoint, and then passing in like a schema ID. But we thought, you know, that was a lot of overhead, and like another endpoint to maintain, and ju…”
Multi-step agentic apps fail at 95% reliability due to compounded errors
“Like if something is 95% reliable, but you're chaining together a bunch of calls, if you magnify that error rate, it makes your like application not work. So that's a really exciting thing here from going from like 95% to a hundred percent. I'm very biased wor…”
Assistants API tools succeeded, while its stateful endpoints need iteration
“I would say the hosted tools aspect has been really successful. Like people love our file search tool and it's like saves a lot of time to not build your own rag pipeline. I think we're still iterating on the shape for the stateful thing. To make it as useful …”
Structured response format is limited to GPT-4o and GPT-4o mini
“Actually, the new response format is only available on two models. It's Foro Mini and the new Foro. So the old Foro doesn't have the new response format. However, for function calling, we were able to enable it for all models that support function calling, and…”
Structured Outputs supports recursive schemas for generating dynamic UIs
“The schemas, we support recursive schemas, and this allows you to do really cool stuff, like, you know, every UI is a nested tree that has children, and so I thought that was super cool. You can use one schema and generate, like, tons of UIs.”
Building custom evals is high leverage for AI application developers
“I think for customers, and we work with a lot of customers, really developing their own evals is super high leverage. Because then you can upgrade really quickly when we have a new model, you can experiment with these things with confidence.”
Swix: OpenAI Structured Outputs cut API costs by 55% vs Instructor
“We use structured outputs for AI news. I use Instructor and I ripped it out and I think it, I saved 20 lines of code, but more importantly, it was like, we cut it by 55% of API costs based on what I measured because of all we saved on the retries.”
OpenAI Structured Outputs enforces schemas in one shot without retries
“We are not retrying, you know, we're doing it in one shot and this is how you save on latency and cost.”
Fine-tuning requires only 100 to 1,000 high-quality examples
“It's actually a lot easier to get started than a lot of people expect. I think they might need Tens of thousands of examples, but even a hundred really high quality ones or a thousand is enough to get going.”
OpenAI's API is its broadest vehicle for distributing AGI
“So I believe that the API is kind of our broadest vehicle for distributing AGI. You know, we're building some first party products, but they'll never reach every niche in the world and kind of every corner in community.”
The API was OpenAI's first commercial product
“The API is actually OpenAI's first product, and the first idea for commercialization, that predates me as well.”
OpenAI's Assistants API increased its file limit to 10,000 files
“Before, we only supported, I think, like, 20 files per assistant, and the way we used those files was, like, less effective. Basically, the model would decide, based on the file name, whether to search a file, and there's, like, not a ton of information in the…”
OpenAI's seed parameter is best-effort and not fully deterministic
“Yeah, the seed parameter is not fully deterministic, and it's kind of a best effort thing. So you'll notice there's more determinism in the first few tokens. That's kind of the current implementation.”
OpenAI aims to become an AI development platform, not just LLM-as-a-service
“We want to do more in this space and not just be an LLM as a service, but kind of AI development platform as a service.”
OpenAI likely to release streaming video API with frame sampling
“Yeah, I think it's very possible that we'll have an API where you stream video in, and maybe, you know, to start, we'll do the frame sampling for you.”
Whisper v2 outperforms v3 at certain tasks, delaying its API rollout
“And so whisper V two is better at some things than whisper V three. And so it didn't seem that worthwhile to ship whisper V three compared to like the other things in our priorities. I think we still will at some point, but yeah, it's just, you know, there's a…”
Standard request-response APIs will not work for speech-to-speech models
“I think just the regular request response probably isn't going to be the right solution.”
Pokrass: LLMs are far more rational 'econs' than humans
“So I think way more than all of us, they are e-cons.”
OpenAI hires applied engineers without prior AI experience
“We've hired people with all kinds of backgrounds, people who have PhD in an ML or folks who have just done engineering like me, and we're really hiring for a lot of teams. We're hiring across the applied org, which is where I sit for engineering, and for a lot…”