Nov 28, 2025 · 53m · a16z
How OpenAI Builds for 800 Million Weekly Users: Model Specialization and Fine-Tuning
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The a16z Show, host Martin Casado interviews Sherman Wu, Head of Engineering for OpenAI's Developer Platform, about model specialization, reinforcement fine-tuning (RFT), AI agent architecture, and platform economics.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The host as informed peer 5.9 Guest teaching 3.4 Guest disagreement 0.9 The host pushing back 2.4
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 27:43 Reframing agents as interfaces rather than separate product lines
Sherman rejects Martin's attempt to isolate agents into traditional API or product categories, reframing them as varied interfaces for underlying model intelligence.
Hardest push from the host ▶ 42:30 Calling dual language and diffusion development an anti-patternMartin explicitly pushes back on combining text and video diffusion models in a single company, labeling it an organizational anti-pattern.
Biggest teaching moment ▶ 41:37 Explaining text vs pixel model training bottlenecksSherman educates Martin on why text models cannot be verticalized as easily as image diffusion models due to heavy pre-training and post-training compute steps.
The host holds their own ▶ 50:38 Detailing pseudocode prompting patterns for NPCsMartin demonstrates expert engineering knowledge by describing how developers pass Python pseudocode in prompts to constrain LLM behavior in game NPCs and regulated fields.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| The a16z Show Title Card | 3 | 2 | 1 | 1 | Martin asks conversational questions about Sherman's career transitions from OpenDoor ML pricing to OpenAI's developer platform, drawing on his own background at Livermore. | |
| Early Career at Quora and MIT Origins | 2 | 1 | 0 | 0 | Sherman shares stories about his MIT externship and Quora's early engineering talent, while Martin reminisces about Quora's legendary founding team. | |
| Internal Strategy: Horizontal APIs vs. Consumer Products | 5 | 3 | 1 | 2 | Martin asks whether offering both horizontal APIs and vertical consumer apps creates internal product conflict, prompting Sherman to explain OpenAI's 800M user scale and mission framing. | |
| Model Retention and Anti-Disintermediation Mechanics | 7 | 2 | 1 | 3 | Martin proposes that AI models act as anti-disintermediation technology because users form sticky behavioral habits that resist traditional software abstraction. | |
| Developer Workflows in a Multi-Model Landscape | 6 | 3 | 1 | 3 | Martin cites his hands-on developer experience using Cursor across multiple models to challenge the early consensus that one single AGI model would rule everything. | |
| Enterprise Customization: Reinforcement Fine-Tuning (RFT) | 6 | 5 | 1 | 3 | Martin probes whether fine-tuning is capitulation on general intelligence, leading Sherman to explain how reinforcement fine-tuning (RFT) unlocks state-of-the-art domain capabilities. | |
| Context Engineering and the Evolution of RAG | 8 | 4 | 1 | 4 | Martin sharply critiques primitive RAG architectures as insulting to superintelligence, which Sherman validates while explaining how reasoning models transform context retrieval. | |
| Architectural Perspective on AI Agents and Interfaces | 5 | 4 | 2 | 3 | Martin attempts to categorize agents into classic product versus API categories, but Sherman reframes agents as varied user interfaces for underlying model intelligence. | |
| AI Platform Economics and Usage-Based Pricing | 7 | 4 | 1 | 3 | Martin analyzes the industry transition to usage-based billing and challenges the feasibility of outcome-based pricing in complex domains, which Sherman agrees with via test-time compute trends. | |
| Open Source Strategy and Community Ecosystem Impact | 6 | 3 | 1 | 2 | Martin highlights that open-weight models do not cannibalize API revenue because running performant inference infrastructure remains the true operational moat. | |
| Exploring Deep Model Verticalization Across Modalities | 6 | 5 | 1 | 2 | Martin compares text models to image LoRAs, prompting Sherman to explain the heavy pre-training and post-training bottlenecks that limit deep text verticalization. | |
| Managing Multimodal Models and World Simulation Infrastructure | 7 | 4 | 1 | 3 | Martin identifies developing both text and video diffusion models under one roof as an organizational anti-pattern, which Sherman confirms while detailing OpenAI's isolated World Simulation team. | |
| The Evolution of AI Agents and SOP Workflows | 6 | 5 | 1 | 3 | Martin brings up developer skepticism regarding low-code agent builders, and Sherman explains why deterministic nodes are essential for procedural standard operating procedures. | |
| Constraining Agent Execution for Games and Regulated Fields | 8 | 2 | 0 | 2 | Martin details technical implementation patterns like passing Python pseudocode in prompts to constrain NPC behavior in games and regulated fields, which Sherman enthusiastically praises. |
Statements from this episode (27)
Sherman Wu: AI industry will shift toward specialized models
“It's, like, becoming increasingly clear that there will be room for a bunch of specialized models. There will likely be a proliferation of other types of models.”
Sherman Wu: OpenAI deployed a local model at Los Alamos National Lab
“We actually do have a local deployment. At Los Alamos National Labs. It's super cool. I went to visit it. It's like very different than what I'm used to. But yeah, in a like, you know, classified supercomputer with our model running there.”
Sherman Wu: OpenAI's API pricing is less sophisticated than Opendoor's
“I don't think we do anything as sophisticated as Open Door.”
Sherman Wu: Early Quora team included future Scale AI and Perplexity founders
“A bunch of the perplexity team was there. Dennis, Dennis was on the feed team with me. Johnny Ho Jerry Ma. And then Alexander, the scale, you know, like was there. He was there between high school and college.”
Sherman Wu: ChatGPT has reached 800 million weekly active users
“The first party app is a really great way to get, you know it was like eight hundred million wows or whatever now.”
Sherman Wu: OpenAI API end-user reach previously exceeded ChatGPT's reach
“One thing we talk about internally sometimes is like, what does our end user reach from the API? Like it's actually, it's like really, really, it's really bright. It might even, it's hard because ChatGPT is growing so quickly, but like it, like at some point i…”
Martin Casado: LLMs naturally resist software disintermediation layers
“Because the models are so hard to abstract away. Like, they're just unruly, right? If you try to, like, have traditional software drive them, they just don't kind of manage very well. So part of me thinks that it's almost like this, like, anti-disintermediatio…”
Sherman Wu: Developer retention on OpenAI's API remains surprisingly high
“For or like the retention of people building on our API is like surprisingly high especially when people thought you could just kind of swap things around. You might have, you know, like even tools that help you swap things around. But yeah, the stickiness of …”
Sherman Wu: OpenAI previously believed a single model would replace fine-tuning
“I remember like, even with an open AI, the thinking was that there would be like one model that rules them all. And it's like, why would you, I mean, like this kind of goes to the fine tuning API product. It's like, why would you even have a fine tuning produc…”
Sherman Wu: OpenAI RFT elevates domain models to state-of-the-art performance
“The reinforcement fine tuning API... Kind of changes the paradigm away from just, like, small incremental, like, tone improvements, which is what SFT did, to actually improving the model to potentially SOTA level on a particular use case that you know about.”
Sherman Wu: OpenAI pilots free training for shared enterprise data
“If you actually build with a reinforcement fine tuning API, you can actually get discounted inference and potentially free training too, if you're willing to share the data”
Sherman Wu: AI development has shifted from prompt to context engineering
“But I think the name of the game now is, is less on like, Prompt engineering as we had thought about it two years ago. It's more of like, it's like the context engineering side where it's like, what are the tools you give it? What is like the data that it pull…”
Sherman Wu: OpenAI's o3 model stands out for diligent tool execution
“One of my favorite models is actually O three. Cause it was like one of the most diligent models. It would just like do all these tool calls and it's like really the intelligence itself trying to like do the, you know, tool calls or reg or anything like that o…”
Sherman Wu: OpenAI views ChatGPT and Sora as interfaces for core intelligence
“At the end, like, at the end of the day, OpenAI is like a, an AGI company. It's like an intelligence company. And so, agents are just, like, one way in which this intelligence kind of be manifested. And so, the way that I'd say we actually think about internal…”
Sherman Wu: OpenAI prices its API services using a cost-plus model
“Internally, one thing we do is, is we always make sure that we actually price our usage-based pricing from a, like, cost-plus perspective. Like, we're actually just, like, trying to make sure that we're being responsible from a margin perspective.”
Sherman Wu: Usage-based pricing naturally approximates outcome-based pricing in AI
“On outcome-based pricing it sounds very appealing, like, if it can work, but one thing that we've started realizing is it actually ends up correlating quite a bit with usage-based pricing, especially with test time compute. Like, if the thing is just, like, th…”
Sherman Wu: OpenAI has seen zero API cannibalization from open models
“Yeah, I mean, to be clear, like, we have not seen cannibalization at all.”
Sherman Wu: Major AI labs generate revenue from two or three models
“Especially for all these major labs, like they're usually like two or three models where like, that is where you're making all of your impact, all of your revenue.”
Sherman Wu: Replicating OpenAI's flagship model inference performance is extremely hard
“Even if we just like, you know, open source, like if we just literally open sourced GPT-V or something, it would be really, really hard to inference it at the level that we are able to get it to do.”
Sherman Wu: Smaller size of image models drives faster iteration and proliferation
“Image models tend to be way smaller and like you can iterate on it a lot faster. Like that's why you get that crazy cool proliferation of like the image model side.”
Sherman Wu: Heavy compute for text model post-training bottlenecks verticalization
“For the text models, there's always going to be this like really big fat free training step that like you have to invest in here. And then even the post training side is like, You know, it's not the, it's not like the easiest thing. Like it's, you know we all,…”
Sherman Wu: Combining language and diffusion models is an anti-pattern
“Yeah, I think you're totally right. It's an anti-pattern. It's pretty tough to pull off.”
Sherman Wu: OpenAI's Sora team has the highest talent concentration he's seen
“For my perspective, I think the biggest thing is I think our like image like our I think called like the world simulation team, like the team that builds Sora and all that under Aditya is just extremely solid. Like they are probably, it's like the highest Conc…”
Sherman Wu: Current AI models cannot reliably execute exact multi-step instructions
“The models today, just like maybe in some future world instruction following would be so good that you just like ask it to do this four step process. And it like always does the four step process exactly. We're still not there yet.”
Sherman Wu: Enterprise AI requires deterministic node workflows for procedural tasks
“There's A huge need on that side to have determinism here. Of which an agent builder with nodes that kind of like helps enforce this thing ends up being very, very helpful. But I think a lot of us, especially in Silicon Valley, don't really appreciate that. Th…”