Jan 17, 2025 · 31m · latent-space
OpenAI o1 isn’t a chat model (and that’s the point)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space podcast, hosts and AI practitioners deconstruct OpenAI's o1 reasoning model, explaining why it requires a departure from conversational chat in favor of structured specification briefs, unified diff coding workflows, and compound system architectures.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 33.4% of the talking time here. How this is scored →
The hosts as informed peer 5.9 Guest teaching 3.4 Guest disagreement 1.1 The hosts pushing back 1.3
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 8:32 Ben dismisses hand-holding prompt engineering
Ben rejects conventional chain-of-thought prompting practices, calling attempts to instruct o1 on how to think patronizing and counterproductive.
Hardest push from the hosts ▶ 24:45 Swyx challenges Ben on search UI balanceSwyx interrupts and corrects Ben's narrative about models intelligently balancing web search by pointing out it was resolved by adding a manual UI button.
Biggest teaching moment ▶ 6:35 Dan educates on the 95% vs 100% code completion barrierDan reframes LLM coding utility by explaining that 95% functional code is practically 0% in software engineering, showing why o1 one-shot completions crossed that threshold.
The host holds their own ▶ 19:10 Swyx explains diff-critique style steering and prompt leakingSwyx demonstrates sophisticated prompt engineering expertise by explaining how to bypass few-shot example leakage through diff critiques and soft prompting concepts.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Initial Experiences and Shifting Mental Models on o1 | 4 | 2 | 1 | 0 | Alessio sets up the premise of the episode around Ben's essay and asks about changing mental models on o1. Both Ben and Dan share their evolving impressions collaboratively without friction. | |
| Transitioning from Chat Interface to Goal-Oriented Prompting | 6 | 2 | 0 | 1 | Alessio demonstrates technical depth by contrasting chat-completion tuning with goal and reward-based post-training. Ben shares how mobile app timeout bugs forced him into batch-style brief writing. | |
| Dan's 100% One-Shot Coding Workflow | 5 | 4 | 1 | 0 | Dan explains his one-shot 100% coding workflow and codebase concatenation method. Alessio mentions community tooling like Manuel's text file merger, confirming familiarity. | |
| Anatomy of an o1 Prompt: Structure and Philosophy | 4 | 5 | 2 | 1 | Ben breaks down his prompt template philosophy, arguing against patronizing chain-of-thought prompt engineering and emphasizing return formatting. | |
| Hidden Reasoning Tokens and Defining Intent | 5 | 5 | 1 | 1 | Alessio probes the tension between specifying intent and output formatting. Ben provides detailed analysis on hidden reasoning tokens and academic tone side effects. | |
| AI Observability and Background Intelligence Tasks | 5 | 4 | 0 | 0 | Alessio asks about production observability and heuristics for o1 at Dawn Analytics. Ben outlines background level intelligence and long compute tasks. | |
| Swyx Joins: Steering Tone via Diff Critiques | 8 | 3 | 2 | 2 | Swyx joins and delivers deep technical advice on using diff critiques and self-critique loops to steer tone while avoiding prompt leaking. | |
| Model Chaining and Compound System Architectures | 6 | 3 | 1 | 1 | Ben reflects on the historical cycle of monolithic models versus compound chained systems. Dan shares his startup's implementation of compound bot architectures. | |
| The Evolution of Intelligent Model Routing | 8 | 2 | 2 | 4 | Swyx notes correcting Ben's draft on o1 reasoning scaling, forecasts the rise of automated model routers, and directly pushes back on Ben regarding the search button UI. | |
| Coding Environments and Model Utilization | 7 | 3 | 1 | 2 | Swyx teases Ben for using copy-paste rather than Cursor. Alessio shares specific founder case studies replacing agency work with detailed o1 briefs. | |
| Bootstrapping Prompts Using Coding Assistants | 7 | 4 | 1 | 2 | Dan demonstrates bootstrapping prompt generation with coding agents. Ben questions prompt caching and context positioning, prompting Swyx to assert that practical token changes dictate placing context at the end. |
Statements from this episode (20)
Hylak: Non-engineers form fixed negative mental models after one failed AI test
“Most people try something once and if it doesn't work, they like They have this like very fixed mental model. They never tried again.”
McAteer: OpenAI o1 is the first model that grows more impressive over time
“O-one is actually the first model where I'm getting more impressed by it the more I use it. So, like, when ChatGPT first came out, right, I think it was GPT-III. And at first it seemed like, oh wow, this is amazing, it can actually create text that sounds like…”
Fanelli: OpenAI o1 is a goal-based reasoning model, not a chatbot
“Like O-one is not a chat model. And I think this is both from a usage perspective, but also ties back to some of the training stuff and post training that we already talked about. Like the previous models were so focused on early chat based on, especially on c…”
Fanelli: Chat interfaces confuse o1 users, who should write specification briefs instead
“It's almost like the chat is still the UX, you know, and I think that's what confuses people. Maybe like if you didn't have a chat interface and it was like write a brief, you know, I think people would maybe be steered more the right way.”
McAteer: o1 is the first model to achieve one-shot codebase implementation
“Using O-one was, it was the first time where I would connect it to my IDE. I would provide the full context of my code base. I'll just create a file that concatenates all my files into one just simple text file, give it to O-one, and then say, hey, based on th…”
Hylak: Do not micromanage OpenAI o1's step-by-step reasoning when prompting
“What I found that worked really well was not telling the model how to think about it. If that makes any sense, like almost more like you're actually interfacing with a person, which is like scary in its own right. But no, if you're working with a like a cowork…”
Hylak: OpenAI o1 struggles to match personal tone and writing styles
“I think that I've had a very hard time getting it to actually write stuff. I know that I've heard of people using it for writing where it's like processing diffs, more like providing critiques or feedback, but at least for myself, I haven't found a good way to…”
McAteer: Asking o1 for full code files instead of diffs is ineffective
“At first I was asking it to generate full files of code, like as I'm making changes. That can get a little bit confusing, and then if you're trying to use like a coding agent to bring it over into your projects. So I feel like that's not the best format for co…”
Hylak: Hidden reasoning tokens create an information asymmetry between OpenAI and developers
“I think that what makes a one even trickier than other models is that there is actually an asymmetric miss to how well open AI understands the model and how well we, for example, the fact that like reasoning tokens are hidden, right? So there's all this stuff …”
Hylak: OpenAI o1 is OpenAI's most capable yet hardest model to use
“We're finding that like, oh, one is the most capable model. I think that opening eye has made. And it's also, I think the hardest to use.”
Hylak: Users willing to wait five minutes for AI will wait an hour
“I think that it's like the number of tasks that you're willing to wait, you know, like 3:05 minutes for it. It's like probably pretty similar to the number of tasks you're willing to wait like an hour for, which is interesting.”
McAteer: Use Claude Sonnet for simple tasks and o1 for complex context
“Anything where it just seems simple and like, you can do one off and you don't need to bring a ton of context into it. You can typically use like sonnet or GPT for that. Anything where I feel like if I was going to try to implement it myself and I would need t…”
Swyx: Diff critiques steer LLM style better than few-shot examples
“And actually I found that that is a better way of doing this than when doing, you know, XML bracket, good example, close bracket, bad example, close bracket. Those examples tend to meet. There's an issue of prompts leaking, example leaking. Where there's a few…”
Hylak: AI development is returning to multi-step chaining of specialized models
“It feels like we're, like, going back Into a time where having these separate steps with almost like varying levels of intelligence becomes increasingly important, like this idea of chaining.”
Swyx: Anthropic and OpenAI will launch automated API model routing
“And so I pick and OpenAI have both keys that model routing on APIs already. And so I think they'll launch them, especially at some point where you can sort of prioritize the three tradeoffs that are in model routing, cost, speed, intelligence.”
Hylak: Claude.com yields better coding results than Cursor, which loops frequently
“I actually generally get better results out of Claude. It's in, you know, just Claude.com versus Cursor a lot of times which is interesting, like, I love being able to apply code, et cetera, But as far as just like, a lot of times I find it getting stuck in so…”
Fanelli: Founder used o1 Pro to replace a $30,000 marketing agency
“One of them, they were trying to get this, like a marketing positioning document on from an agency. They were charging like 30 grand and they were just not getting the right results. And then in like about five, six hours, like the founder created like a 20 pa…”
McAteer: Use coding assistants to bootstrap structured prompts from raw brainstorms
“What I do, and I think what you should do is use other prompts to sort of bootstrap your prompts. So like I have it set up in cursor or in windsurf where I was telling Alessio before we started, you basically, I have like a prompt template in a prompts directo…”
Swyx: Place dynamic context at the end of prompts for efficient caching
“Anyone who's doing, who's working with a ton of context has to put them at the end because they're swapping them out. That's just how it is. I don't see any way around it.”
McAteer: AI reasoning models are probably already superhuman at scientific synthesis
“I think that I think it's hard for one person to take in so much data, so much information and like make connections. And I think that's definitely one of the areas where these models, they're probably already superhuman at doing that.”