Apr 7, 2026 · 1h 17m · latent-space
Extreme Harness Engineering: 1M LOC, 1B toks/day, 0% human code or review — Ryan Lopopolo, OpenAI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
OpenAI's Ryan Lopopolo breaks down extreme harness engineering, detailing how his team maintained a million-line codebase with zero human-written code using Codex models. The interview explores sub-minute build loops, multi-agent orchestration via Symphony, token-efficient CLI design, and enterprise governance on OpenAI Frontier.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 3.9 Guest teaching 4.5 Guest disagreement 0.6 The hosts pushing back 0.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 38:20 Bearish stance on Model Context Protocol (MCP)
Ryan forcefully rejects MCP conventions, outlining how they degrade agent context through unneeded token injection and break auto-compaction loops.
Hardest push from the hosts ▶ 29:30 Challenging universal dependency vendoringThe host challenges the premise of vendoring all external libraries by citing scale limits and Linus's Law regarding open source security inspection.
Biggest teaching moment ▶ 30:45 Debugging directly with raw agent telemetry over human UIsRyan illustrates how an engineer wasted an afternoon building a Next.js trace visualizer when directly feeding raw tarballs into Codex resolved the issue immediately.
The host holds their own ▶ 1:13:35 RL on-policy vs off-policy harness framingThe host synthesizes Ryan's harness architecture using reinforcement learning terminology, distinguishing on-policy integration from fragile off-policy scaffolding.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Host Welcome and Listener Channel Support | 1 | 0 | 0 | 0 | Standard introductory segment featuring housekeeping, channel subscription appeals, and welcoming the guest to discuss his background at OpenAI and previous tech companies. | |
| The Zero-Human-Code Constraint and Build Evolution | 3 | 5 | 1 | 0 | Ryan details the zero-human-written-code constraint and how tooling evolved through model generations. The hosts listen attentively, summarizing the core premises of the article. | |
| Enforcing Sub-Minute Build Loops and Invariants | 5 | 4 | 1 | 2 | The host questions the rigid sub-minute build requirement and cites build times on existing projects. Ryan explains how keeping build times invariant maintains agent parallel velocity. | |
| Inverting Agent Scaffolding and Dev Stack Tooling | 4 | 5 | 1 | 1 | Ryan explains why they inverted the environment setup by letting the agent boot the stack directly via scripts instead of wrapping it in rigid scaffolds. The co-host connects this to reasoning models outperforming rigid frameworks. | |
| Encoding Process Knowledge and Managing Review Agents | 4 | 5 | 1 | 3 | The host raises concerns about over-constraining the agent with persistent guidelines when exceptions arise. Ryan explains how prompt frameworks allow code review and author agents to push back against nitpicks. | |
| End-to-End Autonomous Ownership and Safety Smoke Tests | 4 | 5 | 1 | 2 | The host checks in on whether fully autonomous merging causes safety concerns. Ryan clarifies that releases are gated by human-approved smoke tests rather than direct unreviewed production deployments. | |
| Architectural Governance for Agent-Legible Codebases | 3 | 5 | 1 | 1 | The hosts ask how human teams should adapt when codebases are optimized for agent consumption over human readability. Ryan notes his role shifted from PR reviewer to systems-level tech lead setting architectural primitives. | |
| MVC to Model-View-Claw and Collapsing Knowledge Work | 4 | 4 | 0 | 0 | The host jokingly introduces the concept of Model-View-Claw, and Ryan expands on how coding harnesses generalize across non-coding knowledge work simply by encoding tasks as scripts. | |
| Delegated Git Workflows and Autonomous Conflict Resolution | 5 | 5 | 1 | 2 | The host brings up Cursor's decision to move away from Git worktrees due to merge conflict overhead. Ryan counters that LLMs resolve merge conflicts effectively, making multi-branch worktrees painless. | |
| Vendoring Dependencies and Eliminating OSS Bloat | 5 | 4 | 1 | 3 | The host pushes back against Bret Taylor's claim of vendoring all dependencies, citing security review hygiene and scale testing. Ryan acknowledges scale constraints but argues inlining small dependencies eliminates unnecessary open source bloat. | |
| Unlearning Human UI Habits for Direct Agent Telemetry | 3 | 6 | 1 | 0 | Ryan shares an anecdote where an engineer wasted time building a human-facing trace visualizer instead of giving the raw tarball directly to Codex. He describes how Symphony distributes specs Ralph-style without human UI bias. | |
| Symphony: The Elixir Orchestrator and the Rework State | 3 | 6 | 1 | 1 | Ryan describes why Elixir was chosen for Symphony's supervision trees and details the 'rework state' where unviable PRs are completely trashed and regenerated from updated criteria. | |
| Experimental Tooling, MCP Drawbacks, and 500-Package Repos | 4 | 6 | 2 | 1 | Ryan is explicitly bearish on Model Context Protocol (MCP) due to forced context token overhead and compaction issues, advocating for lightweight bespoke CLI shims across an aggressively sharded 500-package repository. | |
| Linear Integration, Issue Tracking, and Unified Skills | 3 | 5 | 0 | 0 | Discussion centers on using Linear as an agent-driven issue tracker and consolidating process knowledge into a tight set of core skills rather than proliferating workflows. | |
| Centralized Trace Distillation and System Self-Modification | 4 | 5 | 1 | 2 | The host discusses meta-programming and whether agents should be put in boxes. Ryan details ingesting team-wide Codex logs into blob storage to run distillation loops that update guidelines automatically. | |
| Command-Line Supremacy, ASCII Vision, and Symphony Tiers | 4 | 5 | 1 | 0 | Ryan explains how CLI text feeds models better than GUIs and describes rasterizing UI screens into ASCII art for spatial positioning without relying solely on pure pixel computer vision. | |
| Designing Token-Efficient CLIs for Agent Consumption | 5 | 4 | 0 | 1 | The host and guest discuss the nuances of CLI design for LLMs, highlighting how silent flags and error filtering scripts dramatically cut token consumption during test runs. | |
| Symphony Specifications as Adaptable Blueprints | 3 | 4 | 0 | 0 | Ryan emphasizes that the Symphony spec is a flexible, adaptable blueprint rather than a rigid system, allowing developers to plug in alternative trackers and pipelines. | |
| Building Trust with Automated Screen Recordings | 4 | 4 | 0 | 0 | The dialogue explores how automated ffmpeg screen recordings attached to PRs build human reviewer trust and simulate the communication style expected of human teammates. | |
| Comparing Fast Spark Models with High-Reasoning LLMs | 4 | 5 | 1 | 1 | The co-host and Ryan compare high-reasoning models with fast, smaller models like Spark. Ryan admits Spark struggles on deep reasoning tasks by blowing through context compactions before coding, but excels at fast lint conversions. | |
| OpenAI Frontier: Enterprise Agent Governance and Security | 3 | 5 | 0 | 0 | Ryan provides an overview of OpenAI Frontier's enterprise platform, discussing integration with IAM, customizable safety specifications, and IT governance dashboards. | |
| Internal Data Agents, Business Ontologies, and Slack Memes | 5 | 4 | 0 | 1 | The host relates semantic data layers to historical business reporting problems like defining revenue. Ryan highlights embedding business context and cultural memes into agent prompt contexts. | |
| Agentic Organizations and the Progression Toward AGI | 5 | 4 | 0 | 0 | The host links centralized agent supervision to level four and five AGI definitions. Ryan outlines the vision for unified organizational knowledge across ChatGPT and coding harnesses. | |
| On-Policy Harness Design and Long-Term Compatibility | 5 | 4 | 0 | 0 | The host frames harness design in reinforcement learning terms of on-policy versus off-policy compatibility. Ryan emphasizes shipping relentlessly, rapid model releases, and hiring for the Bellevue office. |
Statements from this episode (51)
Lopopolo: OpenAI Frontier Is OpenAI's Platform for Enterprise Agent Deployment
“I work on frontier product exploration, new product development in the space of open AI frontier, which is our enterprise platform for deploying agents safely at scale with good governance in any business.”
Lopopolo: OpenAI Engineers Face No Internal Rate Limits for Development
“It certainly helps that we have no rate limits internally and I can go, like you said, full send at this thing.”
Lopopolo: Coding models and harnesses are now isomorphic to human engineering capability
“The models are there enough. The harnesses are there enough where they're isomorphic to me and capability and the ability to do the job.”
Lopopolo: When coding agents fail, decompose tasks into smaller reusable building blocks
“Whenever the model just cannot, you always pop open the task, double click into it and build smaller building blocks that then you can reassemble into the broader objective.”
Lopopolo: Zero-code harness was 10x slower initially before outperforming any single engineer
“Honestly, the first month and a half was 10 times slower than I would be. But because we paid that cost, we ended up getting to something much more productive than any one engineer could be, because we built the tools, the assembly station for the agent to do …”
Lopopolo: Codex can run background builds while concurrently reviewing code
“It basically means that Codex is able to spawn commands in the background and then go continue to work while it waits for them to finish. So it can spawn an expensive build and then continue reviewing the code, for example.”
Lopopolo: Team halts work to optimize build graph if builds exceed one minute
“No, we just take that as a signal that we need to stop what we're doing, double click, decompose the build graph a bit to get the time back under so that we can enable the agent to continue to operate.”
Lopopolo: Cheap parallel tokens enable continuous automated codebase maintenance
“But because tokens are so cheap and so insanely parallel with the model, we can just constantly be gardening this thing to make sure that we maintain these invariants, which means There's way less dispersion in the code and the SDLC, which means we can kind of…”
Lopopolo: OpenAI Frontier team operates with post-merge or zero human code review
“You know, we, we've moved beyond even the humans reviewing the code as well. Most of the human review is post merge at this point, but it's not even reviewed.”
Lopopolo: Synchronous human attention is the only scarce resource in agentic software engineering
“The model is trivially paralyzable, right? As many GPUs and tokens as I am willing to spend, I can have capacity to work with a code base. The only fundamentally scarce thing is the synchronous human attention of my team.”
Lopopolo: OpenAI inverts harnesses by having Codex spawn dev environments
“One neat thing here is we have tried to invert things as much as possible, which is instead of setting up an environment to spawn the coding agent into, instead we spawn the coding agent, like that's the entry point, just codex, and then we give codex via skil…”
Lopopolo: Reasoning models eliminate need for rigid state-machine scaffolding
“And this I think is like the fundamental difference between reasoning models and the four ones and four O's of the past where these models could not think. So you kind of had to put them in boxes with a predefined set of state transitions. Whereas here we have…”
Lopopolo: OpenAI uses incident pages to update repository reliability rules via Codex
“When we get a page because we're missing a timeout, for example, I can just add codecs in Slack on that page and say, I'm going to fix this by adding a timeout. Please update our reliability documentation to require that all network calls have timeouts. So I h…”
Lopopolo: PR review agents initially caused non-convergence by bullying author agents
“Initially the codex driving the code author was willing to be bullied by the PR reviewer, which meant you could kind of end up in a situation where things were not converging.”
Lopopolo: AI coding agents default to rigid instruction-following without explicit leeway
“Without the context that this is permissible, the coding agents are going to bias toward what they do, which is following instructions.”
Lopopolo: OpenAI Frontier requires human-approved smoke tests before distribution
“So because we are building a native application here, we're not doing continuous deploy. Right. So there's still a human in the loop for cutting the release branch. We require a blessed human approved smoke test of the app before we promote it to distribution,…”
Lopopolo: Codex authors Grafana dashboards and handles on-call incident paging
“Like the dashboard thing you mentioned, we have Codex authoring the JSON for the Grafana dashboards and publishing them, and also responding to the pages, which means when it gets the page, it knows exactly which dashboards are defined and what alerts. What al…”
Lopopolo: Managing coding agents resembles tech leading a 500-person organization
“The mindset is very much that I'm removed from the process, right? I can't really have Deep code level opinions about things. It's as if I'm group tech leading a 500 person organization. Like, yeah, like it's not appropriate for me to be in the weeds on every …”
Lopopolo: Better AI models propose their own code abstractions
“As the models have gotten better, they have gotten better at proposing these abstractions to unblock themselves, which again, lets me move higher and higher up the stack to look deeper into the future on what ultimately blocked the team from shipping.”
Lopopolo: Collapsing product problems into code allows Codex harnesses to solve them
“If you can figure out how to collapse a product that you're trying to build a user journey that you're trying to solve into code, it's pretty natural to use the codex harness to solve that problem for you.”
Lopopolo: AI models are really great at resolving Git merge conflicts
“The models are really great at resolving merge conflicts.”
Lopopolo: OpenAI uses an automated landing skill to delegate PR merges to Codex
“We invoke a dollar land skill and that coaches codecs to push the PR, wait for human and agent reviewers, wait for CI to be green, fix the flakes if there are any Merge upstream if the PR comes into conflict, wait for everything to pass, put it in the merge qu…”
Lopopolo: Harness engineering codifies implicit engineering standards into agent context
“The whole meta of the thing is to basically tease out of the heads of all the engineers on my team, what they think good looks like, what they would do by default. Or what they would coach a new hire on the team to do, to get things to merge. And that's why we…”
Lopopolo: AI models can in-house 2,000-line dependencies in an afternoon
“The level of complexity of the dependencies that we can internalize is I would say low medium right now, right? Just based on model capability. What is medium? I would say like a couple thousand line dependency is a thing that we could in house no problem in a…”
Lopopolo: Codex reviews internalized dependencies with less friction than upstream patching
“When we deploy Codex security on the repo, it is able to deeply review and change The internalized dependencies in a much lower friction way than it would be to like push patches upstream, wait for them to be released, pull them down, make sure that's compatib…”
Lopopolo: Optimizing AI debugging workflows for human legibility is wrong
“Optimizing for human legibility of that debugging process was wrong. It kept him in the loop unnecessarily, when instead he could have just like codex cooked for five minutes and gotten the same.”
Lopopolo: OpenAI used iterative Codex loops to generate Symphony specs
“Like we have taken all the scaffolding that has existed in our proprietary repo, spun up a new one. Ask codex with our repo as a reference. Write the spec. We tell it, spin up a tmux, spawn a disconnected codex to implement the spec. Wait for it to be done. Sp…”
Lopopolo: Coding models have largely solved all tasks except hard and new
“And I think things that are hard and new is still something that the models need humans. Yeah. Drive. Yeah. But I think those other quadrants are largely solved, given the right scaffold and the right thing that's going to drive the agent to completion.”
Lopopolo: BEAM process supervision gives agent orchestration free concurrency
“The process supervision and the gen servers are super amenable to the type of process orchestration that we're doing here, right? You are essentially spinning up little daemons for every task that is in execution and driving it to completion, which means the m…”
Lopopolo: OpenAI engineer PR volume jumped from 3.5 to 5-10 daily
“At the end of December we were at about three and a half PRs per engineer per day. This was before five two came out in the beginning of January. Everyone gets back from holiday with five two and no other work on the repository. We were up in the five to 10 PR…”
Lopopolo: Symphony discards failed PRs entirely to regenerate from scratch
“In Symfony, there's this like rework state where once the PR is proposed and it's escalated to the human for review, it should be a cheap review, right? It is either mergeable or it is not. And if it's not, you move it to rework. The Elixir service will comple…”
Lopopolo: Bearish on MCP due to forced token injection and compaction issues
“MCPs I'm pretty bearish on because the harness forcibly injects all those tokens in the context and I don't really get a say over it. They mess with auto compaction. The agent can forget how to use the tool. There's probably only like, what, three calls in Pla…”
Lopopolo: AI-amplified small teams require extreme package decomposition and strict boundaries
“The structure of the repository is like, 500 NPM packages. It's like architecture to the access for what you would consider, I think, normal for a seven person team. But if every person is actually, like, 10 to 50. Then the, like, numbers on, like, being super…”
Lopopolo: Standardizing codebase structure and skills maximizes AI agent effectiveness
“I do think that there is leverage to be had in making the code and the processes as much the same as possible. If you think that code is context, code is prompts, it's better from the agent behavior perspective to be able to look in a package in directory XYZ …”
Lopopolo: OpenAI Frontier codebase operates on approximately six core skills
“So like in our code base, we have, I think six skills. That's it. And if some part of the software development loop is not being covered, Our first attempt is to encode it in one of the existing setup skills, which means that we can change the agent behavior m…”
OpenAI Frontier runs daily agent loops over team logs to update repositories
“We're actually slurping these up for the entire team into blob storage and running agent loops over them every day to figure out where as a team can we do better? And how do we reflect that back into the repository? Yeah, though, everybody benefits from everyb…”
Lopopolo: PR comments and failed builds indicate AI agents lacked context
“A PR comment, a failed build. These are all signals that mean at some point the agent was missing context. We've got to figure out how to slurp it up and put it back in the reboot.”
OpenAI Frontier grants coding agents full permission to file follow-up tickets
“Like, this thing is also able to cut its own tickets, because we give it full access. Yeah, yeah, yeah. You can make a ticket to have it cut tickets, you can put in the ticket that you expected to file its own follow-up work.”
Lopopolo: Converting UI Images to ASCII Art Improves AI Agent Layout Perception
“If we want to actually, like, make it see the layout, it's almost easier to rasterize that image to ASCII arc and feed it in to the agent.”
Lopopolo: Autonomous Coding Removes Human Language Familiarity Constraints
“No humans in the loop here. So like my, Own personal ability to write or not write Elixir doesn't really have to bias us away from using the right tool for the job, which is just wild.”
Lopopolo: Agent CLIs must suppress passing output for token efficiency
“The CLIs are nice because they're super token efficient, and they can be made more token efficient really easily, right? Like, I'm sure you all have seen, like, I go to Buildkite or Jenkins, and I could just get this massive wall of build output. And in order …”
Lopopolo: Strict success criteria in agent prompts increase deployment reliability
“Fundamentally, the agents are good at following instructions, so give them instructions, right? And it will, you know, improve the reliability of the result, right? Like we, much like the way we use Symphony, we don't want folks to have to monitor the agent as…”
Lopopolo: Coding agents should summarize proof instead of requiring full oversight
“I would expect you to do what you think you need to do to convince me that the code is good and mergeable and compress that full trajectory in a way that is legible to me, the reviewer.”
Lopopolo: GPT-5.3 Spark burned three compactions before coding complex tasks
“I was adapting it to the same sorts of tasks I would use X high reasoning for, and it would blow through three compactions before writing a line of code.”
Lopopolo: Fast Spark models excel at prototyping, docs, and lint healing
“It's very great for spiking out prototypes, exploring ideas quickly, doing those documentation updates. It, Is fantastic for us in taking that feedback and transforming it into a lint where we already have good infrastructure for ES lints in the code base. The…”
Lopopolo: Current AI models cannot go from idea to prototype
“They're definitely not there on being able to go from new product idea to prototype.”
Lopopolo: OpenAI Agents SDK provides an out-of-the-box agent harness
“Agents SDK is a core part of this to enable both
Startup builders as well as enterprise builders to have a works by default harness that is able to use all the best features of our models from the shell tool down to the codex harness with file attachments and …”
Lopopolo: GPT OSS Safeguard supports custom enterprise safety specs
“The GPT OSS Safeguard model, for example.
One thing that's really cool about it is it ships the ability to interface with a safety spec.
Safety specs are things that are bespoke to enterprises.
We owe it to these folks to figure out ways for them to instrument…”
Lopopolo: OpenAI feeds 12-month business vision and customer context to agents
“One thing that's in core beliefs.md is like,
Who's on the team, what product we're building, who our end customers are, who our pilot customers are, what the full vision of what we want to achieve over the next 12 months is.
Like these are all bits of context …”
Lopopolo: Native code guardrails outlast external model scaffolds as AI advances
“If we were building an entire separate Ross scaffold around Codex to restrict its output, that I think would be like additional harness that would be prone to being scrapped. But yeah, if instead we can build all the guardrails in a way that's just native to t…”