Apr 25, 2024 · 31m · no-priors
No Priors Ep.61 | OpenAI's Sora Leaders Aditya Ramesh, Tim Brooks and Bill Peebles
Tim Brooks · 9m spoken Bill Peebles · 7m spoken Sarah Guo · 4m spoken Aditya Ramesh · 3m spoken Elad Gil · 3m spoken
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of No Priors, OpenAI's Sora leadership team—Aditya Ramesh, Tim Brooks, and Bill Peebles—discusses the diffusion transformer architecture, emergent physics simulation, safety guardrails, and creative potential of their generative video model. They explain how scaling video generation marks a foundational milestone on the direct path toward Artificial General Intelligence.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 27.7% of the talking time here. How this is scored →
The hosts as informed peer 4.8 Guest teaching 5.7 Guest disagreement 0.3 The hosts pushing back 1.2
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 27:25 Bill reframes human physics intuition as a deficiency
Bill politely rejects the premise that models should merely match approximate human physics intuition, arguing that human fidelity is deficient and will be superseded by scaled neural world models.
Hardest push from the hosts ▶ 21:18 Elad pushes back on AI platform liability using Photoshop precedentElad challenges the notion of developer liability for misuse by citing historical precedents like Photoshop, where toolmakers are not held responsible for malicious user edits.
Biggest teaching moment ▶ 11:32 Bill details the shift from fixed-resolution crops to 3D spacetime tokensBill systematically educates the hosts on how previous visual models threw away diverse internet data via rigid crops, whereas 3D spacetime patches unlock broad LLM-style generalist capabilities.
The host holds their own ▶ 15:14 Elad draws technical analogy to end-to-end deep learning in AVsElad demonstrates sharp domain insight by connecting Tim's first-principles architecture explanation to the recent paradigm shift from heuristic systems to end-to-end learning in autonomous driving.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Safety Roadmap, Creator Feedback, and Emerging Creative Workflows | 4 | 4 | 0 | 1 | Elad draws thoughtful historical parallels to Pixar's early shorts and computer graphics evolution. The guests explain their deliberate rollout strategy, red teaming, and early artist feedback like the Airhead short and Bling Zoo. | |
| Video as World Models and Diffusion Transformer Architecture | 5 | 6 | 0 | 1 | Sarah prompts Tim to break down diffusion transformers and asks Bill about empirical scaling laws. Tim clearly articulates how iteratively removing noise combines with transformer compute scaling. | |
| Latent Space-Time Patches and First-Principles Architecture | 5 | 6 | 0 | 1 | Sarah and Elad engage deeply on tokenization paradigms and compare Sora's design to end-to-end deep learning in self-driving cars. Bill and Tim explain how 3D spacetime cubes broke away from legacy fixed-crop image extension hacks. | |
| Aesthetic Steering, Model Personalization, and Education | 5 | 5 | 0 | 1 | Sarah shares her personal experience synthesizing stories for her kids and Elad asks about avatar applications. Aditya notes that Sora's aesthetic is emergent and prompt-steered rather than hand-tuned. | |
| Safety Challenges, Misinformation, and Physical Coherence Limitations | 5 | 6 | 1 | 2 | Elad pushes on the liability question using Photoshop as a precedent for user responsibility. Bill transparently outlines current physical coherence failures, such as soccer balls vaporizing during interactions. | |
| The Bitter Lesson, Scaling Laws, and Video as the GPT-1 Moment | 5 | 7 | 1 | 1 | Sarah asks about approximate human physics versus exact simulation, and Tim and Bill explain how the Bitter Lesson dictates scaling raw data prediction over engineered approximations. |
Statements from this episode (14)
Peebles: Generative video models are on the critical path to AGI
“Yeah, we absolutely believe models like Sora are really on the critical pathway to AGI.”
Peebles: Sora will evolve into interactive world simulators with autonomous humans
“And so looking forward, as we continue to scale up models like Sora, we think we're going to be able to build these, like, world simulators, where essentially, you know, anybody can interact with them. I, as a human, can have my own simulator running, and I ca…”
Brooks: OpenAI has no immediate timeline to launch a Sora product
“We don't currently have immediate plans or even a timeline for creating a product.”
Brooks: Generative video will birth entirely new media formats within years
“I do think that, yeah, maybe over the next couple years we'll see people starting to make, like, more and more films, but I think people will also find completely new ways to use these models that are just different from the current media that we're used to.”
Peebles: Training on raw video is essential for future embodied robotics
“So you learn so much about the physical world just from training on raw video that we really believe that it's going to be essential for
things like physical embodiment moving forward.”
Peebles: Sora is the first visual model with LLM-like breadth
“And so this is really the first generative model of visual content that has breadth in a way that language models have breadth.”
Brooks: OpenAI built Sora from scratch to achieve minute-long HD video
“We started from scratch, and we started with the question of, how are we going to do a minute of HD footage? And that was our goal. And when you have that goal, we knew that we couldn't just extend an image generator. We knew that in order to do a minute of HD…”
Brooks: AI teams should build for three-year end-states, not incremental features
“There is this pressure to do things fast because AI is so fast, and the fastest thing to do is, oh, let's take what's working now and let's kind of, like, add on something to it, and that probably is, as you're saying, more general than just image to video but…”
Ramesh: OpenAI did not spend much effort tuning Sora's visual aesthetic
“Well, to be honest, we didn't spend a ton of effort on it for Sora.”
Brooks: Sora represents the 'GPT-1' stage of generative visual models
“I think where we are in the trajectory of Sora right now is like, this is the GPT-I of these, this new paradigm of visual models, and that we're really looking at the fundamental research into making these way better, making it a way better engine that can pow…”
Peebles: Generating long videos with Sora currently takes several minutes
“It's not instant and you have to wait at least like a few minutes for like these really long videos that we're generating.”
Brooks: Sora learned 3D spatial understanding purely from raw 2D video
“We didn't explicitly bake three D information into it whatsoever. We just trained it on video data and it learned about three D because three D exists in those videos.”
Peebles: Video models will eventually surpass humans as physical world models
“We're optimistic that Sora will, you know, supersede that kind of capability and will, you know, in the long run enable it to be More intelligent one day than humans as world models.”
Brooks: The best way to scale intelligence is simple data prediction
“What works really well as you increase scale is just predict data. And that's what we do with text. We just predict text. And that's exactly what we're doing with visual data with Sora, which is we're not making some complicated trying to figure out some new t…”