Jul 3, 2024 · 53m · big-technology
What The Ex-OpenAI Safety Employees Are Worried About
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Host Alex Kantrowitz interviews former OpenAI Superalignment researcher William Saunders and Harvard Law professor Larry Lessig to examine why departing employees are raising safety alarms. The conversation explores how commercial pressure, restrictive non-disclosure agreements, and a lack of specialized regulatory oversight threaten responsible artificial intelligence development.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Alex holds 27.2% of the talking time here. How this is scored →
Alex as informed peer 5.0 Guest teaching 5.5 Guest disagreement 2.5 Alex pushing back 3.9
speaking balance: gold is Alex, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 9:59 Lessig preemptively blocks Kantrowitz's line of questioning
Lessig directly interrupts and objects to Kantrowitz asking Saunders specifically what he witnessed inside OpenAI, citing confidentiality bounds.
Hardest push from Alex ▶ 10:40 Kantrowitz demands clarity on whether whistleblowers saw immediate dangersKantrowitz refuses to let the guests evade whether tangible catastrophic technology was observed, pressing that public alarm requires concrete evidence.
Biggest teaching moment ▶ 21:08 Lessig explains California employment law regarding vested equityLessig educates the host on how vested equity is legally classified as wages in California, making clawback threats for non-disparagement illegal.
Alex holds their own ▶ 41:36 Kantrowitz confronts guests with insider counterarguments against the safety policyKantrowitz demonstrates strong command of internal lab politics by citing specific technical pushback from OpenAI staffer Joshua Achiam.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Alex as informed peer | Guest teaching | Guest disagreement | Alex pushing back | Why |
|---|---|---|---|---|---|---|
| The Titanic vs. Apollo Analogy and Resignation | 5 | 3 | 2 | 4 | Kantrowitz probes Saunders' choice of analogies, asking why he selected the Titanic and Apollo framing rather than the Manhattan Project often invoked around AI. Saunders unpacks his reasoning, explaining the Manhattan Project represents societal impact while the Titanic captures corporate safety negligence. | |
| Defining Superalignment and Technical AI Oversight | 4 | 5 | 1 | 2 | Saunders explains the technical nature of superalignment research, illustrating recursive AI auditing and the difficulty of evaluating systems smarter than humans. Kantrowitz listens attentively with minimal intervention. | |
| The 'Right to Warn' and Internal OpenAI Culture | 6 | 5 | 4 | 7 | Kantrowitz pushes directly on whether the whistleblowers actually saw a dangerous, undisclosed technology at OpenAI to warrant sounding public alarms. Lessig intervenes to guard legal boundaries while Saunders clarifies that the danger lies in trajectory and internal governance rather than immediate catastrophic systems. | |
| Near-Term AI Risks and Monitoring Vulnerabilities | 3 | 6 | 2 | 2 | Saunders educates the host on realistic near-term failure modes, including multilingual disinformation campaigns bypassing small monitor models. The host allows Saunders to lay out technical security and monitoring deficiencies uninterrupted. | |
| NDAs, Equity Clawbacks, and California Employment Law | 5 | 7 | 2 | 2 | Lessig delivers a legal breakdown of California wage law regarding vested equity and non-disparagement agreements, connecting it to Frances Haugen's revelations. Kantrowitz notes OpenAI's claim of never clawing back equity, prompting Lessig to explain the chilling effect of unenforceable clauses. | |
| Whistleblower Frameworks and Specialized Oversight Bodies | 6 | 6 | 3 | 5 | Kantrowitz asks whether existing whistleblower protections cover safety concerns when no laws are broken, leading Lessig to explain SEC jurisdiction limitations and technical illiteracy. Kantrowitz pushes back by noting OpenAI already maintains internal hotlines and safety boards. | |
| Legislative Proposals and Non-Punitive Safety Models | 5 | 6 | 1 | 3 | Saunders and Lessig discuss California SB 1047 and the necessity of non-adversarial reporting mechanisms modeled on medical morbidity reviews. Kantrowitz asks specific clarifying questions about enforcement mechanisms and agency jurisdiction. | |
| Compute Allocation, Model Interpretability, and Research Realities | 6 | 6 | 3 | 6 | Kantrowitz presents online critiques arguing that OpenAI rationally reallocated 20% of superalignment compute toward active products because near-term catastrophe was unlikely. Saunders refutes this, explaining that interpretability research requires long lead times before frontier systems arrive. | |
| Addressing Public Criticisms of the 'Right to Warn' | 6 | 6 | 4 | 6 | Kantrowitz cites OpenAI researcher Joshua Achiam's public pushback claiming the 'right to warn' grants carte blanche to leak secrets and alienates safety staff. Lessig and Saunders systematically rebut Achiam's framing by explaining the multi-tiered whistleblower process. | |
| Sam Altman's Governance and Product Launch Pressures | 5 | 5 | 3 | 3 | Kantrowitz references Saunders' New York Times quote criticizing Sam Altman's governance and presence on oversight committees. Saunders elaborates on the friction between rigid product launch timelines and adequate safety evaluations. | |
| AGI Timelines, Systemic Risk, and the Titanic Analogy | 4 | 6 | 2 | 3 | Kantrowitz summarizes the broader timeline debate and asks about systemic desensitization to AI risks. Saunders and Lessig detail the disparity between fast AI development timelines and the decade-long regulatory process, concluding with the Titanic lifeboat precedent. |
Statements from this episode (16)
Saunders: OpenAI prioritizes shiny products over safety like the Titanic
“OpenAI claimed that their mission was to build safe and beneficial AGI, and I thought that this would mean that they would prioritize, you know, putting safety first. But over time, it started to really feel like the decisions being made by leadership were mor…”
Saunders: AI-assisted critique helps humans detect model errors
“What some of the research that I did at OpenAI was, you know, trying to develop techniques for this, and the simple technique that we tried and showed that worked was just ask a different AI system, or even the same AI system, is there a problem with this answ…”
Saunders: Departure from OpenAI was not caused by witnessing immediate harm
“If there were, if there was a group of people that I knew were being seriously harmed by this technology first, I still really hope that open AI would like do the right thing and address this. If this was like a very clear cut case. I also personally would you…”
Saunders: GPT-4 is safe, but GPT-5 or later might be the Titanic
“I don't think that I was working on the Titanic. I don't think that GPT four was the Titanic. I'm more, I'm afraid that like GPT five or GPT six or GPT seven might be the Titanic in, in, in this analogy.”
Saunders: Most OpenAI employees and alignment efforts are English-focused
“Most of the people at opening I speak English. Most of the alignment work is done in English.”
Saunders: Small model oversight of large models clearly misses violations
“Some systems that the company has talked about involve using a very like small and dumb language model to like monitor what a larger language model is doing. And this will like clearly miss a bunch of things.”
Lessig: California Law Treats Vested Equity as Unconditional Wages
“Equity Is wage in the state of California. If you earn equity, your vested equity, it's like your wages. And when you leave, they can't say, oh, here's a bunch of other additional agree terms you must agree to in order to take what you've already earned. So th…”
Lessig: Ex-OpenAI Staff Remained Silent Fearing Equity Clawbacks
“I mean, I've spoken to ex-employees who've said, look, I've not done X, Y, and Z because I feared the club acts. So they can say, and they never enforced it. They didn't need to enforce it to have the effect, which it had for a significant number of people”
Lessig: Failing to Follow Stated AI Safety Regimes Is Reportable to SEC
“Some agencies like the SEC takes the view that most anything could potentially be the sort of thing you'd have privilege to complain to the SEC about because it could potentially affect the value of the company. And to the extent it's potentially affecting the…”
Saunders: California SB 1047 creates a superior model for AI whistleblowing
“I think that like a model that I think might work, you know, I think I personally think would work better would be like, The model more proposed in California Senate bill, 1047, where there would be like a, you know, the, where the law would create like, you k…”
Lessig: Federal AI whistleblower framework requires new legislation, not just FTC authority
“I mean, you know, agencies like the FTC believe they have lots of inherent jurisdiction that they could set up something that would be close to this, but the kind of thing that would convince people like William would require legislation the way California has…”
Saunders: Many at OpenAI discuss a three-year timeline to AGI
“And you know, what people were talking about at the company in terms of timelines to something dangerous were like, there were people talking, a lot of people talking about similar things to like the predictions of like, Leopold Aschenbrenner, where it's like …”
Lessig: 'Right to Warn' only allows public disclosure if reporting channels fail
“The right to warn we were talking about actually talked about creating an incentive so that no confidential information would be released to the public. If they had this structure, you know, imagine a portal again, where you can connect with the company and wi…”
Lessig: Adopting 'Right to Warn' will give AI labs a talent advantage
“If a company were to embrace the right to warn the way we've discussed it there would be a lot of people like William or others who would say, that's the kind of company I want to work for. And so that company would achieve an advantage of talent that might, y…”