Sep 4, 2024 · 1h 3m · news
Zico Kolter: OpenAI's Newest Board Member on The Biggest Questions and Concerns in AI Safety | E1197 · 20VC with Harry Stebbings
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode, CMU Professor and OpenAI Board Member Zico Kolter joins Harry Stebbings to discuss the technical realities of scaling artificial intelligence, debunking common data shortage myths, while offering a pragmatic, grounded approach to immediate AI safety threats, governance, and the future of public trust.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Harry holds 12.8% of the talking time here. How this is scored →
Harry as informed peer 2.4 Guest teaching 4.1 Guest disagreement 1.3 Harry pushing back 1.4
speaking balance: gold is Harry, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 45:08 Rejecting Nuclear Weapon Analogy
Zico directly rejects the host's premise comparing AI to nuclear weapons, explicitly arguing that nuclear weapons have only destructive utility while AI is a dual-use technology like nuclear power.
Hardest push from Harry ▶ 30:28 Challenging Impact on Scientific ProgressHarry directly challenges Zico's assertion about localized trust, pushing back on whether abandoning objective shared evidence would hinder global scientific advancement.
Biggest teaching moment ▶ 24:33 Explaining Cloud Data vs Model TrainingZico educates the host on enterprise misconceptions, explaining that using APIs or RAG architecture does not train or retain internal corporate data inside LLM weights.
Harry holds his own ▶ 7:32 Correctly Citing Video File SizesHarry demonstrates domain knowledge by instantly citing the precise 6.5 gigabyte file size of their video podcast recording to contrast against text transcript sizes.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Harry as informed peer | Guest teaching | Guest disagreement | Harry pushing back | Why |
|---|---|---|---|---|---|---|
| Guest Welcome and Zico Kolter's Professional Background | 1 | 2 | 0 | 0 | Harry asks for Zico's background and basic AI mechanics. Zico outlines his roles at Carnegie Mellon and OpenAI before explaining how LLMs predict next words. | |
| The Myth of the Data Shortage Crisis in AI | 2 | 4 | 1 | 1 | Harry questions whether AI is facing a data shortage crisis. Zico reframes the premise, noting that public text datasets are tiny and immense reserves of multimodal data remain untapped. | |
| The Computation Overhead and Value of Multimodal Data | 3 | 3 | 0 | 1 | Harry cites exact video file sizes to illustrate data scale. Zico details how compute overheads currently bottleneck the use of video and multimodal inputs. | |
| The Economics of Large General Models vs. Small Models | 3 | 4 | 2 | 2 | Harry references Cohere's views on plateauing model gains. Zico rejects the plateau thesis, attributing perceived stagnation to outdated benchmarks rather than real task performance like coding. | |
| Consolidation and Commoditization in the Model Market | 2 | 3 | 1 | 1 | Harry asks about rapid model commoditization and scaling limits. Zico explains that scaling laws still hold and that compute limits are driven by economic factors rather than technical plateaus. | |
| Defining AGI and its Multi-Decade Arrival Timelines | 2 | 4 | 1 | 1 | Harry asks about commercial incentives versus AGI goals. Zico defines AGI as an entity functioning like a virtual 1-year close collaborator and gives a 4 to 50 year arrival timeline. | |
| Empowering Workforces and the Labor Displacement Cycle | 3 | 5 | 2 | 3 | Harry asks if AI labor disruption is just a standard historical technology cycle. Zico clarifies worker augmentation and corrects widespread corporate misconceptions regarding cloud data privacy and training. | |
| The Frustration with Polarized AGI Debates | 2 | 5 | 2 | 1 | Harry highlights deepfakes and cyber risks. Zico reframes the primary danger of misinformation as a collapse in universal trust rather than people believing fake content. | |
| Scientific Progress and Trust in a Post-Truth World | 3 | 4 | 2 | 3 | Harry challenges whether localized trust slows global scientific progress. Zico notes that human knowledge advanced for centuries prior to objective media records like video. | |
| Social Platforms as the Arbiters of Truth | 3 | 4 | 1 | 2 | Harry questions if governments possess the technical sophistication to regulate AI. Zico advocates regulating downstream application harms using existing legal frameworks rather than underlying code. | |
| The Specification Problem and prompt Injection Risks | 2 | 5 | 1 | 1 | Harry prompts Zico for his primary AI safety concern. Zico explains the specification problem, comparing prompt injection to an unpatchable software buffer overflow. | |
| Jailbreaks as Risk Multipliers for Bad Actors | 2 | 5 | 1 | 1 | Harry clarifies how specification failures act as risk multipliers. Zico explains how jailbreaks drastically lower skill thresholds for bad actors creating zero-day cyber exploits. | |
| The Dual-Use Nature of AI and Open-Weight Regulation | 3 | 5 | 3 | 2 | Harry cites Scale AI's comparison of AI to nuclear weapons. Zico rejects the analogy, arguing dual-use nuclear power is a better comparison, and outlines risks surrounding open-weight models. | |
| Practical Safety vs. Rogue Sci-Fi doom Scenarios | 2 | 4 | 1 | 1 | Zico pivots away from sci-fi rogue AI scenarios to emphasize practical risks like correlated software failures in power grid SCADA systems. Harry connects this to recent CrowdStrike outages. | |
| Pragmatic Optimism and Safety as an Enabler | 3 | 5 | 2 | 1 | During rapid-fire questions, Zico expresses pragmatic optimism and boldly asserts that AI engineering is now post-architecture, meaning specific model structures no longer matter. |
Statements from this episode (45)
Kolter: AI will accelerate trust erosion until people believe nothing they see
“The real negative outcome is that people are not going to believe anything that they see. It didn't even need AI to get there, but AI is absolutely an accelerant for this process.”
Zico Kolter recently joined OpenAI's board of directors
“Also, I am recently on the board of OpenAI which I joined at this point a couple weeks ago, and it's been extremely exciting as well.”
Kolter: LLMs are demonstrably intelligent despite being next-word predictors
“Oftentimes I know people say, oh, well, AI is, it's just predicting words. That's all it's doing. Therefore it can't be intelligent. It can't be. And I think that's just. Demonstrably wrong. What I think is amazing, though, is the scientific fact that when you…”
Kolter: Word prediction intelligence is the top scientific discovery in decades
“You can train word predictors, and they produce intelligent, coherent, long-form responses. This is one of the most notable, if not the most notable, scientific discovery of the past 1020 years. Maybe much longer than that, right?”
Kolter: Public AI models are trained on around 30 terabytes of text data
“Public models are trained on the order of, you know, 30 terabytes of data or something like this, right? So 30 terabytes of text data.”
Kolter: AI progress faces compute bottlenecks rather than a data shortage crisis
“We are nowhere close to hitting the limits of available data in these models. Arguably, we're unable to process it because we don't have enough compute and things like this, but we're nowhere close to data limits in other senses.”
Kolter: Multimodal AI data is abundant but constrained by compute bottlenecks
“There are massive amounts of data available, And I think we have not yet figured out how to properly leverage those due to either limitations of compute. I mean, you have to process all that data and it does take, we don't have current models to do this very w…”
Kolter: Larger AI models perform better on fixed datasets without plateauing
“But it is also true that if you just take a fixed data set and run over it multiple times, if you use a bigger model, it will often work better, right? So I think that we have not really reached the plateau there.”
Kolter: Machine learning algorithms fail to extract maximum data value
“What that means is our current algorithms, we are not yet maximally extracting the information from data we have. And there are way more deductions and inferences and other processes that we can apply to our current data to provide more value.”
Zico Kolter works almost exclusively with the largest available AI models
“I work almost exclusively with the largest models that are available to me because it just works better, and when I don't have a given task that I'm doing over and over, when I want to have that generality, I want to work with the larger models that are availa…”
Kolter: Small models become valuable when specialized for repeated tasks
“Once you have a task, a rote task that you're repeating again and again enough times, And you know a small model can do it. It probably does become valuable to specialize a small model for that task only.”
Kolter: AI model performance is not plateauing in coding tasks
“The domain I use models for most probably is coding, and also doing things like transcribing lectures and stuff like this. On those tasks, I am absolutely not seeing plateauing gains. The latest models, they are notably better than the previous iteration, and …”
Kolter: Perceived AI plateaus stem from limited user imagination, not models
“So I think this perception has more to do with people's limited imagination of what they can do with these models, and less to do with the models themselves. But that will evolve over time. People will start figuring out you can use them for better and better …”
Kolter: AI foundation model landscape will likely undergo market consolidation
“And I think there will be most likely consolidation, but I'm not quite sure how it will play out.”
Kolter: Training custom AI models from scratch will become economically unviable
“Companies that are right now thinking about training their own models and things like this, and it's just sort of the default that of course you would do this, that this must, won't be an economically viable thing to do. In the future, and so it won't happen a…”
Kolter: AI scaling limits are financial and practical, not physical
“I'm not really sure what the rationale is for saying that we've, we plateaued in the compute sense. Most scaling laws that I've seen certainly suggest it can keep going. It's more expensive. You could argue that it's by far just scaling is, may not be the most…”
Kolter defines AGI as equivalent to a year-long human collaborator
“I define AGI as a system that acts functionally equivalent to a close collaborator of yours over the course of about a year long project.”
Kolter predicts AGI will arrive within 4 to 50 years
“And I am massively uncertain as to when this will happen, but a massive shift that I've undergone is I think this will probably happen in my lifetime. I think the answer to AGI has always been in academia, not in my lifetime. And the timeframe I give this righ…”
Kolter: Winning companies will leverage workforce with AI, not fire them
“The winners in this new world Will the companies that say survive and thrive and become dominant in this new world? Not talking about the AI companies for now, talking about sort of the rest of the companies that the ones that people worry that they're going t…”
Kolter: Current AI products fail to let workforces maximize AI potential
“Do we have AI products that are able to be maximally used by workforces? And the answer to this is right now is no. Clearly there is a gap between what People could use these things for and what they're using them for right now.”
Kolter: RAG systems will remain essential despite fine-tuning advances
“RAG based systems are so Are so common here, and so, and probably will remain, even with the advent of fine-tuning availability, they're going to remain a useful paradigm.”
Kolter: Enterprise cloud LLM adoption will mimic cloud storage migration
“This is not in certain use cases, any riskier than just having your data in the cloud to begin with, which all of them typically do. They've all moved that way. So I think this will just happen naturally with progression of time.”
Kolter: Extreme certainty on either side of AGI debate is unwarranted
“The thing that frustrates me most, honestly speaking, is the degree of certainty that some people have about whether we will definitely get there very, very soon or even more on the flip side, that there's absolutely no way that we will ever achieve AGI with t…”
Kolter: Early AI optimists have been proven right by recent progress
“And the people that have been sort of ringing this bell for a while saying, look, this is coming, this is this, you know, they've, in many cases, in my view, been proven right. And I've updated my sort of posterior beliefs based upon the evidence I've seen.”
Kolter: AGI skeptics ignore overwhelming evidence of rapid AI progress
“And so what irks me the most about a lot of people's sort of philosophy of AGI is that, to a certain extent, how little It seems like observable evidence has changed their beliefs one iota. You know, they had certain beliefs about what it would take to get to …”
Kolter: Video will soon cease to be trusted as objective evidence
“Video is a short blip where we sort of think that there's some objective evidence for a hundred years of our history, and that's going to be now, that's no longer true pretty soon.”
Kolter: Detailed AI regulations become outdated within months
“A lot of the details about how those regulations sometimes evolve can be a bit misguided or miss the point or somehow Just when I read them, basically they're going to become dated in, in a matter of months, because they're dealing with things and they're appr…”
Kolter: Existing laws with minor tweaks can regulate downstream AI harms
“We have a much better handle on regulating the downstream uses of AI. Like, when it comes to misinformation, we already have laws that deal with sort of libel and things like this. In many cases, because AI is acting as an accelerator, there are situations in …”
Kolter: Current AI models cannot reliably follow developer specifications
“Right now, the mod, the AI models that we have, for lack of a better phrasing, are not able to reliably follow specifications.”
Kolter: AI models contain an unpatchable buffer overflow vulnerability
“This is sort of like these models have a buffer overflow in all of them that we know about, and most importantly, that we don't know how to patch and fix. We don't know how to fix this yet with models”
Kolter: Existing AI models can already analyze code to find security vulnerabilities
“I think this is essentially already solved in many cases by these models that can already solve and analyze code to find vulnerabilities.”
Kolter: Cyber attacks present a clearer AI risk than biological threats
“Personally, I think cyber attacks are a much more clear and present threat than, for example, bio threats and things like this.”
Kolter: AI's primary risk is lowering skill barriers for malicious actors
“The concern is that not that they can do this sort of autonomously, maybe initially, but that they can lower the bar so far in the skill level required to create these things that effectively it puts them in the hands of a huge number of bad actors.”
Kolter: Comparing AI to nuclear weapons is an overstated, flawed analogy
“The nuclear weapon analogy is actually not a great one, because nuclear weapons have one purpose, which is to destroy things. Maybe a better analogy is sort of nuclear technology, period, because it has the ability to create nuclear weapons, but it also has th…”
Kolter: Open-sourcing models at GPT-4 capability poses low catastrophic risk
“If you look at the current best models that there are right now, so things like GPT-IV, Claude, 3.5 Gemini, things like this, I would not currently be all that nervous about having an open source model that was as capable as these in terms of the catastrophic …”
Kolter: AI models will eventually reach capabilities that make open-sourcing unsafe
“I think there will come a time when a certain capability, a certain ability of these models Reaches the point that should give us pause when it comes to just turning these things over to whoever and what, however, they want to use them.”
Kolter: Closed-source models leading open-weight releases creates a necessary safety buffer
“At least for now, there's a constant stream of closed models that are released sometime before an equivalently capable open, open weight model, right? And I think this is actually a very good thing because my hope would be that, and we've sort of found ourselv…”
Kolter: AI safety should focus on practical risks, not sci-fi doom
“The first is that I think the vast majority of AI safety should not be about these topics. The vast majority should be about quite practical concerns we have on making systems safer, like the kind that I've talked with you about so far.”
Kolter: AI job loss is a more immediate concern than extinction
“I think killing jobs is much more immediate of a concern than killing humans.”
Kolter: We do not understand how AI models work internally
“We do not understand how these things work internally, the possible correlated failures, the possible attack factors, all sorts of things. We don't understand it.”
Kolter: AI safety is a necessary condition for using AI
“I want to develop and improve safety of these tools Because I want to use them. It's not that we have some moral imperative that we have to develop these tools. I mean, maybe there are, or we have to develop AI and AGI. Maybe that's true. But that's not what m…”
Kolter: AI model architectures and Transformers no longer matter
“I, for a lot of my career, was thinking that model architectures really mattered, and by having clever, complex architectures and sub-modules inside architectures, you would have, that was the route to sort of better AI systems. For the most part, I don't beli…”
Kolter: Modern AI progress comes from uncurated web data over manual labeling
“Kind of on the contrary, I thought that data was sort of, you know, data had to be highly curated to be valuable, and the value in data came essentially from very manual labeling of this data and human intensive curation. The big amazing insight Of current AI …”
Kolter: Bret Taylor invited him to join OpenAI board via email
“I got, actually the day before I started as department head, I got an email from Brett, the chair of the board, just saying, hey, do you want to talk about maybe joining the OpenAI board?”
Kolter: AI safety requires global collaboration rather than national competition
“I think there are certain things like, for example, AI safety, where we very much need to work as a world to help set standards and help better the future of everyone here. Because yes, certain things can be done by countries. Capabilities can maybe advance mo…”