Mar 27, 2023 · 39m · another-podcast
GPT-4 is here, now what?
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Tech analyst Benedict Evans and co-host Tony Cameron Brown analyze the launch of GPT-4 and the broader generative AI wave, evaluating technological mechanisms, reliability risks, historical parallels, and economic disruptions.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 86.5% of the talking time here. How this is scored →
The hosts as informed peer 8.0 Guest teaching 0.0 Guest disagreement 0.2 The hosts pushing back 1.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 29:29 Questioning shifting value of non-text content
Brown challenges the trajectory of automated creation by proposing that audio and video will appreciate in value as text output becomes commoditized.
Hardest push from the hosts ▶ 29:46 Instant rejection of multimedia scarcity theoryEvans bluntly counters Brown's suggestion with a direct 'No', arguing that generating video and audio will rapidly become just as trivial as text, citing viral Balenciaga parodies.
Biggest teaching moment ▶ 26:33 The second coming of Gutenberg parallelBrown brings in fresh reporting on copyright litigation framing generative models as Gutenberg's printing press putting publishing power into public hands.
The host holds their own ▶ 23:10 Formulating the error detectability matrixEvans synthesizes the conversation on model hallucinations into a clean 2x2 decision framework based on whether an error matters and whether the user is capable of spotting it.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Three Conceptual Frameworks for Understanding AI's Magnitude | 8 | 0 | 0 | 1 | Evans establishes a comprehensive three-tier analytical framework comparing generative AI to the iPhone/web, the PC/mainframe shift, and existential atomic risk. Brown acts as an agreeable co-host facilitating Evans's deep-dive monologue. Evans recounts pushing back forcefully against skeptical journalists. | |
| AI Infrastructure, Engineering Realities, and Model Economics | 8 | 0 | 0 | 0 | Evans details the economics of model training, RLHF engineering graft, custom silicon on edge devices, and current cloud latency bottlenecks. Brown asks clarifying questions regarding true game-changing use cases without challenging Evans. The dynamic remains collaborative and host-led. | |
| Pattern Generation, Structural Understanding, and Visual Hallucinations | 8 | 0 | 0 | 0 | Evans elaborates extensively on how LLMs and diffusion models generate statistical patterns rather than semantic understanding, detailing his prompt experiments with French sports cars, galleys, and cliffs. Brown actively listens and offers validating interjections. Evans connects error convergence to autonomous vehicle analogies. | |
| The Risk Matrix of AI Errors and Contextual Reliability | 8 | 0 | 0 | 1 | Brown raises concerns regarding the false sense of accuracy in newer AI image iterations. Evans builds on this by formalizing a 2x2 risk matrix analyzing whether errors matter versus whether users can detect them. Evans cites specific examples including his own hallucinated biography and medical query risks. | |
| Scaled Misinformation, Gutenberg Parallels, and Software Stacks | 8 | 0 | 0 | 1 | Brown introduces the historical Gutenberg printing press analogy regarding democratized distribution. Evans acknowledges this and extends the concept by contrasting crypto infrastructure development with AI building directly atop the modern consumer internet stack. Evans quotes Steven Sinofsky and discusses incumbent platform strategies. | |
| Labor Automation, Historical Precedents, and Cognitive Frontiers | 8 | 0 | 1 | 3 | When Brown suggests audio and video content might become uniquely valuable as text generation becomes commoditized, Evans immediately dismisses the premise by pointing out multimodal generation will be just as cheap. Evans draws on 1960s tabulating machine operators and the history of automation to contextualize potential cognitive labor displacement. |
Statements from this episode (12)
Evans: Generative AI is at minimum as big as prior ML wave
“It changes an awful lot of how tech works and what tech products you'll use. And there'll be all sorts of stuff that's happening inside stuff you use you don't even know is using this, which is also what happened with machine learning. So at the minimum, it's …”
Evans: Generative AI does not work for general web search
“I mean, it also doesn't work for general web search, which is what we can come back to, but it seems kind of hilarious, relevant to relative to all the other excitement and attention that's going on.”
Evans: Midjourney is not a product yet, just a Discord channel
“So they've kind of productized mid journey because mid journey basically isn't a product at the moment. It's a discord channel.”
Evans: Rebuilding ChatGPT from scratch takes 6–12 months and tens of millions
“How long would it take you to make a chat GPT again yourself from zero? And it's like six months to a year and however many tens or hundreds of millions of dollars. So, you know, it's not quite, you know, anyone can do that at home, but it's like, it's also no…”
Evans: Generative AI generates patterns without structural understanding of underlying objects
“It's making a pattern without any structural understanding of what the actual thing actually is.”
Evans: Near-zero AI error rates render semantic understanding questions practically irrelevant
“And as the error rate goes down, you get this philosophical conversation, which is what gets you to AI risk, where people are saying, look, you throw another hundred X more compute and 200 X more data at it, and the error rate goes to 90 X percent, 99.98%.
At…”
Evans: AI mistakes are easier to hide in images than in text
“The point is that, that images are both much higher bandwidth, but also much higher noise. And so the mistakes can be kind of hidden in the noise that, like, the clouds aren't quite right. If you're not a meteorologist, you don't know, and you don't, and you a…”
Evans: Closing AI models cannot stop proliferation because machine learning is out
“They basically, they suddenly stopped disclosing anything and they say this is because they worry this is going to lead to producing Skynet. And so they want to keep the whole thing under control, which is, is, is another topic, which is that the whole thing i…”
Evans: AI utility depends on whether errors matter and can be spotted
“This is a two by two matrix, which is, you know, does the error matter and can you spot the error?”
Evans: Crypto builds infrastructure stacks while ML builds atop existing stacks
“Creation of the stack with the enthusiasm for crypto, whereas creation at the top of the stack is the enthusiasm for machine learning.”
Evans: Incumbents treat AI as a feature, but disruption requires new products
“The incumbent's always trying to make the new thing a feature. And so that's very obviously what's happening with Microsoft and Google, including this in PowerPoint and office and of course in search and everything else. But what actually happens is, is then p…”
Evans: AI summarizing news without links raises deeper issues than search engines
“If you're going to, you know, build a new system that will just tell you what the story was without linking or giving you credit, that's very different from you know, what Google and Facebook do, which is you search for a new story and they tell you where to r…”