Apr 25, 2023 · 35m · no-priors
No Priors Ep. 12 | With Noam Shazeer
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of No Priors, Transformer co-author and Character.AI co-founder Noam Shazeer joins Elad Gil and Sarah Guo to discuss the architectural mechanics of deep learning, scaling frontiers, and the creation of customizable conversational AI personas.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 20.3% of the talking time here. How this is scored →
The hosts as informed peer 4.2 Guest teaching 5.3 Guest disagreement 1.9 The hosts pushing back 1.2
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 7:34 Noam dismisses non-text modalities
Noam casually dismisses research excitement around computer vision and multimodality, insisting that pixel data is inefficient compared to dense text representations.
Hardest push from the hosts ▶ 9:56 Sarah challenges compute scaling limits and data exhaustionSarah directly questions the feasibility of unbounded scaling, pushing Noam on undertrained models and internet text exhaustion.
Biggest teaching moment ▶ 4:17 Noam breaks down the fundamental mechanics of attention and parallelismNoam details why RNNs bottleneck modern hardware sequentially and how attention tables enable constant-step parallel backpropagation.
The host holds their own ▶ 0:11 Elad demonstrates granular knowledge of Noam's Google engineering historyElad cites specific Google ad systems written by Noam and George Herrick, demonstrating firsthand institutional knowledge from his own Google tenure.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Noam Shazeer's Early Background at Google | 6 | 5 | 1 | 1 | Elad displays deep domain familiarity with Google internal systems, citing specific early ad clustering architectures developed by Noam and George Herrick. Noam provides foundational context on early deep learning hardware constraints and language modeling formulation. | |
| Architectural Mechanics: Transformers Versus Recurrent Neural Networks | 5 | 7 | 1 | 0 | Noam delivers a lucid masterclass on the operational distinction between sequential RNN computations and parallelizable attention tables across sequence lengths. Sarah adds an illustrative framing on non-linear word mapping in machine translation. | |
| Multimodal AI Applications and Scalability Frontiers | 5 | 6 | 4 | 2 | Elad probes for scaling asymptotes and unexpected modalities like AlphaFold, but Noam dismissively doubles down on pure text, arguing information density per pixel makes images inefficient by comparison. He rejects the premise that an architectural wall exists. | |
| Data Availability, Synthetic Generation, and Model Memory | 4 | 5 | 3 | 2 | Sarah challenges Noam on data exhaustion and undertrained models, but Noam brushes off data scarcity concerns via conversational volume and synthetic AI generation. He playfully dismisses hallucination concerns by declaring them a feature for creative products. | |
| Internal Development of Meena and LaMDA at Google | 4 | 6 | 1 | 1 | Elad and Sarah ask about Google's internal development of Meena and LaMDA and reasons for withholding launch. Noam provides internal history on Daniel de Freitas panhandling TPU compute credits and big tech's asymmetrical risk profile. | |
| Founding Character.AI and Recruiting Philosophy | 3 | 6 | 2 | 1 | Sarah and Elad explore Character.AI's inception and hiring criteria. Noam critiques sanitized single-persona corporate assistants like Siri and Alexa, explaining that universally inoffensive personas are inherently boring to users. | |
| Emerging User Behaviors, Parasocial Dynamics, and Emotional Connection | 3 | 5 | 2 | 0 | Sarah inquires about the depth of emotional coherence in character interactions. Noam demystifies the requirement for high-order linguistic intelligence for emotional bonding, comparing conversational agents to dogs providing emotional support. | |
| Platform Engagement Metrics, Monetization Strategy, and Scaling | 5 | 4 | 2 | 2 | Noam jokes about losing money on every user before laying out subscription monetization plans. Elad provides historical context, noting how early commercial platforms like eBay functioned as informal social networks due to emergent user habits. | |
| Safety Guardrails and the Dual Focus on Product and AGI | 4 | 5 | 2 | 2 | Elad asks whether AGI is an explicit corporate objective or an accidental byproduct. Noam articulates his thesis of building a company that is simultaneously product-first and AGI-first by making core product performance strictly dependent on AI capability. | |
| Founder Advice, Character Design Mechanics, and Hiring Plans | 3 | 4 | 1 | 1 | Sarah and Elad ask for tactical advice on character creation and startup hiring. Noam explains that famous personas require minimal prompting because the model already has strong priors, whereas lesser-known characters require few-shot dialog prompts. |
Statements from this episode (24)
Deep Learning Succeeded Because It Matches Modern Chip Hardware
“Really the key insight is that what makes deep learning work is that it is really well suited to modern hardware where, you know, you have the current generation of chips that are great at Matrix multiplies and, you know, other, other forms of things that requ…”
Shazeer Calls Next-Token Prediction an AI-Complete Problem
“And like, the problem is super simple to define. It's just like predict the next word, the fat cat sat on the, you know, like, okay, well, you know, what comes next? Like it's extremely easy to define. And if you can do a great job of it, like, you know, then …”
Transformers Beat Recurrent Models by Processing Whole Sequences at Once
“The magic of transformer kind of like convolutions is that you get to process the whole sequence at once. I mean, it still talks, you know, it's still a function of like the, you know, the predictions for the later words are dependent on what the earlier words…”
Self-Attention Brought GPU Parallelism to Sequence Modeling
“The insight here was, Hey, you can use the same attention thing to like, look back at the past of the sequence that you're trying to produce. And you know, the beauty is that the, that It runs great on on GPUs and CPUs, and it's kind of parallel to, like, how …”
Shazeer Estimates Text Is 1,000 Times More Information-Dense Than Images
“An image is worth a thousand words, but it's like a million pixels. So like the text is like a thousand times as dense.”
Core AI Intelligence Will Primarily Emerge From Text Models
“But I think that a lot of the core intelligence is going to come from these text models.”
AI Scaling Has Not Hit a Wall and Will Keep Improving
“So, and at the same time, like, I don't think anyone's seen a wall in terms of how good this stuff is. So I think it will just, it's just gonna keep getting better. I don't know what stops it.”
AI Data Needs Scale With the Square Root of Compute
“The data requirements tend to go up like with the square root of the amount of computation, because you're going to train a bigger model and then you're going to throw more data at it.”
AI Models Can Generate Synthetic Data to Avoid Looming Shortages
“I think I'm not that worried about Coming up with data and I feel like we could probably like just generate some more with the AI.”
Character.AI Prioritizes Model Memory So Chatbots Can Remember Users
“Some of the things we want to work on the most are like memory cause you know, our users definitely want their virtual friends to remember them.”
Researchers Will Eventually Solve the AI Hallucination Problem
“Yeah, I think, yeah, there's like a ton of great work going on in you know, in trying to figure out what's real and what's hallucinated, of course. I think we'll solve this.”
Daniel de Freitas Built Google's Meena as a Scrappy 20% Project
“Like he started the thing as like a 20% project where like people are encouraged to spend 20% of their time, like doing whatever they want. So and then he just, like, recruited, like, an army of, like, 20% helpers who were, like, ignoring their day jobs and, l…”
Google's LaMDA Was Originally Just a Renamed Meena Chatbot
“So that was like a renaming of Mina. So I guess I went and helped Daniel on Mina. We got it on some giant language models and then kind of became like an internal, like a viral sensation and then got renamed to Lambda.”
Big Tech Avoids Releasing Open-Ended Chatbots Due to Asymmetric Risk
“I mean, I think just like large companies have concerns around like launching products that can say anything. I mean, it's probably just a matter of like I w I would guess it's just like a matter of how much you're risking versus how much you have to gain from…”
Conversational AI Platforms Should Let Users Define Their Own Use Cases
“This is kind of a technology that that's so accessible that like billions of people can just invent use cases, you know, and like, it's so flexible that, you know, you really just want to put the user in control because often they know way better than you do w…”
AI Assistants With a Single Neutral Corporate Persona End Up Boring
“Like if you're like such a public, you know, trying to present like one public persona that everyone likes, you're going to end up just being boring essentially. And people just don't want boring. You know, people want like the, you know want to interact with …”
RPGs, Anime, and Gaming Personas Dominate Character.AI Usage
“There's like a lot of you know, there, there's a lot of, you know, role-playing, like role-playing games are big, you know, like, you know, like text adventure where it's just like making it up as it goes, there's a lot of like video game characters and anime …”
Many People Turn to Character.AI for Companionship When Lonely
“We also see, like, a lot of
You know, people using it cause they're, you know, they're lonely or troubled and need someone to talk to.”
Providing Emotional Connection Does Not Require High AI Intelligence
“Probably you don't need that high end, like level of intelligence, like
To, you know, to do emotion.”
Active Character.AI Users Spend About Two Hours Per Day on Average
“Somebody who's on the site today is active for about two hours on average today. That's of people who send a message today, which is pretty wild.”
Character.AI Enforces Filters Blocking Pornography, Violence, and Self-Harm
“Like, you know, we don't want to encourage people to, like, harm, you know, hurt themselves or hurt other people or you know, we're blocking porn.”
AGI-First Companies Must Make Product Quality Depend Entirely on AI
“The way you have a company that's both AGI first and product first is that you make the, make your product depend entirely on the quality of the AI. Like the most, the biggest determining factor in the quality of our product is, is how smart things gonna be.”