Sep 20, 2024 · 48m · a16z
“The Future of AI is Here” — Fei-Fei Li Unveils the Next Frontier of AI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The a16z Podcast, AI pioneers Dr. Fei-Fei Li and Justin Johnson discuss the evolution of artificial intelligence and unveil their new venture, World Labs. They explain why spatial intelligence—the ability for AI to perceive, reason about, and interact within 3D and 4D environments—represents the next major frontier beyond large language models.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The host as informed peer 3.7 Guest teaching 4.3 Guest disagreement 0.6 The host pushing back 1.4
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 10:46 Fei-Fei Li playful rejection of Martin's premise on human data
When Martin asserts that self-attention models rely on implicit human labeling, Fei-Fei playfully responds that she knew he would say that, directly pushing back that his point applies far more to language than to pixels.
Hardest push from the host ▶ 10:38 Martin Casado challenges the compute-only narrative using data labeling argumentsMartin positions himself as a naive listener to directly challenge the compute-driven 'bitter lesson' narrative, arguing forcefully that human-labeled structure in CLIP alt-tags and text data is what truly unlocks deep learning.
Biggest teaching moment ▶ 27:08 Justin Johnson explains the fundamental 1D limitation of multimodal LLMsJustin educates the host on model architectures, explaining that LLMs operate on 1D token sequences and shoehorn visual inputs, whereas true spatial intelligence requires 3D representations built into the model's core.
The host holds their own ▶ 9:16 Martin Casado details data-centric unlocks against compute-centric assumptionsMartin demonstrates deep familiarity with AI history by contrasting the 'bitter lesson' compute argument with specific data-centric counterexamples like ImageNet, sentence structure, and CLIP alt-tags.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| Disclaimer and a16z Podcast Opening Bumper | 2 | 2 | 0 | 0 | Martin opens with broad introductory questions inviting Fei-Fei Li and Justin Johnson to share their personal backgrounds in AI. The conversation is welcoming and collaborative without pushback. | |
| Fei-Fei Li's Background and the Genesis of ImageNet | 2 | 4 | 0 | 0 | Fei-Fei Li recounts her transition from physics to computational neuroscience and explains the realization that internet-scale data was the key overlooked driver of model generalization, leading to ImageNet. | |
| Breakthrough Epochs: The Role of Compute and AlexNet | 4 | 6 | 1 | 2 | Martin prompts a discussion on whether breakthroughs come from algorithmic unlocks or compute. Justin illustrates the staggering compute scaling since 2012 by showing AlexNet's training time reduced from six days to under five minutes on modern GPUs. | |
| Supervised Learning vs. Unsupervised Data and Implicit Labeling | 6 | 5 | 3 | 6 | Martin actively challenges the compute-centric narrative by raising data structure and implicit human labeling in CLIP and Transformers. Fei-Fei playfully reframes his argument, noting implicit human labeling holds far more strongly for language than for pixels. | |
| The Evolution of Generative AI and Justin Johnson's PhD Milestones | 4 | 5 | 1 | 1 | Martin asks the guests to trace the shift from predictive computer vision to generative AI. Fei-Fei and Justin map out Justin's PhD milestones spanning image matching, style transfer, and early scene-graph-to-image GAN generation. | |
| Founding World Labs and the Pursuit of Spatial Intelligence | 3 | 4 | 0 | 0 | Martin inquires about the transition from academic research to founding World Labs. Fei-Fei explains her pursuit of visual-spatial intelligence as a fundamental North Star alongside language. | |
| Defining Spatial Intelligence and 3D/4D World Representation | 4 | 5 | 0 | 2 | Martin asks for a precise definition of spatial intelligence and whether it applies to physical or abstract spaces. Justin defines it as perceiving, reasoning, and acting across 3D/4D space-time and details the pivotal impact of NeRF. | |
| Contrasting 1D Language Models with 3D Spatial Representation | 5 | 6 | 2 | 3 | Martin pushes the guests on why multimodal LLMs cannot simply handle spatial tasks. Justin clarifies that LLMs shoehorn data into 1D sequences, whereas true spatial intelligence requires 3D representations front and center. | |
| Use Cases for Spatial Intelligence: Interactive Worlds, AR, and Robotics | 4 | 4 | 0 | 1 | Martin guides the discussion toward concrete commercial applications. Justin and Fei-Fei explore interactive 3D world generation, new media experiences, augmented reality interfaces, and physical robotics. | |
| Deep Tech Platform Strategy and Assembling World Labs' Team | 4 | 3 | 0 | 1 | Martin asks how World Labs balances deep tech platform ambitions with specific vertical applications. Fei-Fei outlines their foundational deep tech positioning and describes recruiting co-founders Ben Mildenhall and Christoph Lassner. | |
| Reaching the North Star and the Expanding Frontier of AI | 3 | 3 | 0 | 0 | Martin asks how the team will evaluate reaching their ultimate goal. Fei-Fei highlights real-world deployment impact, while Justin emphasizes that understanding a 4D universe is an expanding, infinite journey. |
Statements from this episode (15)
Fei-Fei Li: Spatial intelligence is as fundamental to AI as language
“Visual spatial intelligence is so fundamental. It's as fundamental as language.”
Fei-Fei Li: AI is undergoing a Cambrian explosion beyond text
“And now I think we're in the middle of a Cambrian explosion. In almost a literal sense, because now in addition to texts, you're seeing pixels, videos, audios, all coming out with possible AI applications and a model.”
Justin Johnson: AlexNet trained for six days on two consumer GPUs
“That AlexNet was a sixty million parameter deep neural network and it was trained for six days on two GTX five eighties, which was the top consumer card at the time, which came out in 2010.”
Justin Johnson: AlexNet's 6-day training takes under 5 minutes on GB200
“So I ran the numbers last night, like that two week training run, that of six days on two GTX five eighties, if you scale, it comes out to just under five minutes on a single GB 200.”
Fei-Fei Li: AlexNet's architecture mirrored 1980s models, unlocked by GPUs
“The 2012 AlexNet paper on ImageNet Challenge is literally a very classic model, and that is the convolutional neural network model, and that was published in 19 eighties, the first paper. I remember as a graduate student learning that, and it more or less also…”
Fei-Fei Li: Language data contains more implicit human labeling than pixels
“Yes, philosophically, that's a really important question. But that actually is more true in language than pixels.”
Justin Johnson: AI is shifting from analyzing web data to sensor data
“The previous decade had mostly been about understanding data that already exists. But the next decade was going to be about understanding new data.”
Justin Johnson: Ben Mildenhall's 2020 NeRF paper ignited 3D computer vision
“In 2020, you asked about breakthrough moments. There was a really big breakthrough moment from our co-founder Ben Mildenhall at the time with his paper, NERF neural radiance fields. And that was a very simple, very clear way of backing out three D structure fr…”
Justin Johnson: Multimodal LLMs shoehorn visual data into 1D token sequences
“And now the multimodal LLMs that we're seeing now, you kind of end up shoehorning the other modalities into this underlying representation of a one D sequence of tokens. Now when we move to spatial intelligence, it's kind of going the other way. Where we're sa…”
Fei-Fei Li: The 3D world follows physics, unlike human-generated language
“Language is fundamentally a purely generated signal. There's no language out there. You don't go out in the nature and there's words written in the sky for you. Whatever data you feed in, you pretty much can just Somehow regurgitate with enough generalizabilit…”
Justin Johnson: Native 3D AI representations will outperform 2D video generation
“Modeling the two D projections of a dynamic three D world is, is a function that probably can be modeled, but by putting a three D representation into the heart of a model, there's just going to be a better fit between the kind of representation that the model…”
Justin Johnson: Creating interactive 3D worlds currently costs hundreds of millions
“Because we already have the ability to create virtual interactive worlds but it costs hundreds and hundreds of millions of dollars and a ton of development time.”
Fei-Fei Li: Spatial intelligence will be the operating system for AR/VR
“Suddenly this piece of technology is, is going to be the operating system, basically for AR, VR MixR.”