Jan 2, 2019 · 29m · a16z
a16z Podcast | The Dream of AI Is Alive in Go
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the a16z podcast, host Sonal Chokshi and guests Steven Sinofsky and Frank Chen discuss the rapid evolution of artificial intelligence, sparked by AlphaGo's historic victory over Lee Sedol. They explore the transition from expert systems to data-driven deep learning, the underlying hardware infrastructure driving progress, and practical engineering trade-offs in modern AI development.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The host holds 12% of the talking time here. How this is scored →
The host as informed peer 5.0 Guest teaching 3.7 Guest disagreement 1.3 The host pushing back 2.9
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 10:04 Steven's Frustration with Pure Taxonomy Labels
Steven expresses mock exasperation with purist academic debates over deep learning definitions, arguing that rigid classifications miss the practical point of AI breakthroughs.
Hardest push from the host ▶ 23:32 Host Refuses 'This Time Is Different' PremiseSonal explicitly refuses to accept the narrative that AI has solved generalized intelligence, demanding that the guests justify why this wave will not end in another AI winter.
Biggest teaching moment ▶ 4:41 Frank Explains Why Expert Systems FailedFrank delivers a clear history lesson detailing how 1990s expert systems hit a wall due to compute limits and inability to handle edge cases, educating the host on past AI failures.
The host holds their own ▶ 21:06 Sonal Corrects Entity Resolution DefinitionSonal directly intervenes during Frank's technical explanation to correct and expand his definition of entity resolution to cover name variations, earning instant validation from the guest.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| Historical Cycles of AI and Expert Systems | 2 | 5 | 1 | 1 | The host primarily facilitates by prompting the guests to explain why expert systems failed to live up to expectations in the 1990s. The guests educate the host on historical limitations including computational bottlenecks and edge cases in rules-based AI. | |
| The AI Winter and Hardware Evolution | 4 | 4 | 2 | 1 | The host displays context by citing venture capitalist Chris Dixon's theory on technology gestation periods and recognizing Jeff Hinton's role in deep learning. The conversation remains friendly and collaborative as the guests explain hardware supply chain shifts. | |
| Defining the AI Taxonomy: Deep Learning and Neural Nets | 5 | 4 | 1 | 4 | The host takes control of the segment by stopping the conversation and asking the guests to explicitly establish a taxonomy for AI, deep learning, and machine learning. She also demonstrates domain awareness regarding the role of GPUs and brain modeling in neural networks. | |
| Neural Network Variants and Computer Vision Evolution | 5 | 3 | 1 | 2 | The host highlights connections between artificial neural networks and cognitive psychology theories on human memory. The guests walk through specialized neural net architectures like recurrent and adversarial networks in a lighthearted, collaborative tone. | |
| Machine Learning Realities: Debuggability, Self-Driving Cars, and NLP | 6 | 3 | 1 | 2 | The host actively synthesizes the discussion on data-driven models and offers insightful commentary on how hybrid approaches resolve natural language ambiguity. The dynamic is peer-to-peer and constructive as the guests elaborate on debuggability in machine learning. | |
| Combining AI Techniques and Practical Engineering | 6 | 3 | 1 | 3 | The host demonstrates sharp expertise by interrupting to refine the guest's definition of entity resolution to include name variations, which the guest explicitly confirms. The guests discuss how practical engineering and product goals differ from pure academic research. | |
| Why "This Time is Different" in Artificial Intelligence | 7 | 4 | 2 | 7 | The host forcefully challenges the core premise of the episode, stating she remains unconvinced that current AI advances are distinct from previous hype cycles. She also demonstrates technical knowledge by distinguishing Monte Carlo tree search from deep learning and emphasizing data ubiquity. |
Statements from this episode (15)
Sinofsky: Practical AI applications are reclassified as standard software
“The whole thing about AI was always, if you actually could find a practical use for one of the techniques, you cross it off the list of AI techniques, and it's like no longer AI.”
Chen: 1990s expert systems failed due to insufficient hardware compute
“Keep in mind what computational power was back then. We didn't have a lot of memory. We didn't have a lot of disk. We didn't have a lot of CPU cycles. And so the rate at which we could do these calculations and the amount of storage we had, like, it just never…”
Chen: Symbolics collapse triggered an AI winter and venture capital drought
“Not only did venture funding completely dry out, it was embarrassing to be a professor in that field for a long, long time because it just didn't work.”
Chen: Google powers DeepMind algorithms using commodity x86 PC hardware
“Like if you look at an x-a-t-e-s server in a data center, exactly the servers that Google is using to compute deep mind algorithms, they're PCs.”
Chen: DeepMind's AlphaGo uses an ensemble of algorithms centered on deep learning
“It is the heart of the Go algorithms, although interesting to point out, it's an ensemble of techniques that's working for Go.”
Sinofsky: Neural network theory dates back to the 1956 Dartmouth AI Conference
“Neural networks aren't new. In fact, they're actually, if you go back and read the 1956 Dartmouth Summer AI Conference, they're actually mentioned in there as one of the first things that that group of people who basically invented the field. This is like Marv…”
Sinofsky: Stacking neural networks via Geoffrey Hinton's innovations drove modern AI
“What's happening right now, and since the innovations of Jeff Hinton, have been the ability to pile on a bunch of neural nets, one on top of another. And so maybe you dive in that like that, because that's the big math advance.”
Chen: Neural networks are trivially fooled by subtle noise in images
“So one really interesting thing is if you feed Pictures into a neural net and you tune in. You can defeat the categorization fairly trivially by introducing noise in the data, and the really interesting thing is you introduce the noise, you look at the resulti…”
Chen: Deep learning represents the triumph of data over algorithms
“So the big trend is the triumph of data over algorithms. Which is you try to make more and more sophisticated edge detection algorithms, feature recognition algorithms. The big advance with deep learning was, screw all that. I'm not going to try to figure out …”
Chen: Deep neural networks outperform decision trees but are inherently undebuggable
“In a decision tree, you can actually examine the decision tree and understand why a system made any single decision. Very, very easy to debug. The bummer is decision trees don't get you very good results. And so these deep networks get you much better results,…”
Chen: The next major AI breakthrough will combine multiple techniques
“Yeah, and I think this is where we're going to see the next big breakthrough. It won't be one technique in isolation, just like the go, Algorithms, one, on a combination of techniques.”
Sinofsky: Pure deep learning systems will emerge for translation and vision
“And that's an important point about just innovation in general, which is there will be massive innovation, and in fact, I fully expect to see pure deep learning approaches to translation, to image recognition, which is, you know, internet already is that, but …”
Chen: Deep learning enables robots to cook from YouTube videos
“There are systems where robots are learning to cook food by watching YouTube videos of people cooking food. There's systems that can take photos and paint them in the style of Renoir or Van Gogh. There's algorithms that can create paintings that are indistingu…”
Sinofsky: Cloud computing scale and internet data drive modern AI breakthroughs
“That is like, in a sense, the ultimate reason why all of these things are working is basically because of cloud computing, the scale of the architecture of cloud computing, and the internet that brings all that data in.”
Chen: Deep learning generates non-human insights that humans cannot debug
“So I think maybe this is the vector, which is these deep learning techniques, which we can't completely characterize and describe, much less debug, are leading to these flashes of insight, and they might not be human insight.”