Sutskever: Early neural networks failed primarily because they were too small
Ilya Sutskever · No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever · Nov 2, 2023 · at 2:32
OpenAI co-founder and Chief Scientist Ilya Sutskever discusses the historical intuition behind scaling deep neural networks ahead of the breakthrough AlexNet project in 2012.
0:00 / 0:20exact quote · 20.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The reason neural networks of the time weren't good is because they were too small. So like if you try to solve a vision task with a neural network, which has like a thousand neurons, what can it do? It can't do anything. It doesn't matter how good your learning is and everything else. But if you have a much larger neural network, you will do something unprecedented.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ilya Sutskever
Sutskever: AGI causing universal unemployment is 'not impossible'
“If you believe that the technology that we are building, AGI, could potentially be so capable as to do every single task that people do, does it mean that it might unemploy everyone? Well, I don't know, but it's not impossible.”
Sutskever: Transformers can reach AGI; alternatives only offer compute efficiency
“So it's better to think about it in terms of compute efficiency rather than in terms of, can it get there at all? I think at this point, the answer is obviously yes.”
Sutskever: A nonprofit cannot build large compute clusters for frontier AI
“The appetite for compute is truly endless as now clearly seen, but we realized that we will need a lot. And a nonprofit was, wouldn't be the way to get there. Wouldn't be able to build a large cluster with a nonprofit.”
Sutskever: AGI requires massive compute engineering projects, not small research efforts
“Because if you imagine how an AGI should look like, it has to be some kind of a big engineering project that's using a lot of compute, right? Even if you don't know how to build it, what that should look like, you know that this is the ideal you want to strive…”