Neural Nets
topic on 7 shows · 19 statements across 16 episodes
the Y Combinator Startup Podcast 
No Priors 
WTF is with Nikhil Kamath 
Capital Allocators 
Catalyst 
the a16z Podcast 
20VC
19 statements about Neural Nets, every show
Karpathy: Core LLM training algorithm requires only 200 lines of Python
“Training neural nets and LLMs specifically is a huge amount of code, but all of that code is actually complexity from efficiency. It's just because you need it to go fast. If you don't need it to go fast and you just care about the algorithm, then that algorit…”
Adcock: Humanoids Must Run Real-Time Motor Neural Nets On-Board
“We need to run
A decent amount of the neural nets at, ah, kind of like, basically a couple hundred hertz, like a couple hundred times a second.
There's no way you're gonna be able to do that fully off board.
The robot, ah, would just be too slow.”
Sinofsky: Geoffrey Hinton could not get neural net funding in 1989
“In, in, like, in 1989, like, Hinton couldn't get funded trying to do neural nets.”
Karpathy: Tesla Autonomous Stack Will Be Pure End-to-End Neural Net in 10 Years
“And I do suspect that The end-to-end systems for Tesla in, like, say, 10 years, it is just a neural net. I mean, the videos stream into a neural net and commands come out.”
Karpathy: Pure End-to-End Imitation Learning Lacks Sufficient Supervision Bits
“Actually, like, end-to-end driving, when you're just imitating humans and so on, you have very few bits of supervision to train a massive neural net. And it's too Too few bits of signal to train so many billions of parameters. And so these intermediate represe…”
Adcock: Figure robots execute tasks and speech end-to-end via neural nets
“We're doing that fully autonomous end to end on our robots now, all bipedal. And the second is we're doing kind of full consumer level manipulation and like speech to speech reasoning. So we're able to Talk with the robot. It's able to understand what we're sa…”
Beyang Liu: Statistical learning and convex optimization will re-emerge in AI
“I'm still waiting for the statistical learning and maybe some of the convex optimization stuff to reemerge. I wouldn't count it entirely out yet. I feel like the pendulum always swings back the other way. It's swung away from statistical learning and convex op…”
Kelly: Neural Network Analysis Will Function Like Systems Biology
“Like you're about to experience, like the analysis of these neural nets is going to look like systems biology, right? It's going to be like, go in and like, try to back, figure out a thing that you didn't design, my friends.”
Douwe Kiela: Humans will never fully understand large neural network outputs
“So we're not going to be able to really know why a neural net, what does what it does at the scale that neural networks operate at.”
Madaeni: LLMs are not equipped to build and train neural nets for forecasting
“But it's clear that I don't think that LLMs are equipped with, you know, building a whole neural net from scratch and training them and do some form of supervised, unsupervised learning for Forecasting.”
LeCun: Neural network research faced a decade of ridicule starting in 1995
“There was indeed about 10 years when not only nobody was interested in neural nets, but people were even making fun of it, you know, talking about it in sort of disparaging terms.”
LeCun: Deep learning pioneers conspired in early 2000s to revive neural nets
“Joshua, Jeff and I decided in the early 2000 to basically start a conspiracy to You know, revive the interests of the community in, in neural nets by making them work.”
IBM's Deep Blue proved that scaling search works in AI
“We learned that scale really does work. And in that case, it wasn't scaling, you know, training and neural nets, it was scaling search.”
O'Shaughnessy: Almost No Funds Have Pure Machine Learning Models in Live Production
“To be clear, I don't think almost anybody has a lot of that in production today, but those tools are becoming more and more useful.”
Sinofsky: Stacking neural networks via Geoffrey Hinton's innovations drove modern AI
“What's happening right now, and since the innovations of Jeff Hinton, have been the ability to pile on a bunch of neural nets, one on top of another. And so maybe you dive in that like that, because that's the big math advance.”
Chen: Neural networks are trivially fooled by subtle noise in images
“So one really interesting thing is if you feed Pictures into a neural net and you tune in. You can defeat the categorization fairly trivially by introducing noise in the data, and the really interesting thing is you introduce the noise, you look at the resulti…”
Brundage: Offense generally defeats defense in adversarial machine learning competitions
“If you look at the, like, offense and defense and competitions on adversarial examples, like, the offense generally wins. Like, we don't really know how to make neural nets robust against deliberate or even unintentional things that could mess them up.”
Brockman: AI hardware will advance faster than people expect
“Now, if you look forward to what's going to happen over upcoming years is the hardware for these applications for running neural nets really, really quickly are going to get fast, faster than people expect.”
Hwang: Visual and interface designers will see high demand in AI
“I think the second thing that's about to be in really strong demand is thinking about the visual dimension of this, right, which is, like, happens on a couple levels. That's both, like, the interface of how you work with machine learning systems, But also just…”