Deep Learning

“Basically there are three things that we know how to do to speed up deep learning in hardware. Still till this day. Increase the number of cores. Increase the communication between cores. Bring the memory closer to the compute. Those are the three things. That…”

“The world does not understand how to intuit exponentials, and so they missed this one.”

“The big idea of the company that was spot on is that it is possible to augment the CPU to solve problems that otherwise are too difficult to solve. And molecular dynamics is one of them. Image processing is one of them. Inverse physics is another one. And so a…”

“The breakthrough for us was realizing that AlexNet was not AlexNet. That AlexNet was an approach with deep, deep learning that allows you to learn any function.”

“At the time he had been working in bioinformatics, so he hadn't been working on deep learning or the things that we call AI these days.”

“And so everything you're doing with machine learning today, with parametric curves, we should be able to do it. With symbolic models in the future in a way that will be much, much closer to optimality. Much closer to optimality in the sense that you're going t…”

“You could not really get Gradient descent to encode sort of like reasoning style algorithms. It was not because the models could not represent these algorithms. It was because gradient descent could not find them, right? So the problem was that it wasn't about…”

“You have to break the combinatorial wall, and the way to do it is to add deep learning guidance. It's actually very similar to the principles that analyze something like AlphaGo or AlphaZero.”

“If you had thrown the same amount of investment into almost anything else, you would also have seen extremely exciting results, like genetic algorithms, for instance.”

“All of deep learning is, ah, doing correlations. It's not doing causation. Causal models are the ones that are able to do simulations and intervention.”

“I think deep learning is still in the Shannon entropy world. It has not crossed over to the Kolmogorov complexity and the causal world.”

“Especially in deep learning, a negative results doesn't mean something doesn't work. It means you haven't made it work yet often.”

“In deep learning, people laugh that ideas are cheap. Making them work is, is the hard part.”

“It's largely impossible to do good data analysis on multimodal data of this kind, unless you have really fine grained labeling of your images. For example, which is just very, very burdensome, but obviously deep learning, we can let the model figure out its ow…”

“And deep learning has been this miracle that keeps on giving, and we have kept finding, like, breakthrough after breakthrough. Again, when we got the reasoning model breakthrough, like, I also thought that was like, we're never gonna get another one like that.…”

“Some more computational photography, you know, plus on blending and some of these early techniques that you kind of get like 85% of the way there to something awesome, but not a hundred percent. And it wasn't until, you know, we really had deep learning that y…”

“Currently the pace of progress is very fast. Maybe also the ideas tends to work out a little bit more often than they did in the past. Because yeah, deep learning just wants to learn”

“It's not just convolutional neural network. It was also the first time that two GPUs were put together by Alex and his team. And were used for the computing of deep learning. So, it was really the first moment of data, GPUs, and neural network coming together.”

“A lot of the progress of AI can be attributed to some very simple, elegant ideas that have stayed around, and the vast majority of ideas that have been put out there haven't had staying power and haven't added a ton.”

“I actually don't think either is sufficient. I think some merger of the two is what's necessary to get to AGI.”

“So if somehow this whole deep learning paradigm is wrong and we just, like, totally missed the boat somehow, then I could see it happening and that's, I give it a 10, 20%.”

“There's maybe a couple of million deep learning engineers now in the world. In contrast, there's probably maybe only a few hundred like really great program synthesis folks across, across the world.”

“And one thing that is not widely known is that the single most cited paper in all of science is a paper on deep learning from 10 years ago, from 2015, and it came out of Beijing.”

“Meteorology is like something I actually think, like, deep learning should, like, change, right? Like, it sort of makes a lot of sense. Like, you know, deep learning should be very good at, like, you know, predicting, you know, turbulence and things like that.…”

“Deep learning, which is really the foundation of pretty much all of AI today. So basically, neural networks with multiple layers, right? The idea of this goes back to the 19 eighties and backpropagation. That's still the basic foundation of everything we do.”

“These deep learning based systems, they had flattened out in terms of if you added more user data or more parameters, they did not get better like the LLMs. There were no scaling laws. It's just like, it is what it is, and you could move at .2%. There's someth…”

“There is something about betting on deep learning that feels like being on the side of the angels, and you kind of just, it eventually seems to work out, even though you hit some big stumbling blocks along the way, and so like a deep belief in that has been go…”

“The more training data you put into LLMs, the better results you get, and how much better your results are actually predictable. It's kind of like a power law. This comes back from, you know, a paper from Baidu Research from back in 2016, I think, and it seems…”

“Weather modeling is kind of cracked with deep learning, but climate is quite different.”

“Anything can look however you want it to look if you put it on a log scale to a certain extent. And log, we love our log scales and deep learning for various reasons. Everything looks very clean on a log scale until everything looks very flat on a log scale.”

“Deep learning seemed to actually legitimately be working. And two, it got better with scale. We didn't know how predictably it got better with scale at the time, but it was clear that like bigger was better.”

“What I think we're starting to see right now is a the pendulum starting to swing back in the sense that there is a greater understanding that you really need a bit of both. You need that hugely powerful pattern recognition that we get from deep learning, but y…”

“Yeah, I was actually the first company to use deep learning in medicine, so I kind of founded the field.”

“When it comes to science, Just the applications of all this technology that's happening right now are still completely underexplored, whether it's, you know, using deep learning to get approximations of systems faster or figuring out how we can just accelerate…”

“Deep learning is, is amazing. And with that said, it's necessary, but not sufficient component.”

“The reason this whole field is working so well is because Now we have this magic hardware that's great at these dense matrix multiplications. And so you can do them like orders of magnitude faster than you can do anything that involves poking around in memory.”

“At the end of the day, in my mind, that's the one and only thing we know really works. If you want to push deep learning forward is to make it faster and more effective and more efficient on a given piece of Hardware.”

“I don't think GPUs are at the sweet spot when it comes to large-scale deep learning with respect to exactly those trade-offs, and so it may very well be that if we actually try these combinations, we might actually even quickly find something that's better.”

“There are people trying, and I think it's worth trying. I, I'm not super optimistic about that. I think it'll work for some cases, right, where it's simple enough that we can get it. I think there are many cases where it just isn't, right? Like, say, climate a…”

“I think pharma is investing way more in deep learning than people realize.”

“What I think we're starting to see right now is a the pendulum starting to swing back in the sense that there is a greater understanding that you really need a bit of both. You need that hugely powerful pattern recognition that we get from deep learning, but y…”

“Really the key insight is that what makes deep learning work is that it is really well suited to modern hardware where, you know, you have the current generation of chips that are great at Matrix multiplies and, you know, other, other forms of things that requ…”

“You know, you take deep learning out of you know, Google and Meta and a few other companies, and they crumble. I mean, they're completely built around it now.”

“One thing that's interesting to think about is that you can think of DNA almost like a one-dimensional image, and so you can use the exact same technology now to put in DNA sequences, and maybe now you're not identifying a face, you're identifying whether some…”

“A lot of the things that people think of as successes of deep learning are actually successes of combining deep learning with other things.”

“And at the end of the day, you know, you can't, like the European Union, mandate that every model has to be explainable, right? Because then, if you read that law literally, it makes deep learning illegal, right?”

“I agree that deep learning is not about to give us generalized intelligence, but it has really fundamentally changed the way that we interact with unstructured data.”

“These successes are examples of one thing. They're all examples of what a cognitive psychologist would call perceptual classification, and that's part of what we do as intelligent human beings, but it's not all that we do.”

“If a typical person can do a mental task with less than one second of thought, and we can gather an enormous amount of data, of directly relevant data, we have a fighting chance. So long as the test data aren't too terribly different from the training data, an…”

“Deep learning doesn't even really fully understand the relations between parts and wholes. It certainly doesn't understand what a silhouette is, and it gets it wrong.”

“Deep learning, it mostly cares about texture.”

“Deep learning can't do what we call compositionality. It can't put ideas together.”

“I think that deep learning doesn't just fail on reading. It completely, entirely misses the point.”

“Perception is what deep learning does, and actually it only does a small part of perception. There's lots of parts of perception where we use our knowledge about the world that it doesn't capture. And then there are all these other things like common sense and…”

“Deep learning is a better ladder for sure, but a better ladder doesn't necessarily get you to the moon.”

“There is a deep learning component that will make that happen. And those are the areas that we want to invest in, so that when you're in that experience with your friends, You can actually communicate both verbally from the audio cues, but non-verbally as well…”

“Deep learning of moles, cancer. Because that seems like a very intuitive kind of problem, but that's an easy one. We have many, many harder problems in biology medicine, but they take time to learn, so you have to be patient.”

“I think there are other kind of silly things that companies do sometimes that are unnecessary, like say things like physicians are going away, right, with AI and deep learning. Yeah, the greatest way to Make us not want to accept your product, right? That kind…”

“Most ML implementations are not deep learning. They're not neural networks. There's a ton of implementations that are incredibly valuable that don't involve networks.”

“It takes five years to train a radiologist, and in five years, deep learning will get better results than a trained radiologist, so we should stop training them right now.”