Neural Networks

“The way careers work is you want to be kind of novel, but if you're too novel, if you're too far out there, then your papers will get rejected, and that certainly happened to me a lot in the early days, like, 2010 of neural networks for natural language proces…”

“Neural operators enable because they model inputs and outputs as continuous functions that can be infinitely resolved, that can have infinite discretization. And now we can have You know, at inference time, you can give it now inputs and ask for outputs at any…”

“The neural network, the benefit of neural network is, it's made up of a lot of standard GMM, general matrix multiplication, which, you know, is a famous, the GPU. GPU is highly optimized for this kind of operation. But the selection tree, you know, the old gen…”

“Long before the first videos were generated outside that people saw a couple of years earlier inside our labs, we were driving a simulator completely generated by video, and computer, completely generated by neural networks”

“That was the major step in AI translation. Before that, all of the solutions really didn't work. I think that's just general consensus.”

“70 years of people, maybe 80 years now of people saying this stuff is never going to work, never going to scale, going to hit the wall, hasn't hit the wall yet. There's still no wall in sight.”

“The answer to question one is yes, if you can figure out how to change each connection strength, you can make systems that are very smart just by training on data to predict the next word, or to predict the next frame of a video, or to predict something about …”

“The software three-point paradigm is a lot more kind of raw. It just, neural network is doing more and more of the work, and your prompt or context is just the image, and the output is an image, and there's no need to have any of the app in between.”

“This paper showed that you could actually reduce the size of these networks by 90%, and Get the same accuracy out by pruning very large models down to smaller models, and then you can make a selection on which model to run for inference. And by doing this, you…”

“And like, for example, we now know the neural network is the correct architecture.”

“Where if you build equivariance in, basically, once you've trained it in one orientation, it will understand it in any other orientation. So that means you need a lot less data to train these models.”

“We've been able to find, you know, neurons that correspond to very specific concepts, neural circuits that correspond to, you know, keep track of how to do rhymes in poetry, and so we're starting to understand what these models Do, right?”

“There's like actually so many of these kind of more scientific questions that I would like love to explore sometime, but then it really kind of conflicts with like empirical stuff, you know, like unfortunately at any given moment in time, it doesn't seem like …”

“A neural network is actually a stochastic machine, and so why are we using the substrate that is highly precise and deterministic? For something that's actually stochastic and distributed in nature.”

“In fact, one of the longest time we used to always obsess about, oh, how complex do the relationships have to be, or the data model needs to be to capture the essence of an enterprise, right? And it turns out, It's lots of parameters in the neural network with…”

“If you look at the ingredients of what brought chat GPT to the world, technically is still use these three ingredients. Now it's internet scale data. Mostly texts is a much more complex neural network architecture than. But it's still neural network and a lot …”

“I used to be quite skeptical that neural networks were necessarily the ultimate A solution to machine learning. I seem to be quite wrong on this one.”

“If you've got really good interp, you can start to reframe neural networks, not as just a tool for automating things that we already know how to do, but as a tool for discovery, as like a lens through which you can see patterns in data that would otherwise Be …”

“At this point in history. I mean, neural networks were not a respected subtree of AI.”

“And if you make the model bigger and bigger and bigger without the data changing, you just memorize everything. And actually, it starts to get worse again because it never had the opportunity to generalize because the model was so big and there's so many weigh…”

“It's hugely valuable that the artifact you get out of Alpha Evolve is a piece of code, and then you deploy that piece of code. And so before you do that experts, engineers who have worked on that system can visually inspect that piece of code, understand it, a…”

“Well, where my head is drawn to is some of the toy problems in academia, like t-shirt folding, and there are robots and neural networks now that can fold t-shirts.”

“Today you don't need a robot that's repeatable. You need a robot that's adaptable, like powered by neural networks, and if it makes a mistake or doesn't approach this object at exactly the right angle, it doesn't matter. It can correct that.”

“And I think what's changed, and I think is a quite fundamental change, is that neural networks became programmable. With large language models. And so I see this as quite new, unique. It's a new kind of a computer. And so in my mind, it's worth giving it a new…”

“As we made the autopilot better, basically the neural network grew in capability and size, and in addition to that, all the C++ code was being deleted and kind of like was And a lot of the kind of capabilities and functionality that was originally written in o…”

“We are actually not operating an end to end neural network. But we are operating a combination of, yes, C++ and neural network.”

“And I would say the forefathers like Lacan or Hinton, Benjia, they did a lot of work to establish the foundations, but one guy single-handedly, you know, with, of course, with a group of amazing engineers who worked with him, truly made it work. I'd say it's I…”

“If you're training it on the raw stock price, let's say you just have a bunch of numbers of the stock price of Nvidia opening price every single day. Sure, it's not going to be useful on its own, because there are so many other factors that influence the price…”

“There are so many other ways to do machine learning that are like, you know, support vector machines, linear regression, logistic regression, there's like a whole bunch of techniques, but it happens to be that neural networks is the one way to do things when y…”

“If AI was not neural nets, then GPUs wouldn't have mattered. But AI happened to be just basically neural nets at scale. And so all the primitives they built, all the software stack they built ended up being, like, The core foundational building blocks for neur…”

“And there was real insight that was generated there, which was that it really strengthened our belief that scale was the path to improving artificial intelligence. That with Dota two, the secret idea was that we could take huge amounts of experience and feed i…”

“I remember pitching Max Levchin to invest and Max goes, I'm totally in. I'm all for this company. As long as you tell me you're not using neural networks, because those things don't work. And I'm sitting there thinking all we're using is neural networks. And I…”

“I went to school to study AI. But at the time I was working in the AI lab, the one thing that they told you is definitely don't work on neural networks. We tried that. It doesn't work.”

“And then I would say also one last dimension of it is you benefit a ton from like the transfer learning between the different tasks. And in AI, you really want a single neural nut that is multitasking, doing lots of things that's very getting all the intellige…”

“What we finally concluded was that your hand Your hand writing probably had the greatest impact on neurological functioning on the way in which these neural networks formed and were changed than anything else we could do.”

“Outside of this, like, where we don't have good symbolic models, like, synthetic data obviously, like, doesn't make any sense. So synthetic data is not a magic wand where it'll work in all cases, in every case, you know, whatever. It's just where we as humans …”

“And the thing about it is that as you scale these digital brains, they end up with more emergent capabilities that no one who built them anticipated.”

“The breakthrough here was that now we've learned how to decompose a neural network into these Interpretable features when previous approaches focused on interpreting single neurons.”

“That it's very like that in our lifetime, neural networks will become capable of learning anything and everything that we are capable of as humans.”

“Source code is one of the very few things that we generate with neural networks. That we can actually execute an introspect. It doesn't require human feedback to evaluate it. It doesn't require humans to step by step reason through it. We have built compilers,…”

“And in the game worlds themselves can no longer, will no longer be rendered, but actually generated at runtime using neural networks.”

“The reason neural networks of the time weren't good is because they were too small. So like if you try to solve a vision task with a neural network, which has like a thousand neurons, what can it do? It can't do anything. It doesn't matter how good your learni…”

“I think maybe the most surprising, if I had to pick one, it would be the fact that when I speak to it, I feel understood.”

“You didn't always have to have An explanation. You can have a black box that does certain things for you. You just provided input and output and it trains a model for you. That is it works really well.”

“There's a few early models on neural networks and equities, which are starting to sort of show Some promising signs.”

“Neural networks had a bad name cause the hardware wasn't good enough.”

“Frankly, almost everything that is, that matters, Ah, was basically in the past four years. Like, there were a lot of things that fit in academics that were before that, and you know, and they were mostly dealing with models that were under a billion parameter…”

“Yeah, and at some point, there's a Joe Lonsdale neuron that, you know, is out there, you know, and individual people, you can actually find these neurons in the networks.”

“So I think that the thesis behind open AI when I joined in 2017 was that neural networks were the final architecture that could take AI all the way to human level intelligence or AGI.”

“I, at this point, I think we understand neural networks a lot better than we understand the brain. So whenever someone talks to me about the brain, I always think, well, what does a neural network do? And then probably actually the brain does that.”

“If you could figure out how to pour enough compute and pour enough data into a neural network, there's really no limit to how, how good it could get.”

“Like the difference between neural networks and linear regression was really big. And the difference between neural networks now and neural networks two years ago is just a sequence series of tricks.”

“That bottom up Will take you much further than you thought you could go, but will never take you all the way to where you want to go. So Wikipedia, this classic bottom-up thing, over time, there's a more and more top-down editorial control, and eventually, ov…”

“If you take out the planning that's being done in AlphaGo and just use the raw Policy network, the raw neural network, it's actually substantially below top human performance.”

“I'm a little less convinced that the tech, software technologies we have now are, like, on some linear path towards just, like, quote, unquote, AGI, or just, quote, unquote, like, consciousness, like, I don't, It's hard for me to believe that consciousness is …”

“And so then what machine learning does, or neural networks, or deep learning, whatever, or slightly different terms that all kind of get at the same thing, is you say, instead of making it a logic problem, we make it a statistics problem.”

“Identify faces and have a system that recognize, recognizes more faces than any human can with you know, the same level of accuracy. You know, maybe not with the same sort of robustness to you know, different changes in pose and facial hair and things like tha…”

“So in our case, we use neural networks to make a directional prediction, but we have actually taken the neural network and we're using a concept called deep reinforcement learning, which is, you can think of it as a decision framework, but an optimization that…”

“But in a deep reinforcement learning process, you're using that neural network, and it has the ability to learn new relationships where all other optimization functions will only use the parameters or the factors or the framework that the model itself is capab…”

“That was a really interesting time, 2013, 14, because that's when this neural network revolution started, and that is, we were finally at the point where we'd had some research breakthroughs into how neural networks work. We had enough data to feed these, like…”