The Wisdom Wall

“even if they don't work, I think to a large extent, you feel like it's a skill issue. It's not that the capability is not there. It's that you just haven't found a way to string it together of what's available. Like I just don't, I didn't give good enough instructions in the agent's MD file or whatever it may be. I…”

“to get the most out of the tools that have become available now, you have to remove yourself as the, as the bottleneck. You can't be there to prompt the next thing. You're, you need to take yourself outside Um, you have to arrange things such that they're completely autonomous and the more, you know, how can you…”

“A research organization is a set of markdown files that describe all the roles and how the whole thing connects.”

“So if the barrier comes down, then actually you have the Jevons paradox, which is, like, you know, you actually, the demand for software actually goes up. It's cheaper and there's more”

“even with other research, like, OpenAI or, or, you know, uh, Anthropic or these other labs, like, they're employing, what, like, a thousand something researchers, right? These researchers are basically, like, glorified auto, like, you know. They're, like, automating themselves away, like, actively, and this is, like,…”

“Like, if you're inside one of the frontier labs, like, there are certain things that you can't say, uh, and conversely, there are certain things that the organization wants you to say, and, you know, they're not gonna twist your arm, but you feel the pressure of, like, what you should be saying, you know, because,…”

“And I think if you're outside of that frontier lab, uh, your, your judgment fundamentally will start to drift because you're not part of the, you know, what's coming down the line.”

“when people talk about the scaling loss in neural networks, the scaling laws are actually a, um, uh, to a large extent of a property of the transformer. Before the transformer, people were playing with LSTMs and stacking them, etc. You don't actually get like clean scaling loss, and this thing doesn't actually train…”

“I don't think that the neural network architecture is like holding us back fundamentally anymore. It's like not the bottleneck, whereas I think in the previous, before Transformer, it was a bottleneck, but now it's not the bottleneck. So now we're talking a lot more about what is the loss function? What is the data…”

“currently at the current AI capability, I don't think the models are good enough to create a good course. Uh, but I think they're good to become the front end to the student and, uh, interpret the course to them.”

“And the correct answer is mostly like, um, I would say like math, physics, CS kind of disciplines. And the reason I say that is because I think it helps, um, uh, for just thinking skills. It's just like the best thinking skill core. Uh, is, is my opinion.”

“It's just like you can, you can, you can move in much larger macro actions. It's not just like, here's a line of code, here's a new function. It's like, here's a new functionality. And delegate it to agent one. Here's a new functionality that's not going to interfere with the other one. Give it agent two. And then try…”

“this is extremely well suited to anything that has objective, uh, metrics that are easy to evaluate. So for example, like writing kernels for more efficient CUDA, you know, code for various parts of a model, etc., are the perfect fit. Because you have inefficient code, and then you want efficient code that has the…”

“flipping, flipping bits and, and the ability to copy paste digital information is like, makes everything a million times faster than accelerating matter, you know? So, um, so energetically, I just think we're going to see a huge amount of activity in digital space, huge amount of rewriting, huge amount of activity,…”

“training neural nets and LLMs specifically, um, is a huge amount of code, but all of that code is actually complexity from efficiency. It's just because you need it to go fast. If you don't need it to go fast and you just care about the algorithm, then that algorithm actually is the 200 lines of Python.”

“And then I would say also one last dimension of it is you benefit a ton from like the transfer learning between the different tasks. And in AI, you really want a single neural nut that is multitasking, doing lots of things that's very getting all the intelligence and the capability from. And that's also why language…”

“That's the thing with AI. I feel like a lot of them, a lot of these capabilities are just kind of like prompt away. So you always get like demos, but like, do you actually get a product? You know what I mean? So, um, so in this sense, I would say the demo is near, but the product is far.”

“It takes effort, but it's effortful, but it's also kind of fun. And you also have a payoff of like, You feel good about yourself in various ways, right? And I think education is basically equivalent to that. So that's what I mean when I say education should not be fun, etc. I mean, it is kind of fun, but it's like a…”

“You would feel nervous when your GPUs are not running. Like you have GPU capability and you're not maximizing the available flops to you. But now it's not about flops, it's about tokens. Uh, so what is your token throughput and what token throughput do you command?”

“And I do think that the current models are wasting a ton of capacity remembering stuff that doesn't matter. Like they remember SHA hashes.”

“actually, like, end-to-end driving, when you're just imitating humans and so on, you have very few bits of supervision to train a massive neural net. And it's too Too few bits of signal to train so many billions of parameters. And so these intermediate representations and so on help you develop the features and the…”

“I would say the human node aspect is also very appealing because people can teleoperate it very easily. And so it's a data collection thing that is extremely helpful because people will be able to obviously very easily teleoperate it. I think that's usually overlooked.”

“When you're doing synthetic data generation, this is a problem, because you actually really want that entropy. You want the diversity and richness in your data set. Otherwise, you're getting collapsed data sets, and you can't see it when you look at any individual example. But the distribution is, has lost a ton of…”

“a lot of things have this property that, you know, very expensive to come up with, but very cheap to verify. And so in all those cases, things like folding at home or SETI at home or auto research at home will be good fits.”