The Ledger, every show

“I think, personally, Tesla is ahead of Waymo, and I know it doesn't look like that, but I'm still very bullish on Tesla and its self-driving program. I think that Tesla has a software problem, and I think Waymo has a hardware problem, is the way I put it, and …”

“The software three-point paradigm is a lot more kind of raw. It just, neural network is doing more and more of the work, and your prompt or context is just the image, and the output is an image, and there's no need to have any of the app in between.”

“Even if they don't work, I think to a large extent, you feel like it's a skill issue. It's not that the capability is not there. It's that you just haven't found a way to string it together of what's available. Like I just don't, I didn't give good enough inst…”

“I actually think Claude has a pretty good personality. It feels like a teammate and it's excited with you, et cetera. I would say for example, Codex is a lot more dry which is kind of interesting because in Chashi PT, Codex is like a lot more upbeat and highly…”

“So I think the industry just has to reconfigure in so many ways that it's like the customer is not the human anymore. It's like agents who are acting on behalf of humans. And this refactoring will be, will probably be substantial in certain sense.”

“I kind of feel like this kind of stuff that I just talked about, this should be free, like in a year or two or three. There's no bi-coding involved. This is trivial. This is table stakes. This is like any AI, even the open source models, et cetera, can like do…”

“To get the most out of the tools that have become available now, you have to remove yourself as the bottleneck. You can't be there to prompt the next thing. You're, you need to take yourself outside you have to arrange things such that they're completely auto…”

“What I'm more interested in is, like, this idea of recursive self-improvement and to what extent you can actually have LLMs improving LLMs, because I think all the Frontier Labs, this is, like, The thing for obvious reasons. And they're all trying to recursive…”

“A research organization is a set of markdown files that describe all the roles and how the whole thing connects.”

“The story is that we're getting a lot of the intelligence and capabilities in all the domains of society, like, for free as we get better and better models, and it's not, like, exactly fundamentally what's going on, and there's some blind spots, and some thing…”

“I do think that we will, we, I do think we should expect more speciation in the intelligences.”

“A swarm of agents on the internet could collaborate to improve LLMs and could potentially even, like, run circles around Frontier Labs. Like, who knows, you know? Yeah, like, maybe that's even possible. Like, Frontier Labs have a huge amount of trusted compute…”

“So if the barrier comes down, then actually you have the Jevons paradox, which is, like, you know, you actually, the demand for software actually goes up. It's cheaper and there's more”

“Even with other research, like, OpenAI or, you know Anthropic or these other labs, like, they're employing, what, like, a thousand something researchers, right? These researchers are basically, like, glorified auto, like, you know. They're, like, automating th…”

“Like, if you're inside one of the frontier labs, like, there are certain things that you can't say and conversely, there are certain things that the organization wants you to say, and, you know, they're not gonna twist your arm, but you feel the pressure of, l…”

“And I think if you're outside of that frontier lab your judgment fundamentally will start to drift because you're not part of the, you know, what's coming down the line.”

“So there may be, they're behind by like, what is the latest, maybe like eight months, six months, eight months kind of thing right now.”

“I don't actually think it's like structurally, I think there's some systemic risk attached to just having intelligences that are closed, and that's like, that's it. And I think that that's a, you know, centralization has a very poor track record in my view”

“I feel like there's gonna be less of like explaining things directly to people. And it's gonna be more of just like, does the agent get it? And if the agent gets it, they'll do the explanation.”

“Instead of HTML documents for humans, you have markdown documents for agents, because if agents get it, then they can just explain all the different parts of it. So it's this redirection through agents, you know and that's like, so I think we're going to see a…”

“LLMs, like, flip, they flip the direction of technology diffusion, ah, that is usually, ah, present in technology. So, for example, with electricity, cryptography, computing, flight, internet, GPS, lots of new transformative technologies that have not been aro…”

“Like, you may see Waymo's going around, and they look driverless, but, you know, there's still a lot of teleoperation and a lot of human in the loop of a lot of this driving.”

“When I see things like, oh, twenty-twenty-five is the year of agents, I get very concerned, and I kind of feel like, you know, this is the decade of agents, and this is going to be quite some time. We need humans in the loop.”

“Working with valuable LLMs and so on, I would say, you know, it's less Iron Man robots and more Iron Man suits that you want to build. It's less like building flashy demos of autonomous agents and more building partial autonomy products”

“I think even a billion. Billions of ISIS. We'll probably get to that point and the models can be very, very small.”

“If LLINs are about to become much better at coding over time, then I think you can expect that the LLIN could actually do this for any custom application over time. And so the LLINs could act as a kind of compiler What you're interested in, they're gonna do al…”

“The use of Python and PyTorch and everything else is just a crutch, because we humans are finite. We have finite knowledge, intelligence, and attention.”

“Tesla actually does use a lot of expensive sensors. They just do it at training time. So there are a bunch of cars that drive around with LIDARs. They do a bunch of stuff that like doesn't scale and they have extra sensors, et cetera, and they do mapping and a…”

“And I do suspect that The end-to-end systems for Tesla in, like, say, 10 years, it is just a neural net. I mean, the videos stream into a neural net and commands come out.”

“Basically everything transfers and I don't think people appreciate it. Cars are robots and Tesla, I don't think is a car company. I think this is misleading. This is a robotics company, robotics at scale company, because I would say at scale is also like a who…”

“I don't know that there's, like, any individual impediments that I'm, like, really familiar with. I just think it's a lot of grunt work. A lot of, like, the tools are available. Transformers are this beautiful, like, blob of tissue. You can just get just arbit…”

“When people talk about the scaling loss in neural networks, the scaling laws are actually a to a large extent of a property of the transformer. Before the transformer, people were playing with LSTMs and stacking them, etc. You don't actually get like clean sca…”

“I don't think that the neural network architecture is like holding us back fundamentally anymore. It's like not the bottleneck, whereas I think in the previous, before Transformer, it was a bottleneck, but now it's not the bottleneck. So now we're talking a lo…”

“The trajectories in your brain as you're doing problem solving. If we had a billion of that, like AGI is here, roughly speaking. I mean, to a very large extent.”

“So I think basically synthetic data is absolutely the future. We're not going to run out of data, is my impression. I just think you have to be careful.”

“I think transformers are actually better than the human brain in a bunch of ways. I think they're actually a lot more efficient system. And the reason they don't work as good as the human brain is mostly data issue, roughly speaking, is the first order approxi…”

“I think even a 1,000,000,001, billion suffices. We'll probably get to that point, and the models can be very, very small. And I think the reason they can be very small is fundamentally, I think, just, like, distillation works.”

“I think there's a lot of activity in, like, AI, and I think most of it is to kind of, like, replace or displace people, I would say.”

“Currently at the current AI capability, I don't think the models are good enough to create a good course. But I think they're good to become the front end to the student and interpret the course to them.”

“And the correct answer is mostly like I would say like math, physics, CS kind of disciplines. And the reason I say that is because I think it helps for just thinking skills. It's just like the best thinking skill core. Is, is my opinion.”

“I don't think I've typed like a line of code. Probably since December, basically which is like an extremely large change.”

“It's just like you can move in much larger macro actions. It's not just like, here's a line of code, here's a new function. It's like, here's a new functionality. And delegate it to agent one. Here's a new functionality that's not going to interfere with the o…”

“And then I let our research go for like overnight and it came back with like tunings that I didn't see. And yeah, I did forget like the weight decay on the value embeddings and my atom betas were not sufficiently tuned and these things jointly interact.”

“This is extremely well suited to anything that has objective metrics that are easy to evaluate. So for example, like writing kernels for more efficient CUDA, you know, code for various parts of a model, etc., are the perfect fit. Because you have inefficient c…”

“Flipping, flipping bits and the ability to copy paste digital information is like, makes everything a million times faster than accelerating matter, you know? So so energetically, I just think we're going to see a huge amount of activity in digital space, huge…”

“First, there's gonna be a huge amount of unhobbling, and I think there's a huge amount of work there. Then, actually, it's going to move to, like, the interfaces between physical and digital. So, and that's, like, sensors of, like, seeing the world and actuato…”

“Training neural nets and LLMs specifically is a huge amount of code, but all of that code is actually complexity from efficiency. It's just because you need it to go fast. If you don't need it to go fast and you just care about the algorithm, then that algorit…”

“And I think what's changed, and I think is a quite fundamental change, is that neural networks became programmable. With large language models. And so I see this as quite new, unique. It's a new kind of a computer. And so in my mind, it's worth giving it a new…”