The Ledger, every show

“By almost any definition anyone could have written down, let's say five years ago or 10 years ago, certainly when, you know, Tim, you and I started our PhD. We basically have the vision of AGI that, that we had back then. We have things that can write code. Th…”

“You know, we maybe already have AGI or like some form of AGI. And if not, then certainly the next generation of models, the models that today are training already. If they're at all better than what we have today, then we're, we we've already hit something tha…”

“We had this experience over and over again where you could have any, an embedding model of any quality, so you could have a really, really bad embedding model, or you could have a really, really good one by, and by any measure of good, and for the final RAG ap…”

“But if you give an expert programmer This set of tools, they can go 10, 10 times faster than they were able to go before.”

“We basically built a whole library just around this basic idea that all your basic compute primitives should not be a float, but it should be a matrix and everything should just be matrix compute.”

“You're certainly not going to do a giant quadratic attention computation to try to run that.”

“If you look at the deep seek model, for instance, this is one of the best open source models we have out there today. It was trained at the end of 2024. On last generation, kind of nerfed GPUs, H 800 instead of H 100, the 800 is nerfed by all sorts of ways fro…”

“The models that we see today that we can play with today have been pre-trained on clusters that were built out a year or two ago. Because, you know, you need enough time to get the cluster running. You need enough time to do the large pre-training run. And the…”

“At inference time, when the, when you have the model, when it's already been trained, already been post-trained, the hardware utilization is like less than five percent.”

“I think you see a lot more risk taking out of the Chinese labs where you're trying to differentiate the next model of your next open source model.”

“Even if your model is theoretically more efficient, if somebody goes and runs it and it's two times slower one of the things that, that we've learned is that if you're in that situation, it's just going to be dead on arrival. So you want to be designing your a…”

“Nobody is actually putting in a two million context prompt into these models.”

“I'm sure NVIDIA will still do great and still grow beyond their five trillion dollar company or whatever it is at the time of recording. But I think you're going to see a lot more diversity especially around, I think inference of the model.”

“When they are really gung ho about understanding and being able to use the AI agents, there's, they're able to communicate so much better than in the olden days. They're able to level up their level of understanding a lot faster.”

“So if you do the speculative decoding right, you can get, again, two X, three X speed ups over, over, you know, just running a vanilla model.”

“So some of the best audio models in the world are at least partially based on state space models.”

“AI-II trained this hybrid MOE called Jamba that, that, that seems, that is currently the state of the art for these non-transformer architectures.”

“If we decided to change one thing in PyTorch, like one line of PyTorch code is like a week of CUDA code at least.”

“They're companies like Poolside. They're building out tens of thousands of B-two hundred, GB-two hundred chips. You know, there's other folks like Reflection who are who are building out. Tens of thousands of B 200 chips.”

“One of those gated, SSM gated states-based models ended up on the cover of science because a great group of folks went and trained some DNA models. So that's Michael Polley, Eric Yuen from Stanford and the Arc Institute.”