GPU

“Now, you can limit power on GPUs and still get, you can use like half the power and get 80% of the performance. This is a known fact about GPUs”

“And in practice, once you start cascading there, you just saturate the GPU, and that's how it starts being paralysable trained. You no longer need to train in slices like traditional RNNs.”

“At NVIDIA, They know that they're going to win regardless. So they don't care where you get the GPUs from. They're like, they're truly neutral, unlike various sales reps that you might encounter at various like clouds and, you know, hardware companies, et cete…”

“LLM inference will be bigger than training, or multimodal, whatever, blah, blah, blah inference will be bigger than training, you know, probably next year, in fact at least in terms of GPUs deployed,”

“We have close to seven to 8000 GPUs today. It's growing monthly.”

“There is four to five million GPUs that will be sold this year. NVIDIA and others.”

“In the end, like, I don't think VCs will have the return they expected. Like, you know, in these things, but guess who's going to benefit? Like, you know, it's the consumers, right? Like someone's like reaping that the value of this. And that's, I think an ama…”

“It's like pretty easy in Moto, like, fan out to like, you know, at least like a hundred GPUs, like in a few seconds, and you know, if you give it like a couple of minutes, like we can, you know, you can fan out to like thousands of GPUs.”

“From our point of view, demand is not outpacing supply of GPUs. Like we have enough, from our point of view, we have enough GPUs to go around, but that might change for sure.”

“So, I think the, it's all a matter of time. I think time is the biggest bottleneck. It's like, when do you stop training the previous one, and when do you start training the next one? And how do you make those decisions? The data, do you have net new data, bet…”

“The Jaxx implementation just does not work on, on GPUs very well. Like, any naive setup that you do, like, it just won't run out of the box very easily”

“The normal, the like vanilla setup or, you know, these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 10, 24 GPUs instead of 4000. Here we have a larger cluster. As you start to get into the larger clusters…”

“In data centers that for the most part were not built remotely for this kind of power or heat and have been retrofitted for this. Like failures happen on a good day with normal CPUs. And this is not a good day and not a normal CPU for the most part.”

“For GPUs with floating points, and you push it through so many calculations, and so many met miles, the floating points aren't just not gonna be precise. So that's why even if temperature is zero, it's not gonna be the same throughout, ah, for multiple request…”

“Right now we have, like, the human CPU doing a lot of, you know, silicon CPU tasks, and so you really have to, like, redesign the work thoughtfully such that, you know, probably not that different from how it's evolved in computer architecture, where the CPU i…”

“Just for one seed, For one run of these things cost 4000 dollars all in with the GPU plus the tokens. And a bulk of the cost was actually the token, so even if you cut the GPU out, it'll still cost you three grand to run on one preview.”

“They compared a CPU only setup to a GPU setup to a multi GPU setup, and it kind of made no difference really.”

“So both cerebris and GPUs can handle structured sparsity But GPUs are not designed to handle unstructured sparsity, whereas what I've just mentioned before is able to handle this unstructured sparsity.”

“To give you a sense of, like, how I personally think about research budget for each part of the language model pipeline is, like, on the pre-training side, you can maybe do something with a thousand GPUs. Really, you want 10,000. And, like, if you want real es…”

“If you're interested in you know, your, Open replication of what OpenAI's O-one is you're gonna be on the 10 K spectrum of our GPUs.”

“So that means that the best way to make money in GPUs was to do basically exactly what CoreWeave did which is go out and sign only long-term contracts, pretty much ignore the bottom end of the market completely, and then maximize your long-term contracts with …”

“My general prediction is that like by the winter we will be back towards shortage, but then also this very much depends on The rollout of future chips.”

“So if you have a 10% margin increase because you have great software on your billion dollars, the customers are that price sensitive. They will immediately switch off if they can, because why wouldn't you? You would just take that hundred million dollars, you'…”

“Gusto isn't going to make like, you know, five percent more money. They're going to make zero, like literally zero money from every incremental GPU or CPU after a certain point. This is not the case for anyone who is training models. And it's not the case for …”

“Like your eviction policy runs on a CPU. Like that radix hashing algorithm and block hashing and all that stuff happens like primarily CPU. That's really important for performance because if you have latency in these steps, like you're not keeping your GPU uti…”

“I think that we should establish the definition of small model as being a model that a grad student can inference at reasonable time on a single GPU. Which is probably like seven B maybe. I don't think 27 is small under any reasonable.”

“That said, if you do it, you're, you've gotta be one of the most hireable people in the world. Like if you like, Really deeply understand the architecture of the new GPUs coming out and how to control it. You're in a very small handful of people and like every…”

“For most people, one of the things about CPUs that's really nice is with containers, right? You can end up having a single node and you can place many containers on them and all the containers will slowly start eating the compute. It's not really the same with…”

“And I saw one of your customers, they went from. 30 clients to just one single GPU and they cut costs by 97%.”

“Talent is cheaper than GPUs by a dramatic margin, and At the end of the day, it's like, okay, if we're spending this much, they go to the room and they stare in the mirror and you're like, wait, it might not actually be that ridiculous to spend this money on t…”

“The inference, high inference length generations definitely just, like, kind of breaks all infrastructure, because there's just so many tokens, there's more opportunity for out of memory or other things to go wrong. So it's like, just on a default, all of your…”

“And we have 2625 times more memory bandwidth than the GPU does.”

“Now we're in the world where GPUs are expensive, inference costs are high, and free trial abuse or refund abuse or general non-payment abuse, right, you rack up these charges and you never pay, is like existentially threatening for AI businesses.”

“Like five or 10 years ago, you really could train state-of-the-art models in the lab even with just a couple of GPUs. But, you know, because that technology was so successful and scaled up so much, then you can't train state-of-the-art models with a couple of …”

“And if you think about, you know, AlexNet required this jump from CPUs to GPUs, but even from AlexNet to today, we're getting about a thousand times more performance per card than we had in AlexNet days. And now it's common to train models, not just on one GPU…”

“Even for video, if you can't afford the kind of GPUs, pretty many, very kind of, do the kind of parallel inference algorithm. So even you have a lot of object to track, you can still get real-time tracking performance as long as you scale up the GPUs there.”

“Language models themselves are more like compute or GPU a generation ago, where what can we build at the layer above? And in software systems, we've traditionally thought of VMware being a great example. You have the operating system and the underlying archite…”

“In terms of like total spend on GPUs, it can still be a total, a small fraction of your burn. So sometimes it kind of makes sense.”

“I think the best researchers in the world have this dilemma of, okay, I want to go all in on AGI, but it's the product usage revenue flywheel that keeps the revenue in the house to power all the GPUs to get to AGI. And so it does make you know, I think it sets…”

“But we don't have a supply overhang. Like, there's no dark GPUs, right?”

“More or less we've reached the limits of You know, scaling things down, and now we are trying to improve further by new materials, so that's the fundamental materials problem.”

“When we look in 26, Google would buy a lot more TPUs, but they can't ramp production fast enough, right? And so they have to buy tons of GPUs. And we go to 27, it applies again, right? Google simply cannot buy enough TPUs, and they have to buy tons of GPUs.”

“It's only in my hands the one I'm still not naming is only about five times faster than my fastest DFT calculation on a GPU, and it also doesn't work all the time.”

“The model is trivially paralyzable, right? As many GPUs and tokens as I am willing to spend, I can have capacity to work with a code base. The only fundamentally scarce thing is the synchronous human attention of my team.”

“The reason why a lot of people come to us is because GPUs are more expensive than CPUs, right? So you want your GPU running at what? A hundred percent the entire time. And so when you're running runs on CPUs, when the CPU cycle is like down and spinning up the…”

“So the GEMA for model is a E to B. That means that it effectively has two billion parameters loaded into the GPU. It actually has almost five billion parameters, but those three billion parameters can be in the CPU, they can be in the disk, which means that yo…”

“Like the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL is, there's a lot of degrees of freedom, and it is basically a systems problem of Moving me…”

“There it's not about scaling from zero to one, but it's how do we scale really elastically from, like, thousand to 1500 GPUs very quickly in, in a given region.”

“That's why it's something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls off the bus or something happens, we user workloads are not affected.”

“And for reinforcement learning, it's always somewhere, like, around five to, like, six percent. So, said differently, that means that we're getting, like, five percent of the actual GPU computing power that we're paying for.”

“Effectively, all models today are MOE models that are, you know, at least all models large enough that you would care to parallelize them across multiple GPUs.”

“These layers are pretty costly and like that kind of limits what you can do with the architectures. They're not like, Not only are they like costly in terms of compute, they're just like not efficient on modern GPUs either. You have small hidden dimensions, la…”

“To our surprise, we found that it's not only, you know, accurate, it's almost as close to the what the traditional weather models can do accurately, but also tens of thousands of times faster. So what would take a big supercomputer to run can now be run. And w…”

“So lean still has a lot of shortcomings there. It's CPU based and you know, it's not, Like, getting that onto the GPU has a lot of nuances there. So, you know, a lot of work needs to be done. So what we've started with is a framework, you know, making that mor…”

“I think my reaction when I see pictures like this is that it's very impressive graphics design. But I also don't see them building anything beyond just, or trying to build something better than just, you know, a traditional GPU, right? You know, they've made c…”

“We're running you know, frontier level, one of the most intelligent models, right? OpenAI's largest, most capable, most intelligent model right now at 14 times faster than their normal, you know, GPU speeds.”