GPU

“Around 20 12, I guess, and it was because simultaneously Andrew Eng reached out to Bill Daly, our chief scientist to work on a way to get the neural network model that they were working on onto GPU so that they could, instead of using thousands of CPU servers,…”

“The insight here was, Hey, you can use the same attention thing to like, look back at the past of the sequence that you're trying to produce. And you know, the beauty is that the, that It runs great on on GPUs and CPUs, and it's kind of parallel to, like, how …”

“I don't think GPUs are at the sweet spot when it comes to large-scale deep learning with respect to exactly those trade-offs, and so it may very well be that if we actually try these combinations, we might actually even quickly find something that's better.”

“And that was an idea that came from supercomputing that we knew really well, that we could organize this so you could run an arbitrarily large, now a trillion parameter network on a single system.”

“And if you have to spend months doing distributed compute, doing tensor model parallel distributed compute, I mean, if you look at the back of some of these papers, they're crediting 20 or 30 people, sometimes more, who helped on the distributed compute. And i…”

“And the reality right now that the requirements on bandwidth, right? Like people who are training these models right now, they have like a, you know, 800 gigabit connect right between the GPUs, right? So Maybe you have a hundred megabits on between this, usual…”

“The challenges, the GPUs there are like, not the ones that AI folks want to use, right? Like kind of all the AI is really zeroed in on like, how do we get a 100 or H 100 and the GPUs that like folks used for Ethereum mining and like similar is like older ones …”

“And it obviously was dramatically more performant than GPU for a long time.”

“And to me, it was clear that the industry needed That powerful combination of the serial, the scalar competing of these traditional CPU workloads and the massive parallelization that you get from a GPU.”

“Everything maybe besides like the battery cell and the GPU and CPU at this point, it feels like we've done what we're doing.”

“Today to train these large models, you need all of the GPUs co-located because there is enough data transfer between different chips, right? Between your nodes. And there's a physical constraint on that in that you need to get that much power and to a data, da…”

“In AI cloud today, you're kind of forced to get really long-term reservations often three years for a fixed amount of capacity. [944] Jared Quincy Davis: No one really wants 64 GPUs for three years or a thousand GPUs for three years. [949] Jared Quincy Davis: …”

“And so one of the consequences of that is that it's very common now to hold aside 10 to 20% minimum of the GPUs that a team has as buffer, as healing buffer, in case of a failure so you can slot something else in to keep the training workload running, right?”

“Decide to build this 100,000 GPU super cluster, which is, you know, the largest of its kind in, in one unit.”

“We are buying chips. We're not making GPUs. We're taking the hottest chips in the world and we're cooling them down so that you can, we can open the envelope of performance for the system architect. Okay. And so we fit in, we're coming into the world as a serv…”

“Means that cloud, you know, a lot of the cloud capacity is like, you know, the only way to get it is to sign long term commitments, which I think for a lot of startups is really not the right model for how things should be.”

“Human developers are expensive because there's limited supply agents will have infinite supply that, that will only be limited by the amount of compute capacity GPUs available in data centers.”

“Under the Biden era, there was something called the Biden diffusion rule, which basically was a 200 page document, which basically made it illegal for America to export GPUs. It was really hard for you know, if you're Jensen or if you're Lisa Su to really kind…”

“And one of the other things we're doing that is we get our GPUs over, we probably get them to run our models as opposed to models from, you know, another country, and we go from there.”

“We're going to speed run the last 30 years of CPU efficiency gains. In, in the next, you know, five to 10 in, in GPUs.”

“Like the DeepSeq implementation of inference is like a 160 GPUs or something like that, like that's over ten million dollars of Hardware. And then that's just one replica.”

“Meta is literally building these like temporary, like tent structures to put GPUs in because building the building takes too long and it takes too much labor, right?”

“Their ROIC is, like, extremely high on CPU and storage, and to assume that it can, like, translate over to GPUs is, is a bit of a fallacy, which is why a lot of these companies are moving in, right?”

“And I think what got missed was the GPUs themselves were actually like the second, second or tertiary level of collateral in those instruments. The primary collateral was the contracted cash flows from investment grade counterparties.”

“So, In that domain, I think it's happening now-ish. And I think we'll see the same thing too for AI research. That's a slower outer loop because now the experiment isn't just checking some unit tests passing, but it's checking what was the scaling property? Di…”

“And right now we're the fastest at inference, not by a little bit, but by a lot. 1518, 20 X faster than GPUs.”

“Right now, the authentic AI and inference, CPU become, you know, highly in demand. And so, you know, versus one to eight in the training CPU to GPU, now I can see one to four, maybe one to one, and I'm delighted CPU become important.”

“And in fact, as you get to the smaller footprints, where more and more AI is going to take place, that's going to be a sweet spot for Arm, because the CPU's table stakes anyway, you have to have it to do all the things that are required in the edge device. But…”