GPU clusters

“We had a situation where they were trying to train on a GPU cluster and they were at 60 days and it wasn't converging and We stood it up, and three and a half days later, their model converged”

“In March, we put seven GPT models in the open source community. Everybody else was putting one. Why? Because it's really hard to redistribute work across a GPU cluster. For us, it's one keystroke.”

“Even the GPU clusters themselves, like, you know, the full training networking is a very, very important Piece to have networking on the racks themselves with inference and matters a little less because you're doing a little bit more on individual GPUs and les…”

“And if you are meta and you have Somewhere between, you know, 22,000 GPU clusters and 350,000 GPUs available then continuing to train past, like, supposedly optimal points, like, does improve performance apparently and doesn't just fully asymptote as soon as m…”

“I think this actually points the way towards like what the infrastructure future might look like. And I think it looks a lot less like everything requiring these big clusters.”

“And so even for a lot of the more sophisticated orgs running large pre-trainings at scale, The utilization sub-eighty percent, sometimes less than 50%, actually, depending on how bad of a batch they have and the frequency of failure in the cluster.”

“Increasingly, I think that's going to get harder and harder as you move to liquid cooling and larger clusters”