Schmidt: DisTrO will eliminate data centers' InfiniBand dependency before edge AI dominates
Jeff Schmidt · The Quest for Community-Trained Open Source AI Models · Oct 1, 2024 · at 1:09:20
Jeff Schmidt is the co-founder of Nous Research, discussing DisTrO, a decentralized AI training protocol that reduces bandwidth requirements between compute nodes.
0:00 / 1:04exact quote · 64.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think there's, you know, immediately coming out. What you'll see is the ability for even centralized actors who might have multiple data centers to now, like just use them in a more efficient way. Like just have N equals two, you know, like anything. And each one of them is like a whole data centers and node on the network. I mean, and that that's just an unlock because especially like for a lot of these data centers, they have like maybe a hundred gigabit interconnect between their own data centers, right? And it's like just the fact that you could now treat each of these two as one. I think just from like a practical standpoint, probably is like the biggest unlock that starts beginning. But that that's the thing about the method is that it is can be decentralized, but it doesn't necessarily have to be decentralized. And the centralized actors can still You know, reap the benefit because they, for that, or they just don't have to buy InfiniBand anymore, you know, like they just can use a hundred gigabit ethernet or 10 gigabit ethernet when they're laying out their data center. So I think there might be a lot of like, just simple kind of like things, effects that happen quickly, and then sort of the dream, you know, of the full decentralized one may progress slower, but will ultimately eclipse.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jeff Schmidt
Schmidt: AI industry relies on outdated 1990s architecture assumptions ripe for disruption
“It turns out that most assumptions in the AI space right now are a product of that's just how things had been done when there was not nearly as much energy and attention to it. So someone made an assumption maybe in the early nineties that everyone just kind o…”
Schmidt: Relying solely on fine-tuning is an existential threat to open-source AI
“Because that for us, if we're just fine tuning models. Right. That's like an existential threat, right? That's like an actual existential threat because the closed providers will continue to get better and we would be like dead in the water in a lot of sense.”
Schmidt: Sharing only key signals in distributed training yields equivalent model learning
“We know that like what we, what needs to be communicated between these things, the two, the different nodes are just these few key pieces of information. And that is necessary. That is a necessary condition or rather a sufficient condition To get the equivalen…”
Schmidt: Decentralized training will force NVIDIA to redesign chips around VRAM ratios
“What might happen sooner would be a redesign of the types of chips that NVIDIA or someone would make. Okay, under this model, we can dedicate more VRAM versus, there's like this question of how much VRAM versus how much processing power is on a die, and that, …”
Schmidt: Compute chips in NVIDIA's RTX 4090 and H100 are almost identical
“I think people don't actually realize that like a forty-ninety and like an H 100 are in a lot of ways the same card. For the non-gamers in the room, explain the forty-ninety. The chip that's inside of them is almost identical. The chip, the actual compute chip…”
Schmidt: Consumer gaming GPUs will become the sweet spot for distributed training
“Because you're able to distribute it so wide, I think the gaming GPU angle is really going to be like the sweet spot. If, as long as there's continued to be sort of like higher end gaming GPUs, and those are on comparison with the high end training GPUs, even …”