Distro

“I think there's, you know, immediately coming out. What you'll see is the ability for even centralized actors who might have multiple data centers to now, like just use them in a more efficient way. Like just have N equals two, you know, like anything. And eac…”

“I think it still is, it would still be, you know, like a next year sort of environment thing that we would have to do. There are some scaling problems, or not scaling problems, but technical things about how you shard the model, because at that point you get t…”

“What might happen sooner would be a redesign of the types of chips that NVIDIA or someone would make. Okay, under this model, we can dedicate more VRAM versus, there's like this question of how much VRAM versus how much processing power is on a die, and that, …”

“Because you're able to distribute it so wide, I think the gaming GPU angle is really going to be like the sweet spot. If, as long as there's continued to be sort of like higher end gaming GPUs, and those are on comparison with the high end training GPUs, even …”

“We've gone up through seven B now, like seven B models.”

“And we've re-implemented now a third time in their framework, and we're able to reproduce their training run exactly, and then did it again with Distro, got the exact same results we got with Natron and stuff.”

“What we have seen empirically is that as we make it bigger, the differential between distro and MW actually gets wider.”

“We know that like what we, what needs to be communicated between these things, the two, the different nodes are just these few key pieces of information. And that is necessary. That is a necessary condition or rather a sufficient condition To get the equivalen…”

“So with distro, what we found is that rather than bringing everyone back home and averaging it back together, what you want to do is give each of those little nodes that are searching for the lowest point in the lost landscape, the freedom to move around. And …”