Tensor Parallelism

“Well, I mean, like a fused kernel can't save you. Like here with the tensor parallelism, you're the half, the matrix is one GPU and the other half is on another. And if I need the entire matrix in order to do like a non-linear operation on the next step, which…”

“Tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnects like NVLink was, you know, They might be forced to do something like pipeline parallelism, which we're never going to do unless we're doing some kind of mu…”