Fused Kernel
topic on 1 show · 1 statements across 1 episodes
1 statements about Fused Kernel, every show
Fused GPU kernels cannot eliminate inter-GPU communication in tensor parallelism
“Well, I mean, like a fused kernel can't save you. Like here with the tensor parallelism, you're the half, the matrix is one GPU and the other half is on another. And if I need the entire matrix in order to do like a non-linear operation on the next step, which…”