tensor parallelism
2 statements across 1 episodes · 0 bullish · 2 bearish · 2 people on the record · first statement Aug 3, 2026 by Philip Kiely · across every show →
Everything said about tensor parallelism, oldest first
Aug 3, 2026 bearish
Tensor parallelism is a poor fit for local multi-device AI setups
“Tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnects like NVLink was, you know, They might be forced to do something like pipeline parallelism, which we're never going to do unless we're doing some kind of mu…”
Aug 3, 2026 negative
Fused GPU kernels cannot eliminate inter-GPU communication in tensor parallelism
“Well, I mean, like a fused kernel can't save you. Like here with the tensor parallelism, you're the half, the matrix is one GPU and the other half is on another. And if I need the entire matrix in order to do like a non-linear operation on the next step, which…”