Assertion Supported
Stuart: Multi-GPU compilers often produce kernels slower than baseline
“Two is to use compiler-based approaches, but we find these compilers, compilers to be quite suboptimal. They produce kernels that are sometimes slower than non-overlapped baselines”
Assertion Supported
Stuart: Bypassing NCCL intermediate buffers speeds up all-reduce by 80%
“For example, Nickel's default mode forces intermediate buffers which adds extra data movement between the sender and the receiver. And for fine-grained communication, this overhead really accumulates, and by stripping it out, you can speed up an operation as s…”
Assertion Supported
Stuart: ParallelKittens matches hand-optimized kernels in 50 to 100 lines
“And what we find is that with roughly 50 to 100 lines of device code, PK is able to surpass or match hand-optimized kernels that are often hundreds to thousands lines of code.”
Assertion Supported
Stuart: GPU networking can consume up to 50% of LLaMA pre-fill runtime
“For example, networking can still consume up to 50% of total runtime for workloads like Lama's MDB pre-fill.”
Prediction Held up
Stuart: Single NVLink domains will soon scale to hundreds of GPUs
“And you also have a scale of architectures like NBL-L-Seventy-Two, which packs 72 GPUs inside a single NVLink domain, and soon this is going to extend to hundreds of GPUs.”
Assertion Supported
Stuart: Cursor uses ParallelKittens on tens of thousands of Blackwell GPUs
“For example, Cursor is using it to train Composer on tens of thousands of Blackwell GPUs.”
Insight
Stuart: Multi-GPU optimization requires overlapping compute with remote memory prefetching
“For multi-GPU kernels a simple, a similar idea applies, except that you're overlapping computation with communication with other GPUs, such that when the current computation is done, the data for next computation is ready and fetched from remote GPU HPMs.”
Assertion Supported
Stuart: TMA saturates NVLink on Blackwell using roughly 15 SMs
“So on Blackwell, we find that with roughly 15 SMs out of one 48 TMA is able to saturate the NVLink.”
