Stuart Sul, researcher from Stanford and Cursor, presents findings on multi-GPU kernel optimization mechanisms on NVIDIA Blackwell architecture at YC Paper Club.
“So on Blackwell, we find that with roughly 15 SMs out of one 48 TMA is able to saturate the NVLink.”
quote is from the automated transcript, cleaned for reading:
filler sounds and stutters are removed, nothing is rephrased. names can be misheard
(the analysis reads context, assessments check outside sources). how →
More from Stuart (Stu)
AssertionSupported
Stuart: Multi-GPU compilers often produce kernels slower than baseline
“Two is to use compiler-based approaches, but we find these compilers, compilers to be quite suboptimal. They produce kernels that are sometimes slower than non-overlapped baselines”
Stuart (Stu)Jul 29, 2026▶ 10:43Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club · Y Combinator
AssertionSupported
Stuart: Bypassing NCCL intermediate buffers speeds up all-reduce by 80%
“For example, Nickel's default mode forces intermediate buffers which adds extra data movement between the sender and the receiver. And for fine-grained communication, this overhead really accumulates, and by stripping it out, you can speed up an operation as s…”
Stuart (Stu)Jul 29, 2026▶ 18:36Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club · Y Combinator
AssertionSupported
Stuart: ParallelKittens matches hand-optimized kernels in 50 to 100 lines
“And what we find is that with roughly 50 to 100 lines of device code, PK is able to surpass or match hand-optimized kernels that are often hundreds to thousands lines of code.”
Stuart (Stu)Jul 29, 2026▶ 20:38Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club · Y Combinator
AssertionSupported
Stuart: GPU networking can consume up to 50% of LLaMA pre-fill runtime
“For example, networking can still consume up to 50% of total runtime for workloads like Lama's MDB pre-fill.”
Stuart (Stu)Jul 29, 2026▶ 8:30Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club · Y Combinator
PredictionHeld up
Stuart: Single NVLink domains will soon scale to hundreds of GPUs
“And you also have a scale of architectures like NBL-L-Seventy-Two, which packs 72 GPUs inside a single NVLink domain, and soon this is going to extend to hundreds of GPUs.”
Stuart (Stu)Jul 29, 2026▶ 9:43Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club · Y Combinator
AssertionSupported
Stuart: Cursor uses ParallelKittens on tens of thousands of Blackwell GPUs
“For example, Cursor is using it to train Composer on tens of thousands of Blackwell GPUs.”
Stuart (Stu)Jul 29, 2026▶ 20:57Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club · Y Combinator
Made with StarZero
Turn any episode into a week of clips.
This entire site, over 300 episodes transcribed, diarized, checked and made playable,
runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the
moments worth sharing, cuts them, captions them, and reframes them for every feed.
We use essential cookies to make the site work. With your permission we
also use analytics cookies (Google Analytics and Mixpanel) to understand
usage and improve StarZero. See our Cookie Policy.
