CUDA

“Maybe in a year or two, we'll, we'll have compilers that are able to do a lot of these optimizations for you, and you don't have to, for example, spend a couple months writing CUDA to get this stuff to work.”

“There is a thousand flash attentions out there for us to build. You just got to make it easy for us to build them. So like Triton definitely helps. But it's still, Not easy. You know, it still requires kind of really understanding the VPU architecture, writing…”

“I learned, like, we did not make the developer experience very good. It was very complicated like, because we were using, like, local whisper, local models, and, like, getting it to work on CUDA, Mac, Windows. We didn't do a good job, so it was very difficult …”

“PMPP is actually quite good but also, I think, still kind of like mostly on the beginner level, because a lot of the CUDA code that we ended up developing in the lifetime of the LMC project, you would not find those things in, in this book, actually.”

“If we decided to change one thing in PyTorch, like one line of PyTorch code is like a week of CUDA code at least.”

“Basically the single-turn model that was just trained on, like, getting the best result after one turn. It would basically be a little bit, like, too careful, because it couldn't risk writing, like, non-compiling code, whereas, like, the multi-turn model would…”

“In the case of Modular, we go literally, like, we only work at that level, because we get rid of all of CUDA, right? And so we've replaced the entire stack, and so we only do that.”

“That said, if you do it, you're, you've gotta be one of the most hireable people in the world. Like if you like, Really deeply understand the architecture of the new GPUs coming out and how to control it. You're in a very small handful of people and like every…”

“Actually, our initial version of power retention was written in Triton, but we realized quickly that it just didn't offer the flexibility to really squeeze the performance that we wanted out of the GPU. So we took a step back and dove into CUDA.”

“It would be great, but it's not a silver bullet because kernels are also hard to validate. It's hard to have like an eval in kernels. Every time that someone releases like, oh, we created a new eval that measures kernels and we trained a model that generates k…”

“Getting Below Triton or level or anything else down to like CUDA or below there's orders of magnitude less public good kernels at that level. And I think that really shows and models capabilities. So when I try and get a model to do something in CUDA or PTX or…”

“I don't really care if someone knows CUDA kernel writing. If someone does string theory and is really good at understanding complex problems, they will be productive faster than someone who knows CUDA and doesn't really care about trying to get better at it.”

“I think that themes around like KV cache offloading, KV aware routing, and disaggregation are going to be substantially more important in the Rubin era, which means that inference engineering becomes not just a like CUDA kernel problem, but also like a very tr…”