The Ledger, every show

“So they caught up on hardware. Now they've caught up on software. Not a lot of people have sort of discovered that they've caught up on software and we're kind of capitalizing on that.”

“We found that it's great for flash attention to specifically, we were able to be H-one hundred. We also found that like the less time you spend in like dense compute, like the less time you spend in tensor cores specifically, or less time you spend in lower bi…”

“It would be great, but it's not a silver bullet because kernels are also hard to validate. It's hard to have like an eval in kernels. Every time that someone releases like, oh, we created a new eval that measures kernels and we trained a model that generates k…”

“Grok is very inflexible hardware so that you kind of, if you want to do like a Mamba SSM on it, you're going to have a really hard time because instead of having like a low level CUDA compiler, like everything is like Designed it at the hardware level, right?”

“I definitely think things are moving towards you design a model. It's sized the way that fits well on hardware that you can design. You immediately start creating an inference hardware that is custom made to fit the sizes of your model. And then you can also s…”

“And I do suggest that people not try and use the model to offload thinking. It should enhance your thinking or else like you'll, you'll get worse over time and you won't know when the model is quality, whether it's outputting quality or not. If you don't know …”

“I think actually there's a lot of big tech examples where they're kind of pushing really hard for their teams to use more code. They're being evaluated on how much code they're using. And they're just kind of getting more and more slop that nobody understands.…”

“I don't really care if someone knows CUDA kernel writing. If someone does string theory and is really good at understanding complex problems, they will be productive faster than someone who knows CUDA and doesn't really care about trying to get better at it.”

“We recently moved all of our training cluster over to AMD. So we're really going all in on AMD ecosystem.”

“If I was sort of in house, maybe XAI is a good example of this, or Microsoft is another one where they kind of know the model architecture a priori then they absolutely should make an ASIC that is custom designed for that model architecture that they're curren…”

“AI sped me up a bit, but only in specific cases and only when taking a lot of sort of digital hygiene practices.”

“So I personally don't really use tools like cursor because I want total control over the context. I know what models can handle, what prompts, and I know, for example, one thing I mentioned is context rot. So how long the context is before the model chokes on …”

“No AI in the interview, first off. And you gotta watch people's eyes now on what monitors they're looking at behind the screen, which is, that's changed in the last two or three years, unfortunately. No AI allowed.”

“Anyone who's interviewed people can almost always tell, I think, about whether someone's using AI on the other side. How long is the time to first token? Humans are typically faster than a model.”

“Maybe it sounds wrong, but I feel like banding together is not necessarily good by default. There's a lot of different incentives. There's so much noise that no one knows where to focus. I prefer, if anything, I would prefer siloed focused teams who have fundi…”

“We released a Zamba two, which was a hybrid between transformers and a Mamba two blocks. And we were able to be like a Lama three eight B for example, with a seven B model.”

“So when MI 300 X first dropped, if you run like an MLP gem or something, you would get like 400 T flops. Now that number is, you know, more like six, 606 107 650 or so just in like the gem back ends themselves.”

“Most of that is an attention and MOPs, right? So if you have good kernels for attention, MOPs and norms and so on, then it doesn't much matter what the front end to, you know, send tensors to and from those kernels is”

“And then on the AMD side, they have this composable kernel library that does something very similar.”

“Getting Below Triton or level or anything else down to like CUDA or below there's orders of magnitude less public good kernels at that level. And I think that really shows and models capabilities. So when I try and get a model to do something in CUDA or PTX or…”

“Well, kernel is a last resort. So first I go, oh, another kernel. And I try and find some way to go around it.”

“O1's like the initial thinking models were a big deal when I was doing like core academic, like how do I create a performance model for explaining how this kernel behaves? Like, From first principles, that kind of thing was not really in the scope of any model…”

“I think you're just kind of noise unless I tell people, okay, I was the one that got the most speed up in the study.”

“The high performance inference kernels that sort of drive a lot of, you know, anthropic and open AI and stuff, their models, those aren't open source. They're not going to be open source.”

“AMD's actually done this. Like there's some deep seek, like through GPU modes, discord, there have been some like deep seek kernels that they say you have, you know, guaranteed access to compute for, write them.”