The Exchanges

Q People used to say that you would also do Franken merges. Where you would take, like, layers from each model. Does anyone do that anymore?

A Well, to your point previously, when you were mentioning, like, um, the work that goes into supporting a model when it first comes out, like JLM-VII or Minimax M-III or whatever the case is, sometimes you do have to, like, you do have to switch out some things. Like, for instance, the Minimax M-III head uses full attention. And with full attention, you end up with this, like, insane bottleneck inspector because you're doing autoregressive token generation for three tokens, and you're doing this, like, like O of N squared over all of the tokens that are in your sequence. Um, your KV cache is, like, very large because it's not sparse, it's not OK. So we find it better to, like, okay, we're going to replace this, you know, we're going to replace this layer with a layer from another model that's using, like, GQA, for instance. And then just for the right training, you can get it to have the same acceptance rate. So it is, it is very possible to, to retrofit layers from other models, and very much needed, actually. If a layer is, like, inefficient, the training just becomes the challenge. Like, how do you ensure that you train it properly? Which again, to earlier points, is like, the mesh between training and inference. As in, like, you need very good training in order to do fast inference. That's, like, I feel like more and more becoming true.

AI assessment note: “we're going to replace this layer with a layer from another model”

Q Uh, okay. One more thing while this chart is still up. All gather, all reduce is expensive. One of the things that is a movement in Silicon Valley is mega kernels. Just keep fusing kernels. I don't know. Is it that simple?

A Well, I mean, like a fused kernel can't save you. Like, like here with the tensor parallelism, you're the half, the matrix is one GPU and the other half is on another. And if I need the entire matrix in order to do like a non-linear operation on the next step, which is for instance, like if I'm doing attention, I need the soft max or I need to like, like exponentiation, I need to have the entire row. So I need to know what the partial result was from GPU two and what the partial result was from GPU one in order to be able to do the soft class in the next stage. So I, I like, I have to make them communicate with each other, even if I had a fused kernel because of the nonlinearities within each one. Also with, with like mega kernels, like honestly, I'm, I'm, I'm very bearish on, I'll be honest.

AI assessment note: “Well, I mean, like a fused kernel can't save you.”

Q So we talked a lot about LLMs. You have a lot more in the book. What about audio, video? What's the other side of inference engineering? Ali, you're pretty big in video diffusion.

A Video diffusions, I think, are, like, they're just shaped, a lot of the stuff that you can think about, reason about with LLMs, being autoregressive, with video diffusion, it's, it's, it's not the case. For instance, you don't do, you don't do batching. You, every request just comes in on one GPU, and it serves on one GPU. You don't have to shard. The models are a lot, are a lot smaller, like, one two two, for instance, is a twenty billion parameter model. Um, you don't need, To worry about, it's like orders of magnitude smaller than the best LLMs. And it's one of those spaces where the, the open source models are, like with LLMs, we see Kimi K-III is almost comparable to, you know, Mythos or like GPT-Five. The difference between the best open source LLM and the best open, uh, closed source LLM is very small. Like it used to be six months. I don't think it's six months anymore. I think it's like basically almost unparative. Video models are definitely not, there's a huge gap. If you look at the best video that you can generate today with an open source model like QuantoDato versus something like with Kling or video, difference is night and day. So it creates this disparity where media companies will choose to go most of the time to closed source models, for instance, and I were to tell you, hey, I can generate an entire three hour movie for you with this model, and I'll optimiz…

AI assessment note: “with video diffusion... you don't do batching. You, every request just comes in on one GPU”

Q Yeah. I think one thing that is, that is a classic journey, you know, it's basically Vivo is asking the, what happens when you type Google into the browser? Um, tool calling, is that just, you know, you're generating JSON, or is there more complication beyond that?

A Certain customers that we have, they have their own post-trained models, and so they, they demand Uh, tool calling that's not just like, you know, parse a file or, you know, go find the weather. It's, it's something that's very specific and you have to do post training on this. And if the post training on the model is not good, or if the quantization after the post training to get the inference to be fast, the model will struggle reading the JSON file and reading the tool calling, but it doesn't trick, it doesn't require its own like sandbox. It's not like it's going to use that tool calling to like, like escape a sandbox or like, it doesn't have to be contained. It can just be a normal dedicated deployment. The challenge with tool calling more and more seems to be that the companies want certain tool calling, which, which is a very sensitive thing to train. And because you're dealing with all of the JSON outputs, if it doesn't like close the end of the request in a very certain manner, you end up with a model that did the tool calling and like the thinking. And so as a result of that, it didn't see the result and just hallucinated the result as it decoded. That seems to be the most challenging thing with tool calling. Not really the sandbox as well.

AI assessment note: “The challenge with tool calling more and more seems to be that the companies want”