The Exchanges

Q we had a nice launch of the book. Very successful. But before we get into all that, I want to start off with a fun question for you. Ok, you're expert inference engineer. What happens when I send a long query, say, 200,000 tokens into base tens inference? What's the process of query through GPU model routing, balancing, all that? What is all the stuff that we don't think about?

A With a long query specifically, the first thing that I'm going to ask is, have you sent me this query before, or at least part of it? Um, and I really hope you have, because it's going to be a lot easier for me and a lot cheaper for you. So the first thing that we're going to look at is some kind of cache away routing where we're going to see, we probably have a number of instances, a number of replicas up serving whatever model you're hitting. Uh, we want to send this one to Something with number one, available pre-fill workers, and number two, ideally, some cached input already there so that we can skip pre-fill on at least part of these 200,000 tokens. Um, if you're doing 200,000 tokens, it's probably coding or a multi-tone agent or something where you would expect to have that cached. Um, if you don't, we're gonna have to send it to a pre-fill worker. Um, we've, at least on certain models, disaggregated pre-fill and decode. Um, so You're going to have one set of GPUs that's solely going to process the input, create that KV cache, and get you your first token, and then that's going to be passed over to a separate set of GPUs, which is going to run decode. We're going to iteratively make those tokens. Um, we're probably going to have some kind of speculator model in front of that. I'm going to assume that you're doing coding, and because of that, our speculator model, which a…

AI assessment note: “the first thing that we're going to look at is some kind of cache away routing”

Q know, it's the same thing, it's just a model. Can we not just distill a lot of the weights, quantize the speculator, but out of my domain. Um, I guess the, the question that also comes up is, this is all for big server workloads, right? How much of this applies to, say I have this MacBook, I want to run Gemma really efficiently, um, Similar problems? Not the same?

A Pretty different. I talked to Cero, um, about this on, on his podcast a couple weeks ago. The difference between inference engineering for the data center and for production workloads versus inference engineering for local AI is that we start with fundamentally like different constraints and different goals. With local AI, it's how do I fit this model onto my hardware? And then make it less dumb. And with data center inference, it's how do I load this model and then make it less slow? And obviously, you know, we care about less dumb and they care about less slow, but the local AI inference engineering ecosystem, I think actually has a lot for us to learn from in the data center space. They are experts in various forms of quantization, including dynamic quantization that we just kind of don't touch. In the pruning, in the distillation, in the, uh, you know, layer removal. There's removal letters less. Yeah. Yeah, but, but,

AI assessment note: “Pretty different. The difference between inference engineering for the data center and local AI”

Q talked about all these steps of, okay, you gotta do quantization, train your own speculative decoder, run on different hardware. Um, looking at other model providers, Okay, you kicked off a inference speed race. On the consumer end, um, what goes into keeping quality the same across them, right? Sure, you can run benchmarks, but like, how do you determine how much quantization are the standards? What actually goes into?

A There's a few things on quality. Most inference optimizations are lossless. KV caching, for example, you are just recomputing or preventing recomputing the same values. Speculation, of course, If a draft token is wrong, it gets rejected. The main lossy optimization is quantization, um, and that really comes down to number one, data format, uh, number two, uh, which parts of the model you choose to quantize, which layers, and number three, like, doing a lot of calibration on the quantized weights, uh, to ensure that you're sort of preserving all the outliers. There's other sort of tricks that you can do, though. A big one is long context, because one thing you asked right at the beginning is, oh, what's going to happen if I send a 200,000 token request in? So obviously with a long input sequence, you need to, you know, store a lot more information, you need to process a lot more tokens, and so even if a model has a context of a certain length, You might, as an inference writer, choose to build an API with a shorter context length, and of course a full length one as well, because if someone doesn't need the full million token context, for example, you can get them better performance. I don't know if that's exactly like quality of the model. The way that I think about quality is to what degree are we faithfully serving the original model? If you think of a sort of golden implement…

AI assessment note: “The main lossy optimization is quantization, um, and that really comes down to”

Q And how far does that get you? And how, how easy is that for the average person to do? So say right now, I want to throw the weights of GLM five two on a node of B 200. How easy is it to find speculative decoder, decoder model or already quantized model? How much work goes into it?

A If you're doing it up front, it's quite a lot of work. If you're doing it today, there's going to be people who have published Things that you can just, you can just grab some NVFP four weights. You can grab a speculator. Uh, yeah. If we're thinking about like, what are the two X's we're stacking? Going from, uh, BF-sixteen to NVFP four is, it's not quite a two X, right? It's like, I think it's about like 30, 30 to 40%, um, from 16 to eight, and then another 30 to 40% multiplied from, um, eight to four. So that doesn't quite get you a two X, but like roughly a two X. Speculator, roughly a two X. Disag on top of that, if you're able to get enough hardware and put enough traffic through it, another roughly a two X. And then you add in some, you know, double digit percent increase from having just a, a better run time with, you know, the, the latest kernels and stuff behind it. Um, and that, that's kind of how it stacks up. Um, so building each of those, like building the, Um, quantized weights is, for someone who really knows what they're doing, hours to days of work, um, building the speculator, again, like, hours to days of work, and the, um, dis-ag setup, hours to days. Well, uh, ok, once, once you have it, yeah,

AI assessment note: “If you're doing it up front, it's quite a lot of work.”

Q Yeah. Anything else on the support side, when you, when you say, like, get it to fully production ready?

A Yeah, I think that, There's also a question of just, you know, we can test a model to a pretty extensive degree, but we're trying to get it out quickly, and then you see a bunch of other people test it, and you get interesting results. There was an issue with, um, GLM briefly, where we had some, like, mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures, like, Once you expose an endpoint to, to the real world, there's going to be, you know, so many more varieties of, of things given to it that, that you're able to, you know, discover and, and patch things. So it's not just a, you know, day zero process. It's then like for the first week, for the first month, if a model remains popular, like how do you both fix bugs and then continue to push the envelope on performance?

AI assessment note: “There was an issue with, um, GLM briefly, where we had some, like, mode collapses”

Q Yeah. Just, just, just, you know, you have more compute than me. Just go, just go try it. Yeah. Any other upcoming trends in inference engineering that we didn't cover? Like right now, you know, you, cause you guys are so close to it, you can obviously see it that the rest of the world doesn't know about.

A The big ones are obvious. Models get bigger. Hardware gets more powerful. Users get used to a certain level of speed and demand a higher one. I think that some things I'm excited about are systems level, you know, We, we still have a lot to think about in terms of composing multiple models together. If you think about a voice agent, there's three to five models involved in that and the communication between those models. There's a lot of new modalities that are coming out. Uh, there's like the cosmos, the new world model. Um, there's more research. Uh, speech to speech is still like not entirely a thing, uh, but it's, it's getting, it's getting closer. Uh, there's gonna be just a lot of new modalities to build around, which is gonna be exciting. And then, yeah, I think that the other thing to solve, which is something we've been solving for a long time and I'm not done with yet, is, is just going to be continuing to operate at another 10 X, another 10 X, another 10 X scale as an industry. If you think about the Degree of usage that AI has worldwide compared to, you know, some of the more mature technologies, both on, on consumer and business. It's, it's pretty clear that there could be multiple 10 X's more of demand. If you look at the infrastructure work industry wide, obviously it's been stood up very, very quickly to meet a unprecedented spike in demand. And that is like not…

AI assessment note: “I think that some things I'm excited about are systems level”