Inference

“As models are getting cheaper and cheaper, more and more of that actually can just be done using inference instead of using determined, like things like graphs or things like embeddings.”

“Like, you'll, you'll know that influence is pretty much solved when researchers start publishing about how they got one percent faster at something.”

“And with training, it's more of like a math, like you can run the math and see the flops and maximize it. With inference, it's more of like an auto-tuning, like GPU kernel auto-tuning... You shadow the same traffic, like real traffic, and you just see which co…”

“If, like, inference is going to commoditize, then, of course, the next best thing is for the model providers to be moving up the application layer in order to own more of the application stack so that they ensure that they're not commoditized and they're getti…”

“We will have so much usage of our models that we do not need to be a gigantically high margin business to be able to afford model training. Like, so much of our future compute plans Will be used to sell inference to customers that even if we can enjoy a modest…”

“Inference is a big, perhaps even the majority on compute.”

“We don't like to make a distinction between Training and infants, because a lot of training is now infants. So when we train a new model, we are generating synthetic data, for example. That's inference. When we train a new model, we are doing post-train, and t…”

“And meanwhile, we sort of know that you have positive gross margins on inference alone of sort of 40, 50%, but you've got the cost of building the next model, and you don't know where the cost The cost will move, or the cost of the next model will move, and yo…”

“Every day there's over a billion dollars of revenue in this category, and a lot of it's inference.”

“When you hear about all these companies, and you hear about all of these businesses that are going not like, you know, one to three to nine to 20, like they used to, but are going one to 20 to a hundred, or one to 30 to 300, all of that can be traced back to t…”

“And because of where we deploy systems, we often are in places where We don't have to pay for the space or the power, which allows us to then pass those savings on to customers. So it can be significantly more cost effective to use inference with us than it …”

“The whole big bet of open AI, you know, trying to build these target and other You know, big data centers is not that they just want to train on this, because that cost is a one-time cost, right? So you train ones, you sort of run inference for a lifetime. So …”

“On bare metal level, you have maybe a dozen of the customers in the world that you can work with. On managed infrastructure, there are hundreds. On inference, there are thousands. On agentic, there will be tens of thousands of new developers that build it, rig…”

“Anthropic I'm sure is at very positive gross margins on inference today.”

“And the disaggregation of inference means that I think these GPUs are going to have 10 or 15 year lives.”

“If we're doing reinforcement learning on the model, it's basically inference within a sandbox with a reward function, right? And so if the model's better at more efficient inference, that RL is more efficient as well.”

“The open source models have crossed some sort of chasm in terms of their baseline. Capability, and then I think RL techniques and post-training is for specialized models has become mainstream enough, and, you know, there's enough examples of its work, of it wo…”

“GPUs as a service is not sticky. I think that's been seen. Like, customers generally just see that as commodity. Inference with the software layer included is incredibly sticky.”

“Even if there's AGI, all that's left is inference.”

“I always think of inference as the monetization of the investment in artificial intelligence.”

“Yeah, I mean, I think it's got to be three orders of magnitude more inference we run in the next five years.”

“Our product is both training and inference, but I think the first sales will be an inference. That's mostly just a market effect where It's easier to buy, like, it's not as big of a risk to go to buy an inference cluster than as a training cluster.”

“Data center is the one that's the most under-invested categories today. When you look at inference needing to run for 24 hours a day for most of the knowledge worker population, and it running for like one percent of knowledge worker population today, I'm like…”

“For simplistic folks, for founders, I say inference is the new sales and marketing.”

“The GPU, for example, has a lot of capacity of memory, but it's really slow. All right, and that's a huge bottleneck in inference. It's why they can't be fast.”

“I'm sure NVIDIA will still do great and still grow beyond their five trillion dollar company or whatever it is at the time of recording. But I think you're going to see a lot more diversity especially around, I think inference of the model.”

“Primarily inference that funds the free usage of the platform and then also hiring, of course, headcount. We have an office, you know. That's an SF.”

“If you must update weights, then, like, you know, weights become stateful, and today, like, inference is not stateful.”

“I'm of the belief that 50% of your inference and your queries will be on device. They will not be on device going to the cloud. They will be on device. Locally hosted on models that are either etched into the silicon or using flash memory.”

“It does seem as if most compute gets spent on inference that companies don't so far regret like using to offer their products.”

“To move people off GPUs in inference, and the number of people doing inference is vastly higher than the number of people doing training.”

“The more inference you have, as mentioned before, the more you need to train the model to optimize for the inference. And the more training you have the more inference you want to deploy to optimize for the cost of that training, to amortize the cost of the tr…”

“Over 40% of your revenue today is inference.”

“When you think about the total cost of ownership of these models, it's gonna be very, very heavily weighted towards inference. It's all inference.”

“What I do expect is I expect the cost of inference to stay the same or go up as we allow users to do much more complicated queries over much larger data sets.”

“We believe that a lot of inference is going to happen on your end device, but there will always be some model which is too big or too resource intensive. And in that case, the next best place to run it is going to be on at the inside the network at the edge.”

“Our thesis is inference is not the business. We, yeah, we want to give the end user total transparency into price, into, which I think is like incredibly important to, you know, even get comfortable with the idea of spending as much money as you do. I think th…”

“And I mean, we have Jensen Wang, the CEO of Nvidia saying inference is going to take a hundred times more compute than traditional LLM inference.”

“In a GPU, most of the time it's doing inference, it's five or seven percent utilized. That means it's 95 or 93% wasted.”

“The fundamental architecture of the GPU with off-chip memory is not great for inference. Now, they will continue to do well in inference, but it can be beaten, and I think they know it.”

“The thing is these chips are on the market. They're here. I can, you know, out tab on Chrome and get one. That is something that, you know, I don't take lightly. Availability that is right. So I think Nvidia is used to stay at least if not for the H-one hundre…”

“If you go from one GPU to two, you don't get twice the performance. Maybe you get 10% better performance. Yeah, that's the dirty secret nobody talks about. I'm talking inference, right? So, so you go from, let's say, a hundred to a 110 by doubling the amount o…”

“I think now what we're seeing is that inference in the aggregate is actually probably a much larger draw than the training in the long run.”

“They clearly did a bunch of optimizations to make the code run faster. And there's just no question that inference is cheaper. Like that is true.”

“I think, 95%.”

“The shift away from training to inference as a fundamental, like, almost macro shift in how people deploy AI. I actually think that will destabilize slightly the dominance of NVIDIA chips.”

“Traditionally, most of modal has always been inference. Like that's been our main use case, but we're really interested also in training. So in particular, like probably focused more on these like shorter, like very bursty sort of experimental training runs, n…”

“NVIDIA's moat in, in inference is actually A lot smaller on software but it's a lot bigger on, hey, they just have the best hardware.”

“In fact, there's more inference in training than there is updating the model weights, because you have to generate hundreds of possibilities And then, oh, you only train on a couple of them, right?”

“Our prediction is for those kind of applications, the inference is much more important than training. Because inference scale is proportional to the upliminal world population. And training. Training scale is proportional to the number of researchers.”

“Microsoft has said of its ten billion dollars in AI revenue, it's almost all inference.”

“We know Nvidia has said half of their revenue is inference”

“40% of their revenues are already inference.”

“What we know is that the cost of inference has fallen by 90% over the course of last year.”

“People used to think that pre-training was hard, and inference was easy. Now everything is hard.”

“And I'm sure I said it would be greater.”

“As you move forward, generative AI honestly becomes one of the compute building blocks that you think about. You're going to need storage, you need compute, you need databases, you need inference, if you will, for your application, largely. And I think that's …”

“Over time, applications are going to want efficient inference, and, like, really large models are impossible today to serve for the vast majority of use cases from a cost and speed perspective”

“AI is really two markets, training and inference is going to be a hundred times bigger than training.”

“NVIDIA is really good at training and very miscast at inference.”