inference

“The shift away from training to inference as a fundamental, like, almost macro shift in how people deploy AI. I actually think that will destabilize slightly the dominance of NVIDIA chips.”

“I think, 95%.”

“The thing is these chips are on the market. They're here. I can, you know, out tab on Chrome and get one. That is something that, you know, I don't take lightly. Availability that is right. So I think Nvidia is used to stay at least if not for the H-one hundre…”

“If you go from one GPU to two, you don't get twice the performance. Maybe you get 10% better performance. Yeah, that's the dirty secret nobody talks about. I'm talking inference, right? So, so you go from, let's say, a hundred to a 110 by doubling the amount o…”

“The fundamental architecture of the GPU with off-chip memory is not great for inference. Now, they will continue to do well in inference, but it can be beaten, and I think they know it.”

“In a GPU, most of the time it's doing inference, it's five or seven percent utilized. That means it's 95 or 93% wasted.”

“The more inference you have, as mentioned before, the more you need to train the model to optimize for the inference. And the more training you have the more inference you want to deploy to optimize for the cost of that training, to amortize the cost of the tr…”

“To move people off GPUs in inference, and the number of people doing inference is vastly higher than the number of people doing training.”

“For simplistic folks, for founders, I say inference is the new sales and marketing.”

“Data center is the one that's the most under-invested categories today. When you look at inference needing to run for 24 hours a day for most of the knowledge worker population, and it running for like one percent of knowledge worker population today, I'm like…”

“Yeah, I mean, I think it's got to be three orders of magnitude more inference we run in the next five years.”

“On bare metal level, you have maybe a dozen of the customers in the world that you can work with. On managed infrastructure, there are hundreds. On inference, there are thousands. On agentic, there will be tens of thousands of new developers that build it, rig…”

“If, like, inference is going to commoditize, then, of course, the next best thing is for the model providers to be moving up the application layer in order to own more of the application stack so that they ensure that they're not commoditized and they're getti…”