AI inference

“When we started, the first misconception, which people don't hold anymore, is that training was more expensive than inference.”

“In terms of infra, probably the number one thing that is the number one difference between these two is the need for interconnect. So if you do, you know, production, you, if you can avoid to have interconnect between, you know, let's say a cluster of GPUs, of…”

“And that's number, probably the number one thing that, you know, gives you a lot of efficiency in terms of spend. Like we're talking, you know, multiples, like, you know, five, you know, sometimes 10 X, you know, improvement.”

“In inference, it's not real at all. There's no CUDA lock-in in inference. None. Well, you can move from OpenAI on an NVIDIA GPU to Cerebrus to Fireworks Service on something else to Together to perplexity with 10 keystrokes. I mean, anybody who actually uses A…”

“They are making a bunch of architectural changes to GPUs to make them better and better inference.”

“I believe inference itself is not such a hard algorithmic solve that, you know, you need to pay someone else to do it for you, but clearly training a model is”

“If we lower what we charge 50%, people are gonna buy twice as much. They're spending as much as they're making because whatever they spend increases the quality of the output.”

“The greater growth of inference is the number of people who use it, Times the frequency of use. Times the amount of compute needed per use. Right? It is three different variables multiplied by each other. The problem is they're all growing fast.”

“When you have demand like this, like you have for the initial GC, like the initial hyperscaler clouds. And I think we're seeing an even greater cohorted demand curve for AI inference. Sometimes you just got to shut your mind up and invest with the momentum.”

“Whether it happens widely in knowledge work, By 20, 26, I think is, you know, I'll take a bet with you on that. Certainly for coding. I think in a lot of places we're already there, right? I mean, you have like the Ralph Wiggum stuff that, that popped up over,…”

“Why do we believe that inference will be any different? There'll be zero marking for slowing them.”

“One is rare disease research, which I think is one of those things that has been intelligence bottleneck or really just the inference bottleneck. Like it involves like trying out lots of ideas and seeing if they work.”