inference

“I think actually, I think from a computation perspective, it's very inefficient to have like a trillion parameters and have to actually load them all and add and multiply by them. Each time you make an inference, because they're just encoding knowledge, most o…”

“Inference, like inference already dominates open AI compute usage, right?”

“And so we're going to be down at three and at six billion and at thirteen billion, because that's, I can get pretty good, pretty darn good, and not break the bank with free inference.”

“I think an inference is really interesting because we do need so much more compute for inference than we need for training, right? Like it's a very interesting like economy of scale. You train once, like Lama trained once and then everybody runs it everywhere.”

“Resiliency and reliability matters a lot more. You know, downtime is unacceptable from an input perspective. Nodes get terminated all the time from a training perspective.”

“Even the GPU clusters themselves, like, you know, the full training networking is a very, very important Piece to have networking on the racks themselves with inference and matters a little less because you're doing a little bit more on individual GPUs and les…”

“Over time, applications are going to want efficient inference, and, like, really large models are impossible today to serve for the vast majority of use cases from a cost and speed perspective”

“As you move forward, generative AI honestly becomes one of the compute building blocks that you think about. You're going to need storage, you need compute, you need databases, you need inference, if you will, for your application, largely. And I think that's …”

“Traditionally, most of modal has always been inference. Like that's been our main use case, but we're really interested also in training. So in particular, like probably focused more on these like shorter, like very bursty sort of experimental training runs, n…”

“We believe that a lot of inference is going to happen on your end device, but there will always be some model which is too big or too resource intensive. And in that case, the next best place to run it is going to be on at the inside the network at the edge.”

“Even if there's AGI, all that's left is inference.”

“The open source models have crossed some sort of chasm in terms of their baseline. Capability, and then I think RL techniques and post-training is for specialized models has become mainstream enough, and, you know, there's enough examples of its work, of it wo…”

“GPUs as a service is not sticky. I think that's been seen. Like, customers generally just see that as commodity. Inference with the software layer included is incredibly sticky.”