AI inference

“I don't find, I find the DevOps for inference to be relatively easy. It doesn't feel that different than, you know, I think we had thousands and thousands of servers at Mixpanel just for dealing with the API had such huge quantities of volume that I didn't fin…”

“If you only push on one direction, you are going to reach diminution return really, really quickly. Yeah, there's only that much you can do on the system side, only that much you can do on the algorithm side. And since the only big thing that's going to happen…”

“It only works if you really know which chip you're going to do. If you don't, then it's a little harder. So it makes, in my head, it makes more sense for inference where you've already established it, but for training there's so much, like, experimentation.”

“The cost when you are local first and your typical consumer is on a free plan or a Like, 20 dollar a month paid plan limits the amount of high quality inference you can do, and the scale or volume of inference you can do per, like, outcome.”

“Training scales the size of your research team. Inference scales the size of your customer base.”

“And so things like PyTorch came on the scene and I think PyTorch gets all credit for democratizing model training, right? It's taught to pretty much every computer science student that graduates. That's a huge deal, but nobody democratized inference. Inference…”

“I think that there is today, like. 10 times as much AI inference that could exist than is existing right now, just Purely with projects that are like sitting in the proof of concept stage and have not been deployed because there's like huge bucket of those. An…”