The Ledger, every show

“So I, I'm very bullish on software engineers. I think it would take a lot to sort of destroy that demand. I think people look at a lot of like AI as like a kind of fix something, but in my opinion, it's like, No, it's just gonna unlock more latent demand for m…”

“I do think it's kind of fascinating how it's like kind of going in the same direction as like crypto mining towards more like specialized hardware.”

“So we built our own, we threw out Kubernetes out the window. We threw out Docker out the window. We built our own File system in order to optimize for how container images are distributed. We built our own scheduler in order to maintain this pool of workers an…”

“It's obvious that like in the long run, like inference spend will dominate just for like the obvious reason that like, that's where you make the money, right? Like, you know, when you're training a big model, that's like, you know, you, you're spending a lot o…”

“In the end, like, I don't think VCs will have the return they expected. Like, you know, in these things, but guess who's going to benefit? Like, you know, it's the consumers, right? Like someone's like reaping that the value of this. And that's, I think an ama…”

“I think there's so many benefits of this multi-tenant model in terms of capacity management that to me, it is very clearly like a big part of the future of AI is like running a big pool of compute and slicing it very dynamically.”

“Like, so, I don't know, I'm a big believer that, like, you know, in the long run, like, people will train a lot of different custom models.”

“For a while, like, each 100 prices were, like, almost, like, going up, and then they were kind of still for a while, and then now they're, like, actually going down quite a lot, like, you know, what I'm seeing in the market, right?”

“You look at the developers are, I've been getting like probably 10 X more productive every decade for the last four, four decades or something that was kind of crazy. Like on an exponential scale, we're talking about 10 X or is there a 10,000 X like, you know,…”

“In many cases, like users who do inference on those platforms or those clouds even though we charge a slightly higher price per, per GPU hour, a lot of users like moving their large scale inference use cases to model, like end up saving a lot of money because …”

“I'm sure their VCs at some point are gonna force them to reconsider.”

“I think a lot of these companies over in the long run, like, you know, they're accepted maybe super big ones, like, you know, the Facebook and Google, they're always gonna build their own ones, but like everyone else, like some extent, you know, I think they'r…”

“It's like pretty easy in Moto, like, fan out to like, you know, at least like a hundred GPUs, like in a few seconds, and you know, if you give it like a couple of minutes, like we can, you know, you can fan out to like thousands of GPUs.”

“Margins are gonna go up for sure, but I don't know if prices will go up, because like, GPU prices have to drop eventually, right? So like, you know, like in the long run, I still think like prices may not go up that much but certainly margins will go up.”

“We run a very big compute pool, like thousands of GPUs and CPUs, and we make it very easy to get, you know, if you need a hundred GPUs, we can typically get you that within seconds.”

“The more native, like AI native storage solution would be, you put text in, you put, you know, video in, you put image in, and then you can search by that. Like to me, that would be like a more sort of native, AI native sort of storage solution. So that's like…”

“Meteorology is like something I actually think, like, deep learning should, like, change, right? Like, it sort of makes a lot of sense. Like, you know, deep learning should be very good at, like, you know, predicting, you know, turbulence and things like that.…”

“Suno which is AI generated music. And, you know, they have a big cluster. They train their own models outside of modal and then they use modal for the inference side.”

“I think the general wind is like blowing in the direction of like multi-tenancy and like, you know, security moving away from the sort of network layer into the app layer and a bunch of different other things.”

“At the end of the day, like EC two is like what, like, you know, 50, 60, 70% margins. It's actually, you know, not as much of a commodity as like people think.”

“Margins on Lambda is much higher than margins on, on GPU serverless products.”

“Yeah, so, I mean, they're using model for, like, production infrastructure, like, they have their own, like, custom model, like, custom code and custom weights, you know, for AI-generated music, Suno.ai.”

“So, so one example of a customer I think is super cool building really amazing stuff is Suno. Which does AI generated music. So they run all their inference on modal very large scale.”

“No one told me who my manager was for the last year, for the first year of Spotify.”

“Yeah, I mean, the company, like, grew from, like, 10 people when I joined to 10,000, now it's back to a thousand, and, but yeah, they actually went public a few months ago, kind of crazy.”

“So that was, like, the first, like, sort of stuff we started working on was, like, let's build this, like, container file system, And, you know, coupled with like, you know, just using RunC directly. And that actually enabled us to like get to this point of li…”

“You know, the resource allocation, like turns out like that actually, like you can phrase that as a mixed integer programming problem. Like we now have that running in production, like constantly optimizing how we allocate cloud research.”