The Exchanges

Q Yeah. Yeah. Um, and, and then, um, so that was the start of your data career. You also wrote a couple of popular sort of open source, uh, tooling, uh, while, while you were there. Um, and then, and then you, is that, is that correct or?

A No, that's right. I mean, I was at Spotify for seven years. This is a long stint. Uh, and, and Spotify was a wild place early on. And I mean, the data space is also a wild place. I mean, it was like Hadoop cluster in the like foosball room on the floor. Um, and you know, so, so like it was, It was a lot of crude, like, very basic infrastructure, and I didn't know anything about it, and, and, like, I was hired to kind of figure out data stuff, and I started hacking on a recommendation system, and then, you know, got sidetracked into a bunch of other stuff. I fixed a bunch of reporting things, and, and set up A-B testing, and started doing, like, business analytics, and later got back to music recommendation system, and a lot of the infrastructure didn't really exist. Like, there was, like, Hadoop back then, which is kind of bad, and I, I don't miss it, but spent a lot of time with that. Uh, as a part of that, I ended up building, uh, Workflow engine called Luigi, which is like briefly like somewhat like widely ended up being used by a bunch of companies. Sort of like, you know, kind of like Airflow, but like before Airflow, um, I think it did some things better, some things worse. Uh, I also built a vector database called Annoy, which is like for a while it was actually quite widely used, uh, in 2012. So it was like way before like all this like vector database stuff ended up ha…

AI assessment note: “I ended up building, uh, Workflow engine called Luigi”

Q How do you think of that compared to the traditional past history? Like, uh, you know, yeah. Yeah. Then you had Eroko, then you ran the railway.

A Yeah. I mean, I think they're all, those are all like great. Like, I think the problem that they all faced was like the graduation problem, right? Like, you know, Heroku or like, I mean, like also like Heroku, there's like a counterfactual future of like, what would have happened if Salesforce didn't buy them, right? Like that's a sort of separate thing, but, but like, I think what Heroku, I think always struggled with was like eventually companies would get big enough that you couldn't really justify running in Heroku. So they would just go and like move it to, you know, whatever AWS or, you know, in particular, uh, and, and, you know, that's something that keeps me up at night too. Like, like what you, what, what does that graduation risk like look like for modal? I always think like the, the only way to do, to build infrastructure, uh, to build a successful infrastructure company in the long run in, in, in the cloud today is, You have to appeal to the entire spectrum, right? Or, or at least like the enterprise, like you have to capture the enterprise market. And, ah, but the truly good companies capture the whole spectrum, right? Like I, I think of companies like, I don't like Datadog or Mongo or something like that where like, they both captured like the hobbyists like, and, and, um, and, and, and acquire them, but also like, you know, have very large enterprise customers. …

AI assessment note: “I think the problem that they all faced was like the graduation problem”

Q Um, how, what's some of the fun stuff you're working on to get a higher number there?

A Yeah, I think on the inference side, like that, that, that's where like, you know, like from a cost perspective, like utilization perspective, we've seen, you know, like very, Very good numbers. And in particular, like it's our ability to start containers and stop containers very quickly. And, ah, that means that we can, you know, we can auto scale extremely fast and scale down very quickly, which means like we can always adjust the sort of capacity, the number of GPUs running to the exact, you know, the, the, the traffic volume. And, um, so in many cases, like that actually leads to a sort of interesting thing where like we obviously run our things on like the public cloud, like AWS GCP, we run on Oracle. Uh, but in many cases, like users who, who, who do inference on those platforms or, or those clouds, uh, even though we charge a slightly higher price, uh, per, per GPU hour, a lot of users like moving their, their large scale inference use cases to model, like end up saving a lot of money because we only charge for like with the time the GPU is actually running. And, and that's a hard problem, right? Like if you go, you know, if you have to constantly adjust the number of machines, if you have to start containers, stop containers, That's a very hard problem. That, that, you know, and starting containers quickly is a very difficult thing. Uh, I mentioned we had to build our o…

AI assessment note: “we've implemented recently CPU, uh, memory checkpointing, so we can take running containers”

Q So, so, okay. Any, any other broader lessons, you know, just broadening out from, from like the single use case of fine tuning, like, um, what are you seeing people do with, uh, fine tuning or just language models on Modo in general?

A Yeah. I mean, I think language models is interesting because so many people get started with APIs and that's just, you know, they're just dominating a space in particular, open AI. Right. And, and, and that's not necessarily like a place where we aim to compete. I mean, maybe at some point, but like, it's just not like a core focus for us. And I, I think sort of separately, sort of question if like there's economics in that long term, but, but like, so we, we tend to focus on more like the areas, like the, around it, right? Like fine tuning, like another use case we have is a bunch of people, ramp included is doing batch embeddings on model. So, so let's say, you know, you have like a, actually we're like writing a blog post, like we, where we, we take all of Wikipedia and like, uh, parallelize embeddings in 15 minutes and, and, and produce vectors for each article. Uh, so, so those types of use cases, I think model suits really well for, uh, I, I think also a lot of, like, custom inference, like you have, like, you know, structured output, guided, uh, generation, or, or, uh, or, or things like that, we have, you want more control, like, those are the things, like, we see a lot of users using model for, uh, but for a lot of people, it's like, you know, just go use, like, GPT-IV, and, like, you know, that's, like, a great starting point, and we're not trying to compete necessari…

AI assessment note: “a bunch of people, ramp included is doing batch embeddings on model”

Q Yeah. Um, just, we can just broaden out for modal a little bit, but you still have a lot of, you have a lot of great tweets, so it's very easy to just kind of, uh, go through them. Um, why is Oracle underrated?

A I, I love Oracle's GPUs. Um, I mean, I don't, like, I don't know why, you know, what the economics looks like for Oracle, but, um, like, I, I think they're great value for money. Like, we, we run a bunch of stuff in Oracle and, um, They have bare metal machines with, like, two terabytes of RAM. They're, like, super fast SSDs. Uh, and yeah, like, compared to, you know, I mean, I mean, we love AWS and AGCP, too. We have great relationships with them. Uh, but I, I think Oracle, surprisingly, like, you know, if you told me, like, three years ago that I would be using Oracle Cloud, like, I'd be like, what? Wait, why? Uh, but now I'm, you know, I'm a happy customer.

AI assessment note: “They have bare metal machines with, like, two terabytes of RAM.”

Q Yeah, um, and so, like, um, obviously it's very hard to get hired at Modo, but, like, what is, like, uh, what is it like, um, to work with, like, such a talent density, like, you know, how is that contributing to the culture at Modo?

A Yeah, I mean, I think Humans are the root cause of like everything at a company, right? Like, you know, bad code is because it's bad human or like whatever, you know, bad culture. So like, I think, you know, like talent density is very important and like keeping the bar high and like hiring smart people. And, you know, it's not always like the case that like hiring competitive programmers, it's the right strategy, right? If you're building something very different, like you may not, you know, but we actually end up having a lot of like hard, you know, complex challenges. Like, you know, I talked about like the cloud Uh, you know, the resource allocation, like turns out like that actually, like you can phrase that as a mixed integer programming problem. Like we now have that running in production, like constantly optimizing how we allocate cloud research. There's a lot of like interesting, like complex, like scheduling problems and like, how do you do all the bin packing of all the containers? Like, so I, I, you know, I think for, for, you know, for, for what we're building, you know, it makes a lot of sense to hire these people who like, like those very hard problems.

AI assessment note: “makes a lot of sense to hire these people who like, like those very hard problems”