The Exchanges

Q to make this, uh, broadly, uh, interesting to a large group of people, what, what is an ML developer? I think this, this, this, this concept that, um, you know, there's a lot of people like working on machine learning, uh, but there's actually different flavors of like people actually building the models, people deploying the models. Um, so what is an ML developer and who's your target audience specifically?

A Yeah. I mean, we, we, we say ML, um, practitioner in our, um, internal mission statement and stuff because we want to keep it a little bit vague. Like I've watched, um, I think I was originally hired as like a research scientist and then became data scientist and now might be ML engineer. And so I think these titles change a lot and in different industries, um, titles mean different things. So I don't mean like ML developer, like the title, I mean it more by, The function. So, um, we want to help the people that are literally just trying to get these models into production and do something. And there's different flavors, right? There's like typically like MLOps teams that are more, um, you know, operational and more like, how do we make this like a reliable, repeatable process? And, you know, we love them. Um, we also love the more kind of like research scientist oriented, um, you know, Person who's trying to get the last, you know, uh, bit of accuracy out of their model. And, and now we see a whole new, um, kind of archetype where it's like a software developer that's, um, you know, just really interested in machine learning and wants to, you know, get, get, um, you know, models working inside their company, maybe using like a third party API or something like that. So, um, You know, we're not too precious about, um, you know, which version of that you are. I think we need to …

AI assessment note: “people that are literally just trying to get these models into production”

Q So, so tracking what goes into the model, is that data? Is that, uh, what, what, what is it? Is that the model parts of the data part?

A Totally. I mean, I think there's like more things than you'd think, right? So, you know, like, uh, um, you know, when a model gets trained, you know, the data is super important, right? The training data that it's learning from, um, often there's like upstream pre-processing steps that are happening to the data before it's fed into the model. Um, there's things called, uh, hyperparameters, which might be like, you know, famous one is learning rate, sort of how fast does the model react to new data, but there's typically, you know, thousands of these, um, You know, kind of hyper parameters that, that people are always trying to tune. Um, there's sort of the architecture of the model, and that's changed a lot, um, over the years. So, you know, there's like broad categories of model, but even if you think about like neural nets, there's like different ways you can, wildly different ways you can, um, you can set that up. But then, you know, there's actually a lot of other things that go into it. Like, um, you know, what GPUs did you train your model on? Like what, um, you know, what infrastructure You know, were you using, like, what libraries were installed on your, on your system when the, the model changed, and what we kind of realized was that people wanted to keep track of all the stuff, but they didn't want to, like, explicitly write it down, and so one of our core tenets is …

AI assessment note: “training data that it's learning from, um, often there's like upstream pre-processing steps”

Q And, uh, so what, what kind of companies do you work with?

A Well, we work with Bloomberg. I was just here, yeah. Um, we work with, we work with any company that has a data science team, typically. So, that tends to be, um, retail companies, and, uh, financial services, and technology companies. So, um, you know, we, we tend to work with bigger companies because I think they're, they tend to, um, be more early adopters of, of data science teams. Um, but, but basically our, our, if, if you have somebody at your company with the title data scientist, then we probably have a good solution for you. And if you have somebody at your company with the title chief data officer, then we definitely have a good solution for you.

AI assessment note: “we work with any company that has a data science team, typically.”

Q Yeah. Actually, there's a little bit of a tangent on, on, on this. What are you seeing in terms of, um, the, the, the most sophisticated organization that you work with? Like rough estimate, like how many models do they actually have in, in, in production with the caveat that you just described? Um, just to give us a sense for like the, the state of maturity of this industry.

A Totally. Well, I'll tell you one funny thing, right? So we have a model registry where we can, like, count how many models people have, um, you know, in production. But, you know, the most common question that I get when I go into, like, an organization and talk about the model registry is, like, what should be, like, the, the model here, right? Like, because, because, like, you know, in the real world, you have typically, like, a model that's relying on classifications from, like, upstream models. So, you know, we have organizations that you might think of as having one real, Like application, but inside our model registry, they have hundreds of, um, models that are actually like tracking separately and deploying, um, you know, separately. And so when I see these like, um, you know, crazy, like, you know, studies where they ask like CIOs like, okay, how many models do you have in production? I have a feeling they have like no idea or like, you know, I, I wonder if you ask the same CIO in two different days, like how, how different the answer would really be. Cause I don't think it's like, um, It's like sort of saying, like, how many, like, applications do you have in production? I think it really, you'd probably get a lot of different answers, um, you know, inside an organization. Um, I mean, that said, we certainly have Um, you know, customers that, uh, you know, will have, l…

AI assessment note: “OpenAI has been, like, a longtime customer... they have a pretty small number of models”

Q Um, so this is the developer kind of bottoms up part. Um, but you mentioned that most of your customers are global, 2000 customers. So like, I, I have you guys presumably flipped at some point where you also need a top down, like a sales force that goes talk to those people, or I guess what, what does the go to market motion, the full go to market motion?

A Uh, yeah, yeah, totally. Um, So I think that's what I actually think this is a lot of ML companies struggle with this. Um, and my board was like complaining about this for a long time. Maybe they still secretly harbor resentment. I don't know. They're listening. Um, but, but I think like the challenge that we have is like, we're not like, uh, um, you know, like a Figma where like, we just totally spread virally within an organization. But the good thing is people do like use us in, in pockets, um, for free quite a bit. Inside everywhere. And, and, you know, ML, most of ML is inside of enterprise. That's where like most interesting work happens because that's where most of the data is. And so we have a lot of usage in most of the, you know, uh, what'd you call it? Global 2000, but it really unlocks it to have a salesperson, um, talk to them. So you can totally buy us with a credit card and I really want to make it so you can do a managed cloud deploy, um, with a credit card. We haven't, I haven't quite gotten that yet, but we've tried to make that like, you know, one click, boom, you have it. Um, and so, but, but we've really found that once we have some traction inside an organization, if we go in and we talk to the MLOps team and kind of get them on our side, then we can get companies to standardize on us. So, you know, for example, we'll have like credit card usage inside, I …

AI assessment note: “once we have some traction inside an organization, if we go in and we talk to the MLOps team”

Q count a couple of years, but like, obviously that's the big sea change of the last few months. How, how was that transition for you guys, given that You know, yet build this whole platform pre-explosion of LLMs. Do you view that as just like a plug and play? We have a broad platform. Here's a set of new capabilities, or is that, or does it change more than that?

A It changes more than that. I mean, I think broadly the workflow has a lot of similarities, right? Like I think, you know, if you've, if you've used LMs, I'm sure a lot of your audience, um, has, and you're trying to get them into production. If you're an engineer, there's some things that surprise you, right? Like these things are very non-deterministic. Um, it's very hard to tell if they're like working or not. Right. And, and your workflow is very like experiment oriented. I think where, Um, you know, like with coding, it's so satisfying, right? Cause you kind of write down these steps and then you debug it and then it's like out and it's like this deterministic thing that does what you think it does. Whereas the LMs, it's much more like this exploratory process where you try it in different ways and, and see what happens. And so I think that exploratory workflow is something that we understand well and are like really passionate about, um, you know, supporting. I think when you have that exploratory workflow, you really want an easy way to track everything that happens and, You know, kind of see if weird things are happening in production. So those things are the same, but then, um, there actually are some big, big differences. Like, I think that, um, you know, the, even the people Working with a lot of these APIs, um, you know, are, like, less, um, huge math nerds than, you…

AI assessment note: “It changes more than that. I mean, I think broadly the workflow has a lot”

Q Yeah. Not interesting. And, uh, if, uh, this one of LLM engineering AI is, is, uh, nascent and emerging, what is sort of the bread and butter in terms of like use cases that, uh, weights and biases in terms of where you see your customers, uh, use the platform. What, what do they do in terms of actual machine learning?

A Well, you know, I mean, so like, I would say the majority of users on a platform still are building their own models. Um, you know, I think that, It is kind of amazing when we look at, like, I think like one thing that surprises people when they, when they look at the company is how broad our, um, our customer base is, right? So like, you know, like a lot of people come in, they're like, Hey, what, look, it's like, why don't you like, you know, verticalize your like messaging, like you sell into so many different places, but it's like, well, no one vertical is more than like, you know, 15% of our, um, usage or revenue or anything like that. So it's kind of cool, right? Like ML is like just really, really broadly. Applicable. You know, there's like a lot of like manufacturing use cases of like, just like quality control from images. There's like, you know, making like cool characters inside video games. There's like, you know, looking at like health records and trying to predict that someone's going to get, um, you know, sick. I mean, there really are just so many different, um, you know, applications of ML. And I think when one gets like really kind of cookie cutter, then you get a company that does a great job with it. And people just buy the model from that, Um, you know, company that's like offering a complete packaged, um, solution. So like sort of wild west. I mean, even, …

AI assessment note: “manufacturing use cases of like, just like quality control from images.”

Q the, the, the big tech, uh, companies, the Ubers of the world are the more sophisticated ones, more willing to experiment and that, that, you know, uh, propagates through smaller startups and then, uh, to the global 2000. Is that, is that sort of what you're finding or are you, are you seeing that the global 2000 are already pretty sophisticated, When it comes to deploying machine learning and production?

A Well, I mean, I say this, I, I, most of our revenue comes from the global 2000. So it's not like, um, we only sell to like the Ubers and Apples and, you know, open-ended the world. We do. Um, I think, I mean, I, I actually think that Machine learning works so well and so obviously well for a lot of applications right now. I think people are kind of surprised to learn that I bet every might be strong. I bet 99% of the global 2000 is using machine learning for something that they actually really care about. Like, like, I, I think there's way more applications than people, um, realize, like, you know, like supply chain predictions, like sales forecasting, and, and like, you might say, well, why do people build like custom models for all this stuff? But it's like, why do people build their own tech stack? It's like, everybody has like a different, slightly different set of priorities, slightly different set of metrics they're trying to optimize. People think this stuff is core, so they don't want to outsource it. So, um, I, I think that our, our TAM is, I mean, I'm talking to VC, I guess, I don't know if you believe this, but I, I genuinely believe, I'm not pitching you, but I think our TAM is like basically all the global 2000, they all, um, you know, should be using wisdom biases in my opinion.

AI assessment note: “99% of the global 2000 is using machine learning for something that they actually really care about”

Q Who have you found for a machine learning tools company is a good, um, archetype for a salesperson, especially the first few? Are they, are they engineers turned salespeople? Are they salespeople who are more technical, or do you have a pot approach where you have, like, an AE and a technical person with them?

A Ooh, I think a lot about this. I've been, like, thinking about this for, like, 15 years now. Um, I mean, I took a lot of my good reps from my last company, because I think they, you know, kind of saw who was good at this. I, I'll tell you, my, when I interview, like, an AE, I'm kind of asking myself, like, one question, which is just, like, if my good friend was trying to buy weights and biases, Would I, like, hand them to this person? And there's, like, so many different ways that someone could be good at that, right? Like, I think if they're, like, nice and, like, humble, and they'll just, like, actually, like, listen to what the person's saying and, like, you know, treat them well, like, that's, like, a great way, and you don't have to be super technical. If they are, like, I mean, some of these reps are, like, amazingly Technically deep. Um, like, I don't know, we have this like VP of business development, and I feel like you ask him like a random open source software, like he'll give me the one sentence on what it does. It's like amazing. Like, I mean, I always feel bad. Like I should be on top of this stuff. I don't know how he, he does that. Right. But that's like actually like a pretty useful, um, thing. So I, I think it's like, it's really important that they bring, um, a kind of like humility and curiosity. Uh, to the table, but it's like hard to put your finger on. I…

AI assessment note: “I've just seen so many different ways to be successful as a sales rep.”

Q Um, so it's a centralized place where regardless of where you are in the organization or what office or what have you, you can, uh, just go check out what, what models we've people will say, well, how many mission learning models do we actually have in production? That's, that's where you can find out the answer.

A Exactly. And, and like, you know, in production is always it's complicated thing and like a real organization, right? There's often like candidate models or ones getting like a B tested and, um, you know, there, there's, there's just, and, and sometimes you'll have some models in production that flow into other models in production. And so, um, you know, it ends up like actually supporting real world workflows ends up being kind of, uh, a tricky thing that we're really passionate about. Um, So that's, uh, that's something we're really proud of that's been really popular lately. And then, and then you mentioned prompts is kind of our latest thing, you know, that we launched, which is all around, um, essentially using LLMs, um, which was, which was fun to make because, um, you know, I mean, I feel so proud of this. Like, oh, I think all of the, the major LLMs out there, almost all, um, were trained using weights and biases. So we had, like, seen all these come out, um, But then, um, we even, we were, like, shocked by how much adoption they've been getting lately, and, and, um, you know, people, I think, want to compare, um, you know, asking GPT-IV to, um, training your own model, and we, of course, want to support that.

AI assessment note: “Exactly. And, and like, you know, in production is always it's complicated thing”

Q know, selling a broad horizontal platform is awesome when you, when you get there and when you do have all those global 2000 customers, uh, but it can be pretty daunting. You mentioned, you know, the temptation to verticalize or, or, or, or, you know, or not like, how do you, I guess, how do you think about it? How do you sell, uh, what's your go to market motion?

A Yeah. And I will say it's like, it took us a really long time to get this working and there was like never like a moment where we felt like good about it, you know? So I do, it's funny to go back to like the early graphs. I remember like getting excited because like we were like at 30 weekly active users and like telling my early investors and they were just like, like, it's like, what are you doing? Like, what are you talking about? But it's actually kind of funny because the one thing about the graph was that it was like basically always going up. Yeah. So that, that gave me some hope. Um, But it was really small for a really long time, and there wasn't, people talk about like an inflection point, but that really hasn't been our, um, experience. It's been sort of like, um, you know, kind of like slow exponential growth that when you zoom out, it looks amazing, but when you're actually in it, it doesn't, you know, feel, um, that amazing. And, um, and it's been, um, the biggest driver of new users has always been Mystery. We call it like direct. It's like they just jump to the website, and they just use the product, and I think like we've tried to like triangulate, and this is a zillion different ways, and I think it's basically word of mouth, so I, I think we actually have a product that people like, and we know this because we measure NPS in a bunch of different ways religiou…

AI assessment note: “biggest driver of new users has always been Mystery. We call it like direct.”

Q What else moves the needle? The quality of documentation have you found, uh, makes a difference?

A Well, you know, it's funny. I always feel bad about the quality of our documentation. Sometimes people tell me it's good, and I don't believe them. Sometimes people tell me it's bad, and I get really, like, really anxious, but I don't know how to measure that. Like, I think it really matters. I think it really, really matters. Um, but, uh, I don't have any metric that like proves it. Um, and, and I think like, it'd be interesting to actually look at our NPS and see people specifically on the docs, what they think. Uh, but some people say it's great. Some people hate it. Um, I think, uh, yeah. Yeah. What else has mattered? Um, yeah, I don't know. It's funny. I feel like we, um, Oh, you know, lately, actually, um, it's kind of funny. We got our start. I was teaching some ML classes, or that was one of the things that started us. I actually think this was an amazingly good experience because I was teaching classes on ML and I was using weights and biases, and so what would happen was I would actually just watch, like, 80 students on board all at once, and a lot of them would fail, and that would, like, really stress me out because I'm literally, like, in front of the class trying to get them to use, like, my buggy products, and they're, like, mad at me, you know? I know, I know, exactly, and I would get so anxious, and then you'd sort of realize, like, You know, where people get s…

AI assessment note: “I think it really matters. Um, but, uh, I don't have any metric”