The Exchanges

Q Interesting. Let's switch to the team, the machine learning and data science team at DoorDash. How big is it? How many people? And, um, what, what kind of people do you have on the team in terms of function?

A We, so a year and a half ago when I joined, um, we had six, five or six people on the team. We're now at, uh, almost 30 people a year and a half later, and this year we want to double, so to get to, uh, 50 plus people. Um, these data scientists, I, I wrote a blog post recently, um, that went live, I think last Monday, um, we're looking for, of course, Technical brilliance. You know, we have people who are typically have a master's or PhD in a, um, quantitative subject, um, plus some years of industry experience. Um, so that's a given and we have a very rigorous, um, challenge before you come on site and have, uh, one-on-one interviews with us at the test that, but what we, what we really look for in the interviews as well is an ability to, or, A real motivation to solve business problems rather than a desire to build cool models. Um, I think that's well, and it's okay if you want to build interesting models and write interesting algorithms. Um, that's probably best saved for academia or some research labs. For us, we're really interested in moving business metrics, and that's what we look for. We look for people who will do anything to move those business metrics, and over the course of a year, Most of your time will be spent building interesting models to serve that purpose. Um, but, um, we have found a lot of people that, um, are very technically gifted, but don't, don't imme…

AI assessment note: “We're now at, uh, almost 30 people a year and a half later”

Q And, um, you had this interesting theme around the democratization of, of machine learning, um, and I understand you created a machine learning council, I think you guys called it. Uh, can you, can you talk to that?

A Yes, some of the, some of the tensions I'd seen at previous places, and I'd also heard from others, is who, who should work on ML? You know, should it be data scientists, or ML engineers, or the AI lab? And, um, I found a lot of these discussions, um, I thought I could make these, these discussions and these conflicts unnecessary by essentially creating A set of principles for machine learning at DoorDash whereby we say machine learning is a tool and we encourage everyone to be able to use it. We don't try and put restrictions on what people can do. What we do do is put, um, is, is to put hard, hard, uh, lines, um, hard boundaries on what people are accountable for. So it is perfectly acceptable for anyone at DoorDash to Build a machine learning model, but if they want to put it into production, they need approval or a review from someone like a data scientist or machine learning engineer. So we try to divorce what someone can work on from what they're accountable for. The machine learning council was a way to bring different flavors of machine learning together to co-build and co-strategize. And again, to diffuse some of the territorialness that can develop at larger companies. And so we have people from engineering, From the ML platform, from data science, and we'll add other people in the future, and it's, it's a place for people to discuss ideas, discuss strategy, discuss h…

AI assessment note: “The machine learning council was a way to bring different flavors of machine learning together”

Q Maybe compare and contrast this overall infrastructure against what you saw at Airbnb and Lyft. Are you seeing a convergence of the architectures being more or less the same, um, or is it, um, you know, or do you see significant differences in, in terms of like how people build those machine learning sort of pipelines?

A The themes are definitely similar. Um, I would say at all three places, there was a desire for a centralized stack so that, um, iterative improvements sort of helped all models. Um, for the feature store, for example, um, there's a real tension between the data we use to train models offline and the actual data hitting the models when a prediction needs to be served. And, uh, there can be divergence between those two things. So you could imagine if I'm, um, pulling my data from a warehouse to create a feature that I train a model. And then in real time, I'm trying to compute that feature. They can actually diverge based on, I don't know, timestamps, rounding errors, precision, um, formats. And so a way to create a feature store whereby we can consolidate the training data with the, The prediction serving time data is a, is a common sort of holy grail for, for all the companies I've been at in their pursuit. Um, another thing is being able to house models that can make batch predictions. So I can send a set of, um, for example, if I want to make a prediction for someone's life expectancy, I can send a hundred sets of features for a hundred people and make a Batch prediction in one go. So make a hundred predictions versus in sequence, sending, um, a person one at a time and making a prediction in serial. And so there's a real, again, holy grail is how do you create an ML platform…

AI assessment note: “The themes are definitely similar. Um, I would say at all three places”

Q like you guys looked at like a bunch of different things and then sort of converge towards a smaller subset of models. So just, uh, again, reading through some of the stuff I've, I've seen XGBoost, LightGBM, CatBoost, obviously TensorFlow and PyTorch on the, on the neural network, um, side. Are those, is that, is that the, the core of it or are you always Experimenting, trying new, new things?

A Yeah, the, the motivation at the start was to, um, how do we get everybody at the company onto the same centralized machine learning platform stack? So, uh, when I joined a year and a half ago, we'd already had models in search, fraud, logistics, et cetera, and they're all on their own, um, own stack. And so the first thing was, how do we bring everything onto one stack? So the first question was, okay, how do we build something quickly, which solves most people's use case? And so we landed on, um, using a framework that enables tree-based models. And we picked light GBM for that after trying a few different, um, packages and also deep learning. And for that, we then used PyTorch. And so we started with those two, um, core libraries. And we basically said to everyone, if you want to use the platform, you have to build to this. And yes, over time, we will add capabilities for other things. So it wasn't, it wasn't so much a constraint or restriction. It was more of a prioritization. How do we enable the most people to migrate to a platform as quickly as possible?

AI assessment note: “we started with those two, um, core libraries.”