DoorDash runs Databricks on AWS S3 and ingests data from Snowflake
Alok Gupta · Fireside Chat: Alok Gupta (Head of Data Science & ML, DoorDash) with Matt Turck (Partner, FirstMark) · Feb 1, 2021 · at 21:24
Alok Gupta, Head of Data Science & ML at DoorDash, clarifies DoorDash's data warehouse and machine learning infrastructure architecture.
0:00 / 0:07exact quote · 7.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We run a Databricks on S-III, but we pull data into Databricks from Snowflake, so we have a connection between the two.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Alok Gupta
DoorDash aims to offer white-label infrastructure services modeled after AWS
“I think we would like to think of ourselves as an infrastructure company. That's right. Where we build products and services to help us in the things we're trying to do, but ultimately we get them to a stage of maturity and robustness through experimenting on …”
Problem framing is the hardest part of the machine learning lifecycle
“I think right at the start of a project is probably, and probably the hardest and most impactful part of the life cycle of the model. I think I prefer to call it sort of data-driven software because it could ultimately be it could be something other than a mod…”
DoorDash commits raw training scripts instead of pickled machine learning models
“Instead of building a model and pickling it or whatever a conversion package you want to use, we commit the actual training script to our model library.”
Deploying ML models to production at DoorDash requires data science approval
“It is perfectly acceptable for anyone at DoorDash to Build a machine learning model, but if they want to put it into production, they need approval or a review from someone like a data scientist or machine learning engineer.”
Training dedicated models for distinct cohorts beats fitting general models
“If there's a strong enough use case for a different cohort or segment like new users, typically it'll make more sense just to train a new model rather than trying to shoehorn it into a general model.”
Grocery expansion turns DoorDash from a three- to four-sided marketplace
“The food delivery is a three-sided. You're right. There's the merchant, the consumer, and the dasher. As we start to branch out into grocery delivery, convenience delivery, it becomes a four-sided where we introduce a picker as well.”