Apr 5, 2021 · 33m · mad
Fireside Chat: Dave Burgess (Head of Data Engineering, Pinterest) w/ Matt Turck (Partner, FirstMark)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Data Driven NYC fireside chat hosted by Matt Turck, Dave Burgess, Head of Data Engineering at Pinterest, details Pinterest's scale, big data infrastructure, real-time streaming architectures, machine learning platforms, and open-source initiatives like Querybook.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 25.1% of the talking time here. How this is scored →
Matt as informed peer 3.7 Guest teaching 4.4 Guest disagreement 0.0 Matt pushing back 0.3
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 10:22 Dave gently corrects host's assumption about Redshift
In a completely non-combative interview, this represents the mildest pushback where Dave corrects Matt's assumption about using AWS Redshift by clarifying they use Oracle for financial data.
Hardest push from Matt ▶ 19:57 Matt questions reliance on commercial open-source vendorsMatt pushes past the general architecture discussion to ask whether Pinterest depends on third-party commercial vendors like Confluent for Kafka or manages the code bases directly.
Biggest teaching moment ▶ 11:07 Dave explains schema-level PII identificationDave educates the host on how Pinterest handles PII detection at the source schema level rather than relying solely on downstream scanning tools.
Matt holds his own ▶ 2:12 Matt highlights Yahoo's historical importance in big dataMatt displays deep industry familiarity by pointing out Yahoo's massive influence as an incubator for the modern big data ecosystem.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Dave Burgess's Career Path and Background | 4 | 3 | 0 | 0 | Matt demonstrates industry context by highlighting Yahoo's foundational role in the big data ecosystem. Dave warmly agrees and expands on Yahoo's contributions to Hadoop and early ML. | |
| Announcement and Overview of Querybook | 3 | 4 | 0 | 0 | Matt asks clarifying questions about Querybook to frame it as a collaboration layer and query execution interface. Dave confirms and explains its role within Pinterest. | |
| Pinterest Data Engineering Stack and PII Handling | 5 | 5 | 0 | 1 | Matt demonstrates domain familiarity by prompting a definition of Parquet, referencing author Julien Ledem, and guessing Redshift usage. Dave clarifies they actually use Oracle for financial metrics and details their S3 data lake and PII tagging. | |
| Machine Learning Use Cases and Infrastructure at Pinterest | 4 | 5 | 0 | 0 | Matt offers commentary on Pinterest's ad quality and clean user environment before pivoting to ML frameworks. Dave details Pinterest's ML platform for glueing TensorFlow, PyTorch, and MLflow. | |
| Real-Time Data Streaming with Kafka, Flink, and Druid | 4 | 5 | 0 | 1 | Matt asks about streaming adoption and presses gently on whether Pinterest relies on commercial vendors like Confluent or manages OSS directly. Dave explains why direct management ensures maximum uptime. | |
| Data Team Organizational Structure at Pinterest | 3 | 4 | 0 | 0 | Matt inquires about organizational structure and transitions into rapid-fire ecosystem questions. Dave outlines how platform data engineering supports embedded ML teams across Pinterest. | |
| Audience Q&A: Infrastructure, Monitoring, and Career Advice | 3 | 5 | 0 | 0 | Matt moderates audience questions on Druid, monitoring tools, Airflow, and career paths. Dave provides detailed explanations including Pinterest's custom Goku time-series database. |
Statements from this episode (14)
Apache Kafka was built on prior engineering work at Yahoo
“Things like Kafka eventually came out of LinkedIn that was based on work that we had been doing at Yahoo”
Yahoo used ML and behavioral targeting in ads over 15 years ago
“We were doing machine learning and behavioral targeting back 15 years ago, more than 15 years ago in advertising.”
Pinterest runs about 1,000 parallel A/B experiments at any time
“And so we have about a thousand experiments running in parallel at any point of time.”
Pinterest deploys machine learning across roughly 80 distinct use cases
“So we have about 80 different use cases of machine learning.”
Pinterest visual search beats Google and Bing in head-to-head tests
“You'll find that Pinterest is usually the one that comes out of top.”
Pinterest platform deploys ML models across thousands of servers within hours
“So we, what we can do is within a few hours, you could do create a model, train a model, and then deploy it to our production service with thousands and thousands of servers by using this MLflow repository.”
Pinterest uses 50 machine learning models in its pin safety pipeline
“And so, there's a, that whole pipeline has got like about 50 different machine learning models in itself.”
Pinterest manages open-source data tech in-house to limit downtime
“Yeah, we do. I mean, we're fortunate to have the number of engineers to do that, and one of the reasons why we work directly either with the open source community or the companies that are also working on that is that we want Pinterest to be up all the time, a…”
Burgess outlines Pinterest's online serving and batch data engineering stack
“So data engineering at Pinterest comprises the serving part, all the online systems. So that includes the online databases like MySQL and key value stores like RocksDB and HBase, and also includes Druid platform. So we have our online systems. We also have all…”
Pinterest embeds machine learning engineers directly within product teams
“With machine learning is actually distributed everywhere. We have Many, many machine learning engineers and many, many use cases. And those mission machine learning engineers are embedded within each organization.”
Pinterest wants non-coding data scientists to deploy ML models to production
“And to get to a point where we can have people that are data scientists that don't even code, that can just build models and be able to deploy those to production. So that's really what we want to get to within Pinterest as well.”
Pinterest reduced maintenance costs, latency, and infrastructure expenses by migrating to Druid
“And so we decided to migrate that to Druid. And so the maintenance cost has gone way down. The latency has gone way down. The actual cost of running the infrastructure way down. So really, really happy.”