Burgess outlines Pinterest's online serving and batch data engineering stack
Dave Burgess · Fireside Chat: Dave Burgess (Head of Data Engineering, Pinterest) w/ Matt Turck (Partner, FirstMark) · Apr 5, 2021 · at 22:07
Dave Burgess, Head of Data Engineering at Pinterest, details how Pinterest structures its data engineering team and infrastructure across online serving databases and batch platforms.
0:00 / 0:24exact quote · 24.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So data engineering at Pinterest comprises the serving part, all the online systems. So that includes the online databases like MySQL and key value stores like RocksDB and HBase, and also includes Druid platform. So we have our online systems. We also have all the batch systems and analytics and experimentation platforms.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dave Burgess
Pinterest visual search beats Google and Bing in head-to-head tests
“You'll find that Pinterest is usually the one that comes out of top.”
Apache Kafka was built on prior engineering work at Yahoo
“Things like Kafka eventually came out of LinkedIn that was based on work that we had been doing at Yahoo”
Pinterest manages open-source data tech in-house to limit downtime
“Yeah, we do. I mean, we're fortunate to have the number of engineers to do that, and one of the reasons why we work directly either with the open source community or the companies that are also working on that is that we want Pinterest to be up all the time, a…”
Pinterest embeds machine learning engineers directly within product teams
“With machine learning is actually distributed everywhere. We have Many, many machine learning engineers and many, many use cases. And those mission machine learning engineers are embedded within each organization.”
Pinterest wants non-coding data scientists to deploy ML models to production
“And to get to a point where we can have people that are data scientists that don't even code, that can just build models and be able to deploy those to production. So that's really what we want to get to within Pinterest as well.”