Pinterest reduced maintenance costs, latency, and infrastructure expenses by migrating to Druid
Dave Burgess · Fireside Chat: Dave Burgess (Head of Data Engineering, Pinterest) w/ Matt Turck (Partner, FirstMark) · Apr 5, 2021 · at 27:25
Dave Burgess is the Head of Data Engineering at Pinterest. He details the technical outcomes of migrating Pinterest's analytics and experimentation infrastructure from HBase to Apache Druid.
0:00 / 0:12exact quote · 12.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And so we decided to migrate that to Druid. And so the maintenance cost has gone way down. The latency has gone way down. The actual cost of running the infrastructure way down. So really, really happy.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dave Burgess
Pinterest visual search beats Google and Bing in head-to-head tests
“You'll find that Pinterest is usually the one that comes out of top.”
Apache Kafka was built on prior engineering work at Yahoo
“Things like Kafka eventually came out of LinkedIn that was based on work that we had been doing at Yahoo”
Pinterest manages open-source data tech in-house to limit downtime
“Yeah, we do. I mean, we're fortunate to have the number of engineers to do that, and one of the reasons why we work directly either with the open source community or the companies that are also working on that is that we want Pinterest to be up all the time, a…”
Pinterest embeds machine learning engineers directly within product teams
“With machine learning is actually distributed everywhere. We have Many, many machine learning engineers and many, many use cases. And those mission machine learning engineers are embedded within each organization.”
Pinterest wants non-coding data scientists to deploy ML models to production
“And to get to a point where we can have people that are data scientists that don't even code, that can just build models and be able to deploy those to production. So that's really what we want to get to within Pinterest as well.”