Pinterest

“As part of QBOL, we run, ah, very large Hadoop clusters on the cloud for, you know, companies like Pinterest Quora, MediaMath, DataZoo, and so on and so forth.”

“At Pinterest we use Redshift. We're big users of Amazon's Redshift product. That's been a pretty substantial game changer for us moving away from Hive.”

“There might be five people in this room, the people who raised their hands, who could sit down and just derive this from first principles. I am certainly not among those people, and I am a working data scientist.”

“Pinterest of course only uses AWS, as they've mentioned.”

“And so we decided to migrate that to Druid. And so the maintenance cost has gone way down. The latency has gone way down. The actual cost of running the infrastructure way down. So really, really happy.”

“Yeah, we do. I mean, we're fortunate to have the number of engineers to do that, and one of the reasons why we work directly either with the open source community or the companies that are also working on that is that we want Pinterest to be up all the time, a…”

“And to get to a point where we can have people that are data scientists that don't even code, that can just build models and be able to deploy those to production. So that's really what we want to get to within Pinterest as well.”

“And so we have about a thousand experiments running in parallel at any point of time.”

“So data engineering at Pinterest comprises the serving part, all the online systems. So that includes the online databases like MySQL and key value stores like RocksDB and HBase, and also includes Druid platform. So we have our online systems. We also have all…”

“You'll find that Pinterest is usually the one that comes out of top.”

“And so, there's a, that whole pipeline has got like about 50 different machine learning models in itself.”

“We're still using the APIs for the existing system so that we could migrate, but we have over 2000 workflows running in production.”

“We have more than 400 petabytes of data.”

“So we, what we can do is within a few hours, you could do create a model, train a model, and then deploy it to our production service with thousands and thousands of servers by using this MLflow repository.”

“So we have about 80 different use cases of machine learning.”

“With machine learning is actually distributed everywhere. We have Many, many machine learning engineers and many, many use cases. And those mission machine learning engineers are embedded within each organization.”