MapReduce

“He also implemented the first version of AdWords, built AdSense, rewrote the core search pipeline five times, co-invented and implemented Bigtable MapReduce, TensorFlow, and Gemini.”

“But you know, nowadays, like, you know, I've got a Mac M two laptop. It's two years old. It's like probably an order of magnitude to two orders of magnitude faster than the server machines were back when, you know, MapReduce came out and people started buildin…”

“This was the genesis for them to create MapReduce, which they wrote a paper about. Which was a way to parallelize and break into pieces all the little bits of crawling and re-indexing the web.”

“And the way that all that stuff worked was my co-founder would write a MapReduce script behind the scenes when someone would set up a campaign. They would write in plain English what they wanted the campaign to do. We would manually look at the data that they …”

“MapReduce initially came out of Google, where it was used for web indexing, and the whole point was, I will run this giant job every night, and in the morning, it's built a new index of the web.”

“Interestingly, a lot of people may not realize that MapReduce was designed to be slow.”

“So other than search, everything I described is batch processing. We use standard Hadoop. We use MapReduce.”

“What we learned is that, ah, stream processing is in fact much more than a faster MapReduce layer. It, in fact, most of the applications that do stream processing look much more like a microservice of an application, and less like a faster version of a batch o…”

“I think the percentage of cycles spent on MapReduce in Hadoop clusters generally is going to asymptotically approach zero. That's not because there will be less MapReduce happening, but because there will be so much of the other stuff happening.”