Hadoop

“When I came to Spotify, We already had one of these large data sets of music playlists, and so there were some very talented people there, specifically a guy named Erik Bernadsson, who at the time, this is 2009, was doing machine learning something called coll…”

“We were on SQL server before, and that thing would never run with the data that was coming in. And we tried to build it ourselves, failed. And then Hadoop came out. And we were like, why don't we build on Hadoop?”

“Enterprises really started moving to cloud for their data in twenty-twenty. There was like Hadoop stuff going on before that, but that like never really achieved any level of maturity or real penetration. And yes, Redshift and stuff like did exist prior to, an…”

“One of the things, one of the transitions that Joe and I went through, which I think a lot of people in this room went through, was the transition from the Hadoop world, from the previous big data world, into this new, like, cloud-based data engineering snack,…”

“So the business models were not robust in the sense that really, it was really selling services masqueraded as software.”

“The people in Amplab that were doing machine learning, the math folks, they had to use this thing called Hadoop, which was just terrible.”

“Well, you can say maybe we should use Hadoop or BI. Unfortunately, these are kind of older generation tools. They require specialized skills and abilities to be able to use them. And so they need armies of analysts to use. These are really only affordable for …”

“With all the investment in Hadoop and this and Hadoop that, you know, most companies are still data bankrupt. You know, Hadoop or Data Warehouse or whatever is a place where data goes to die”

“Some of the most frequently used kind of data analytics tools extremely basic, because they actually look and feel like sheet of paper, like two-dimensional sheet of paper, and you know, so that's the problem with analytics, that on one hand, we have, you know…”

“Spark doesn't subsume data warehousing. Hadoop doesn't subsume, you know, streaming. So they're just like different technologies for different jobs.”

“I was a PhD student at UC Berkeley, and we actually started working with Hadoop users back in 2007. And I did, for example, an internship at Facebook when Facebook was only about 300 people and they were just starting to set up Hadoop.”

“So in particular you know, one of the things we saw is many of the projects that were built on top of Hadoop, such as Hive, which is a SQL processing at scale and Pig and Mahout for machine learning are starting to run on top of Spark as well, so that users of…”

“We don't necessarily get smarter over time. It's just that certain technologies get cheaper. They get, they become more available. So machine learning algorithms have always been around. The data that exists that you could collect has always been around. But i…”

“So I think Hadoop is, is, is a past technology. I think it's, although it's still gonna, people will still use it still has a place, I think it's not an area where there's gonna be a lot of incremental additional investment.”

“So other than search, everything I described is batch processing. We use standard Hadoop. We use MapReduce.”

“Most things, Hadoop, Spark, Storm, they think they're running by themselves. And so they compete for resources in really interesting ways.”

“Can you actually hire a good Hadoop DevOps engineer? Is it easy? I mean, you saw somebody stand up here saying they're recruiting. There's a reason, and it's because it's really hard to find these people, right?”

“We went from an ETL model, where we scraped from, like, the original source, transformed the data and loaded to Vertica, to, like, just an EL model, where we just, like, just copy the data as soon as possible into, like, Hadoop, and all the transformation can …”

“They're a Y Combinator company, built the third largest instance of Hadoop in the world, a real time predictive ad serving tool.”

“And to me it feels like, Big data. Maybe six or seven years ago where companies were real waking up and realizing we have all these data assets. We need to do something with them. And that led to the rise of Hadoop and the Hadoop vendors and then, you know, a …”

“You know, well, before we started Mapper, I spoke to, like, about 40 or 50 people who were using Hadoop. 50 companies.”

“Every company now has a Hadoop budget item. Almost every company.”

“Well, it's unfortunate, what I think is one of the most unfortunate thing that happened in the Hadoop space is kind of the introduction of SQL on top of Hadoop.”

“That's a data center OS, and I really think that's the next Hadoop killer.”

“I think one of the great innovations of Hadoop is this idea of schema unread, but that's not realized through to the front end, to the user itself”

“So at that point, in big data space we there was Hadoop just started, but of course that was, by, back then it was mostly, you know, batch, computation, so you could do historical analysis, but not much more than that.”

“If you look at the machine learning, it's, fundamentally, it's an iterative algorithm, and every iteration is turned into a Hadoop job. So between the iteration, you write the data and read the data from HDFS, so that's why it's very slow.”

“And Spark was, ah, you know, we targeted first some workloads which are not covered by Hadoop, and from all this experience I mentioned earlier, we look at iterative, iterative computations to support machine learning, as well as interactive computation, right…”

“Hadoop is still a very great, ah, batch engine.”

“This is the key thing that we learned from Hadoop and Google's MapReduce framework, which is You want to ship the code to where the data lives, not the other way around.”

“I wouldn't assert that it replaces Hadoop. In fact, it's based on UEMA. It's an Apache project.”

“What we didn't understand at the time, and it's been a pretty common feeling, is Hadoop wasn't built to solve the problem we'd been solving with relational databases. It was designed to solve a new problem, and it turned out that new problem was going to be ve…”

“Hadoop, this big scale-out, shared-nothing architecture, is good at much, but that architecture doesn't easily translate into OLAP or OLTP workloads.”

“I think the percentage of cycles spent on MapReduce in Hadoop clusters generally is going to asymptotically approach zero. That's not because there will be less MapReduce happening, but because there will be so much of the other stuff happening.”

“We have a 16 node Hadoop cluster currently, and I have on my proposed budget for 2015, a 200 node Hadoop cluster so that we can really get our hands on all that raw data of the thirty billion impressions we're transacting daily.”

“I think it will happen.”

“There's gonna be a lot of you know, broken hearts and tears are gonna fall, because I don't think that, that those valuations are warranted, are sustainable and that's too bad.”

“They just run long-running Hadoop clusters, which completely defeat the purpose of You know, how, how you can leverage the cloud to be dynamically adaptable to your workloads and things like that.”

“If you just take a normal Hadoop distro and try to run it in the cloud, the chances are at 10 nodes it'll work fine as you start growing and, you know, as you start growing and growing and growing further. Things will start breaking because, you know, compute …”

“And in particular, this sort of, the database engine itself is being replaced to a certain degree with things like Hadoop.”

“We were thinking about this problem back in 2005, right? So that was pre-Hadoop”

“This is where the market's going. There's going to be this convergence of, you know, sort of relational database technology and Hadoop, and this is the future, and”

“Sears actually, there's been some interesting things written about Sears going in that direction. Which you think of, you know, major retail, you wouldn't think they would be, you know, compared to Facebook, but they are, and they're making huge investments in…”

“Actually we found that Hadoop and most, kind of, no, no SQL solutions are not very good for, kind of, quantitative data when you need to aggregate and, kind of, go across these things.”

“Hadoop I think will be a definitive platform for a lot of big data workloads.”

“In each one of these, kind of, criteria, or vectors, or themes, there's a handful of billion dollar startups Ah yet to be, ah, yet to be built.”

“The world doesn't need You know, another 10 companies trying to solve the problems of Hadoop.”

“I think Hadoop, or HBase, which is a Hadoop subproject, that's more of a, that's something that's more of an alternative or competitive with Mongo, where you would look at A versus B”

“Facebook built Hive, right, because they needed a tool to sit on top of Hadoop, you know, to allow their business analysts to kind of sequel interface to this big data platform.”

“I think the problem with Mahoot is that anything, it's, many of these things are, if you run in Hadoop, you're slow. You need to be able to run in an environment that's fast”

“So everybody's gonna realize sooner or later that Hadoop is really complicated to install.”

“All it is is a structure for running queries in parallel against data that you store in a redundant file system, and so it is entirely impractical for doing a real-time product.”