Hadoop

“All it is is a structure for running queries in parallel against data that you store in a redundant file system, and so it is entirely impractical for doing a real-time product.”

“So everybody's gonna realize sooner or later that Hadoop is really complicated to install.”

“I think the problem with Mahoot is that anything, it's, many of these things are, if you run in Hadoop, you're slow. You need to be able to run in an environment that's fast”

“Facebook built Hive, right, because they needed a tool to sit on top of Hadoop, you know, to allow their business analysts to kind of sequel interface to this big data platform.”

“I think Hadoop, or HBase, which is a Hadoop subproject, that's more of a, that's something that's more of an alternative or competitive with Mongo, where you would look at A versus B”

“The world doesn't need You know, another 10 companies trying to solve the problems of Hadoop.”

“In each one of these, kind of, criteria, or vectors, or themes, there's a handful of billion dollar startups Ah yet to be, ah, yet to be built.”

“Hadoop I think will be a definitive platform for a lot of big data workloads.”

“Actually we found that Hadoop and most, kind of, no, no SQL solutions are not very good for, kind of, quantitative data when you need to aggregate and, kind of, go across these things.”

“Sears actually, there's been some interesting things written about Sears going in that direction. Which you think of, you know, major retail, you wouldn't think they would be, you know, compared to Facebook, but they are, and they're making huge investments in…”

“This is where the market's going. There's going to be this convergence of, you know, sort of relational database technology and Hadoop, and this is the future, and”

“We were thinking about this problem back in 2005, right? So that was pre-Hadoop”

“And in particular, this sort of, the database engine itself is being replaced to a certain degree with things like Hadoop.”

“If you just take a normal Hadoop distro and try to run it in the cloud, the chances are at 10 nodes it'll work fine as you start growing and, you know, as you start growing and growing and growing further. Things will start breaking because, you know, compute …”

“They just run long-running Hadoop clusters, which completely defeat the purpose of You know, how, how you can leverage the cloud to be dynamically adaptable to your workloads and things like that.”

“There's gonna be a lot of you know, broken hearts and tears are gonna fall, because I don't think that, that those valuations are warranted, are sustainable and that's too bad.”

“I think it will happen.”

“We have a 16 node Hadoop cluster currently, and I have on my proposed budget for 2015, a 200 node Hadoop cluster so that we can really get our hands on all that raw data of the thirty billion impressions we're transacting daily.”

“Hadoop, this big scale-out, shared-nothing architecture, is good at much, but that architecture doesn't easily translate into OLAP or OLTP workloads.”

“I think the percentage of cycles spent on MapReduce in Hadoop clusters generally is going to asymptotically approach zero. That's not because there will be less MapReduce happening, but because there will be so much of the other stuff happening.”

“What we didn't understand at the time, and it's been a pretty common feeling, is Hadoop wasn't built to solve the problem we'd been solving with relational databases. It was designed to solve a new problem, and it turned out that new problem was going to be ve…”

“I wouldn't assert that it replaces Hadoop. In fact, it's based on UEMA. It's an Apache project.”

“This is the key thing that we learned from Hadoop and Google's MapReduce framework, which is You want to ship the code to where the data lives, not the other way around.”

“And Spark was, ah, you know, we targeted first some workloads which are not covered by Hadoop, and from all this experience I mentioned earlier, we look at iterative, iterative computations to support machine learning, as well as interactive computation, right…”

“So at that point, in big data space we there was Hadoop just started, but of course that was, by, back then it was mostly, you know, batch, computation, so you could do historical analysis, but not much more than that.”

“Hadoop is still a very great, ah, batch engine.”

“If you look at the machine learning, it's, fundamentally, it's an iterative algorithm, and every iteration is turned into a Hadoop job. So between the iteration, you write the data and read the data from HDFS, so that's why it's very slow.”

“I think one of the great innovations of Hadoop is this idea of schema unread, but that's not realized through to the front end, to the user itself”

“That's a data center OS, and I really think that's the next Hadoop killer.”

“Well, it's unfortunate, what I think is one of the most unfortunate thing that happened in the Hadoop space is kind of the introduction of SQL on top of Hadoop.”

“You know, well, before we started Mapper, I spoke to, like, about 40 or 50 people who were using Hadoop. 50 companies.”

“Every company now has a Hadoop budget item. Almost every company.”

“And to me it feels like, Big data. Maybe six or seven years ago where companies were real waking up and realizing we have all these data assets. We need to do something with them. And that led to the rise of Hadoop and the Hadoop vendors and then, you know, a …”

“They're a Y Combinator company, built the third largest instance of Hadoop in the world, a real time predictive ad serving tool.”

“We went from an ETL model, where we scraped from, like, the original source, transformed the data and loaded to Vertica, to, like, just an EL model, where we just, like, just copy the data as soon as possible into, like, Hadoop, and all the transformation can …”

“Can you actually hire a good Hadoop DevOps engineer? Is it easy? I mean, you saw somebody stand up here saying they're recruiting. There's a reason, and it's because it's really hard to find these people, right?”

“So other than search, everything I described is batch processing. We use standard Hadoop. We use MapReduce.”

“So I think Hadoop is, is, is a past technology. I think it's, although it's still gonna, people will still use it still has a place, I think it's not an area where there's gonna be a lot of incremental additional investment.”

“The people in Amplab that were doing machine learning, the math folks, they had to use this thing called Hadoop, which was just terrible.”

“One of the things, one of the transitions that Joe and I went through, which I think a lot of people in this room went through, was the transition from the Hadoop world, from the previous big data world, into this new, like, cloud-based data engineering snack,…”