Unmodified on-premise Hadoop distributions fail in the cloud beyond 10 nodes
Ashish Thusoo · Ashish Thusoo, Qubole // Data Driven #26 // April 2014 (Hosted by FirstMark Capital) · May 27, 2014 · at 22:34
Ashish Thusoo, CEO of Qubole, explains why early-stage startups should not try to run traditional Hadoop distributions on cloud infrastructure themselves.
0:00 / 0:14exact quote · 14.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“If you just take a normal Hadoop distro and try to run it in the cloud, the chances are at 10 nodes it'll work fine as you start growing and, you know, as you start growing and growing and growing further. Things will start breaking because, you know, compute and storage are separate on the cloud.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ashish Thusoo
Public cloud does not make financial sense at Facebook scale
“So that's an excellent question, and, ah, there is definitely an, there, ah, you know, there is an inflection point where people, ah, think that, ah, the total cost of ownership, ah, you know, once you're, ah, so suppose you're a Facebook skill, right, of cour…”
Thusoo: Centralized enterprise data teams usually become operational bottlenecks
“More often than not, this, ah, data team eventually becomes a bottleneck, ah, and, ah, you know, it sort of delays, ah, delays the, ah, availability of data and what people can do with that data.”
Static Hadoop clusters in the cloud defeat the purpose of elasticity
“They just run long-running Hadoop clusters, which completely defeat the purpose of You know, how, how you can leverage the cloud to be dynamically adaptable to your workloads and things like that.”
Thusoo: On-premise big data deployments take 6 to 18 months
“If you are, you know, going with a traditional approach of, you know, standing this up in your data centers, you know, ah, I think some of the estimates are it takes between six to 18 months to get to production.”