Facebook data infrastructure capacity planning consistently lagged behind demand
Ashish Thusoo · Ashish Thusoo, Qubole // Data Driven #26 // April 2014 (Hosted by FirstMark Capital) · May 27, 2014 · at 11:00
Ashish Thusoo, co-founder and CEO of Qubole and former head of Facebook's data infrastructure team, describes the operational friction of hardware capacity planning in on-premise environments.
0:00 / 0:29exact quote · 29.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Having, you know, run big clusters at Facebook, I remember conversations with the procurement team where it was like a big capacity planning exercise trying to predict the future, and yes, I know predictive analytics works to predict the future really well, but we couldn't do that, and at every point we were lagging behind the demand for infrastructure. So infrastructure would The demand for the infrastructure would outstrip anything that we could put in, ah, into those clusters and so on and so forth.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ashish Thusoo
Public cloud does not make financial sense at Facebook scale
“So that's an excellent question, and, ah, there is definitely an, there, ah, you know, there is an inflection point where people, ah, think that, ah, the total cost of ownership, ah, you know, once you're, ah, so suppose you're a Facebook skill, right, of cour…”
Unmodified on-premise Hadoop distributions fail in the cloud beyond 10 nodes
“If you just take a normal Hadoop distro and try to run it in the cloud, the chances are at 10 nodes it'll work fine as you start growing and, you know, as you start growing and growing and growing further. Things will start breaking because, you know, compute …”
Thusoo: Centralized enterprise data teams usually become operational bottlenecks
“More often than not, this, ah, data team eventually becomes a bottleneck, ah, and, ah, you know, it sort of delays, ah, delays the, ah, availability of data and what people can do with that data.”
Static Hadoop clusters in the cloud defeat the purpose of elasticity
“They just run long-running Hadoop clusters, which completely defeat the purpose of You know, how, how you can leverage the cloud to be dynamically adaptable to your workloads and things like that.”