Stoica predicts big data infrastructure will eventually unify around one system
Ion Stoica · Ion Stoica, Databricks // Creating Apache Spark // Data Driven NYC (FirstMark Capital) · Apr 2, 2015 · at 21:53
Ion Stoica, co-founder of Databricks and creator of Apache Spark, speaks at Data Driven NYC about whether Spark will coexist with or replace traditional data warehouses.
0:00 / 0:19exact quote · 20.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Now, I do think that looking forward you are going to see more and more of this unification. This happens in many other industries and technologies. I do think this will happen in big data because it's so much easier if you have only one system than if you need to deal with many.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ion Stoica
Stoica: Apache Spark won the TerraSort benchmark processing data out of memory
“Just October last year, we had this we won this kind of TerraSort benchmark. And in those, in that benchmark, the data, it's not in memory. Right? It's SSDs and so forth.”
Stoica: Apache Spark supports all major data workloads with one engine
“While we spark, You can use only one engine and only one API to support all these workloads.”
Stoica: Hadoop's HDFS read/write cycle crippled early iterative machine learning
“If you look at the machine learning, it's, fundamentally, it's an iterative algorithm, and every iteration is turned into a Hadoop job. So between the iteration, you write the data and read the data from HDFS, so that's why it's very slow.”
Stoica: Hadoop remains a very great batch processing engine
“Hadoop is still a very great, ah, batch engine.”
Stoica: Spark Streaming and Storm have roughly similar throughput
“They are roughly, you know, in terms of the throughput, they are roughly similar.”
Stoica: Spark Streaming micro-batching is unfit for high-frequency trading latency
“It's very hard to, you know, if you want millisecond latency from the time you, ah, the data entered in the system until you get the result, it's very hard to get. You can get latencies of several hundreds of milliseconds, but milliseconds, very hard. Ah, you,…”