Apache Spark

“Spark, ah, there's a little known fact, Spark was actually the first, ah, application that was ever built on top of Mesos, and it was sort of the, you know, the demo app for Mesos in the beginning.”

“And Spark was, ah, you know, we targeted first some workloads which are not covered by Hadoop, and from all this experience I mentioned earlier, we look at iterative, iterative computations to support machine learning, as well as interactive computation, right…”

“We don't have our own distribution. We provide only the service.”

“We exceeded 500 contributors. It is the most active big data project right now, Spark.”

“As originally was built to run on top of Mesos. Today is working on Yarn, working, you know, standalone, and is working also in addition to HDFS, you know, imports and exports data to many other data sources.”

“A lot of contributions, still the majority of contributions comes from Databricks.”

“While we spark, You can use only one engine and only one API to support all these workloads.”

“Just October last year, we had this we won this kind of TerraSort benchmark. And in those, in that benchmark, the data, it's not in memory. Right? It's SSDs and so forth.”

“Now, I do think that looking forward you are going to see more and more of this unification. This happens in many other industries and technologies. I do think this will happen in big data because it's so much easier if you have only one system than if you nee…”

“At the time we made that bet, Spark was barely working. I mean, seriously. It was a really promising technology, very complete programming model. We loved the fact it was in memory, but we couldn't find a single customer that could keep their cluster running f…”

“One of the things that you find with things like Spark is that they really failed to shrink down and, Effectively do computing at the single node scale.”

“You can use spark at the single node scale as an alternative to pandas through the koalas interface, but you'll find that for many workloads, it's simply slower than pandas, which is not super impressive.”

“We use SG as so like the storage layer for our data warehouse, and we use Spark, Presto, Snowflake as our query engines.”

“I think initially just picking up Kafka, Spark, those are still so heavily in use that even if they go out in the next three to five years, it's going to take time to migrate and things of that nature.”