Apache Spark

“It turned out that, you know, it's easier to go from that bod thing that's really good at the scale and ingesting and super low cost and create versions in it that have the speed and features of the, you know, super easy to use, like smaller data for business …”

“When I joined, we had hundreds of thousands of spark free users.”

“I think initially just picking up Kafka, Spark, those are still so heavily in use that even if they go out in the next three to five years, it's going to take time to migrate and things of that nature.”

“With open source, you have people in the room that will raise their hand and say, I know what Spark is. I've already used it. I downloaded it on my laptop at home. I know how this Delta, I'm a big fan. I know what this is. So that moves things along much faste…”

“From early on, we said, look technology comes and goes. Spark in 10 years will have probably aged and hopefully we'll come up with other innovations, so let's pick a name that separates the company from the open source technology, and hopefully we can continue…”

“We use SG as so like the storage layer for our data warehouse, and we use Spark, Presto, Snowflake as our query engines.”

“One of the things that you find with things like Spark is that they really failed to shrink down and, Effectively do computing at the single node scale.”

“You can use spark at the single node scale as an alternative to pandas through the koalas interface, but you'll find that for many workloads, it's simply slower than pandas, which is not super impressive.”

“So actually, Mesos was the first project we developed. This is what started the stack. And Mesos was by design to support multiple cluster computing framework. We started with Hadoop. And actually, one of the reasons we designed Spark, it's To show that it's m…”

“The place where Apache Spark was born, UC Berkeley's Amplab has not just created a major open source software platform, it's spun out more than its share of groundbreaking companies.”

“One of the coolest ones I saw was a talk from Toyota about how they use Spark to improve, you know, to basically look at social media feedback, what people are writing about their cars, and figure out things like, oh, is there a problem with the brakes on the …”

“So in particular you know, one of the things we saw is many of the projects that were built on top of Hadoop, such as Hive, which is a SQL processing at scale and Pig and Mahout for machine learning are starting to run on top of Spark as well, so that users of…”

“So I think even beyond the activity happening in Spark itself, these projects on top and on the side are one of the most valuable things for the users.”

“It's the same Spark that anyone else gets in the open source. All the libraries, all the improvements we put into the engine, you can just download them and run them yourselves. Or if you want you know, you can talk to a vendor that provides support. Support o…”

“At the time we made that bet, Spark was barely working. I mean, seriously. It was a really promising technology, very complete programming model. We loved the fact it was in memory, but we couldn't find a single customer that could keep their cluster running f…”

“Here's the new algorithm, you know, again, implement on Spark. It's that little red box there. It took about a couple hundred seconds to get it, And the answer quality is better than the bootstrap after 15,000 seconds.”

“And Spark was, ah, you know, we targeted first some workloads which are not covered by Hadoop, and from all this experience I mentioned earlier, we look at iterative, iterative computations to support machine learning, as well as interactive computation, right…”

“As originally was built to run on top of Mesos. Today is working on Yarn, working, you know, standalone, and is working also in addition to HDFS, you know, imports and exports data to many other data sources.”

“Just October last year, we had this we won this kind of TerraSort benchmark. And in those, in that benchmark, the data, it's not in memory. Right? It's SSDs and so forth.”

“While we spark, You can use only one engine and only one API to support all these workloads.”

“We exceeded 500 contributors. It is the most active big data project right now, Spark.”

“A lot of contributions, still the majority of contributions comes from Databricks.”

“We don't have our own distribution. We provide only the service.”

“Now, I do think that looking forward you are going to see more and more of this unification. This happens in many other industries and technologies. I do think this will happen in big data because it's so much easier if you have only one system than if you nee…”

“Spark, ah, there's a little known fact, Spark was actually the first, ah, application that was ever built on top of Mesos, and it was sort of the, you know, the demo app for Mesos in the beginning.”