The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 14 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Apache Spark failed to effectively shrink down to single-node scale
“One of the things that you find with things like Spark is that they really failed to shrink down and, Effectively do computing at the single node scale.”
Early Apache Spark was so unstable that clusters crashed within two hours
“At the time we made that bet, Spark was barely working. I mean, seriously. It was a really promising technology, very complete programming model. We loved the fact it was in memory, but we couldn't find a single customer that could keep their cluster running f…”
Stoica predicts big data infrastructure will eventually unify around one system
“Now, I do think that looking forward you are going to see more and more of this unification. This happens in many other industries and technologies. I do think this will happen in big data because it's so much easier if you have only one system than if you nee…”
Running Apache Spark on a single node is slower than Pandas
“You can use spark at the single node scale as an alternative to pandas through the koalas interface, but you'll find that for many workloads, it's simply slower than pandas, which is not super impressive.”
Stoica: Apache Spark supports all major data workloads with one engine
“While we spark, You can use only one engine and only one API to support all these workloads.”
Stoica: Apache Spark won the TerraSort benchmark processing data out of memory
“Just October last year, we had this we won this kind of TerraSort benchmark. And in those, in that benchmark, the data, it's not in memory. Right? It's SSDs and so forth.”
Rogojan: Enterprise migration away from Spark and Kafka will take years
“I think initially just picking up Kafka, Spark, those are still so heavily in use that even if they go out in the next three to five years, it's going to take time to migrate and things of that nature.”
Stoica: Apache Spark has exceeded 500 active open-source contributors
“We exceeded 500 contributors. It is the most active big data project right now, Spark.”
Stoica: Databricks still provides the majority of Apache Spark open-source contributions
“A lot of contributions, still the majority of contributions comes from Databricks.”
Netflix uses S3, Spark, Presto, and Snowflake for data querying
“We use SG as so like the storage layer for our data warehouse, and we use Spark, Presto, Snowflake as our query engines.”
Stoica: Apache Spark was created for iterative machine learning and interactive queries
“And Spark was, ah, you know, we targeted first some workloads which are not covered by Hadoop, and from all this experience I mentioned earlier, we look at iterative, iterative computations to support machine learning, as well as interactive computation, right…”
Stoica: Databricks offers only cloud services, not a custom Spark distribution
“We don't have our own distribution. We provide only the service.”