Commercial entities are necessary to scale and sustain open-source projects
Wes McKinney · Fireside Chat: Wes McKinney (Founder & CEO, Ursa Computing) with Matt Turck (Partner, FirstMark) · Feb 1, 2021 · at 19:15
Wes McKinney, creator of Pandas and Apache Arrow, explains why he transitioned from open-source consortium Ursa Labs to commercial startup Ursa Computing.
0:00 / 0:17exact quote · 17.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It became clear to me and to many people that that to have more of a commercial engine behind Arrow and the Arrow ecosystem was important for enabling the ecosystem to continue to grow for us to be able to pour a lot more resources into the open source project.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Wes McKinney
Apache Spark failed to effectively shrink down to single-node scale
“One of the things that you find with things like Spark is that they really failed to shrink down and, Effectively do computing at the single node scale.”
Over 90% of Python tabular data passes through Pandas
“I'd say, you know, nine, more than 90% of the data that's coming into structured data processing in, in the Python ecosystem, tabular data processing is passing through pandas at some point at some point in its lifetime.”
McKinney predicts every data warehouse will soon support Apache Arrow
“So I think that, that, you know, in the course of the next few years you know, pretty much every database system, every data warehouse Is going to support aero based import and export in some format.”
Running Apache Spark on a single node is slower than Pandas
“You can use spark at the single node scale as an alternative to pandas through the koalas interface, but you'll find that for many workloads, it's simply slower than pandas, which is not super impressive.”