Everything Wes McKinney said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Commercial entities are necessary to scale and sustain open-source projects
“It became clear to me and to many people that that to have more of a commercial engine behind Arrow and the Arrow ecosystem was important for enabling the ecosystem to continue to grow for us to be able to pour a lot more resources into the open source project…”
Apache Spark failed to effectively shrink down to single-node scale
“One of the things that you find with things like Spark is that they really failed to shrink down and, Effectively do computing at the single node scale.”
Over 90% of Python tabular data passes through Pandas
“I'd say, you know, nine, more than 90% of the data that's coming into structured data processing in, in the Python ecosystem, tabular data processing is passing through pandas at some point at some point in its lifetime.”
McKinney predicts every data warehouse will soon support Apache Arrow
“So I think that, that, you know, in the course of the next few years you know, pretty much every database system, every data warehouse Is going to support aero based import and export in some format.”
Running Apache Spark on a single node is slower than Pandas
“You can use spark at the single node scale as an alternative to pandas through the koalas interface, but you'll find that for many workloads, it's simply slower than pandas, which is not super impressive.”
Pandas never had a significant corporate sponsor
“Pandas never really had a significant corporate sponsor who was you know putting in the majority of contributions. Like it really was a community project almost, you know from the get go.”
ODBC and JDBC were never designed for bulk data transfer
“Protocols like interfaces like ODBC and JDBC were never designed
Or intended for bulk data transfer, like on the order of gigabytes, for example.”
Apache Arrow strictly complements rather than competes with Databricks
“It's neither it's neither a competitor or a replacement, so it's strictly a complimentary technology.”
Pandas has had well over 2,000 open-source contributors
“I know there've been well over 2000 contributors at this point.”
Apache Arrow was built to bridge database and data science developers
“For me, one of the primary motivators was to create a technology which could you know, proverbially tie the room together and enable that, that cross-pollination between database developers and data science developers that had just never never existed because …”
Snowflake and Google BigQuery support exporting query results to Apache Arrow
“Snowflake exports supports exporting query results to arrow format. So it is big query.”
The term 'data frame' originated in the R programming language
“Data frame is a term that arose from originally in the R programming language which was based on the S and S plus programming languages.”
the MAD Podcast 3