The Exchanges

Q of the Python world. Um, especially when you started all of this, it wasn't super clear that Python was going to be the dominant language for, uh, data analysis and data science, but it's, it's seems to be very much the case now. Like what, what, why do you think that happened? And, and do you think that's a permanent state or, or could it, uh, could other languages emerge?

A Yeah, it certainly, it certainly wasn't clear, like it wasn't, uh, predetermined or inevitable that Python was going to become the number one data language that, that it is, uh, that it is today. Um, I think a part of that was a relatively small group of very passionate open source developers building the essential projects, which, which enabled, um, the people who are now called data scientists to do complete workflows, uh, in Python. And so we needed Uh, the array computing that you have in NumPy, we needed the tabular data manipulation, data cleaning, data loading that you have in pandas. We needed to be able to do machine learning that came from scikit-learn, which was developed around the same time. We needed a nice, uh, user interface, a programming environment that, that came from IPython and the Jupyter project. We needed to be able to, to visualize data. So it was kind of this perfect storm of these different tools coming together And, uh, and enabling a productive workflow. I think another thing, uh, that was a catalyst for, for Python was the fact that so many companies were started, uh, to, to get value out of all of the new data that was being generated by smartphones and by mobile browsing. Um, and so there, the time to market and the speed at which you could develop systems that, that created actionable insights on data and production systems that would deliver, …

AI assessment note: “it was kind of this perfect storm of these different tools coming together”

Q Your focus for the last few years has been Arrow. When, when, when, when and how did that start?

A Well, it was, ah, I, I would describe it as this, um, collective, ah, realization in, in the 2015, ah, ah, time frame that, that the, the entire data ecosystem was facing this grand data interchange and, and data interoperability crisis, and that's partly because there were all these different independent groups of people that built different, ah, computing engines, ah, data processing systems, ah, There are a lot of different file formats. There was the clouds that people were starting to build the first data lakes, um, in the cloud. Uh, and so there was all this angst about like, how do we move data and transport data between different systems? How do we get access to all these different data formats, uh, efficiently? Um, and all that being said, like the hardware underneath our feet, like disk drives, networking, everything was getting a lot faster. And so we found ourselves really limited and held back by, Um, the speed of getting access to data and the costs associated with moving data around. Um, so, uh, I was at Cloudera at the time. So, uh, you know, I was interested in this problem and I'd experienced it from the perspective of pandas and the Python ecosystem, like wanting to build bridges from Python into all of these other, all of these other systems. And so we started, uh, folks from Cloudera. So people from the Impala team, from the Kudu team, They'd worked with, u…

AI assessment note: “collective, ah, realization in, in the 2015, ah, ah, time frame”

Q Why is it so hard to extract data from a database? Can't you just use an ODBC, JDBC connector, you get the data out and you're done?

A Well, it's, uh, we could spend 20 minutes or an hour just talking about this, but, uh, it's, there's many problems. So, so part of the problem is that the way that, that people consume the results of database queries is so diverse. And so, uh, not, not everything is a data frame library or, or an analytical tool. Um, so, uh, so protocols like interfaces like ODBC and JDBC were never designed Or intended for bulk data transfer, like on the order of gigabytes, for example. So they are best used for relatively small amounts of data that are the results of database queries. So all the work happens in the database. The results are relatively small and they come into your application through something like ODBC or JDBC. There are also row oriented interfaces, which makes them an awkward fit when you need to convert to a column oriented tool like pandas. Um, and so there's just all this, there's this impedance or this, this, uh, conversion penalty, uh, going from, going from things like ODBC into, into pandas. And so, you know, it's kind of as, as a, as a user of these tools, it's, you kind of have this feeling of, you know, drinking a thick milkshake through a straw that's too small. So you're just trying really hard to like extract the data out of the database so that you can do, uh, do work on it, but, but you end up bottlenecked on that, on just the, the transfer, uh, cause databa…

AI assessment note: “interfaces like ODBC and JDBC were never designed Or intended for bulk data transfer”

Q And so how does Arrow work and how does it solve the problem?

A Well, we, uh, we designed a, uh, language independent, uh, standardized, uh, data format that, that is column oriented, that can be used for bulk data movement, um, data transfer, uh, but that is also an efficient, uh, data format, uh, for doing analytics. So if you get a bunch of arrow data out of a processing system, out of a database into memory, you can immediately go to work doing, um, doing further data processing on it without a need To convert it into some other data format. So traditionally data comes into your, uh, into your Python interpreter, into your, into your R interpreter, your process, and you immediately have to convert it into, um, into some other, into some other format. And so arrow removes the, the need for, for, for that, for that conversion. So aside from, you know, transporting bytes over, over the network, um, the, uh, the data is, is essentially ready for, uh, ready for analysis as soon as it gets into your hands.

AI assessment note: “arrow removes the, the need for, for, for that, for that conversion.”

Q incredibly, uh, complicated, expensive and time consuming to, to do this. So our, uh, as a very strategic place in the ecosystem in, in that precisely enable all those pieces to, to work together. Um, so just to, uh, just to reiterate, um, so, so precisely to this point about, uh, data warehouses, uh, what, what's the status of, uh, the integration, uh, with a Snowflake or BigQuery or others?

A Yeah, I mean, that's been, that's been one of one of the most successful use cases for the project, um, is as a, a standard, uh, uh, medium or, or data format for, for bulk interchange with, uh, with data warehouse systems. So Snowflake exports, uh, supports exporting, uh, query results to, to arrow format. So it is big query. Uh, Microsoft has a number of internal, uh, projects in their, uh, cloud infrastructure. For, uh, direct to arrow, uh, export. Um, and part of, part of what's motivated the, the data warehouses to support arrow is because we build a really, uh, a really efficient bridge between arrow and pandas. And so if their goal is to get gigabytes of data out of their data warehouses, into the hands of data scientists, rather than building a custom connector that they have to design themselves and build and maintain and optimize, they can go through arrows. They have one thing to think about. So they get access to all of this ecosystem of tools that, that now support arrow after five years of, uh, you know, five years of development work, um, that makes things much simpler for the data warehouse vendors. They have one thing to think about and we can, on the Python side, we can deal with like, okay, you know, I know how to optimize getting data into pandas so we can maintain that for them and their developers don't have to solve that problem, um, themselves. So I thin…

AI assessment note: “Snowflake exports, uh, supports exporting, uh, query results to, to arrow format. So it is big query.”

Q Okay. Very good. All right. Just a couple of questions from the audience and then we'll switch to Julia and Datakin. Um, let's see. How is a question from Raghu. How is our leverage or compared with Databricks product stack, Delta format, Delta engine?

A Uh, it's, it's, it's neither, uh, it's, it's neither a competitor or a, or a replacement, so it's, it's strictly a, a complimentary technology. Spark, uh, Spark supports Arrow as a, as an interchange format, and it's used heavily in the interface with Python and R, for example. Um, so, uh, I'm not up to date on, on what, so what else Databricks has done to, uh, to interoperate with Arrow, but I know that they're an active user. Uh, of, uh, of Apache Arrow in, in, uh, the Delta Lake, uh, uh, Python interface, uh, in Spark, uh, Core Spark itself, um, and probably in other parts of, of Databricks that, uh, that I, I'm not aware of. So, uh, so Arrow essentially enhances interoperability with file formats like Parquet, which are an essential part of the, the Databricks platform.

AI assessment note: “it's neither a competitor or a replacement, so it's strictly a complimentary technology.”