The Exchanges

Q So why do those, um, issues occur at a very simplistic level? I mean, what's the range of things that tend to go bad leading to the quality issue?

A Um, it's an enormous range of things. I, I think the way I would describe it is actually to kind of like pick up on Nick's language a little bit. He talked about how, you know, 80, 90% of, uh, time spent by data scientists, data engineers is, you know, quote unquote data cleaning. When you look at what's happening in that data cleaning process, people are taking raw data, and raw means somebody else generated it, and you don't, you know, often don't know exactly where it came from. So in the data cleaning process, you're trying to form a mental model of, okay, where did this data come from? And therefore, um, you know, what should I do with it in order to make it appropriate for my use case? And so these are decisions like, oh, maybe I'm going to drop outliers, or I'm only going to pay attention to certain columns, or I'm going to say that certain events are good and certain events are bad, or lots and lots of little decisions. And when you look at kind of the work that goes into data science and data engineering, a huge amount of it is just making those Very small granular decisions about how you interpret this, you know, lump of bits that came to you and what you believe that that means about the upstream world. And now any time when one of those decisions changes, either in your pipeline or because somebody upstream is doing something different, that could break things downs…

AI assessment note: “any time when one of those decisions changes... that could break things downstream”

Q Yep. There is a concept of extensibility for the assertions. What does that mean?

A Uh, it means that every, every one of these expectations is its own little module, and over time we actually expect for a lot of these to be built. Um, one of the, one of the things that we're asked about most frequently is, hey, can we extend the library of great expectations to do geographic data, expect point to be within region, or time series data, expect trend to be increasing by x percent. Um, we've done a bunch of work to get the core of the library in good shape, And one of the things that we're really excited to do, uh, that now that we're through our Series A round is, uh, kind of resource better, uh, collaboration with the community to build out that library. Great.

AI assessment note: “it means that every, every one of these expectations is its own little module”

Q level startups like Calm and Komodo Health, um, but now public companies like Vimeo or like more traditional like Fortune, 1000, uh, companies like Heineken, um, like presumably those are like pretty different types of data, um, Stacks or, or maybe not. I don't know. Um, does the product work with like any kind of environment or, uh, is it particularly appropriate for like a certain type of, of customers?

A So, I mean, I would love to say it works everywhere. Uh, the truth is there are some things that are, are fully supported and some things that are more experimental. Um, so for example, Dask, uh, we, we know that there are a number of companies that have deployed and use great expectations on Dask. There are a lot of Prefect users, for example, who are using both tools together, and we don't currently, as part of our testing infrastructure, run all of the checks against Dask, so it's possible that at some point, you know, we'd accidentally break something. Now, that's the thing that we plan to fix eventually, but so I just want to point to this gray area of places where the tool is being used, but we as maintainers of the open source project haven't fully shouldered the burden yet of maintaining it. Um, now that said, we've always believed that it was very important to have, ah, kind of a global view into what is going on in your data. And, I mean, as we've heard from Nick and also from Daveris, like, data engineering is like a multi-application, like, multi-complicated stack world. And so having tests that can travel with you across that is important. Uh, the primary backends that Great Expectations runs on are Python pandas. Which allows you to bring in a lot of notebooks, a lot of machine learning type workflows. Uh, we also do Spark data frames, um, and then SQL, uh, throug…

AI assessment note: “there are some things that are fully supported and some things that are more experimental”

Q And then, so you have like, I've been like a library of expectations that like different people can collaborate on and, uh, you can add rules, remove rules. Is that, is that how it works?

A Yeah, that's exactly right. And I, just to go a little bit deeper, you said that expectations defined kind of the schema of data. And I just want to emphasize, yes, you can do schema for it, especially if you're in a world like CSV files where the schema itself isn't rigidly defined, then having that is very helpful. Um, Even in other places, like within, um, typed data frames in Spark or, you know, your choice of data warehouses, being able to do sets, ranges, regular expressions, uh, distributions, uh, correlations, uh, things like that start to be really valuable because you, you get beyond the kind of just schema into the content and the texture of the data.

AI assessment note: “Yeah, that's exactly right. And I, just to go a little bit deeper”

Q Wonderful. Uh, thanks for the, for the link. So, um, we've talked a lot about the open source project. Um, let's talk a little bit about the, the commercial companies for conductive and then, um, Great Expectations Cloud. What, what, uh, what does that or what will that do?

A So superconductive is the, you know, the formal corporate entity behind the company. We actually started in healthcare data analytics and, uh, deliberately pivoted the company to open source about two years ago now, when great expectations started to just really take on a life of its own. And it became obvious that there was something really interesting happening around data quality. Uh, so just a little bit of backstory. So everybody understands why the names are different. Um, Great Expectations Cloud is the SaaS offering that we are building on top of open source Great Expectations. I'm going to be slightly less cagey than Nick here, although we, we haven't formally unveiled and kind of like showing everybody what it can do, but we're currently working with a small number of design partners on it. Uh, the, the way that we think of it is it's a collaboration layer that sits on top of the open source project. Uh, so we've You know, have committed, will always commit to keeping everything that's in open source, great expectations open. And there are a bunch of things that, uh, you can do once you have all the power of, you know, SAS at your disposal, and it's not just a Python library, uh, that, that will lend themselves to much better data collaboration.

AI assessment note: “it's a collaboration layer that sits on top of the open source project”

Q Okay. Anyway, while you do that, so how's that gonna work?

A So, I mean, our goal for the open source project is, is really to create a shared open standard around data quality. And to do that, we need input from the community. Like we don't believe that it is a thing that we should solely govern. Um, we, we think it's important for a superconductive as an entity to be able to kind of bring together ideas, help prioritize, help move things forward. So there's sort of a PMing role to be done there. Um, But as we build integrations to more back ends, as we tie into additional systems that you can kick off actions in post validation, building more expectations, building more profilers, there's actually a lot of surface area there. And, um, I think we're unusual as an open source project in the sense that the same people who use it can also contribute directly to it. Um, it's not the case for things like Docker, say, where it's, Sort of an infrastructure layer that a lot of people use, but it's at a lower level of abstraction than most of the developers are developing. Great Expectations is actually a place where lots of people want to develop custom expectations and extend that library, and so we want to make that as easy as possible, and community roadmap meetings are a way of kind of facilitating that. Like I said, we're doing this for the first time next week. We'll be figuring out the format A little bit as we go along. Um, and, you kno…

AI assessment note: “We'll be figuring out the format A little bit as we go along”