The Exchanges

Q think you said, um, if you look at the evolution of the data space that, um, a lot of the, um, sort of early wins in the data space over the last few years have already been around managing scale, uh, and that we're now switching to a phase where, um, the primary challenges are higher in the stack around productivity testing integration. Is that, is that, is that fair?

A I think that's fair. Um, you can think of it sort of like Maslow's hierarchy of needs. Uh, and when the amount of data started to explode, there was a very critical need that you couldn't even process it on a technical level. Like you could not scale out compute. And that's why, you know, in the big data revolution, you called it big data. You started with You know, Hadoop and then spark and now the cloud data warehouse. So now if you have petabytes of data, you are able to process that efficiently at scale, but now the problems are kind of higher up on Maslow's hierarchy, so to speak. So now it's okay. We can actually process the data. Okay. What is it? Does it have high quality? Are the people that actually build the computations that process that data productive? Can you track that data throughout the ecosystem? And so on and so forth. So, you know, you know, an amazing engineering achievement happened in the early, you know, throughout the 2010, which was solving these massive pure technical scale problems. But now we're talking about organizational scale, uh, dealing with complexity and dealing with developer productivity and any number of other dimensions.

AI assessment note: “I think that's fair. Um, you can think of it sort of like Maslow's hierarchy”

Q I'd love to, um, continue down that path and, and, um, maybe double click on, um, how Dexter works, like some high level easy concept that would say, for example, a concept of, of a solid, which is, uh, your, I think your atomic units. Do you want to talk about that and, um, how Dexter works in general?

A Sure. So Dexter works is an open source Python project. So, you know, if you're, uh, You know, if you're a programmer, you can just type a simple command, pip install dagster, and you're off to the races. You can write a little bit of code. And then, you know, what you effectively do is you build these, these functions, we call them solids, which define like a computation, meaning like a step in the factory to keep on that analogy. And then you can construct graphs out of that. Um, and the moment that you use our APIs and structure Your code in this way, you immediately have access to all sorts of tooling. So without any infrastructure on your laptop, you don't need to deploy anything. You can just type daggett, which is the name of our graphical tool, and load it, and you get this beautiful rendering of like what this graph is prior to actually executing it. Um, and then you can also use that tool as almost like a local IDE for these graphs. Um, and Then you effectively, you know, one of the things that we really focus on at Daxter is being able to execute these computations, these programs in different environments. So you can like develop on your laptop and then also deploy it to a deployed, you know, piece of infrastructure and not have to change the core business logic. Um, and this is really something we focused on, um, and that's a critical piece of this. So, you know, y…

AI assessment note: “what you effectively do is you build these, these functions, we call them solids”

Q Who's a good, um, customer or type of user for Daxter? Like, do you need to have a certain type of infrastructure in place? Do you need to be a certain size? Do you need to have, like, a specific type of, of talent on the team? Who's a good sort of, like, profile?

A Yeah, so, you know, we've really found two classes of user who, um, really gravitate to the system. One is what we consider an emerging title. Of, uh, data platform engineer. A lot of people self title that way. Um, a lot of data engineers act as data platform engineers, but what they see is that, Hey, inside every company, there's a data platform, whether they acknowledge it or not. And this data platform is where all these people come together and all the different, like the data engineers can work with the data scientists. All this stuff can execute on time. You can have like one single management Um, uh, all the important data and all the heterogeneous tools. And so that user, maybe they'll start out and they'll say, okay, all we need is like an ingest tool, like Fivetran. We need something like dbt over the, over the warehouse and maybe a so-called reverse ETL or operational analytics tool to jam that back. And this is all kind of terminology laden and whatnot, but the moment they expand beyond those, that fairly prescribed what's called the modern data stack. I need to do anything outside of that. Um, they need an orchestrator, um, and they want an orchestrator that's kind of in line with the values of those tools, and a lot of those users have, like, really gravitated towards DAX or, like, one of our users said, like, what, what dbt did for our SQL, um, DAX did for our P…

AI assessment note: “we've really found two classes of user who, um, really gravitate to the system.”

Q the data world, there are, um, other folks, um, we had, uh, Jeremiah from Prefect, for example, at a prior event. So there's Airflow and then there's like, All the historical ones, like Luigi, um, we had Kedro as well at the event, like, so for folks, what's, uh, what are the bright lines in terms of, like, thinking of Daxter, um, in comparison to some of those other folks?

A It's a great question, and as you know, positioning is always an evolving art, uh, but I think the primary difference, and I'll focus on Airflow and Prefect, and she started with those, and, you know, Airflow is definitely the dominant incumbent In the space as it's traditionally defined is that they, you know, in terms of the, they don't consider the full life cycle of developing data products. They view their mandate is very narrow and purely these operational use cases of ordering. This comes after that comes after that. And then there's operational complexity in that. So you need to know how to retry things and so on and so forth. But You know, we think the graphs that the orchestrator encodes are, one, complex enough that they need a full local development lifecycle that's fully thought out, and two are, in fact, in some ways, the structure of the applications themselves, especially the ones that are written in Python, and so we really think about the dev and test lifecycle. We also want to be data aware, and so that means that one, we have data dependencies that are encoded, meaning that You know, not only do we order the machines, but we know that upstream that, you know, this input comes in this output, and without getting into details, this is a much, what we believe is a much more natural program model for practitioners. You know, the, uh, Daxter really also embraces …

AI assessment note: “I think the primary difference, and I'll focus on Airflow and Prefect”

Q No. And I, of course that, uh, leads into the question, like what, what's, what's your take on an orchestrator? What, what kind of orchestrator are you guys building with Daxter?

A Right. So I said that the, the one example where there's one assembly line and three stations, well, that's not the way that ends up working. You end up having Thousands of assembly lines, sometimes with thousands of stations, and they have all sorts of crazy interconnections. And then what's even more kind of interesting about it is that I described the interactions between one person to the next. It happens both on a macro scale, meaning like they also interrelate teams, but also the individual practitioner Will build their own little assembly line because it makes sense for them to do it. You know, as they're figuring out what their data is, they might say like, oh, this is an intermediate data product that will be generally useful. And I kind of want to have like a, a checkpoint right in the process. So what ends up happening is that the, these systems explode In complexity, and you need to be able to take the assembly line offline, test it on test data, put it back. If something goes wrong, you need to hold the assembly line, maybe start it from a certain point, move it around. Um, and then it also makes sense for this assembly line to be aware of and track the things that are actually coming out of it. So like, okay, I have this widget over here. It came from this previous intermediate, which came from this, et cetera. So it's, we think this like orchestrator is really th…

AI assessment note: “we think this like orchestrator is really the central leverage point that makes sense”

Q Thank you. So what's next? Um, you guys are a, a thriving startup. Uh, like I know you did some, Integrations with dbt with, uh, great expectations, uh, that we're going to be going to speak with, you know, in a few minutes, uh, what, what's next on the roadmap for the next, uh, you know, year or so?

A Yeah. So, you know, a year's a long time as a startup. And so you can never predict the future. Um, you know, we are a, you know, I think we'll get to this in, uh, we will get to this, but Um, you know, it is a venture backed commercial company. So at some point we will need a revenue model of business models. So I think it would be, uh, without a spoiler alert, we are, uh, we are working on that aspect of the business, but I can't talk about that too much detail. Um, you know, on the open source roadmap, I think there's two things. One is that, you know, we've been, what I'll, what I'll call in sort of a open Applied R&D phase of the company where we've been really working with a set of targeted design partners and not hyper-focused on growing adoption and, uh, too quickly because we didn't want to have too many partners because we knew the technology was still changing a bit. Um, we're about to land, you know, we're, we're, we're about to land some changes that will really set us up to for kind of a pre-one point release. And we've learned a ton from the last, you know, Year and a half of work, and we really are feel like we're settling on a set of abstractions and concepts will serve the foundation of technology for years and years to come. So in the short term, that is really our focus. And I think beyond that, you know, once we have that stable core that we feel can be the…

AI assessment note: “on the open source roadmap, I think there's two things”