The Exchanges

Q Jumping into this, do you want to talk about Kafka and, like, explain for, like, the non-technical part of this group? What's Kafka? What's the big deal? And what, I guess, what it does?

A Right. So Kafka is an open source technology, which is a message broker. So if you have, if you have Various systems that are producing data. You have various other systems that are consuming data. You can put Kafka in the middle in, and you can have the producers of data publish various feeds, and you can have consumers of data in a fairly, In a fairly decentralized fashion, choose which feeds of data or which topics as they call them in Kafka to subscribe to. And what this means is that you can sort of create this, um, real time clearing house infrastructure in your data In, in, in your company, um, to allow various teams that may not even be coordinating or, or, or with otherwise in the absence of a system like Kafka have to, you know, essentially coordinate and talk to each other and figure out a way to, to, to move that data and closely integrate their systems. They don't actually have to, um, you know, do that coordination and they can just sort of subscribe to those data feeds that are available and, and potentially come up with use cases. Um, that make use of that data in real time. You could imagine how this would happen in batch, right? So, so everybody at the end of the day would, would put all their data in, in a data lake, and then tomorrow anybody can pick up that data and do something useful with it. Kafka basically moves this, um, into real time, um, and allows …

AI assessment note: “Kafka is an open source technology, which is a message broker.”

Q Great. Let, let, let, let's dive into, um, materialize, uh, a bit more with a bit more granularity. Uh, so it's, it's based on an open source, uh, protocol timely data flow. Uh, what is that? What is the history there?

A Right. So, um, Materialize, um, despite being a, you know, a relatively young company where we're a little over two years old, um, Is based on, you know, close to a decade of, of, of stream processing research, primarily driven by my co-founder and materialized as chief scientist, Frank McSherry. So Frank, um, has done a lot of, he was an academic, he, he worked at Microsoft Research for a while, where he made several contributions to various parts of information theory, uh, data privacy, and also big data computing. Um, but he, Led a project to build a next generation stream processor, which he then developed as an open source project called Timely Dataflow that he had written, um, in Rust, which back then was, was still a programming language that was under development in a sort of pre, pre-one point O state. Um, I was, I was a close follower of this technology because I was a PhD student at Penn. Um, in, uh, distributed systems, distributed computing, um, and it was sort of the first, what I would describe as The very first stream processor that could do everything that batch processors could do. Like, in before, in this sort of pre-timely data flow world, there existed stream processors like Apache Storm, but they fundamentally posed a trade-off. They said, you can do some things in real time, you can do some things incrementally, but you can't do everything that you can do…

AI assessment note: “led a project to build a next generation stream processor, which he then developed”

Q And, um, As you build, uh, materialize, you, you, you chose to go the SQL route. Um, can you walk us through the thinking that led to this?

A Yeah, so, um, I think SQL is, it's incredibly underrated as a standard for describing computations and describing, um, the queries that one executes over all these, um, Data sets. It's incredibly long lived. It's, it's, you know, three, four decades old. Um, and, and, and, and if you think of many, many mature organizations, the corpus of SQL queries and analytics workflows that they've defined over decades is very, very rich. Right. And so, um, Any, any pitch. I'm, I'm generally very skeptical where you can tell folks, you can have all these great new benefits of low latency or whatever it is, but you got to start all over from scratch, right? You have to throw everything out there and you're going to rebuild everything in some new language. I think those efforts are largely doomed. SQL is also a great standard for interconnectivity. So you have all sorts of different tools. Any, any company that has That is any scale or even, even no scale at all. Usually it's using a dozen or hundreds of tools. These tools all talk to each other using standards like SQL. So a good example of this is a BI tool like Looker is pulling or is issuing queries to a, uh, Data warehouse like Snowflake, which is pulling data from a source of truth database like CockroachDB, right? So you take these three, they're all, they're all able to speak to each other because they, um, speak that common language…

AI assessment note: “SQL is, it's incredibly underrated as a standard for describing computations”

Q Now, where is there a need for a streaming database?

A Right. So, um, The, the biggest reason I would say is that today's analytics databases, while very sophisticated and very powerful are fundamentally designed around the batch paradigm. So existing at, so, so the best way to contrast this is a lot of batch workloads are, are very pull oriented. So you show up, you say, I've got this database and I wish to recompute or compute fresh a bunch of queries. And these questions can be very complicated. They can join merge, you know, many, many data sets. Very, very large data sets and things like that. Um, this paradigm doesn't really work in the streaming world where, um, you may be getting, you know, in aggregate still very large data, very big data that in, in, in, in, to use the phrase, um, but every few milliseconds you get another row of data, right? So, so stopping everything and recomputing from scratch is not really a framework that scales to these lower and lower latencies, which is why fundamentally a lot of analytics databases today Um, including some of the more famous ones, they, they would prefer it if you would just batch up all your data over a few minutes and then rerun the computation, right? So, so they introduced latency because the paradigm in which they perform their computations is not suited for incremental recomputation. Um, and it's really, you know, um, another way we, we, you know, we thought about Describi…

AI assessment note: “stopping everything and recomputing from scratch is not really a framework that scales”

Q Where do you think, um, SQL came back, right? There was this whole evolution over the last, um, few years from, from, from traditional SQL databases, and then like the next bit was, uh, everybody was saying, well, it's all known SQL, and then SQL came back. Like, what, what was the, what was the journey there?

A Right. So, um, that's a great sort of It's a great case study, um, you know, um, for, for those of us veterans, um, who've been around for a while. I mean, um, a couple decades ago, I mean, SQL was done, right? There was this big movement of NoSQL and Hadoop saying, you know, you're, and the fundamental message there was, was that the data sets were growing so large and your SQL databases were never going to scale and your SQL date, like if you wanted to process these Ever-growing data sets. You had to throw everything out, and you had to start over by, by using horizontally scalable technologies like NoSQL on the transactional side, and Hadoop on the analytical side, right? And, and, you know, this, this obviously had some uptake. This had some, um, um, adoption, but I would classify that as As, as adoption among folks who could build and maintain, and we're starting in a fairly clean slate manner. So the tech companies, the, the, the first sort of, I guess the first and second waves of sort of web companies would adopt Hadoop. Um, but by and large, didn't really get to a lot of mainstream adoption. Um, It was only when, when, when, when sort of, I, you had this new sequel that came that, uh, of, of, of vendors and technologies that came about that took those same, you know, ultimately correct ideas of horizontal scalability, cloud native, um, sort of architectures, but then p…

AI assessment note: “packaged it up such that the interface was still this very same sequel”

Q And you have, you have a concept of materialized views. Uh, and I'm, I'm reading from my notes here is some, um, Yeah, it's because they have like a big trigger and a big granularity. Do you want to dive into those and explain?

A Yeah. So, so materialized, the reason we called the company materialize, um, is, is that, is that what we really see, um, happening is us delivering on, on, on a feature that hasn't really, uh, a longstanding database feature, the materialized view that hasn't really lived up to its potential or its promise. And, and, and the way to think about a materialized view is, um, Let's imagine you're writing a query and you're rerunning it over and over again. Um, and, and, and the data may change, the data may not change. Right. So, so Sometimes, sometimes you're refreshing to see if that package is still in Memphis, right? That's a great example of a query that we've all been rerunning all the time. Um, a materialized view basically is a register to the database. I'm interested in the results of this query. Please recompute it for me so that it's fresh, right? So when I come and ask for it, I want it to be top of mind for you. Um, and if any of the underlying data has changed, please recompute the query at that point. So, so Basically triggering the computation, the recomputation when an actually relevant update happens, rather than when I ask for it, right? So if the database starts doing work when you ask the query, if it's a very large data set, or it's a very complicated query, you may have to wait minutes or hours before you get the answer. But if it's already recomputed that, u…

AI assessment note: “a materialized view basically is a register to the database. I'm interested in the results”

Q You, uh, just announced a partnership with, um, DBT and we had, uh, uh, Tristan, the CEO of Fishtown Analytics, the company behind, uh, DBT, speak at this event, um, uh, what, two, three months ago, uh, now. So curious, what does that do? What's the idea here?

A So DBT is this amazing tool that is, um, Is gaining a lot of adoption. dbt is the modeling layer for you to, the way I like to think about it, and then, and Tristan may not like this, but I think of it as like GitHub for all of your SQL, right? It's like there is workflows, there's continuous integration, testing, there's, you know, processes that you want to have to stay sane when your company or has, you know, thousands or tens of thousands of lines of SQL just sort of Spread across all, and with all of these interdependencies between the SQL query depends on this data set, which is computed by the SQL query, which depend on this data set, et cetera. Um, that is where, in my view, most of the world is moving to in terms of writing and defining that SQL. And the bulk of that today is of course down on batch systems. And I think the thing that is, that will truly unlock the potential of streaming is when, you know, A company can go from batch to streaming by taking that existing SQL that's already being written and defined in these DVC models and then sort of flip a switch and say we're now moving all of this recomputation to a streaming pipeline without really having to rewrite that SQL without having, because dbt does a wonderful job sort of extract, abstracting that away from the underlying execution engine in a way that the semantics are perfectly preserved. So, um, I think…

AI assessment note: “flip a switch and say we're now moving all of this recomputation to a streaming pipeline”

Q And so, um, what, what are the pieces that you built on top, um, then to, um, to help achieve that?

A Yeah. So the timely data flow project also includes this other part that, uh, called differential data flow, which, which builds, um, reusable, um, components that can be assembled and into these data flow graphs that executed, but materialized builds, um, all of the SQL planning, the optimization, uh, The integrations to tools like Kafka, uh, the integrations to, to, to, to pull batch data from S three. Oftentimes, um, you know, even when you do have streaming data, you often need to mix streaming and batch. That ends up being another challenge, uh, because again, not everybody's throwing out all their batch data or their batch tools and technologies overnight and switching to streaming. Oftentimes there's a long phase where you dealing with both batch and streaming data, um, in parallel. And, and, and, and the integrations, the, um, SQL execution, and then finally, um, you know, the, the, the tools that make it easy to operate and scale, um, and deploy the system in the cloud.

AI assessment note: “materialized builds, um, all of the SQL planning, the optimization, uh, The integrations”