The Exchanges

Q Great. Let, let, let, let's dive into, um, materialize, uh, a bit more with a bit more granularity. Uh, so it's, it's based on an open source, uh, protocol timely data flow. Uh, what is that? What is the history there?

A Right. So, um, Materialize, um, despite being a, you know, a relatively young company where we're a little over two years old, um, Is based on, you know, close to a decade of, of, of stream processing research, primarily driven by my co-founder and materialized as chief scientist, Frank McSherry. So Frank, um, has done a lot of, he was an academic, he, he worked at Microsoft Research for a while, where he made several contributions to various parts of information theory, uh, data privacy, and also big data computing. Um, but he, Led a project to build a next generation stream processor, which he then developed as an open source project called Timely Dataflow that he had written, um, in Rust, which back then was, was still a programming language that was under development in a sort of pre, pre-one point O state. Um, I was, I was a close follower of this technology because I was a PhD student at Penn. Um, in, uh, distributed systems, distributed computing, um, and it was sort of the first, what I would describe as The very first stream processor that could do everything that batch processors could do. Like, in before, in this sort of pre-timely data flow world, there existed stream processors like Apache Storm, but they fundamentally posed a trade-off. They said, you can do some things in real time, you can do some things incrementally, but you can't do everything that you can do…

AI assessment note: “led a project to build a next generation stream processor, which he then developed”

Q And, um, As you build, uh, materialize, you, you, you chose to go the SQL route. Um, can you walk us through the thinking that led to this?

A Yeah, so, um, I think SQL is, it's incredibly underrated as a standard for describing computations and describing, um, the queries that one executes over all these, um, Data sets. It's incredibly long lived. It's, it's, you know, three, four decades old. Um, and, and, and, and if you think of many, many mature organizations, the corpus of SQL queries and analytics workflows that they've defined over decades is very, very rich. Right. And so, um, Any, any pitch. I'm, I'm generally very skeptical where you can tell folks, you can have all these great new benefits of low latency or whatever it is, but you got to start all over from scratch, right? You have to throw everything out there and you're going to rebuild everything in some new language. I think those efforts are largely doomed. SQL is also a great standard for interconnectivity. So you have all sorts of different tools. Any, any company that has That is any scale or even, even no scale at all. Usually it's using a dozen or hundreds of tools. These tools all talk to each other using standards like SQL. So a good example of this is a BI tool like Looker is pulling or is issuing queries to a, uh, Data warehouse like Snowflake, which is pulling data from a source of truth database like CockroachDB, right? So you take these three, they're all, they're all able to speak to each other because they, um, speak that common language…

AI assessment note: “SQL is, it's incredibly underrated as a standard for describing computations”

Q actually let's use that as a segue. So I'd love to like dive into like four different areas and just like get like a bit more granularity, but since we, um, uh, started talking about data, we're sort of, let's talk about the data engineering part of the, of the house. Um, what, what's, what's the stack? What does, uh, the data organization at, at, uh, at Reddit run on?

A Yeah, so, you know, Reddit's been around, as you mentioned, you know, Reddit's been around for 15 years, so we have some sort of, uh, some stuff with some cobwebs on it and some stuff that's, uh, that's pretty, uh, pretty state of the art, and it really depends on what part of the organization you're looking at. From the top view, I'd say we're, we're AWS prod, um, with originally a lot of Postgres, um, For a variety of prod systems, we actually transit almost all that stuff over to GCP, and we use TensorFlow and BigQuery for our analytics and our model concerns for a variety of reasons that we could spend all of our time just talking about. We use a lot of open source related tools outside of that. Like most people, we have a love-hate relationship with Airflow and things of this nature. Pretty heavy, pretty heavy Kafka shop, pretty heavy Kubernetes shop, um, and, uh, and we're, oh sorry, please.

AI assessment note: “we're AWS prod, um, with originally a lot of Postgres”

Q So actually several very good questions. Um, so let me pick maybe one, possibly two before we wrap. Uh, TJ says that, um, uh, he or she has been tasked with the responsibility to, um, ensure that their AI ML efforts are not biased. How do you, how would you recommend another data leader to think about that?

A I, I love that question. Um, so, um, we could spend the whole time talking about that. I spend a ton of my time working on that topic. One thing I could say is there's an amazing book to expose you to an early part of this process. So, uh, a writer, Carolyn Criado Perez, uh, wrote a book called Invisible Women. And this book is about data bias. And what it talks about is, okay, let's say you're in an environment where Uh, you are using data to create policy. That's fantastic. People want to be data-driven, data-driven New York City. People want to be data-driven, but what happens when the data that you're using to drive that policy has bias in the collection of the data? This is a very real reality, and many times that bias, uh, Directly impacts a variety of marginalized communities, either communities of people of color or communities of women or both. A book is fantastic in that regard, but I would say look at all of the steps, the way that data is collected, the way that it is aggregated, the way it's, uh, reported on, the way it's generated, you know, what's happening in data quality. I would say breaking out all of the steps from collection all the way through We'll help you identify the risk factors in each area. One other thing I could, I could recommend is LinkedIn has done some really interesting publishing about how in their experimentation tool, they are reporting on…

AI assessment note: “breaking out all of the steps from collection all the way through”

Q talk about Metaflow, uh, specifically, but, um, maybe before that, uh, could you, to the extent you can talk about it, could you give us a sort of a broad, uh, overview of the data stack, uh, including the machine learning Part, but not, not just that, like including analytics, what are some of the tools that I use, some of the open source project that I used across Netflix?

A Yeah, sure. Um, so when it comes to say the stack that's available to our data centers, you know, like, uh, first and foremost, it starts off with our data platform and Netflix has built a very comprehensive data stack. Uh, we are an AWS shop. Uh, so we use SG as Uh, so like the storage layer for our data warehouse, and we use Spark, Presto, Snowflake, uh, as our query engines. And, uh, we have invested pretty heavily in data discovery and data cataloging. Metacat is a Netflix project, uh, that's involved in, uh, data cataloging. So all of the data stored, um, so it gets illuminated over there. Then when it comes to, uh, query engines, as I mentioned, uh, Spark, Presto, Snowflake, Uh, those are sort of like, uh, some of the gradients that people, uh, prefer using, uh, in terms of compute, uh, our container orchestration platform is called Titus, which is, uh, yet another open source project. So, uh, from stock of containers, our, uh, internal users, uh, they'll essentially launch either their batch workloads or their services, uh, on top of Titus. And, uh, then when it comes to say workflow orchestration, uh, as with any big company, we have a bunch of different workflow schedulers. Uh, they're used internally for most of our ETL workloads. Uh, we have a workflow scheduler called Mason.

AI assessment note: “first and foremost, it starts off with our data platform and Netflix has built”

Q Okay. Thank you. Um, so let, let's jump into the product. Um, so when you guys started initially, I believe the initial product was focused on, uh, helping secure open source projects, right? Is that, is that, was, was that the origin from a product standpoint?

A It's a, it's a, there was an interesting kind of, um, uh, ambiguity with our name, which I think might've even kind of served us, uh, well, which is we were open source security. So on one hand you feel like it's securing open source. It's the, it's security that is open source. In practice, it was securing your use of open source. So you're consuming open source libraries and there's an ownership paradigm problem there, which is you download this stuff from the internet. Nobody's really got your back. You know, you need to know what you are using. You need to make sure it's vulnerability free and maintain that over time. Um, so That's what we started with. Uh, I think the, when I think about reusable lessons here, you know, uh, what was interesting for us is we set up to build a developer tooling company following the developer tooling playbook, but we're attacking the security industry. So when we started, we started with a, uh, very much the dev tooling playbook, which was pick a stack and deliver, you know, pick, pick something fairly narrow and deliver a great developer experience on that stack win over that community. And so we build open source security or securing Uh, open source usage in Node.js. So mostly NPM packages, uh, which we felt was a, uh, a niche that was big enough and small enough, big enough to care, to sort of have some commercial viability, but also smal…

AI assessment note: “In practice, it was securing your use of open source. So you're consuming open source”

Q Very interesting. Um, The vulnerability, uh, database also sounds super interesting. And that's something you guys have been doing for, for a while, or is that, is that more recent? What is it? What does that do?

A Yeah. From the beginning. So I've kept harping on the developer first angle, but in practice we combine developer first and security expertise. Uh, in the context of open source vulnerabilities, I can have an amazing system that tells you which libraries you're using, but I need to tell you whether they're vulnerable. I need to know which vulnerabilities are known. Uh, and what happens is, uh, in, in the world of open source dependencies, uh, open source projects, uh, an open source maintainer would fix a vulnerability would save much in the release notes. And that would sort of disappear into the ether of GitHub. And so we built actually from day one, uh, a threat Intel system, uh, that listen. So we have a team now about 25 odd people that have a system that, that reads a source of information, just like these like social feed torrents and things like that. We actually also technically read those social feeds. We read. Uh, open source projects like Apache Jira projects and GitHub and many other sources. Uh, and we identify candidate vulnerabilities. That system has gotten very sophisticated over the years and it uses machine learning to identify, uh, uh, candidates. So some of it is simple, straightforward, like bring the NVD database in, uh, and we'll, uh, we'll identify. And, uh, and some of it is, is much more, uh, natural language processing and other sort of combination …

AI assessment note: “Yeah. From the beginning.”

Q of the Python world. Um, especially when you started all of this, it wasn't super clear that Python was going to be the dominant language for, uh, data analysis and data science, but it's, it's seems to be very much the case now. Like what, what, why do you think that happened? And, and do you think that's a permanent state or, or could it, uh, could other languages emerge?

A Yeah, it certainly, it certainly wasn't clear, like it wasn't, uh, predetermined or inevitable that Python was going to become the number one data language that, that it is, uh, that it is today. Um, I think a part of that was a relatively small group of very passionate open source developers building the essential projects, which, which enabled, um, the people who are now called data scientists to do complete workflows, uh, in Python. And so we needed Uh, the array computing that you have in NumPy, we needed the tabular data manipulation, data cleaning, data loading that you have in pandas. We needed to be able to do machine learning that came from scikit-learn, which was developed around the same time. We needed a nice, uh, user interface, a programming environment that, that came from IPython and the Jupyter project. We needed to be able to, to visualize data. So it was kind of this perfect storm of these different tools coming together And, uh, and enabling a productive workflow. I think another thing, uh, that was a catalyst for, for Python was the fact that so many companies were started, uh, to, to get value out of all of the new data that was being generated by smartphones and by mobile browsing. Um, and so there, the time to market and the speed at which you could develop systems that, that created actionable insights on data and production systems that would deliver, …

AI assessment note: “it was kind of this perfect storm of these different tools coming together”

Q Your focus for the last few years has been Arrow. When, when, when, when and how did that start?

A Well, it was, ah, I, I would describe it as this, um, collective, ah, realization in, in the 2015, ah, ah, time frame that, that the, the entire data ecosystem was facing this grand data interchange and, and data interoperability crisis, and that's partly because there were all these different independent groups of people that built different, ah, computing engines, ah, data processing systems, ah, There are a lot of different file formats. There was the clouds that people were starting to build the first data lakes, um, in the cloud. Uh, and so there was all this angst about like, how do we move data and transport data between different systems? How do we get access to all these different data formats, uh, efficiently? Um, and all that being said, like the hardware underneath our feet, like disk drives, networking, everything was getting a lot faster. And so we found ourselves really limited and held back by, Um, the speed of getting access to data and the costs associated with moving data around. Um, so, uh, I was at Cloudera at the time. So, uh, you know, I was interested in this problem and I'd experienced it from the perspective of pandas and the Python ecosystem, like wanting to build bridges from Python into all of these other, all of these other systems. And so we started, uh, folks from Cloudera. So people from the Impala team, from the Kudu team, They'd worked with, u…

AI assessment note: “collective, ah, realization in, in the 2015, ah, ah, time frame”

Q Why is it so hard to extract data from a database? Can't you just use an ODBC, JDBC connector, you get the data out and you're done?

A Well, it's, uh, we could spend 20 minutes or an hour just talking about this, but, uh, it's, there's many problems. So, so part of the problem is that the way that, that people consume the results of database queries is so diverse. And so, uh, not, not everything is a data frame library or, or an analytical tool. Um, so, uh, so protocols like interfaces like ODBC and JDBC were never designed Or intended for bulk data transfer, like on the order of gigabytes, for example. So they are best used for relatively small amounts of data that are the results of database queries. So all the work happens in the database. The results are relatively small and they come into your application through something like ODBC or JDBC. There are also row oriented interfaces, which makes them an awkward fit when you need to convert to a column oriented tool like pandas. Um, and so there's just all this, there's this impedance or this, this, uh, conversion penalty, uh, going from, going from things like ODBC into, into pandas. And so, you know, it's kind of as, as a, as a user of these tools, it's, you kind of have this feeling of, you know, drinking a thick milkshake through a straw that's too small. So you're just trying really hard to like extract the data out of the database so that you can do, uh, do work on it, but, but you end up bottlenecked on that, on just the, the transfer, uh, cause databa…

AI assessment note: “interfaces like ODBC and JDBC were never designed Or intended for bulk data transfer”

Q And so how does Arrow work and how does it solve the problem?

A Well, we, uh, we designed a, uh, language independent, uh, standardized, uh, data format that, that is column oriented, that can be used for bulk data movement, um, data transfer, uh, but that is also an efficient, uh, data format, uh, for doing analytics. So if you get a bunch of arrow data out of a processing system, out of a database into memory, you can immediately go to work doing, um, doing further data processing on it without a need To convert it into some other data format. So traditionally data comes into your, uh, into your Python interpreter, into your, into your R interpreter, your process, and you immediately have to convert it into, um, into some other, into some other format. And so arrow removes the, the need for, for, for that, for that conversion. So aside from, you know, transporting bytes over, over the network, um, the, uh, the data is, is essentially ready for, uh, ready for analysis as soon as it gets into your hands.

AI assessment note: “arrow removes the, the need for, for, for that, for that conversion.”

Q incredibly, uh, complicated, expensive and time consuming to, to do this. So our, uh, as a very strategic place in the ecosystem in, in that precisely enable all those pieces to, to work together. Um, so just to, uh, just to reiterate, um, so, so precisely to this point about, uh, data warehouses, uh, what, what's the status of, uh, the integration, uh, with a Snowflake or BigQuery or others?

A Yeah, I mean, that's been, that's been one of one of the most successful use cases for the project, um, is as a, a standard, uh, uh, medium or, or data format for, for bulk interchange with, uh, with data warehouse systems. So Snowflake exports, uh, supports exporting, uh, query results to, to arrow format. So it is big query. Uh, Microsoft has a number of internal, uh, projects in their, uh, cloud infrastructure. For, uh, direct to arrow, uh, export. Um, and part of, part of what's motivated the, the data warehouses to support arrow is because we build a really, uh, a really efficient bridge between arrow and pandas. And so if their goal is to get gigabytes of data out of their data warehouses, into the hands of data scientists, rather than building a custom connector that they have to design themselves and build and maintain and optimize, they can go through arrows. They have one thing to think about. So they get access to all of this ecosystem of tools that, that now support arrow after five years of, uh, you know, five years of development work, um, that makes things much simpler for the data warehouse vendors. They have one thing to think about and we can, on the Python side, we can deal with like, okay, you know, I know how to optimize getting data into pandas so we can maintain that for them and their developers don't have to solve that problem, um, themselves. So I thin…

AI assessment note: “Snowflake exports, uh, supports exporting, uh, query results to, to arrow format. So it is big query.”

Q Interesting. Let's switch to the team, the machine learning and data science team at DoorDash. How big is it? How many people? And, um, what, what kind of people do you have on the team in terms of function?

A We, so a year and a half ago when I joined, um, we had six, five or six people on the team. We're now at, uh, almost 30 people a year and a half later, and this year we want to double, so to get to, uh, 50 plus people. Um, these data scientists, I, I wrote a blog post recently, um, that went live, I think last Monday, um, we're looking for, of course, Technical brilliance. You know, we have people who are typically have a master's or PhD in a, um, quantitative subject, um, plus some years of industry experience. Um, so that's a given and we have a very rigorous, um, challenge before you come on site and have, uh, one-on-one interviews with us at the test that, but what we, what we really look for in the interviews as well is an ability to, or, A real motivation to solve business problems rather than a desire to build cool models. Um, I think that's well, and it's okay if you want to build interesting models and write interesting algorithms. Um, that's probably best saved for academia or some research labs. For us, we're really interested in moving business metrics, and that's what we look for. We look for people who will do anything to move those business metrics, and over the course of a year, Most of your time will be spent building interesting models to serve that purpose. Um, but, um, we have found a lot of people that, um, are very technically gifted, but don't, don't imme…

AI assessment note: “We're now at, uh, almost 30 people a year and a half later”

Q And, um, you had this interesting theme around the democratization of, of machine learning, um, and I understand you created a machine learning council, I think you guys called it. Uh, can you, can you talk to that?

A Yes, some of the, some of the tensions I'd seen at previous places, and I'd also heard from others, is who, who should work on ML? You know, should it be data scientists, or ML engineers, or the AI lab? And, um, I found a lot of these discussions, um, I thought I could make these, these discussions and these conflicts unnecessary by essentially creating A set of principles for machine learning at DoorDash whereby we say machine learning is a tool and we encourage everyone to be able to use it. We don't try and put restrictions on what people can do. What we do do is put, um, is, is to put hard, hard, uh, lines, um, hard boundaries on what people are accountable for. So it is perfectly acceptable for anyone at DoorDash to Build a machine learning model, but if they want to put it into production, they need approval or a review from someone like a data scientist or machine learning engineer. So we try to divorce what someone can work on from what they're accountable for. The machine learning council was a way to bring different flavors of machine learning together to co-build and co-strategize. And again, to diffuse some of the territorialness that can develop at larger companies. And so we have people from engineering, From the ML platform, from data science, and we'll add other people in the future, and it's, it's a place for people to discuss ideas, discuss strategy, discuss h…

AI assessment note: “The machine learning council was a way to bring different flavors of machine learning together”

Q And in layman's term, what is it actually a transformation? What are some examples of operations that one does?

A Yeah. Um, the, the one that I, uh, always like to talk about is, uh, we have, Uh, a self-service SAS product. Uh, we use Stripe for our payment processing. Stripe gives us invoices. The invoices have start date and end dates. Um, but if you just add up all of the invoice totals for a given month, you don't actually get monthly recurring revenue. In order to get monthly recurring revenue, you have to take your invoices and amortize them over the period that they are for. So if you have An annual subscriber. You need to recognize that revenue over the course of 12 months. And so a data transformation is, and it's actually like reasonably a complicated one is to take your invoices and amortize them into monthly revenues such that you can just have this table that every month you just add up the numbers and that, that gives you your monthly recurring revenue.

AI assessment note: “take your invoices and amortize them into monthly revenues”

Q many layers, I guess, I guess sites and started in. I believe. So this, uh, it's been a lot of product bills, um, over the years. So as of, as of today, um, so maybe let's start with the cloud stuff since you, you mentioned it. So is, is part of the idea that this can work, work in a hybrid environment, like on-prem and cloud, multi-cloud is that correct?

A Uh, that is correct. So, so we want to fit within our custom customers changing technology ecosystem. Um, I would say over the years, so, so Sisense came out of stealth mode and with their first version of their product in. And as they evolved the product, which was analytics at its core, they noticed this gap in performance within organizations. There wasn't a lot of, um, A lot of big data handling going on within the platforms that we're currently in the market. And so they introduced something called the elastic cube, which is a very intelligent and powerful in memory engine. So now fast forward a few years, um, you start to see redshift, big query, snowflake, and people are investing heavily in those. And so what Sisense always did a great job of was seeing into the future. And I unfortunately take no credit for this. I just am lucky enough to join at this, at this stage. And so now what size sense does is they hit that data live. So keep your data in snowflake and we can go work on top of that. Um, but you know, Matt, as well as I do, everybody's data is not in one place. So, so where we are today is we have this great powerful in memory engine. If you want to leverage that, we also can hit your data live and you can do it all within our platform. And so as people migrate data, they can use us as they start to move their entire business to the cloud. We're completely micro…

AI assessment note: “Uh, that is correct. So, so we want to fit within our custom customers”

Q And what happens after the data warehouse, um, in terms of analysis? I mean, they, there is, there's, I guess the BI world and there's a machine learning world. Like what, what, what do people do, um, sort of, uh, yeah, after the, the data warehouse in the, in the pipeline?

A So data warehouses are database management systems. And so fundamentally you can do anything you can do with data you can do with a data warehouse. Uh, in practice, the most common use of data warehouses is to support, uh, business intelligence dashboards. So these are dashboards. You've probably seen them if you've ever worked at a big company that tell you they have bar charts and line charts and things like that. They tell you what's going on, you know, How many support tickets were filed this week? How much, you know, how many bookings has the sales team done? What is the, you know, average response time of the website? Um, or if you're in, you know, the automobile industry, you might care about what is the average, you know, uh, value of our total inventory from our suppliers, which is something we're trying to minimize. It's always very business specific what your key metrics are. Um, but those are, uh, Generated from data in the data warehouse and then they're presented in a dashboard of a BI tool like Tableau or Looker or Microsoft Power BI or you name it. Um, so that, that's definitely the most common use case of data warehouses, but then you can really do anything with them. Um, we have customers who run billing out of their data warehouse. We have customers who, um, There's all kinds of use cases, uh, that can happen with data warehouses, because at the bottom of it,…

AI assessment note: “most common use of data warehouses is to support, uh, business intelligence dashboards.”

Q We talked about transformation, uh, a minute ago. Um, I see you, you have a product now, right? Your five trend transformations. What, um, is that correct? And, uh, I guess, how do you, how do you go about it?

A Yeah, that's right. So when Fivetran delivers the data, it's going to be in a normalized schema that is like a sensible schema. The data is clean. Uh, there's not like duplication of the same information across tables. Everything's up to date, but that schema is not going to be customized to your particular needs, right? It's going to be like the native schema of whatever the data source is. And for you to do anything useful, With that data, you're going to need to transform it typically into a dimensional schema is what most companies will do. They'll, they'll turn it into a dimensional schema, which if you've never heard of it, it's basically a, A simplified view of the data where you make some simplifying assumptions knowing what kinds of analysis you want to be able to support later. So everyone's going to have a different dimensional schema. Uh, and somehow you need to orchestrate this transformation, right? As a practical matter, an analyst is going to write a bunch of SQL queries that transform from one schema into another, but somehow you need to, like, store those SQL queries somewhere. You need to Keep track of them and review changes to them. Uh, and then you need to actually run them. Uh, and so in order to, uh, how to do this has been somewhat of a, like an open question for the last few years. We're not the only ones who have been pitching this ELT modern data sta…

AI assessment note: “Yeah, that's right. So when Fivetran delivers the data”

Q the first round was in the 2018, right? And then you did multiple rounds, like back to back, like ABC sort of like in compressed timelines. So there was like a period of like five years, right? Where, where you were sort of, uh, two years building and then like, you know, several years into rating to get to that stage. Is that the kind of timeline we're talking about?

A Yeah, we, that's right. We raised a modest amount of money from angel investors right way back at the beginning. Um, so a few 100,000 dollars, and that's what sustained us for those first couple years. We did not pay ourselves a lot. Uh, and, uh, and we didn't spend a ton of money on AWS, and we only had one other person who joined the company in that, in that first phase. Um, The, uh, and then we raised, uh, the, the first significant round was a seed round in, in 2017, um, from a family office called CEAS, and then there was the series ABC in fairly short succession because we started to grow so fast. One of the funny things I learned about fundraising from that was that, um, if the company is growing really quickly, this funny thing happens, even if you don't spend the money, that same pile of money, Look smaller and smaller compared to the size of the company and the amount of money that just goes whooshing through every month. So no matter how capital efficient you are, you end up the, the faster you grow, the sooner you need to raise again, unless you want to just sit there and have, you know, one month's payroll in the bank account, which I don't think you want to when you have a lot of employees. So it is this funny paradox of fundraising, uh, that like the timing between rounds kind of doesn't matter. It's if it's a big Series A or a small Series A, you're going to do …

AI assessment note: “Yeah, we, that's right. We raised a modest amount of money from angel investors”

Q That's me. All right. So switching gears to, um, the product itself. Uh, so at, at, at pretty granular levels, but how does, how does that work? So you, you basically have a platform that has little tentacles in every repository and tracks the data or how do you, how do you get the data into the system so you can mind the processes as you define earlier?

A Yes. Perfect. Great question. So, um, so we basically, there's different types of data sources, right? So one there's the transactional systems, SAP, Oracle, Salesforce, et cetera, and you typically connect to either to the database or through an API, right? Um, so you, like, if we connect to SAP or if we connect to Oracle, we pull thousands of tables from, from these systems. And so, so we have also, we have Over a hundred pre-built connectors, uh, to, to, to, to transaction systems. You know, we've sort of really built, um, standards around this, so it's, it's easier for customers to get started. Uh, then there's task data, which happens on a user's desktop and is not, uh, actually recorded in any transaction system. That would be, you know, opening an email or looking at an Excel sheet. And we can also track that data through basically little tentacles that we have into the desktops. Anonymize the data, you know, Get rid of all the private data automatically, and then pull it into our system to complement this transactional data with task data. And then, uh, there's some extra day that we might pull from a Nielsen or like an external data source. You can pull that into B through API. Yeah. Benchmark data between our customers, you know, so that's easy because the data is already in our cloud. It's a fully cloud-based platform. Um, but, but, but I would say the data extractio…

AI assessment note: “connect to either to the database or through an API”

Q Thank you. I think it'd be really interesting for the audience, you know, you know, in a somewhat educational way, um, to talk about your, your role. What does that actually mean, uh, chief product officer at a database company? What's your role? What are your responsibilities?

A Yeah, so I'm responsible for a few things. The first is the product strategy, right? What are the investments that we're going to take on? What are the opportunities that we're going to pause on in order to build our competitive advantage? So fend off Amazon and some of our other competitors were also attacking this, uh, this cloud native SQL market. Um, the other big thing I do is I'd say Professionalize the process of innovating and taking innovations to market. So building out a predictable way to take inputs from our field and from our customers and translate that into outputs in the form of a prioritized roadmap, the way the product managers work with the engineering team and how they enable the broader team to take those innovations to market. And so the, the black box that translates those inputs to outputs is really where I spend my time.

AI assessment note: “Yeah, so I'm responsible for a few things. The first is the product strategy”

Q And so who is in your team, for example, what, what are the roles, like how does a product organization look like, uh, you know, in a competitive stage?

A Yeah. So the product team right now is about, uh, 25 people. So There are about eight product managers, so there are the four leads, and they're generally organized around user journeys or jobs to be done. This is not an obvious organization for a PM team at a database company. It's much easier to be organized around layers of the architecture components of code, but, you know, similar to the last talk, we're talking about how can we deliver value consistently, and how can we organize the PMs around particular Customer pain point. So, um, what's involved in running a query? What's involved in, um, observing that, uh, query and understanding if it's slow or not versus what's involved in, uh, recovering from a disaster. So those are some of the ways we organize the, the product management team.

AI assessment note: “product team right now is about, uh, 25 people. So There are about eight product managers”

Q product and a managed offering, what, what, what do you do with the open source, um, project? Is that something that continues to create value in terms of like sort of testing out the products and giving you ideas? Uh, how do you think about community? Uh, what, what, yeah, what, how do you, How do you think about the pure open source part of what you do these days?

A Yeah, the, the open source business is very important for us, or at least the community is very important for us because it's one of the ways that we Improve the efficiency of our enterprise sales. So one of the things we measure when we're looking at enterprise sales is, you know, what percentage of opportunities have already at least evaluated CockerDB, if not already running in production by the time our sales team reaches out and the open source product is one of the ways that we support that. So yes, it doesn't have all of the enterprise features, but you can go a long way with just that open source product. So, so that's really important. And even for the enterprise features, the source code is still available. It's under a different license, but you can still see how everything works under the hood. And this has been really important as we've started to approach more sophisticated customers. So some of the, the streaming companies are the same day, uh, delivery services that are using cockroach. Oftentimes they'll say, Hey, you know, you have great docs. We understand how your resilience model works, but we want to dig a little bit deeper. Like how exactly does this feature work? And our engineers can point them to the code. They can read it. They can get comfortable with how it works. And then they're more confident when they're, they're going into production. But the, …

AI assessment note: “it's one of the ways that we Improve the efficiency of our enterprise sales.”

Q Interesting. All right. So switching to, uh, the product itself, um, especially for people who have never been exposed to cockroach, um, what is cockroach DB and what is the fundamental value proposition?

A Yeah. So copper to be, it's a SQL database and it makes it really easy to build resilient, scalable, and low latency applications and services. And one of the things that makes this database so special is that It really makes it possible to build the types of applications and the types of architectures that you'd see coming out of an Amazon or a Microsoft or a Google, but it makes that technology available to every developers. Now any developer can build a multi-region application or a global application. And when we think about the, the major value propositions for Cockroach, we bucket it typically into three categories. So there's the resilience aspect, Right. So how do I build an always on application and how do I have a database that can survive failures, whether they're planned or unplanned? So, uh, an unplanned failure in the cloud can be a virtual machine restarting, which would result in downtime in a traditional database. So it could be a data center going down or a region going down. But, uh, there's also planned downtime, which is a pain in the neck for relational databases. Like, okay, Hey, I want to upgrade my database, or I want to do a schema change and traditional relational databases. You're going to have to take the app offline. You're going to have to lock a table. And that is really no fun for the developers and the operators. The other big Value proposition…

AI assessment note: “it's a SQL database and it makes it really easy to build resilient, scalable”

Q And as you were saying, one of the key or interesting things in the book is that there's not one, one culture that wins, right? Different cultures, what worked for Amazon didn't work for Apple, I think.

A Yeah, no, right, right. So yeah, one of the things I pointed out in the book is, Look, Amazon was famous for its frugality, you know, frugality, like, when you, and there, when the first employees came in, you know, your desk was a door with two by fours holding it up and that kind of thing, so like, we're not even gonna buy a desk, like, we're gonna get an old door and put it up, because that's how much we want to save money, but that was very key to their strategy, because the Amazon strategy, if you guys remember his original chart, um, was to be the low price leader in everything, Uh, because that way, as a consumer, you wouldn't have to price compare, you just know you could get it at the best price at Amazon, and that, um, meant that you had to really control cost if you were gonna do that. That was, it was integral to the overall strategy of being the everything store. Apple, like, so you could say, oh, frugality, work for Amazon, I'm gonna do it, but like, Apple doesn't have that value. Um, and nor should they, or nor can they, because they, like, their strategy is to have the best, most beautiful tools in the world for their customers, and so they spare no expense. In fact, Steve Job got fired, uh, originally for sparing no expense, and that was exactly what got him fired from Apple in the beginning, um, and, but that was, you know, in their campus, even their campus i…

AI assessment note: “You can have, you know, two different cultures”

Q very cool. Do you, do you want to talk a little more, uh, about the specifics to the extent you can? Actually, I don't know if you, if you can, but like, how does that work? And does that work? I mean, AI means lots of different things, like from, you know, deep learning and LP. Like, does it work? Is, is a model a model for your purposes or?

A Yes. We, I mean, we primarily see, uh, like our focus is on tabular business data, computer vision, NLP. Um, those are the ones that we see the most regularly. Uh, there's, there's plenty of other examples, but that covers a huge part of the market. And, uh, Um, and, you know, we are kind of agnostic. Like, people can build their models. Like, what we see in large organizations, they have this really heterogeneous mix of modeling environments. They're using DataRobot, and H-to-O, and open source, and cloud providers like AWS SageMaker, and things like that. And, ah, that's great. Like, I think it's important to allow this kind of highly sought after data science talent to be free to use whatever tools they want. Um, but you, you don't want to have to check 87 different places just to understand kind of what the overall Uh, health of your, of the AI across your organization is.

AI assessment note: “Yes. We, I mean, we primarily see, uh, like our focus is on”

Q That people, um, use is differential privacy. Can you, can you maybe just, um, educate us on the nuances?

A So, you know, the arsenal of tools that we have, so to speak, for, for enabling private analytics and analytics on sensitive data, is that there are a number of techniques. Uh, differential privacy is one, which is, you know, kind of like a real high-level hand-waving version of it, is that you add a bunch of noise to the data, so you can, Kind of, um, uh, obfuscate the actual privacy sensitive information. Uh, the challenge with that, it actually doesn't work very well if you need really fine-grained analysis as you would, for example, uh, genome-wide association studies. So it, you know, uh, there are several other techniques such as secure multi-party computing where multiple participants each have a share of the data and they run some sort of, uh, joint computation, but it's very bandwidth heavy. Uh, and, and there are a couple other different techniques. And, and there is a broad set of Privacy enhancing, privacy protecting analytics technologies, of which Homework for encryption is one of them, and it does solve some problems nicely, and of course there's others that it doesn't, but it just so happens to be that this is what we focus on.

AI assessment note: “differential privacy is one, which is... you add a bunch of noise to the data”

Q And then you, you found your way into Shopify, which is actually, by the way, interesting for anyone, um, that's, uh, a startup in trying to recruit, uh, top data scientists. You, you found your way to Shopify through a hackathon?

A Yeah, so there was this hackathon called, uh, Random Hacks of Kindness, and, uh, basically companies, uh, or people get together to solve problems for non-profit companies where they don't have a large tech team. So I went to the Shopify office, they were holding the hackathon, and I started talking to people there, and they said, yeah, we are growing a data team, and why don't you, They said, why don't you come in for a talk? So I went in for a talk, which was the interview, and then, uh, then I got an offer, so I never actually submitted a resume, and I think that's, like, one of the strengths of Shopify in hiring is that it, we don't go super conventional ways. We just try to get to know people, know about their experiences, and excite them about problems we are trying to solve.

AI assessment note: “Yeah, so there was this hackathon called, uh, Random Hacks of Kindness”

Q And how did you, uh, get the flywheel started on that side initially? Who were the first few, uh, that said the, that they were happy to partner?

A Um, so I think American Express was the first, uh, bank, if you can call them a bank, um, that we launched, uh, I think we kind of went top down from largest to smallest. Um, the, the deeper partnerships though, you know, Chase is the one that we've been working on the longest, and if you look at our history in the press, you can see that we had good times with Chase, bad times with Chase, now very good times with Chase. Um, but we really did go top down. We spend most of our time and attention focused on the largest banks, um, and the technical integrations we'll do with the smaller banks as well. Um, but our business development team, which is a team unique to, to Plaid that only talks to banks, um, that, uh, that, uh, that, that is entirely focused on the largest banks.

AI assessment note: “American Express was the first, uh, bank, if you can call them a bank”

Q So you started talking a little bit about the company. Do you want to finish that part? I guess, where are you as a company in terms of development? How many people? How long have you been around?

A Yeah, sure. So we've been around about 18 months. What's different about this one, well, actually a lot of things, it's funny. This company is both the same and very different at the same time compared to Hidapp. So Hidapp was You know, a sequel engine. This is again, a sequel engine. They're very similar from a technical perspective, but fundamentally the businesses are so different. This is obviously an open source project. We have very wide adoption. You saw some of the logos of some of the public users. It's a very global business. Um, the other thing that's very unusual about this one is, uh, we haven't raised any venture capital thus far. Um, and in fact, we're actually profitable, and we've been running, uh, profitably from day one, and And I will say we cheated, and I'll explain how we did that. Um, we were working at Teradata, as I mentioned, and we collectively left Teradata at the same time to start this business, and in doing so, worked out a deal that allowed us to continue to work with the number of customers we were working with around Presto. So we basically started with customers, started with revenue, and in that sense, this has been kind of like starting, uh, you know, with a head start, right, with a running start, and so that's been Really exciting. We're, uh, roughly 25 people today, uh, and growing organically thus far. Um, I won't say that we'll rule out…

AI assessment note: “we've been around about 18 months... We're, uh, roughly 25 people today”