Dec 11, 2020 · 1h 3m · mad
Fireside Chat: Jeremiah Lowin (Prefect), Tristan Handy (dbt) with Matt Turck (Partner, FirstMark)
Jeremiah Lowin · 25m spoken Tristan Handy · 19m spoken Matt Turck · 9m spoken Jack Cohen · 1m spoken
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Host Matt Turck brings together Tristan Handy (Founder/CEO of dbt) and Jeremiah Lowin (Founder/CEO of Prefect) for a Data Driven NYC panel discussing the evolution of the modern data stack, ELT data transformation, workflow orchestration, open-source commercialization, and emerging trends in operational analytics and data governance.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 17.1% of the talking time here. How this is scored →
Matt as informed peer 2.7 Guest teaching 3.6 Guest disagreement 0.5 Matt pushing back 1.0
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 36:30 Tristan challenges Jeremiah's podcast posture
Tristan playfully calls out Jeremiah regarding his appearance on Invest Like the Best, noting that Jeremiah presented his commercialization thesis as if he had all the answers when reality is much less certain.
Hardest push from Matt ▶ 8:52 Matt defends his two-box landscape taxonomyWhen Jeremiah rejects Matt's proposed categorization of analytics versus data science, Matt pushes back directly by asking if Jeremiah believes reality cannot be fitted into structured boxes.
Biggest teaching moment ▶ 41:45 Jeremiah critiques managed open source business modelsJeremiah educates the audience on why selling managed hosting for open source code is a weak business model, arguing that successful commercial entities must offer distinct value beyond running cloud servers.
Matt holds his own ▶ 20:25 Matt articulates dbt's philosophy and recent momentumMatt displays clear domain expertise by articulating dbt's core ethos of empowering analysts with software engineering habits while correctly referencing their rapid Series B financing timeline.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Jeremiah's Perspective on Modern Data Stack Interoperability | 0 | 2 | 0 | 0 | Matt does not speak in this segment monologue where Jeremiah expands on Tristan's definition of the modern data stack. Host scores are set to zero. | |
| Data Engineering versus Data Science Workflows | 5 | 4 | 3 | 4 | Matt proposes a binary taxonomy of analytics versus machine learning stacks. Jeremiah politely reframes Matt's model around job status versus data transformation, playfully teasing Matt about his landscape diagram. | |
| The Impact and Evolution of Cloud Data Warehouses | 2 | 5 | 0 | 1 | Matt asks a introductory question about cloud data warehouses. Tristan educates the room on how Redshift democratized OLAP technology by making it available for $160 per month. | |
| Comparing Data Warehouses and Data Lakes | 1 | 4 | 0 | 0 | Matt asks the guests to explain the difference between a data lake and a data warehouse. Tristan and Jeremiah explain compute coupling and schema timing analogies. | |
| The Evolution of Data Transformation and ELT | 3 | 4 | 0 | 1 | Matt asks Tristan to explain the shift from ETL to ELT and requests a practical example. Tristan details how SQL standards evolved and gives a clear breakdown of amortizing Stripe subscription revenue. | |
| Defining the Role and Skill Set of the Data Analyst | 2 | 3 | 0 | 1 | Matt asks Tristan to define the role and technical skill set of a modern data analyst. Tristan frames analysts as business problem solvers who adopt technology out of necessity. | |
| The Philosophy, Origins, and Growth of dbt | 5 | 2 | 0 | 1 | Matt demonstrates clear familiarity with dbt's core philosophy of bringing software engineering principles to analysts and highlights their recent Series B funding. | |
| Prefect's Origin Story and Core Mission | 3 | 2 | 0 | 0 | Matt draws parallels between Tristan and Jeremiah's founder journeys. Jeremiah describes building Prefect to solve his own data science and risk management pain points. | |
| Workflow Automation and the Concept of Negative Engineering | 1 | 5 | 0 | 0 | Matt prompts Jeremiah to explain workflow automation. Jeremiah introduces his signature concept of negative engineering and positions Prefect as defensive risk management software. | |
| Differentiating Prefect from Apache Airflow | 4 | 4 | 1 | 1 | Matt asks Jeremiah to compare Prefect directly with Apache Airflow, noting Jeremiah's blog post. Jeremiah explains his background as an Airflow maintainer and why Airflow could not address these new needs. | |
| Audience Q&A: Eliminating Time-Wasting Tasks for Data Teams | 0 | 3 | 0 | 0 | Jack presents an audience question about time-wasting tasks. Jeremiah highlights incident management while Tristan emphasizes analysts getting cross-functionally blocked. Matt does not host this segment. | |
| Building and Monetizing Open Source Data Companies | 4 | 5 | 3 | 3 | Matt probes open source monetization strategies. Tristan gently calls out Jeremiah's podcast confidence, while Jeremiah gives a strong thesis on why hosted open source is a flawed business model. | |
| Audience Q&A: Integrating dbt with LookML and Cube.js | 3 | 3 | 0 | 0 | The panel answers technical audience questions about dbt integration with LookML and GraphQL usage. Matt poses the GraphQL query, prompting Jeremiah to weigh API flexibility against database performance. | |
| Audience Q&A: The Evolving Role of Data Engineers | 5 | 4 | 0 | 3 | Matt challenges whether data engineers will be automated away and raises future trends like streaming and governance. Tristan points to reverse ETL and operational analytics as the next frontier. | |
| Audience Q&A: Data Masking, PII, and Regulatory Compliance | 1 | 3 | 0 | 0 | Jack reads an audience question about PII data masking. Tristan discusses warehouse data retention risks and Jeremiah explains Prefect's hybrid metadata-only architecture for compliance. | |
| Audience Q&A: Balancing Data Democratization with Governance | 4 | 5 | 1 | 1 | Matt asks about balancing data democratization with quality governance. Tristan reframes the premise by drawing an analogy to modern software engineering CI/CD pipelines. |
Statements from this episode (32)
Handy defines the modern data stack across four distinct functional layers
“When we talk about the modern data stack, we think about what's really four layers. So there's data ingestion. There's the data warehouse. There's data transformation or like taking all that raw data and like turning it into something valuable. And then there'…”
Handy: Amazon Redshift's launch sparked a rewrite of data stack products
“Those set of technologies have really been completely rebuilt. I think over the past seven years, really, it was like the introduction of Amazon redshift with In, in, in, that sparked kind of a rewrite of all of the products in that space.”
Lowin: Modern tools let analysts query data warehouses directly over static CSVs
“If we think back to BI tools, say five, certainly 10 years ago, you would sort of beg for a CSV and God help you if the data, if the insight you weren't, the insight you were looking for was not in that CSV, you were sort of screwed. But now thanks to, you kno…”
Handy: dbt serves batch data workloads, not sub-100ms real-time applications
“There's a lot of data science that's done Internally where like you're doing predictive analysis that, you know, the data latency can be a day or it can be like last quarter's data is fine. And those data science workloads are often run on the exact same stack…”
Handy: Amazon Redshift introduced affordable $160-per-month OLAP databases
“And so Amazon Redshift in, for the first time released an OLAP database, a database designed for large scale query processing that you could purchase for 160 dollars a month.”
Handy: Data lakes offer more flexibility but require more effort
“Where we are today, the data lake can kind of do anything. But it also probably takes more work to do anything. Whereas the data warehouse is, has a more constrained set of use cases, but it is much easier to get up and running for those constraints set of use…”
Jeremiah Lowin: Every data system eventually requires defining a schema
“You're going to have a schema. It's just a question of whether you define it upfront or you figure it out later.”
Handy: Data transformation has shifted inside data warehouses using SQL
“What's happening now or over the past, you know, five or so years, the, Transformation step now happens inside the data warehouse and it happens in SQL. And because of that, it is now accessible to a dramatically larger number of people.”
Handy: Data analysts learn technology to solve problems without identifying as technologists
“I think that a data analyst is somebody who answers business questions with data and they have a, they frequently will have a business or econ degree. They like are interested in solving business problems, but they're also not afraid of technology and they oft…”
Handy: dbt's philosophy empowers data analysts to think like software engineers
“Yes, that is completely true.”
Handy: Depending on data engineering backlogs destroys data analyst productivity
“That is like the death of the data analyst as like a productive member of your team. They will get frustrated. They will leave. They, they're, they don't have great career paths. All of these like negative outcomes.”
Handy: dbt originated as an internal tool for a consulting firm
“First time analytics, my company that is the maintainers of dbt we started as a consulting business and dbt was the tool that I wanted to be able to do this work.”
Handy: The dbt community reached over 8,000 members by late 2020
“We've built this community of over 8000 people today who you know, have bought into this as like the way that they want their careers to look.”
Lowin: PyMC3 developers will revitalize the Theano open-source project
“Because I think the IMC three folks are going to sort of revitalize it, which I think is awesome.”
Lowin: Prefect acts as insurance, delivering value when data pipelines break
“That's why we frequently describe our product as an insurance product is because we deliver value mainly when things go wrong. And this will sound crazy maybe to say, but if everything goes the way one of our users expected it to go, they really don't Need our…”
Lowin: Negative engineering is defensive work ensuring code actually runs
“And so we've named this problem, the negative engineering problem, because it's not about what you're trying to achieve. It's about all the defensive work you have to do to make sure that it actually took place, in fact, took place.”
Lowin would have built Prefect within Apache Airflow if possible
“And if I were able to do the things that Prefect does in airflow, I would have.”
Lowin: Prefect's functionality is a superset of Apache Airflow's
“Now it happens that our functionality is a superset of Airflow's functionality.”
Lowin: Saving time during outages offers higher leverage than routine orchestration
“If you imagine saving someone one hour orchestrating something, just getting that code to run versus saving a team one hour in a production crisis incident, the leverage of that hour, it's the same hour, but the leverage, the impact, the emotional burden, the …”
Handy: Data analysts spend 50% of their time blocked by engineering
“Like if you take a team at a like very good, like the data forward organization, a team of 20 data analysts, my guess is that they spend 50% of their time blocked.”
Lowin: Very few open-source projects turn into successful commercial businesses
“The number of successful companies that have actually iterated on this open source, The number is very, very, very small, and it's especially small relative to the denominator, which is much larger because the cost of entering the market is so low.”
Lowin: Managed hosting for open-source tools is a bad business model
“I think that that is a bad business model. And I sort of, I'm on the record of that. And for a very simple reason, which is that A company that is solving a problem needs to find the correct way to express its knowledge and its solution to that problem.”
Handy: Batch transformation is not LookML's core focus or innovation
“The part of LookML that does batch based transformation is, is not the primary focus of their product. It's not like the core innovation that they have.”
Lowin: GraphQL is fabulous for exploratory work and well-defined queries
“What we've found is that it's fabulous for exploratory work. Or ironically well defined queries, which is sort of the opposite of the purpose of GraphQL.”
Handy: Data engineers should focus on scalability, not business logic
“Data engineers shouldn't want to be spending their days expressing business logic. Like how do you amortize revenue across multiple periods? They are technologists. They should want to be thinking about platforms and scalability and all of these like hard tech…”
Handy: Stitch Fix's 2016 blog post best frames the data engineering role
“There's a guy at Stitch Fix who wrote a blog post called Engineers Shouldn't Write ETL back in 2016, and I still think that that is the best point of view on this topic today.”
Lowin: Mapping is Prefect's most popular feature
“It's by far the most popular feature of Prefect. And we can see that in our data. And that's because it solves a real problem that I experienced with data science, which I have a bunch of things I need to work with, but I don't know about it until Until runtim…”
Handy: Reverse ETL will automate workflows and multiply the data market opportunity
“There's a class of tools that takes the data that is in your data warehouse and pushes it back to operational systems. And I think that you will, as soon as that starts happening, you can automate the entire process and all of these technologies become, you kn…”
Handy: Once sensitive data lands in a warehouse, it resists elimination
“If you land data in your warehouse, it is very hard to ever have it go away completely.”
Handy: Turnkey solutions for in-flight data masking did not exist in 2020
“The easy solution to this, I don't think by and large exists yet.”
Handy: Software engineers pushing to production grew over 10x in a decade
“The population of software engineers that exist today and push code to production applications. That has, I don't know what, oh, more than 10 X in the past 10 years.”
Handy: CI/CD guardrails enable both software democratization and governance
“What you do is you have mature CICD processes. You have like DevOps workflows that, so you like build these guardrails that create high quality processes around code releases to production where you have your cake and eat it too. You have democratization. You …”