Mar 15, 2021 · 24m · mad
Fireside Chat: Arjun Narayan (Founder & CEO, Materialize) with Matt Turck (Partner, FirstMark)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Data Driven NYC fireside chat, host Matt Turck interviews Arjun Narayan, Founder and CEO of Materialize, about the evolution of streaming data architectures, the power of SQL-based real-time analytics, and the core technology driving Materialize.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 12.1% of the talking time here. How this is scored →
Matt as informed peer 2.6 Guest teaching 5.4 Guest disagreement 1.3 Matt pushing back 0.3
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 8:20 Dismissal of non-SQL rewrite efforts
Arjun forcefully rejects the premise of database tools that require rewriting systems from scratch, stating that such efforts are 'largely doomed'.
Hardest push from Matt ▶ 1:47 Challenging the streaming hype timelineMatt politely challenges the guest by noting that despite annual claims of streaming becoming dominant, adoption took much longer than expected.
Biggest teaching moment ▶ 13:40 Explaining fundamental trade-off breakthroughs in Timely DataflowArjun educates the host on how previous stream processors forced trade-offs between complex batch computations and low latency, whereas Timely Dataflow solved both.
Matt holds his own ▶ 9:20 Framing the NoSQL to SQL historical trajectoryMatt displays strong industry context by accurately summarizing the multi-year trajectory from relational databases to NoSQL hype and the subsequent return to SQL.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Understanding Streaming Data vs. Batch Processing | 3 | 5 | 1 | 1 | Matt opens with a broad question on streaming and demonstrates basic industry knowledge by pointing out how the expected timeline for streaming adoption lagged behind industry hype. Arjun politely thanks Matt and explains the fundamental difference between batch processing and streaming using real-time financial use cases. | |
| Demystifying Apache Kafka as a Core Infrastructure Component | 2 | 6 | 1 | 0 | Matt acts as an audience surrogate, asking Arjun to define Apache Kafka for non-technical listeners. Arjun provides a clear explanation of message brokers, pub-sub architecture, and how Kafka enables real-time microservices without direct coordination. | |
| The Need for Streaming Databases and Incremental Computation | 2 | 6 | 1 | 0 | Matt asks why a streaming database is necessary given existing tools. Arjun educates Matt on why legacy pull-based batch analytics engines cannot scale to millisecond-level incremental recomputation. | |
| The Resurgence and Power of SQL in Modern Data Stacks | 4 | 5 | 2 | 1 | Matt shows good domain understanding by outlining the historic industry cycle from SQL to NoSQL back to SQL. Arjun validates this context, strongly criticizing tech pitches that force users to throw away SQL, while explaining how NewSQL and cloud data warehouses restored SQL dominance. | |
| Timely Dataflow and the Engine Driving Materialize | 3 | 6 | 1 | 0 | Matt mentions Timely Dataflow by name and asks about its technical origin. Arjun gives a detailed breakdown of Frank McSherry's research and contrasts Timely Dataflow with older stream engines like Apache Storm using an engine vs car analogy. | |
| Unpacking Materialized Views, Trigger Logic, and Update Granularity | 2 | 6 | 1 | 0 | Matt asks questions directly from his notes regarding materialized views, trigger logic, and granularity. Arjun clarifies the mechanics of pushing computation upon data changes rather than query request time. | |
| Integrating with dbt to Bridge Batch and Streaming Workflows | 3 | 5 | 2 | 0 | Matt connects Materialize's dbt integration with a previous event speaker. Arjun playfully offers a slightly provocative reframe of dbt as 'GitHub for SQL' while detailing how dbt bridges batch and streaming workflows. | |
| Future Roadmap: Hosted Cloud, Tiered Storage, and High Availability | 2 | 4 | 1 | 0 | Matt prompts for the future roadmap and cleanly wraps up the session. Arjun explains upcoming developments in hosted cloud services, tiered S3 storage, and database replication. |
Statements from this episode (9)
Arjun Narayan: Streaming data processing has expanded beyond niche financial applications
“And what we've been seeing is that streaming has started to become over the, over several decades and particularly in the past few years, much, much more broadly applicable. Beyond those small niche use cases as more and more businesses and use users benefit f…”
Narayan: Apache Kafka is essential for building and operating microservices
“It has been to, in my opinion, a key enabler of microservices. I think it's pretty difficult to build and operate a decentralized set of microservices without first adopting something like Kafka in your organization to just move the data between all of these v…”
Arjun Narayan: Standard analytics databases add latency by lacking incremental computation
“So, so stopping everything and recomputing from scratch is not really a framework that scales to these lower and lower latencies, which is why fundamentally a lot of analytics databases today including some of the more famous ones, they would prefer it if you …”
Narayan: Software requiring code rewrites in new languages is largely doomed
“Any pitch. I'm generally very skeptical where you can tell folks, you can have all these great new benefits of low latency or whatever it is, but you got to start all over from scratch, right? You have to throw everything out there and you're going to rebuild …”
Arjun Narayan: Snowflake succeeded by wrapping cloud architecture in familiar SQL
“Snowflake is a fantastic example of, under the hood, they, you know, they have a very modern microservice architecture, but it's all sort of very neatly wrapped up with a bow on top, such that it looks like a SQL database, and that's much, much more attractive…”
Narayan: Timely Dataflow was first stream processor to match batch processing capabilities
“It was sort of the first, what I would describe as The very first stream processor that could do everything that batch processors could do.”
Narayan: Materialize provides 99.9% of batch database functionality
“And we really think materialize the product is the first database that gives you know, I don't want to say literally all the functionality, but to the morally speaking, you know, 99.9% of the functionality that you can write in a batch database.”