StarZero All shows Try the podcast clipper →

Jun 16, 2016 · 25m · mad

A Kafka-Powered Real-Time Streaming Platform // Neha Narkhede, Confluent [FirstMark's Data Driven]

Neha Narkhede · 20m spoken Matt Turck · 1m spoken
0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

Neha Narkhede, Co-founder and CTO of Confluent, outlines how Apache Kafka, Kafka Connect, and Kafka Streams enable modern enterprises to transition from legacy batch processing to centralized real-time streaming platforms. The talk covers foundational architectural concepts, ecosystem components, and real-world deployment cases, concluding with a fireside Q&A on commercial strategy and enterprise adoption.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 4.5% of the talking time here. How this is scored →

Matt as informed peer 0.6 Guest teaching 0.6 Guest disagreement 0.4 Matt pushing back 0.4
05100:0010:0020:000:00–4:35 · Matt as informed peer 0/10 Title Cards and Data Driven NYC Opening This segment is a presentation monologue by guest Neha Narkhede introducing streaming data and event-centric thinking. Because the host does not speak, all host-related scores are zero.4:35–7:30 · Matt as informed peer 0/10 Event-Centric Thinking and Central Streaming Architecture Neha continues her presentation on data integration challenges and central streaming architectures. The host is inactive during this presentation segment.7:30–10:38 · Matt as informed peer 0/10 Apache Kafka Core Architecture and Global Scale Neha details Apache Kafka's core architecture and adoption scale at LinkedIn. The host is absent from the dialogue in this monologue segment.10:38–15:34 · Matt as informed peer 0/10 Capturing Streams with Kafka Connect Neha explains Kafka Connect and clarifies misconceptions regarding stream processing paradigms versus batch processing. The host does not participate in this segment.15:34–25:47 · Matt as informed peer 3/10 Kafka as an Enterprise Streaming Platform and Confluent Resources Host Matt Turck enters to facilitate Q&A, asking targeted questions regarding Confluent's commercialization strategy and enterprise adoption trends before turning to audience questions.0:00–4:35 · Guest teaching 0/10 Title Cards and Data Driven NYC Opening This segment is a presentation monologue by guest Neha Narkhede introducing streaming data and event-centric thinking. Because the host does not speak, all host-related scores are zero.4:35–7:30 · Guest teaching 0/10 Event-Centric Thinking and Central Streaming Architecture Neha continues her presentation on data integration challenges and central streaming architectures. The host is inactive during this presentation segment.7:30–10:38 · Guest teaching 0/10 Apache Kafka Core Architecture and Global Scale Neha details Apache Kafka's core architecture and adoption scale at LinkedIn. The host is absent from the dialogue in this monologue segment.10:38–15:34 · Guest teaching 0/10 Capturing Streams with Kafka Connect Neha explains Kafka Connect and clarifies misconceptions regarding stream processing paradigms versus batch processing. The host does not participate in this segment.15:34–25:47 · Guest teaching 3/10 Kafka as an Enterprise Streaming Platform and Confluent Resources Host Matt Turck enters to facilitate Q&A, asking targeted questions regarding Confluent's commercialization strategy and enterprise adoption trends before turning to audience questions.0:00–4:35 · Guest disagreement 0/10 Title Cards and Data Driven NYC Opening This segment is a presentation monologue by guest Neha Narkhede introducing streaming data and event-centric thinking. Because the host does not speak, all host-related scores are zero.4:35–7:30 · Guest disagreement 0/10 Event-Centric Thinking and Central Streaming Architecture Neha continues her presentation on data integration challenges and central streaming architectures. The host is inactive during this presentation segment.7:30–10:38 · Guest disagreement 0/10 Apache Kafka Core Architecture and Global Scale Neha details Apache Kafka's core architecture and adoption scale at LinkedIn. The host is absent from the dialogue in this monologue segment.10:38–15:34 · Guest disagreement 0/10 Capturing Streams with Kafka Connect Neha explains Kafka Connect and clarifies misconceptions regarding stream processing paradigms versus batch processing. The host does not participate in this segment.15:34–25:47 · Guest disagreement 2/10 Kafka as an Enterprise Streaming Platform and Confluent Resources Host Matt Turck enters to facilitate Q&A, asking targeted questions regarding Confluent's commercialization strategy and enterprise adoption trends before turning to audience questions.0:00–4:35 · Matt pushing back 0/10 Title Cards and Data Driven NYC Opening This segment is a presentation monologue by guest Neha Narkhede introducing streaming data and event-centric thinking. Because the host does not speak, all host-related scores are zero.4:35–7:30 · Matt pushing back 0/10 Event-Centric Thinking and Central Streaming Architecture Neha continues her presentation on data integration challenges and central streaming architectures. The host is inactive during this presentation segment.7:30–10:38 · Matt pushing back 0/10 Apache Kafka Core Architecture and Global Scale Neha details Apache Kafka's core architecture and adoption scale at LinkedIn. The host is absent from the dialogue in this monologue segment.10:38–15:34 · Matt pushing back 0/10 Capturing Streams with Kafka Connect Neha explains Kafka Connect and clarifies misconceptions regarding stream processing paradigms versus batch processing. The host does not participate in this segment.15:34–25:47 · Matt pushing back 2/10 Kafka as an Enterprise Streaming Platform and Confluent Resources Host Matt Turck enters to facilitate Q&A, asking targeted questions regarding Confluent's commercialization strategy and enterprise adoption trends before turning to audience questions.

speaking balance: gold is Matt, purple is the guest (3 minute bins)

0:00 · Matt 0% · guest 100%0:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%15:00 · Matt 16.8% · guest 83.2%15:00 · Matt 16.8% · guest 83.2%18:00 · Matt 12.6% · guest 87.4%18:00 · Matt 12.6% · guest 87.4%21:00 · Matt 0% · guest 100%21:00 · Matt 0% · guest 100%24:00 · Matt 16.9% · guest 83.1%24:00 · Matt 16.9% · guest 83.1%
Sharpest disagreement ▶ 23:10 Rejecting faster MapReduce framing

Neha firmly reframes a questioner's premise, asserting that stream processing is an event-driven microservice architecture rather than merely a faster MapReduce layer.

 Hardest push from Matt ▶ 17:22 Querying commercial open source model

Matt presses Neha on Confluent's monetization strategy, asking directly whether they rely on hosted services, consulting, or proprietary software built on open source.

 Biggest teaching moment ▶ 24:30 Differentiating Kafka from proprietary platforms

Neha educates an audience member on system architecture, contrasting MapR's proprietary converged file system with Kafka's open streaming ecosystem.

 Matt holds his own ▶ 18:27 Probing non-tech sector enterprise adoption

Matt demonstrates industry knowledge by steering the discussion beyond Silicon Valley tech companies to ask which traditional enterprise verticals lead production adoption.
the scores for every segment, with the reasoning behind each
ChapterTopicMatt as informed peerGuest teachingGuest disagreementMatt pushing backWhy
Title Cards and Data Driven NYC Opening 0000 This segment is a presentation monologue by guest Neha Narkhede introducing streaming data and event-centric thinking. Because the host does not speak, all host-related scores are zero.
Event-Centric Thinking and Central Streaming Architecture 0000 Neha continues her presentation on data integration challenges and central streaming architectures. The host is inactive during this presentation segment.
Apache Kafka Core Architecture and Global Scale 0000 Neha details Apache Kafka's core architecture and adoption scale at LinkedIn. The host is absent from the dialogue in this monologue segment.
Capturing Streams with Kafka Connect 0000 Neha explains Kafka Connect and clarifies misconceptions regarding stream processing paradigms versus batch processing. The host does not participate in this segment.
Kafka as an Enterprise Streaming Platform and Confluent Resources 3322 Host Matt Turck enters to facilitate Q&A, asking targeted questions regarding Confluent's commercialization strategy and enterprise adoption trends before turning to audience questions.

Statements from this episode (9)

Assertion Not checkable as stated
Narkhede: Companies are shifting from batch processing to real-time data
“A lot of companies are making a fundamental shift towards leveraging data in real-time, and moving away from that style computing Ah, which is essentially once a day data processing.”
Neha Narkhede Jun 16, 2016 ▶ 0:49
Insight
Neha Narkhede: All enterprise data can be represented as event streams
“My bold claim here is that all your data can be represented as event streams.”
Neha Narkhede Jun 16, 2016 ▶ 3:04
Assertion Supported
Narkhede: Apache Kafka powers over 1.2 trillion written messages daily at LinkedIn
“You know, Kafka powers more than, ah, 1.2 trillion messages, ah, written per day. It is, ah, it powers more than 3.4 trillion, ah, messages, ah, read per day. All that amounts to more than one petabyte of streaming data. And that is across thousands of produce…”
Neha Narkhede Jun 16, 2016 ▶ 9:48
Assertion Supported
Narkhede: Apache Kafka is used by thousands of companies worldwide
“Since we open sourced it, you know, roughly five years ago, Kafka is used in thousands of companies worldwide, from Uber, and LinkedIn, and Netflix, all the way to traditional enterprises like eBay, and PayPal, and Cisco, and Goldman Sachs.”
Neha Narkhede Jun 16, 2016 ▶ 10:13
Insight
Neha Narkhede: Stream processing generalizes request-response and batch paradigms
“Stream processing is often thought about as something that is just real time in nature, but it is really a generalization of these two extremes, request, response, and batch.”
Neha Narkhede Jun 16, 2016 ▶ 14:17
Opinion
Narkhede: Non-Kafka stream processing systems are complex and limited to niche problems
“Kafka Streams is, Extremely powerful, but very simple because it builds on top of primitives in Kafka. A lot of other systems are powerful, but, ah, not so simple. They're only applicable to a niche set of problems.”
Neha Narkhede Jun 16, 2016 ▶ 15:16
Assertion Not checkable as stated
Narkhede: Banks are leading non-tech enterprise adoption of Apache Kafka
“In fact, banks is are definitely leading the way in terms of putting Kafka To you know, very sort of ambitious applications.”
Neha Narkhede Jun 16, 2016 ▶ 19:10
Opinion
Narkhede: Central streaming platforms like Kafka replace legacy enterprise service buses
“Fundamentally, companies want to collect all sorts of data, and there isn't just a database and a warehouse anymore. There are lots and lots of distributed systems, which means that we need to move to a sort of platform-centric approach, and this will, this is…”
Neha Narkhede Jun 16, 2016 ▶ 20:23
Insight
Narkhede: Stream processing behaves like microservices, not faster MapReduce jobs
“What we learned is that, ah, stream processing is in fact much more than a faster MapReduce layer. It, in fact, most of the applications that do stream processing look much more like a microservice of an application, and less like a faster version of a batch o…”
Neha Narkhede Jun 16, 2016 ▶ 24:16
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 400 conversations transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more