Narkhede: Stream processing behaves like microservices, not faster MapReduce jobs
Neha Narkhede · A Kafka-Powered Real-Time Streaming Platform // Neha Narkhede, Confluent [FirstMark's Data Driven] · Jun 16, 2016 · at 24:16
Neha Narkhede, co-founder and CTO of Confluent, explains the architectural differences between Kafka Streams and traditional MapReduce/Hadoop processing systems.
0:00 / 0:16exact quote · 16.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“What we learned is that, ah, stream processing is in fact much more than a faster MapReduce layer. It, in fact, most of the applications that do stream processing look much more like a microservice of an application, and less like a faster version of a batch or MapReduce job.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Neha Narkhede
Narkhede: Central streaming platforms like Kafka replace legacy enterprise service buses
“Fundamentally, companies want to collect all sorts of data, and there isn't just a database and a warehouse anymore. There are lots and lots of distributed systems, which means that we need to move to a sort of platform-centric approach, and this will, this is…”
Neha Narkhede: All enterprise data can be represented as event streams
“My bold claim here is that all your data can be represented as event streams.”
Narkhede: Non-Kafka stream processing systems are complex and limited to niche problems
“Kafka Streams is, Extremely powerful, but very simple because it builds on top of primitives in Kafka. A lot of other systems are powerful, but, ah, not so simple. They're only applicable to a niche set of problems.”
Narkhede: Companies are shifting from batch processing to real-time data
“A lot of companies are making a fundamental shift towards leveraging data in real-time, and moving away from that style computing Ah, which is essentially once a day data processing.”