Reddit is leaning heavily into becoming a Kafka shop for streaming
Jack Hanlon · Fireside Chat: Jack Hanlon (VP Data, Reddit) with Matt Turck (Partner, FirstMark) · Mar 15, 2021 · at 17:00
Jack Hanlon, VP of Data at Reddit, discusses Reddit's evolving streaming data pipeline architecture.
0:00 / 0:09exact quote · 9.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So, you know, I think we're ending up leaning heavily into being a Kafka shop, building more of the stuff into base plate, into more of these core services”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jack Hanlon
Data startups push horizontal platforms, but enterprise buyers want single features
“The companies I see, it feels like a number of the startups I see in the space get pressure to expand to be more horizontal solutions. And then frankly, like we want them for one thing, but not for the three other things that they want to do.”
Google, Facebook, Microsoft, and OpenAI used Reddit data for AI models
“Google, Facebook, Microsoft, and OpenAI all used Reddit's data to train their conversational AI models.”
Hanlon: AWS Redshift's core architecture has fallen behind competitors
“Redshift feels way behind, whereas a number of years ago, it was fantastic. The core architecture there has not kept up.”
Hanlon: Reddit ingested 55-60B daily events managed by one person in 2019
“I came in, we were doing probably about 55 or sixty billion events a day into a data warehouse that one person was managing.”