Kafka

“You can kind of, you know, you're onto a big problem when the number of use cases just keeps, you know, going up by like an order of magnitude that kind of pop out and where the problem you're solving seems really fundamental.”

“Amazon had a system that was actually built in, I believe imitation of Kafka called Kinesis. So it was incompatible, but kind of looked roughly like it. And that was popular. And we were like, oh man, that's going to take the opportunity. And if it doesn't, Th…”

“And that's an architecture I would say that we're seeing a lot of out there is Kafka as opposed to more traditional batch-oriented ETL.”

“In the past, when we were doing data platform, there was no Kafka, for example, we created a Kafka-like system ourselves, but we are lucky that, yeah, we created an HBase, sort of HHive-like system ourselves with hundred petabytes of size of data, right?”

“Would it have just been feasible if we started with a SaaS only offering? And the answer is probably not because, you know, one of the most important things an open source company needs to do is quickly establish yourself as the market leader, quickly establis…”

“The whole fraud and risk space came to mind because it's one of the largest use cases of Kafka on a dollar basis.”

“There was one other, maybe problem space again, sort of drawing on my experience, which is a helping companies build like a customer three 60 product where you have like a central platform to tap into, to understand your customer usage patterns, your customer …”

“We spent a lot of time focusing on convincing the company that they didn't need a team managing Kafka or their team could be much smaller. Effectively, you're dealing with a, you know, target customer base that just wants to do it themselves, right? So the DIY…”

“So Confluent which is the company behind Kafka which is like the real-time sort of message bus, like move, move data around in real-time had a wonderful IPO. They just checked today. There are like seventeen billion market cap.”

“It essentially gives you the ability to introspect streams, and it's super, super granular, super detailed, and it gives you know, better, better information than what you would get natively from Kafka.”

“Things like Kafka eventually came out of LinkedIn that was based on work that we had been doing at Yahoo”

“It has been to, in my opinion, a key enabler of microservices. I think it's pretty difficult to build and operate a decentralized set of microservices without first adopting something like Kafka in your organization to just move the data between all of these v…”

“So, you know, I think we're ending up leaning heavily into being a Kafka shop, building more of the stuff into base plate, into more of these core services”

“Today we're running about 300 different microservices, which are kind of piping together various Kafka topics and reading and writing, transforming this data, getting it to where it needs to go and our backend is primarily written in Go.”

“Like, I don't think the popularity of Kafka means that, that, that databases, you know, go away.”

“So the scale that we're operating at, we're pushing about a hundred megabytes per second through Kafka.”

“One of our clients is one of the largest social media companies in the world, and we actually pull in streaming data via Kafka.”

“Kafka was, in general, like, a nice way where people used to pipe the results of, like, their Spark jobs. But often cases, what they do is, like, they hit Kafka hard and bring Kafka down because they're trying to, like, actually send data from, like, hundred e…”

“You know, Kafka powers more than, ah, 1.2 trillion messages, ah, written per day. It is, ah, it powers more than 3.4 trillion, ah, messages, ah, read per day. All that amounts to more than one petabyte of streaming data. And that is across thousands of produce…”

“Since we open sourced it, you know, roughly five years ago, Kafka is used in thousands of companies worldwide, from Uber, and LinkedIn, and Netflix, all the way to traditional enterprises like eBay, and PayPal, and Cisco, and Goldman Sachs.”

“In fact, banks is are definitely leading the way in terms of putting Kafka To you know, very sort of ambitious applications.”

“Fundamentally, companies want to collect all sorts of data, and there isn't just a database and a warehouse anymore. There are lots and lots of distributed systems, which means that we need to move to a sort of platform-centric approach, and this will, this is…”

“All of our events flowing through Kafka, they've been populated into Cassandra, which then we run Spark instances that kind of model on top of the data.”