The Exchanges

Q actually let's use that as a segue. So I'd love to like dive into like four different areas and just like get like a bit more granularity, but since we, um, uh, started talking about data, we're sort of, let's talk about the data engineering part of the, of the house. Um, what, what's, what's the stack? What does, uh, the data organization at, at, uh, at Reddit run on?

A Yeah, so, you know, Reddit's been around, as you mentioned, you know, Reddit's been around for 15 years, so we have some sort of, uh, some stuff with some cobwebs on it and some stuff that's, uh, that's pretty, uh, pretty state of the art, and it really depends on what part of the organization you're looking at. From the top view, I'd say we're, we're AWS prod, um, with originally a lot of Postgres, um, For a variety of prod systems, we actually transit almost all that stuff over to GCP, and we use TensorFlow and BigQuery for our analytics and our model concerns for a variety of reasons that we could spend all of our time just talking about. We use a lot of open source related tools outside of that. Like most people, we have a love-hate relationship with Airflow and things of this nature. Pretty heavy, pretty heavy Kafka shop, pretty heavy Kubernetes shop, um, and, uh, and we're, oh sorry, please.

AI assessment note: “we're AWS prod, um, with originally a lot of Postgres”

Q So actually several very good questions. Um, so let me pick maybe one, possibly two before we wrap. Uh, TJ says that, um, uh, he or she has been tasked with the responsibility to, um, ensure that their AI ML efforts are not biased. How do you, how would you recommend another data leader to think about that?

A I, I love that question. Um, so, um, we could spend the whole time talking about that. I spend a ton of my time working on that topic. One thing I could say is there's an amazing book to expose you to an early part of this process. So, uh, a writer, Carolyn Criado Perez, uh, wrote a book called Invisible Women. And this book is about data bias. And what it talks about is, okay, let's say you're in an environment where Uh, you are using data to create policy. That's fantastic. People want to be data-driven, data-driven New York City. People want to be data-driven, but what happens when the data that you're using to drive that policy has bias in the collection of the data? This is a very real reality, and many times that bias, uh, Directly impacts a variety of marginalized communities, either communities of people of color or communities of women or both. A book is fantastic in that regard, but I would say look at all of the steps, the way that data is collected, the way that it is aggregated, the way it's, uh, reported on, the way it's generated, you know, what's happening in data quality. I would say breaking out all of the steps from collection all the way through We'll help you identify the risk factors in each area. One other thing I could, I could recommend is LinkedIn has done some really interesting publishing about how in their experimentation tool, they are reporting on…

AI assessment note: “breaking out all of the steps from collection all the way through”

Q Yeah. Actually, I love this, uh, uh, chip, the speaker after you is asking questions in the, in the chat. So like, I'm going to relay some of those, uh, in real time. How many roles do you have for great expectations? So do you want to maybe like explain what great expectation is like, uh, for, for, uh, we're just getting started with that.

A So, uh, so I, I would say I know the least of the things I mentioned, of course, chip identified the one I leafed about, but, um, but what it is, is it's a method for, um, you know, we have a lot of, it's effectively unit tests for things that are going to output data. So what should the data output look like? Um, and there's different, there's different feelings about how you might achieve this. And, and so developers are constantly, um, shipping code that's going to change data outputs. A lot of times we're looking at, um, the events that we're sending through off of new features, right? And saying like, are these events firing the right way? Are, is the data output what we're expecting? And, you know, you can both write the test there to identify it there. You can do, Stuff for event validation to say, ah, events that don't match certain things get dropped and sort of get put in the penalty box, which we've done and said, okay, developers, you, you can't actually join these to anything. You can't actually use them if they sort of get penalty boxed. You just have to go fix this, then we will send the stuff back through. And then you could also do anomaly detection, which we're doing, which is, okay, stuff has made it through and is wrong. And data quality, there's, there's simply no area where you're going to solve it. You know, I think you have to look at where are the areas…

AI assessment note: “it's effectively unit tests for things that are going to output data.”

Q And for the machine learning and data science part, like a bunch of Jupyter notebooks, that type of thing?

A Yeah, yeah. And I mean, Colab, certainly, because we're on, we're on Google, the Colab notebooks and Jupyter notebooks work well for us there. You know, other things we didn't mention, um, you know, we've built a bunch of in-house stuff for, um, classic Plumbing issues with relation, you know, when we're talking about, um, schematization, or we're talking about sand testing, like a lot of the stuff that has to do with data quality, we've had to build. Uh, and meanwhile, we just started working with a, a small company a couple of months called Comanomalo, does anomaly detection on data quality and measurement stuff. I built that stuff before at Jet. We built that stuff by hand. And so I know the investment it takes. Um, and, but they have been, uh, they've been a really good partner.

AI assessment note: “Yeah, yeah. And I mean, Colab, certainly, because we're on, we're on Google”

Q Very good. Um, and when you mentioned open source software, as you mentioned Airflow, so open source, um, uh, scheduler, uh, like any other?

A Yeah, we're, we're using, um, um, Amundsen, the Lyft data dictionary, and we're trying to contribute to that project as well. You know, I think there's an opportunity for, I think there's an opportunity for that project to be pretty exciting, where, um, where, you know, you could use something like Like great expectations to do sort of the data unit tests and output the rules you write into that, uh, into Amundsen so that Amundsen has a collection, not only of sort of the data that's there and who's using it, but also the rules around how it should be generated and the success or failure of tests. And so I think with a couple other pieces around it, you could start to build something that ends up being pretty beefy. In terms of helping your team be successful on what is an area that is just still, in my mind, what are the hardest things to solve for at a company when you reach this crazy scale? You know, with the scale of the data, the diversity of the data, everything else, when we get into what do we have, what's working, what's breaking, you know, where do you find things, these problems, you know, they get really, really difficult.

AI assessment note: “Yeah, we're, we're using, um, um, Amundsen, the Lyft data dictionary”

Q And do you have something about data lineage to precisely track where those issues appear?

A Yeah, I mean, I think that's an area where we're more building into it. Um, and it, you know, because it's interesting, it's, you know, You would know more about this than me, Matt, but, um, but the, the companies I see, it feels like a number of the startups I see in the space get pressure to expand to be more horizontal solutions. And then frankly, like we want them for one thing, but not for the three other things that they want to do. And so it's like, uh, are there players out there who could do this for us, plug and play? Yes. But like, what's the lock-in look like for these other things? And if their solution's a little off on something else, What does that mean for us? And so, uh, it's been a little mixed as you get further into sort of the quality lineage infrastructure area in terms of how much we can lean on other people versus build.

AI assessment note: “that's an area where we're more building into it.”

Q How does, how does, how are the decisions on what to prioritize? How does it make?

A Yeah, I mean, that's a hard question normally, but I'd say for us it's been even trickier. When I, when I came into Reddit, I came in in November of 2019, so, you know, 15 months ago, uh, 1516 months ago, and at the time there had never been a real data organization at Reddit, and there had never been a senior data leader at Reddit, which is why I became an advisor there. They wanted to make investments in data, but they weren't sure where to make them, and so originally I was just going to make recommendations on how they might scale that organization out. Then the opportunity Became so, uh, compelling that I ended up joining, but, uh, but as a result, you know, in terms of prioritization, not only are we sort of doing the prioritization that you would think of on data science, which I think of is maybe like a, uh, doctor or a physical trainer model where it's like, ah, so you come to us with a problem and we can give you guidance and you have to actually participate in that. Like I could do pushups in front of you, but you're not going to get me stronger. So, like, how do we, how do we work with you to help you get to the right place that you need to get to? So there's, of course, that relationship, but there's also just building basic capabilities, and when I got to Reddit, there was one person working on data engineering.

AI assessment note: “in terms of prioritization, not only are we sort of doing the prioritization that you would think of”

Q Um, So that was data engineering. Let's talk about the sort of machine learning or data science or whichever one of those. So use cases. What do you actually use data science for?

A So, so I'd say they're, they end up being quite different for us, the two teams. So if I started, if I started machine learning, I'll say, you know, Reddit has the greatest conversational text corpus in the world. Um, you know, and I can say that with confidence, both because You know, you can see the type of research that comes out of it, but also, you know, Google, Facebook, Microsoft, and OpenAI all used Reddit's data to train their conversational AI models. You know, we're talking about some companies in the world who have access to the, you know, greatest data companies in the world who chose to use Reddit's data to train those models because there's just a general understanding that if you get into conversation in terms of depth of conversation, breadth of conversation, That Reddit's data is, is, uh, the gold standard there. So with that in mind, what's amazing, you know, I started, so I got to Reddit and I think we probably had about three people working on machine learning in the whole company. So all of a sudden, all of the other people in the space are actually getting more leverage out of Reddit's data than we were. So, uh, part of our objective has just been, you know, how do we build up these core foundational capabilities to do things? But, you know, that could be a long journey, so if I'm just saying, ah, we're gonna go on this five-year journey to get to core ca…

AI assessment note: “last year was the first year that every Reddit surface saw personalization”

Q Um, What, what do you guys use for ETL or ELT?

A Um, this is one of those ones where I think we're also thinking about what, what it, what we want it to look like as well. You know, I'd say, um, in general, because there's also streaming, you know, we're also thinking a lot about where streaming needs to be the path versus where Bastion needs to be the path. So, you know, I think we're ending up leaning heavily into being a Kafka shop, building more of the stuff into base plate, into more of these core services, um, and thinking about then, yeah, we, we could, we could talk another day about what the path forward is on this. I think this one is, uh, in motion, let's say.

AI assessment note: “leaning heavily into being a Kafka shop, building more of the stuff into base plate”