The Exchanges

Q you worked at Facebook, um, and you also wrote in one of your posts about the reality, uh, of data engineering at companies outside of Silicon Valley. Can you, can you maybe compare and contrast, uh, if I'm a data engineer, do I do different things if I'm at Facebook or Meta or Google, uh, or Uber versus, uh, what I would do in a company outside of Silicon Valley?

A Yeah, I don't think necessarily the goal changes. I think there's, there are aspects of the fact that there are different tools. One, um, when you work at, Companies like Facebook, uh, and other big tech companies, their infrastructure is very mature, right? Like they've, they've generally done at least most of the ones that I've either seen or worked at and case with Facebook, they've done a good job of like creating a centralized set of tooling that everyone's using and everyone's on. Um, which makes your job very easy. Uh, when I worked at, and when I've worked at companies outside of Facebook, you've spent a lot more time kind of just setting up and configuring whatever tools you need to manage. Uh, at Facebook, you essentially have your internal airflow that a different team runs, and you can just push files to it, uh, and it picks up the data pipelines, you know, automatically. Whereas in most other companies, you might be the person that has to even manage, you know, your airflow instance, which is managing all your data pipelines. Uh, you go into a different company, they're gonna have Six different, uh, like data storage or data platforms. One team's gonna be using Databricks. One team's gonna be using, uh, Snowflake. One team's still somehow using a SQL server instance. Um, you've got all these mergers and acquisitions happening, which also cause kind of further layer…

AI assessment note: “there's less support if you're a data engineer outside of Silicon Valley”

Q folks, so, um, hopefully we can cover some key concepts and definitions and all those things, as well as going into more technical stuff so that there's a little bit for, for everyone. Um, But maybe before we get into this, a little bit of your background, your story. So you're Ben, but you're also known as Seattle Data Guy. So do you want to go into all of this?

A Yeah, sure. No, great. Thank you for, for the, the light intro. So yeah, like you said, uh, Seattle Data Guy. Now, now I live in Denver, so it's currently just snowed, I think the other day. So it's white outside. Uh, that's new for me from Seattle. Uh, I've been kind of in the data engineering space for nearly a decade at this point. Started at a hospital doing a combination of like data science, data analytics, data warehousing work. Um, I think like most people around 20 12, 20 15, I got bit by the data science bug and I was like, I want to do data science. It's, you know, the sexiest job, uh, of the 21st century. So I got into that, but eventually found data engineering and really enjoyed it. Uh, from there I worked for a healthcare analytics startup, um, that did everything from like fraud detection, some stuff with like opioid monitoring and a few other Uh, things around like population health, uh, again, around data engineering and building kind of a massive, uh, data warehouse essentially of, of pretty much like, I think it was like over half the US, uh, in terms of like who, who existed in that data warehouse. And then eventually did that at Facebook as well, data engineering. Um, uh, all throughout that time, I've been consulting as well and, and putting out a ton of content. So I think I started putting out content somewhere around in Um, so I've been writing for a l…

AI assessment note: “I've been kind of in the data engineering space for nearly a decade”

Q where there's like every year like a more, you know, even more gajillion, you know, new tools kind of thing. Why is it so Complicated. What, why, from your perspective, why are they, why do you need to put together those, like, chains of, like, tools, and sometimes it works, sometimes it doesn't. You mentioned, like, broken pipelines, that seems to happen all the time. Like, why are we here?

A That's something I've definitely been thinking about more recently, because when I first started in the data world, the common approach for, for a lot of companies I worked with would be, like, okay, we'll set up, um, Some cron jobs to run some SQL scripts essentially, or some Python scripts that call SQL scripts, or PowerShell scripts that call SQL scripts. And we'll just time it all correctly so that they, you know, run in a specific order or set up dependencies somehow. And then what a lot of people would end up doing is just essentially building Airflow, um, but internally. And so we knew that the problem, like we've run into this problem where it's like these things don't talk to each other and that causes issues. So then when Airflow I think came out, I think that's why it was popular for, for a lot of people. It was like, okay, cool. Now we can like write scripts, uh, and not have them bump into each other or have them, you know, make it very easy to backfill and all these things that, you know, we find painful, uh, as data engineers. And then somehow with the MDS, the modern data stack kind of push in We kind of ended up here again where it's like, okay, we have one tool that does extract. Okay, we have one tool that from extract does You know, transform. Okay, from there, you know, you kind of can keep parsing it down. We have another tool that does data quality. Um, p…

AI assessment note: “part of this works because I think it grew the pie”

Q Yeah, and I gotta say, as somebody who's been following your content for years, you're very, uh, yeah, consistent, and, um, you know, you produce a lot of content, which I think is the, the, the, as you said, like, a key to a lot of the things. Uh, so let's start from the top. What is data engineering?

A Yeah, I mean, my, my view, or at least a big portion to me of what data engineering is, is, like, Taking, uh, all this data and making it usable, uh, for, for people and now machines. I think that's something I liked out of Joe Reese's recent kind of talk with data modeling. We're really just trying to take data from multiple disparate sources that maybe don't naturally talk to each other and put them into a format that other users, whether they're analysts, whether they're, you know, data scientists can easily work with and have that in a repeatable fashion, right? Like we don't just want to have it work once we have to have this system, these, these data pipelines, these, These tables that exist that people can rely on. Um, and so that's really kind of the core of it. It's like, we're just trying to make data more usable, um, for, for people who are maybe, uh, you know, don't want to have to go through the core systems and, and you can have those standards and things of that nature set up.

AI assessment note: “Taking, uh, all this data and making it usable, uh, for, for people”

Q And so you mentioned SQL, um, so how proficient do you need to be at SQL? Is that the core of the job?

A I, I'd say that's where like a lot of companies use SQL as their kind of de facto transform, um, tool. Uh, obviously there's some UI based tools, but I think SQL's probably become the de facto tool in most cases. Um, so I, I'd say the, the actual like syntax of SQL doesn't take significantly long to learn. There's a few things that maybe I see every once in a while people struggle with to learn, such as like window functions, if you want to get specific, but That won't take you long. What's going to take you longer, and this almost happens at every company, is like getting a sense for how the data is shaped, right? Like, okay, when I interact with it via SQL, how is it shaped? Uh, you know, when I join, how is it, you know, where am I going to run into accident, like places where I can actually or shouldn't be joining things together, or what do I need to filter out, or how's, how's the logic set up? Um, so I often find that it's more about the data and less about SQL, because again, SQL you can learn quickly. What's going to take time is just getting a sense for how data Operates and flows and, and, and data sets in general.

AI assessment note: “I often find that it's more about the data and less about SQL”

Q So what, what else do I need to know as a budding, uh, data engineer? Like, do I need to, uh, no, I don't know. Spark or Kafka or like any of those frameworks, what would you recommend next once I have my language, I have my SQL, uh, what do I do next?

A Yeah, once you've kind of built that base of like SQL, Python, whatever your programming language is, understand kind of how to work with the terminal and the command line, I'd say then you can start getting comfortable. Yeah, I'd pick a few tools that are popular and companies are hiring for. I think that's obviously what you want to do. So things, like you said, Spark and Kafka, I think are a good place to start. Um, also getting familiar with interacting with cloud, or the cloud, however you want to call it, you know, AWS. GCP I think are always a good place to start. Obviously Azure exists. Uh, I have, I have my own qualms with Azure every time I use it. So, uh, I just like AWS and GCP slightly better. Um, but, um, yeah, so pick up, pick up some of those tools, kind of get comfortable and then start building some projects on that. I think you should have enough that you can kind of start testing our projects. You're gonna run into more tools as you're doing that. You're gonna be like, oh, I want to maybe test out dbt or, um, SQL mesh or something like that for transforms. You know, you can play around with that. Uh, but yeah, I think initially just picking up Kafka, Spark, those, those are still so heavily in use that even if they go out in the next three to five years, it's going to take time to migrate and things of that nature. Uh, and then pick, pick some clouds just to…

AI assessment note: “Spark and Kafka, I think are a good place to start.”

Q And beyond the technical stuff, what would you recommend? Uh, did I engineer learn to be successful at the job?

A Yeah, I get, I start getting comfortable, like, understanding and talking to the business and, and kind of understanding what their problems are and what you're actually doing. Because I think early on in your career, I think it's totally cool to just focus on tech and, like, you know, explore it, figure out what you like, figure out what you don't like, figure out how to, you know, improve performance of queries, do that. But you eventually need to start poking around and being like, okay, but why am I Building this data model. Like, what is it actually helping? Who is it? Where's it going to? Like, what is this data actually being used for? I think asking those questions, you know, the earlier, the better, but you know, if you're just focused on tech for, for the first year, I think that's fine. But once you can just start asking those questions and start trying to understand beyond just like, okay, I'm building a data pipeline, but it's like, what is this for? Um, cause I think that makes you a better strategic partner as you kind of grow. And then that lets you, you know, open up new opportunities in terms of like, you're not just this person that gets told what to do. You can start Interacting with the business and being like, I think we should do X, Y, Z, you know, or these, I think we should build this type of, uh, a data model, or we should start collecting this data ov…

AI assessment note: “start getting comfortable, like, understanding and talking to the business”

Q So if you're a business person in the company, or if you're, you know, somebody who's considering a, a career in data engineer, and I both, both of those kind of people are wondering What a data engineer does every day. So what's a day in the life? What are some of the tasks that you tackle during a standard day as a data engineer?

A I'd say, you know, you're going to spend some time trying to figure out exactly what the business needs, or if not the business, likely the analysts and data scientists kind of talking to them, trying to understand what problems they're trying to solve. Personally, I think it's better to also get closer to the business, understand, okay, well, why are the data scientists and data analysts asking me to build this table? Once you've kind of Gained an understanding of the business, maybe some of the requirements they're actually asking. You're gonna spend time, um, planning and, and data modeling. So taking whatever that raw data is, whatever it looks like, uh, trying to create it into a, a format that again, analysts and data scientists in the future can parse, uh, and actually use in their models or in their, you know, reports. Um, and the other place from there is obviously you're gonna build data pipelines that actually push to those tables, um, and, and actually kind of put it all together. And on top of that, you're probably gonna be spending some time fixing, uh, broken data pipelines here and there. Cause things, things break all the time. Uh, no matter how much you try to kind of get around it, people make changes to tables. You know, we've got some methods we're trying to, I think, implement depending where you are, whether it's like data contracts or something of that n…

AI assessment note: “You're gonna spend time planning and data modeling... build data pipelines... fixing broken data pipelines”

Q And you mentioned data modeling. Do you want to define what that is and just like maybe go into some, some detail there?

A Yeah. So, you know, there's a couple different ways, uh, you can essentially approach data modeling. Most people are probably accustomed to their, what they learned in their database course, which is like transactional systems, which tend to be normalized to like third normal form, depending on, on how far you want to take it. Um, obviously now we can be a little more loose with that because there's so many ways you can kind of Handle data that's maybe a little less structured. But from there, like, that's generally not very good for analytics. Uh, you run into various problems, and also you're generally trying to centralize maybe multiple, or multiple different sources, uh, into one location, which is your data warehouse. And so we've kind of come around, uh, and there's, there's a few different ways you can model for what we call analytics. So people often reference that as like OLAP or Online Analytical Processing. The, the way most people start is through the standard kind of, uh, data warehouse, which is facts and dimensions. Uh, and that's just creating data or putting data in such a format that again, it's just easier for someone who's maybe less technical to come at and be like, okay, I can kind of see the fact tables in the middle. If I need anything descriptive, uh, that's in the dimension tables around that. And so that's what we're really trying to do at the end of …

AI assessment note: “putting data in such a format that again, it's just easier for someone”

Q this, uh, quite, quite a bit, um, uh, where, where basically there is that, um, Uh, constant risk for miscommunication or misunderstanding and, uh, to play it back if I got this right. I mean, what you're sort of suggesting is that the technical people should learn how to talk business a bit more, and the business people should learn how to talk technical a bit more. Is that fair?

A Yeah, I think there's definitely that aspect. I think in general, I found that if you're the data team, you're going to probably have to do a little more of the, the lift, um, just because it's, Maybe a little easier for you to, to make that leap. Um, or maybe you just have more time. I don't know why, but it just feels like generally when my experience that, uh, on the data side tends to be the one that, that has to kind of bridge that gap. You also have, I think, an interesting view of the data. It's a, not data, sorry, the business. You have this kind of derivative view that is the data itself. So you, you can kind of see the, the data and it represents the business to a degree. Um, so you might see spikes before the business even sees it. So if you can kind of understand it, uh, that can help you Proactively just come to the business and be like, hey, we saw this spike and we didn't just tell you we saw the spike. We dug into it. We think it's from whatever store A or product B or feature A, right? Like we think that's causing this problem. So instead of waiting for them to tell you, hey, we're seeing a decline in churn, you can already have proactively looked into it.

AI assessment note: “Yeah, I think there's definitely that aspect.”

Q I mean, as an industry, we've accumulated enough institutional knowledge that, uh, we should know how to do this. Like, we should know, okay, well, this is what self-service analytics looks like, and, you know, maybe you can do it not, you can't do it in the specific circumstances of your company, but at least the goal should be clear. So why, why is that? Why are we not here?

A I think there's probably a few reasons. I think one, we lose a lot of, of like just, you know, brain, like a lot of people leave, right? Like a lot of, a lot of people who leave the industry before they, you know, a certain point. And so you have a lot of new people come in. I think we, we get, I, I, I say this as someone who, when I started got very wooed by the idea of like self-service analytics, right? Like I came in, Tableau was like self-service analytics. And I was like, that's great. So I think that, You know, pushes a lot of the, the industry. Same thing with like, again, on the business side, they hear a term, they're excited about it. Um, they think it's going to work. And so we kind of look past all the stuff that actually matters. So things like, okay, but do we have the processes in place? Do we have, you know, the right people in place to actually make this happen? Um, I think we're getting better. I think we're starting to consolidate some of this knowledge, and I think that's the other part, is for a long time, all this knowledge was just, like, everywhere, right? Like, yes, we had it all, but it was never, like, consolidated into a playbook. Um, it, it just kind of exists, right? Like, Joe Reese only put out his book recently on data engineering, and I think the reason it did so well was because it was kind of the first book that consolidated information. You …

AI assessment note: “I think there's probably a few reasons. I think one, we lose a lot”

Q modern data stack was an attempt at sort of consolidating, like having a sort of default set of technologies that work together. Uh, so do you view more of that? Do you view, uh, you know, the hyperscaler, um, you know, whatever Microsoft Fabric or being able to provide, or Databricks trying to provide all the things for all people? Um, is that, is that a likely future for you?

A I think if they execute well, it's possible, right? I think that's been the big thing. Like when I use Snowflake, for example, They have some integrations with things like HubSpot and other toolings. And it's nice. Like I I've used it for a couple of clients. It makes it considerably simpler, but then there's always just like little things like, okay, Snowflake has tasks, but how I set it up, they're kind of getting a UI now. So they're kind of getting to the point where it's like, okay, now you're getting into this like dbt Airflow-esque space where maybe, yeah, you could be everything. And Databricks has already kind of been there where they have their, their data flows that you can automate, whether it's notebooks or something else. Um, so they've had that. So I think it's going to come down to execution where it's like, if they can do it well, sure. Uh, people will go for it, but if they don't do it well enough to where it's like, look, I prefer to use air flow. Like, of course we're going to stick with air flow or, you know, whatever tool you're using, uh, for, for your orchestration. And if you don't have a good orchestra or not, uh, extract tool, you're going to use whatever they can find out of the box. Uh, if they don't have enough engineers.

AI assessment note: “I think if they execute well, it's possible, right?”

Q You alluded to some of this, but what are some of your favorite tools? So not an Azure fan, not a Redshift fan. But conversely, like, what do you, what do you really like?

A I do think I tend to, to have things I dislike more than I like. Like, there's a lot of tools. I'm, Snowflake makes it really easy, and I'd say BigQuery does too, but Snowflake makes it really easy. Like, if you do not have a ton of people to manage things, I think it's hard to go against it. Similar with BigQuery, I think Databricks makes sense to me still more from an ML standpoint. I know they like the data warehousing side. I know they, that's where they're going to pitch themselves, but I just tend to see, again, like I referenced earlier, like the migration from SQL Server to Snowflake has worked really well. Uh, from what I've seen for a lot of clients where the way I translate it goes really well is like we, when we migrated, they were only doing a certain amount of reporting. Um, and by the time we ended migrating, they're already bringing in more data. They're already asking more questions cause it's just easier for them to interact with. Uh, so I use that as kind of my baseline where I'm like, okay, I'm seeing people just do more. Uh, and that to me means that, you know, it works out well for them. So, um, I, I have a soft spot always for airflow, just because that's kind of the first tool I ran into that made my life easier. Uh, cause again, I came from this world of cron schedule jobs. Uh, it just made things easier. Um, so I think that always has a soft spot.

AI assessment note: “I have a soft spot always for airflow, just because that's kind of the first tool”

Q data engineer, so I guess two, two, two scenarios. One, I'm a, I'm a data team manager, so I'm a technical person managing other technical people. Uh, and then as a second, uh, sort of scenario, if I'm a business person and somehow I need to interact or possibly sort of manage a data team, what, what, what do I need to do for the, uh, collaboration to be successful?

A Yeah, I mean, I, I'd, I'd say one thing is probably hopefully you can spend some time understanding some of the language, uh, especially if you're working with more early career Uh, data engineers just cause they're probably not going to make that, that they're not going to bridge that gap initially. So you need to kind of help them figure out where maybe they're talking or using language that if there may be interacting with a stakeholder and they're starting to use language that could be, uh, just a little too deep, how to kind of handle those situations and better, better kind of bridge the gaps. I think that's, that's one of the spaces that like I noticed even from my own self when I was, especially early on, it's like you start getting too deep in the weeds. Um, and, and kind of losing people, and so I think that's a big place that, that always can be helpful. So if you can call out someone when they're like, hey, you're going a little deep, Is there a way you could say this, uh, that they could understand instead of thinking about it from, from just your perspective? Like how would you talk to someone who's never heard of even what a data warehouse is? Like there, there are cases where maybe they don't know what a data warehouse is, or, you know, when you say data pipeline, maybe they, they don't understand, or when you reference Spark, they're like, I, you know, why do I…

AI assessment note: “spend some time understanding some of the language”

Q scientists, and you know, again, that may be, uh, abundantly clear for like anybody that's, uh, in the field, but it's always, uh, kind of like surprising to me once you leave that exact world, like how people just cannot Tell the difference between those different jobs. So, um, you know, what, what's a data analyst versus again, a data engineer and what's a data scientist versus a data engineer?

A These roles still get very much, uh, amalgamated into one. I still see people and I still talk to people who are like, yeah, we're hiring a data scientist to build data pipelines and things of that nature. Um, and so in many ways, uh, if you're at a small enough company, if you're in specific orgs in a company, you might be just be doing everything. Uh, in, in, in that one role, but, uh, in a perfect role, right? Like data scientists are likely doing things that involve a little more research, uh, are a little heavier on the statistics side. So they're trying to find and create, uh, models or, or insights, uh, using, using different tools that, that maybe analysts would use. Analysts tend to, I think lean more heavily purely on SQL and maybe digging deep into those specific aspects using SQL, whereas data scientists might be using, you know, tools that Uh, are a little more statistically heavy. You know, they're using, um, models like something like K nearest neighbor or something, uh, to try to find insights, um, versus just maybe an analyst generally is like telling you and being very descriptive, like, Hey, here's what happened. Uh, we can kind of tell you why it happened and, and tell you maybe a little less in terms of like what could happen in the future, uh, how you can maybe build models to predict around that. So you're, you're building, you're more of just describing …

AI assessment note: “data scientists are likely doing things that involve a little more research”

Q you think, I guess everything is code, uh, ultimately, but, uh, uh, do you think there are some like chunks of the chain that could be automated? Like, I don't know the whole world of, uh, ETL, ELT, You know, why do you need to have a bunch of companies writing connectors to other sources, moving data around? Like who could, do you see a future where that gets automated?

A I can, but it's, it's weird because that's also something that I've thought about for a little bit where it's like, we've spent, it feels like billions and billions of dollars every year just trying to extract data and put it into things like Snowflake or some other data warehouse. And we're still, it still feels like we're not Like, great at it. Like, sure, good, but like, you'd assume we'd be better, uh, in some cases. Like, it'd either be faster or cheaper, um, and in some cases, it's still not. Like, I still run into people who, you know, for certain EL tools, they're like, it is way too expensive, right? Um, it's like, it's costing me way more than Snowflake. So, I, I do still run into that. I'm like, wow, that's wild. You'd think that by this point, it wouldn't be, but I still have people who are like, yeah, I'm only pulling for one data source, but because it's so many rows or something, You know, 20,000 dollars a month for it.

AI assessment note: “I can, but it's, it's weird because that's also something that I've thought about”