Data Engineering
topic on 5 shows · 17 statements across 13 episodes
We Live to Build 
Latent Space 
the MAD Podcast 
the a16z Podcast 
Big Technology
17 statements about Data Engineering, every show
Gupta: Analytics roles will shrink as data jobs shift to engineering
“A lot of jobs should shift towards the left hand side with the engineering side. The analytical side should become thinner.”
Swyx: AI engineering will professionalize like cloud and data engineering
“I had seen basically front-end engineering become its own professionalized fields with dedicated conferences, dedicated influencers, and tech stacks and all those things, and I've seen the same thing for cloud engineering and data engineering. And all that. An…”
Anugas: Modern data engineering and warehousing are overly complex and slow
“And now we have meaningful data engineering and data warehousing, but it's super complex and it takes too long.”
Ben Rogojan: Most data engineers transition laterally from analyst or developer roles
“I see a lot of people move laterally into data engineering, often from either data analyst or software engineer. Data analyst, because I think there's just the sheer quantity of data analytics jobs are More.”
Rogojan: Understanding data flow takes longer than learning SQL syntax
“SQL you can learn quickly. What's going to take time is just getting a sense for how data Operates and flows and data sets in general.”
Rogojan: Early startups should hire fractional data engineers and full-time analysts
“I do think like if you're early starting out, There's no problem in probably bringing on some sort of consultant to do maybe more of the data engine work and then bring on a full-time maybe analyst to kind of work on top of that is what I'd imagine would be go…”
Chavez: Most enterprise companies have done a terrible job with data engineering
“Getting your single source of truth right, that data engineering problem, I think a lot of companies have done a terrible job of it.”
Tristan Handy: Building a production data system is building a software system
“One of the core beliefs about the, that we have about the profession is that data and software are not that different. And when you're building a production data system, you're building a production software system, which means that we should be taking lessons…”
Housley: Core data engineering issues haven't changed in 20 years
“A lot of the core issues in data engineering actually haven't changed that much in 20 years, and so people got so excited about technology in the transition in the 2000, the internet bubble, rise of Google, the rise of Facebook, that they lost sight of that”
Reis predicts the data engineering job title will face title dilution
“You know, and in fact, I think, candidly, data engineering will suffer the same thing. It will happen. We write about this in the last chapter of our book, in the fact that data engineering as a title could morph into something else, but that's what titles do.”
Reis: Aspiring data engineers must focus on rigorous software engineering skills
“If you're really good at the data part, then you got that covered, right? But what you probably lack is, like, rigorous software engineering skills. That's what I would focus on, hands down.”
Dehghani: Data engineering and data science will become basic engineering skills
“I think one of the big changes would be, we'll move from this specialized and specialization to generalization. So some of the things that we consider specialization today, like data engineering, a large portion of what we call data science becomes basic engin…”
Data challenges have shifted from compute scale to developer productivity
“An amazing engineering achievement happened in the early, you know, throughout the 2010, which was solving these massive pure technical scale problems. But now we're talking about organizational scale, dealing with complexity and dealing with developer product…”
Handy: Depending on data engineering backlogs destroys data analyst productivity
“That is like the death of the data analyst as like a productive member of your team. They will get frustrated. They will leave. They, they're, they don't have great career paths. All of these like negative outcomes.”
Perret: Data engineering is significantly harder than data science at Plaid
“We see ourselves very much as a data company, and we thought that data science in the early days would be a really key part of our strategy, and it is, but the data engineering challenge is so immensely harder that kind of data science kind of, in terms of tea…”
Separating data engineering from data science ties expensive employees' hands
“If you're hiring a bunch of data scientists and then making them dependent on a different organization for engineering is basically Hiring some really expensive people, and then tying their hands behind their back, and you know, that's not a smart thing to do,…”