Data Engineer
topic on 4 shows · 13 statements across 13 episodes
the Neon Show 
No Priors 
the Official SaaStr Podcast 
the MAD Podcast
13 statements about Data Engineer, every show
Sankar: Data engineers spend roughly 80% of time debugging
“In fact, like, 80% of time of data engineers were spent on debugging things.”
Ramaswamy: Data engineers will transition to AI orchestrators within two years
“I think there's going to be a lot more cursor style coding of these data engineering workflows. I think things like being able to extract metadata so that the data set that you extract is almost self describing so that AI can get to work on it. I think that is…”
Rogojan: Fully replacing data engineers with AI is still years away
“So I think we're a few years away from just like replacing data engineers, which has been the goal for what it feels like a decade or plus now.”
Tigani: Prompting tools will reshuffle data engineering and analytics roles
“I think the only thing that you know, I think is clear is that people will be doing things differently. Their jobs will be different. I think, for example, like The data space has sort of started to aggregate into these different job titles, analytics engineer…”
Matei Zaharia: Every software engineer will become an ML and data engineer
“And I think over time, like I increasingly think that basically, especially because of the capabilities of these AI models, every software engineer will need to become an ML engineer and a data engineer also. As they build their application and we'll, we'll fi…”
Housley: Moving from Hadoop to cloud data stacks has been very tough
“One of the things, one of the transitions that Joe and I went through, which I think a lot of people in this room went through, was the transition from the Hadoop world, from the previous big data world, into this new, like, cloud-based data engineering snack,…”
Seeing manipulated data directly increases data engineer productivity
“I just think that as a data engineer, if you have to actually go ahead and maintain a pipeline, being able to see the actual data manipulated really increases your productivity.”
Kim: Analysts and engineers make up 70% of Select Star's user base
“So today, majority of our user base, I would say about 65 to 70% are data analysts and data engineers.”
Data engineering complexity is caused by bad tooling obfuscating the process
“If you really break down what a data engineer is, at the end of the day, that their job is to get data from one place to the next, right? So all you really need to know is three things. Where my data's coming from, where my data's going, and what format does i…”
Handy: Data engineers should focus on scalability, not business logic
“Data engineers shouldn't want to be spending their days expressing business logic. Like how do you amortize revenue across multiple periods? They are technologists. They should want to be thinking about platforms and scalability and all of these like hard tech…”
Fraser: In-warehouse transformation compute is cheaper than data engineering time
“The additional cost of compute and storage to replicate the extra data, to do those steps, those transformation steps inside the warehouse Are so small now, you know, they're less than what it's going to cost you to pay your data engineer for a week to build y…”
Elsamadisi: A working single time-series table eliminates need for 20-30 data engineers
“If a single time series table worked, you wouldn't have 20 or 30 data engineers building tables to answer questions.”