StarZero All shows Try the podcast clipper →

Oct 24, 2022 · 34m · mad

Fundamentals of Data Engineering | Joe Reis and Matt Housley

Joe Reis · 14m spoken Matt Housley · 10m spoken
0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

At a Data Driven NYC event, authors Joe Reis and Matt Housley present key insights from their book 'Fundamentals of Data Engineering.' They break down the data engineering lifecycle, critique industry tool obsession, and offer practical guidance on data architecture and career development.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →

Matt as informed peer 0.0 Guest teaching 0.0 Guest disagreement 1.0 Matt pushing back 0.0
05100:0010:0020:0030:000:13–2:24 · Matt as informed peer 0/10 Speaker Welcome and Audio Check The speakers welcome the room, perform an audio check, and give a shoutout to host Matt Turck for inspiring their book via his data landscape diagram. Because this is a keynote presentation without host participation, host-side scores are zero.2:24–5:12 · Matt as informed peer 0/10 Core Motives and Industry Challenges in Data Engineering Joe and Matt outline why they wrote the book, focusing on the lack of standardized definitions and excessive vendor hype in data engineering. They push back against tool-centric definitions of the field.5:12–7:17 · Matt as informed peer 0/10 Defining Data Engineering and Avoiding Data Swamps The speakers poll the audience for definitions before offering their own simple definition of data engineering. Matt points out how chasing big data tools without fundamentals caused data lakes to become unusable data swamps.7:17–12:52 · Matt as informed peer 0/10 Business Outcomes, Data Lifecycles, and Recovering Data Scientists The presentation turns to business outcomes and data lifecycles, with audience interaction from members like Alex and Tony. Joe and Matt share their experience as recovering data scientists who lacked necessary data infrastructure.12:52–18:11 · Matt as informed peer 0/10 The Data Engineering Lifecycle Diagram and Undercurrents Matt and Joe explain the data engineering lifecycle diagram and undercurrents like security, data management, and ops. They discuss how industry pendulum swings are bringing discipline back to data management.18:11–34:30 · Matt as informed peer 0/10 Audience Q&A Session on Careers, Architectures, and Paradigms The speakers take Q&A from audience members on career transitions, tool lock-in, and architectural paradigms. Joe provides animated commentary on title inflation and dogmatic data modeling debates.0:13–2:24 · Guest teaching 0/10 Speaker Welcome and Audio Check The speakers welcome the room, perform an audio check, and give a shoutout to host Matt Turck for inspiring their book via his data landscape diagram. Because this is a keynote presentation without host participation, host-side scores are zero.2:24–5:12 · Guest teaching 0/10 Core Motives and Industry Challenges in Data Engineering Joe and Matt outline why they wrote the book, focusing on the lack of standardized definitions and excessive vendor hype in data engineering. They push back against tool-centric definitions of the field.5:12–7:17 · Guest teaching 0/10 Defining Data Engineering and Avoiding Data Swamps The speakers poll the audience for definitions before offering their own simple definition of data engineering. Matt points out how chasing big data tools without fundamentals caused data lakes to become unusable data swamps.7:17–12:52 · Guest teaching 0/10 Business Outcomes, Data Lifecycles, and Recovering Data Scientists The presentation turns to business outcomes and data lifecycles, with audience interaction from members like Alex and Tony. Joe and Matt share their experience as recovering data scientists who lacked necessary data infrastructure.12:52–18:11 · Guest teaching 0/10 The Data Engineering Lifecycle Diagram and Undercurrents Matt and Joe explain the data engineering lifecycle diagram and undercurrents like security, data management, and ops. They discuss how industry pendulum swings are bringing discipline back to data management.18:11–34:30 · Guest teaching 0/10 Audience Q&A Session on Careers, Architectures, and Paradigms The speakers take Q&A from audience members on career transitions, tool lock-in, and architectural paradigms. Joe provides animated commentary on title inflation and dogmatic data modeling debates.0:13–2:24 · Guest disagreement 0/10 Speaker Welcome and Audio Check The speakers welcome the room, perform an audio check, and give a shoutout to host Matt Turck for inspiring their book via his data landscape diagram. Because this is a keynote presentation without host participation, host-side scores are zero.2:24–5:12 · Guest disagreement 1/10 Core Motives and Industry Challenges in Data Engineering Joe and Matt outline why they wrote the book, focusing on the lack of standardized definitions and excessive vendor hype in data engineering. They push back against tool-centric definitions of the field.5:12–7:17 · Guest disagreement 1/10 Defining Data Engineering and Avoiding Data Swamps The speakers poll the audience for definitions before offering their own simple definition of data engineering. Matt points out how chasing big data tools without fundamentals caused data lakes to become unusable data swamps.7:17–12:52 · Guest disagreement 1/10 Business Outcomes, Data Lifecycles, and Recovering Data Scientists The presentation turns to business outcomes and data lifecycles, with audience interaction from members like Alex and Tony. Joe and Matt share their experience as recovering data scientists who lacked necessary data infrastructure.12:52–18:11 · Guest disagreement 1/10 The Data Engineering Lifecycle Diagram and Undercurrents Matt and Joe explain the data engineering lifecycle diagram and undercurrents like security, data management, and ops. They discuss how industry pendulum swings are bringing discipline back to data management.18:11–34:30 · Guest disagreement 2/10 Audience Q&A Session on Careers, Architectures, and Paradigms The speakers take Q&A from audience members on career transitions, tool lock-in, and architectural paradigms. Joe provides animated commentary on title inflation and dogmatic data modeling debates.0:13–2:24 · Matt pushing back 0/10 Speaker Welcome and Audio Check The speakers welcome the room, perform an audio check, and give a shoutout to host Matt Turck for inspiring their book via his data landscape diagram. Because this is a keynote presentation without host participation, host-side scores are zero.2:24–5:12 · Matt pushing back 0/10 Core Motives and Industry Challenges in Data Engineering Joe and Matt outline why they wrote the book, focusing on the lack of standardized definitions and excessive vendor hype in data engineering. They push back against tool-centric definitions of the field.5:12–7:17 · Matt pushing back 0/10 Defining Data Engineering and Avoiding Data Swamps The speakers poll the audience for definitions before offering their own simple definition of data engineering. Matt points out how chasing big data tools without fundamentals caused data lakes to become unusable data swamps.7:17–12:52 · Matt pushing back 0/10 Business Outcomes, Data Lifecycles, and Recovering Data Scientists The presentation turns to business outcomes and data lifecycles, with audience interaction from members like Alex and Tony. Joe and Matt share their experience as recovering data scientists who lacked necessary data infrastructure.12:52–18:11 · Matt pushing back 0/10 The Data Engineering Lifecycle Diagram and Undercurrents Matt and Joe explain the data engineering lifecycle diagram and undercurrents like security, data management, and ops. They discuss how industry pendulum swings are bringing discipline back to data management.18:11–34:30 · Matt pushing back 0/10 Audience Q&A Session on Careers, Architectures, and Paradigms The speakers take Q&A from audience members on career transitions, tool lock-in, and architectural paradigms. Joe provides animated commentary on title inflation and dogmatic data modeling debates.

speaking balance: gold is Matt, purple is the guest (3 minute bins)

0:00 · Matt 0% · guest 100%0:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%15:00 · Matt 0% · guest 100%15:00 · Matt 0% · guest 100%18:00 · Matt 0% · guest 100%18:00 · Matt 0% · guest 100%21:00 · Matt 0% · guest 100%21:00 · Matt 0% · guest 100%24:00 · Matt 0% · guest 100%24:00 · Matt 0% · guest 100%27:00 · Matt 0% · guest 100%27:00 · Matt 0% · guest 100%30:00 · Matt 0% · guest 100%30:00 · Matt 0% · guest 100%33:00 · Matt 0% · guest 100%33:00 · Matt 0% · guest 100%
Sharpest disagreement ▶ 32:50 Data modeling cults critique

Joe Reis forcefully mocks the rigid dogmatism in data modeling discussions, comparing proponents of Kimball and Data Vault to clashing religious sects and calling them crazy.

 Hardest push from Matt ▶ 30:22 Audience challenge on tool-based definitions

An audience member challenges the speakers' refusal to define data engineering around tools, pushing back by pointing out the cyclical nature of storage and compute trade-offs.

 Biggest teaching moment ▶ 6:34 Explaining data lake failures into swamps

Matt Housley educates the room on how hype around big data tools without focus on fundamental principles resulted in data lakes turning into unmanageable data swamps.

 Matt holds his own ▶ 28:10 Tracing historical origins of ETL and ELT

Joe Reis demonstrates deep domain expertise by tracing the evolution of data integration paradigms from Bill Inmon's 1989 mainframe ETL model to modern cloud ELT.
the scores for every segment, with the reasoning behind each
ChapterTopicMatt as informed peerGuest teachingGuest disagreementMatt pushing backWhy
Speaker Welcome and Audio Check 0000 The speakers welcome the room, perform an audio check, and give a shoutout to host Matt Turck for inspiring their book via his data landscape diagram. Because this is a keynote presentation without host participation, host-side scores are zero.
Core Motives and Industry Challenges in Data Engineering 0010 Joe and Matt outline why they wrote the book, focusing on the lack of standardized definitions and excessive vendor hype in data engineering. They push back against tool-centric definitions of the field.
Defining Data Engineering and Avoiding Data Swamps 0010 The speakers poll the audience for definitions before offering their own simple definition of data engineering. Matt points out how chasing big data tools without fundamentals caused data lakes to become unusable data swamps.
Business Outcomes, Data Lifecycles, and Recovering Data Scientists 0010 The presentation turns to business outcomes and data lifecycles, with audience interaction from members like Alex and Tony. Joe and Matt share their experience as recovering data scientists who lacked necessary data infrastructure.
The Data Engineering Lifecycle Diagram and Undercurrents 0010 Matt and Joe explain the data engineering lifecycle diagram and undercurrents like security, data management, and ops. They discuss how industry pendulum swings are bringing discipline back to data management.
Audience Q&A Session on Careers, Architectures, and Paradigms 0020 The speakers take Q&A from audience members on career transitions, tool lock-in, and architectural paradigms. Joe provides animated commentary on title inflation and dogmatic data modeling debates.

Statements from this episode (7)

Assertion Not checkable as stated
Housley: Moving from Hadoop to cloud data stacks has been very tough
“One of the things, one of the transitions that Joe and I went through, which I think a lot of people in this room went through, was the transition from the Hadoop world, from the previous big data world, into this new, like, cloud-based data engineering snack,…”
Matt Housley Oct 24, 2022 ▶ 1:03
Insight
Housley: Core data engineering issues haven't changed in 20 years
“A lot of the core issues in data engineering actually haven't changed that much in 20 years, and so people got so excited about technology in the transition in the 2000, the internet bubble, rise of Google, the rise of Facebook, that they lost sight of that”
Matt Housley Oct 24, 2022 ▶ 4:39
Insight
Housley: Many big data era data lakes turned into data swamps
“In many cases, I mean, I, I've worked in companies like this, it just turned into a data swamp, and that's because, in spite of the amazing technology we had, we kind of lost sight of these fundamental things.”
Matt Housley Oct 24, 2022 ▶ 6:55
Insight
Housley: Statistics and ML skills cannot overcome poor data inputs
“No matter how good you are at statistics or machine learning, it's hard to make sense of data without having some quality inputs.”
Matt Housley Oct 24, 2022 ▶ 12:07
Prediction Not checkable as stated
Reis predicts the data engineering job title will face title dilution
“You know, and in fact, I think, candidly, data engineering will suffer the same thing. It will happen. We write about this in the last chapter of our book, in the fact that data engineering as a title could morph into something else, but that's what titles do.”
Joe Reis Oct 24, 2022 ▶ 22:28
Insight
Reis: Aspiring data engineers must focus on rigorous software engineering skills
“If you're really good at the data part, then you got that covered, right? But what you probably lack is, like, rigorous software engineering skills. That's what I would focus on, hands down.”
Joe Reis Oct 24, 2022 ▶ 24:45
Assertion Not checkable as stated
Reis: Data engineering paradigm is swinging back from ELT to ETL
“But what's interesting right now is the discussion is actually moving back to ETL. I, I'm starting to see more and more discussions about how ELT is a bad paradigm, and again, remember when I said talk about pendulums? The pendulum again is swinging back to ET…”
Joe Reis Oct 24, 2022 ▶ 29:33
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 400 conversations transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more