Housley: Many big data era data lakes turned into data swamps
Matt Housley · Fundamentals of Data Engineering | Joe Reis and Matt Housley · Oct 24, 2022 · at 6:55
Matt Housley, co-author of Fundamentals of Data Engineering, explains why high-profile data lake projects failed during the big data boom.
0:00 / 0:10exact quote · 10.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In many cases, I mean, I, I've worked in companies like this, it just turned into a data swamp, and that's because, in spite of the amazing technology we had, we kind of lost sight of these fundamental things.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Matt Housley
Housley: Moving from Hadoop to cloud data stacks has been very tough
“One of the things, one of the transitions that Joe and I went through, which I think a lot of people in this room went through, was the transition from the Hadoop world, from the previous big data world, into this new, like, cloud-based data engineering snack,…”
Housley: Core data engineering issues haven't changed in 20 years
“A lot of the core issues in data engineering actually haven't changed that much in 20 years, and so people got so excited about technology in the transition in the 2000, the internet bubble, rise of Google, the rise of Facebook, that they lost sight of that”