Jan 2, 2019 · 28m · a16z
a16z Podcast | From Data Warehouses to Data Lakes
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the a16z Podcast, host Scott Cooper and guest Gaurav Dhillon examine the technological evolution of enterprise software, detailing the shift from rigid on-premises integration and data warehousing to self-service cloud integration and predictive data lakes.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The host as informed peer 5.2 Guest teaching 4.8 Guest disagreement 1.4 The host pushing back 1.4
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 12:52 Gaurav rejects the 'statistics' label
Gaurav playfully but directly dismisses Scott's suggestion that data science is just statistics, joking that using that term cuts $100,000 off a data scientist's salary.
Hardest push from the host ▶ 5:30 Scott introduces the CIO governor theoryInstead of letting the guest drive the narrative, Scott steps in to offer his own specific thesis on how SaaS removed the CIO as a bottleneck to application proliferation.
Biggest teaching moment ▶ 6:11 Gaurav reveals 10X enterprise SaaS miscalculationsGaurav educates the host on enterprise realities, explaining that companies routinely underestimate their SaaS usage by an order of magnitude because employees treat web apps as simple websites.
The host holds their own ▶ 22:55 Scott's Roman cities architectural metaphorScott showcases deep domain knowledge by framing legacy tech persistence through an evocative historical metaphor of Roman cities layered over ancient foundations.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| The Evolution to Web-Based Applications and SaaS Proliferation | 6 | 5 | 1 | 2 | Scott demonstrates strong domain insight by proposing that SaaS proliferation was caused by eliminating the central CIO governor on software deployment. Gaurav enthusiastically agrees and expands on this, schooling the host on how enterprises underestimate their SaaS app count by 10X because users view SaaS as mere websites rather than formal applications. | |
| Modern Integration Challenges and the Push for Self-Service | 4 | 4 | 1 | 1 | Scott asks guiding questions about modern integration challenges and data formats. Gaurav explains the shift from legacy row-and-column data to web-native document models like JSON, alongside the transition of integration tasks out of dark IT basements into self-service tools. | |
| The Transition from Data Warehouses to Predictive Data Lakes | 5 | 6 | 3 | 2 | When Scott jokingly equates modern data science to old-school statistics, Gaurav humorously pushes back, noting that calling it statistics deducts $100,000 from a candidate's salary. Gaurav then explains how modern compute power enables heuristic algorithms and predictive analytics far beyond traditional regression models. | |
| Architecture and Mechanics of the Modern Data Lake | 5 | 5 | 1 | 1 | Scott frames the contrast between old-school data warehouses and modern data lakes, later accurately summarizing the modern data stack architecture. Gaurav uses an IKEA furniture analogy to illustrate how raw data lakes are curated into purified and bottled assets for predictive consumption. | |
| The Future of Cloud Data Platforms and Organizational IT Shifts | 6 | 4 | 1 | 1 | Scott demonstrates high expertise by offering the metaphor of Roman cities built upon layers of older ruins to describe enterprise IT retrofitting. Gaurav expands on the future shift toward cloud-based data lakes and changing C-suite relationships between CIOs, CTOs, and CMOs. |
Statements from this episode (11)
Dhillon: Replacing enterprise finance software is like open-heart surgery
“I mean, it's open heart surgery for the enterprise to replace finance. It's not simple. It's in many cases, you have to report earnings quarterly. So you really have to get this window just right.”
Dhillon: Enterprises underestimate their SaaS application usage by 10x
“If you bet companies a dollar that they're using X number of SaaS apps, They're off, but not by half. They're off by, like, 10 X, you know, because somebody in marketing is using something, and they go, it's not an application.”
Dhillon: Modern web apps require new plumbing for document models
“One is the data types fundamentally being different require new kinds of plumbing. You know, this is digital plumbing we're talking about, but you're no longer using rows and columns. You're using a document model. The way the worldwide web works, the way brow…”
Dhillon: Historical business intelligence offers zero value to modern tech companies
“This rear view mirror historical perspective is, is no longer of incremental value to a technology company or to an investment bank.”
Dhillon: Enterprise data architecture is shifting from data warehouses to data lakes
“What we're seeing is a trend away from legacy data warehouses into data lakes, which are then consumed both by people using modern visualization products, like say a Tableau, and also by lots and lots of data scientists”
Dhillon: Capital One built a massive business using early data science
“Capital One, who is, I would say, the original data science company, figured out tens of billions of dollars of business, giving credit cards to people who others had denied, and making it profitable using Data science.”
Dhillon: Data warehousing was originally an organizing principle, not a product
“Inman and Kimball came up with an organizing principle, you know, data warehousing was never a product. You couldn't go and buy one. It didn't exist, but it was an organizing principle to, and to corral Marshall and get benefits from the data that you had in y…”
Dhillon: Enterprise architecture is shifting from batch processing to real-time streaming
“First, you're going away from a batch architecture of the nineties to a real-time streaming architecture. Like, we want our stuff now. This is 2016, and you know, we want to see a movie now. So, so you're going to go through a real shift towards streams of dat…”
Dhillon: Enterprise IT architecture is always a retrofit job
“The enterprise is a retrofit job. It always has been.”
Dhillon: You can buy network bandwidth, but latency comes from God
“For some machine data, factory floor data, power plant data, it may not go just because, you know, bandwidth you can buy, but latency you get from God.”
Dhillon: Tension between CMOs and CIOs has largely resolved
“There was this tension between the chief marketing officer and the CIO. I think they've largely kissed and made up because A, the CIOs are getting to be more business people and the marketing people are getting more technical. So that tension has disappeared i…”