Google BigQuery's largest customers never ran 100-terabyte queries, says Jordan Tigani
Jordan Tigani · The Death of Big Data and Why It’s Time To Think Small | Jordan Tigani, CEO, MotherDuck · Oct 24, 2024 · at 2:17
Jordan Tigani, co-founder and CEO of MotherDuck and former founding engineer on Google BigQuery, notes that industry database benchmarks focus on query sizes far larger than real-world enterprise usage.
0:00 / 0:20exact quote · 20.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The query sizes they were using was a hundred terabytes, and I remembered back from my time at BigQuery you know, we had some of the, Largest customers in the world. We had Walmart, Home Depot, Equifax, HSBC, you know, like, and nobody was using anything, you know, running queries anywhere near that because it would have actually fallen over at the time.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jordan Tigani
Distributed data systems impose a massive complexity tax over single-node hardware
“There's just this huge tax that you pay to have to build a distributed system that scales out and can do, like, you know, distributed transactions and shuffling data and, you know, if you were gonna design something for kind of modern hardware, you could make …”
MotherDuck executes queries in milliseconds compared to BigQuery's 400-millisecond overhead
“You know, we can do queries in sort of single digit milliseconds. And you know, in BigQuery, we were very, very happy when we got kind of the overhead down to like, 400 milliseconds.”
Open data formats like Apache Iceberg will pressure incumbent database margins
“I think that's going to be hard for the incumbents, and I think it's going to be a net benefit to the, you know, kind of the people that are coming in with new with new tools and new ways of doing things, and it's going to put pressure on margins which again i…”
Tigani: DuckDB is amazing software, but not a data warehouse
“DuckDB is an amazing, amazing, amazing piece of software, but it's not a data warehouse.”
Network latency makes 60 FPS data visualization impossible in traditional cloud architectures
“This is how you get the 60 frame per second kinds of you know, visual, visual visualization speeds that are literally impossible in more traditional architecture, because if the, if like, if I'm talking to a data center that is, you know, on the other side of …”
Modern laptops are 10 to 100 times faster than MapReduce-era servers
“But you know, nowadays, like, you know, I've got a Mac M two laptop. It's two years old. It's like probably an order of magnitude to two orders of magnitude faster than the server machines were back when, you know, MapReduce came out and people started buildin…”