Benn Stancil: Data orchestration should be deadline-driven rather than task-triggered
Benn Stancil · Separating Data Hype From Substance | Fireside Chat with Mode Co-Founder Benn Stancil · Oct 10, 2022 · at 26:12
Benn Stancil, Co-Founder and Chief Analytics Officer at Mode, discusses architectural flaws in Directed Acyclic Graph (DAG) based data orchestration systems like Airflow.
0:00 / 0:19exact quote · 20.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The way we've built DAGs are kind of like, when do I start the, like, when do I kick over the first one, and then we line it up such that we hope the thing arrives at the end. And the way it would make more sense to me is you just tell the thing, I need this thing to be here by five AM. You figure out what has to happen beforehand, and then like kick over the dominoes when they need to be kicked over.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Benn Stancil
Stancil: Modern data stack startups face an inevitable market reckoning
“Pretty quickly you're gonna realize they're all stepping on each other's toes, and that, that fallout has to go somewhere. Like, not all of these companies can be the next Figma that they all now say that they are, and so it's what happens then, and I think it…”
Stancil: dbt's SQL-based semantic layer is not a real semantic model
“They're moving in, like, the semantic layer tool, so basically they're saying, give us raw data, and we will tell you, like, apply semantics to it. The way that they do that now is through SQL. So, like, semantics are kind of, like, air quote semantics. It's b…”
Stancil: dbt's initial ease of use may become its downfall
“Dbt is, I think, really popular because it's so easy to get up and running, but it may also eventually be, like, if it had an undoing, to me that would be the undoing, is the thing that was really easy to get up and running doesn't actually solve the real prob…”
Stancil: Native SaaS warehouse connections could undercut Fivetran's core revenue
“You know, it doesn't undercut everything for Fivetran, but it potentially undercuts the big sources, which I imagine are the things that are, like, the real drivers of revenue for them.”
Stancil: Enterprise data market won't support 1,000 unicorns
“The median outcome is the data philosophies of Facebook and LinkedIn and all these kind of early tech companies gets adopted by the enterprise, and so all these sort of modern data tools that tech companies buy today go off and get sold to Coca-Cola and Caterp…”
Stancil: Companies should hire social science graduates for data analytics roles
“If you want to find these folks, there's a really good place to look which is hiring social scientists. So social scientists are trained to think this way. They are analytical thinkers without some of the technical and quantitative background that a lot of STE…”