StarZero All shows Try the podcast clipper →

Oct 10, 2022 · 23m · mad

Modern Data Orchestration | Astronomer Co-Founders Pete DeJoy & Viraj Parekh

Pete DeJoy · 11m spoken Viraj Parekh · 8m spoken Matt Turck · 16s spoken
0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

In this presentation at Data Driven NYC, Astronomer co-founders Viraj Parekh and Pete DeJoy explore the evolution of modern data orchestration, demonstrating how their enterprise platform, Astro, empowers data teams to build, run, and observe Apache Airflow pipelines efficiently.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 1.3% of the talking time here. How this is scored →

Matt as informed peer 0.3 Guest teaching 2.1 Guest disagreement 0.1 Matt pushing back 0.3
05100:0010:0020:000:10–3:18 · Matt as informed peer 0/10 Speaker Introductions and Audience Survey The guests open with introductions, conduct an audience poll, and present a slide illustrating data orchestration architecture. The host does not speak during this presentation block.3:18–6:27 · Matt as informed peer 0/10 Apache Airflow Overview and Ecosystem Growth Viraj outlines the evolution of Apache Airflow and details the distinct needs of data analysts, data engineers, and data scientists. The host remains silent throughout the presentation segment.6:27–9:33 · Matt as informed peer 0/10 Extending Airflow & Introducing the Astro Platform Pete and Viraj explain Astronomer's strategic vision, introducing the product pillars of Build, Run, and Observe. The host does not interject.9:33–14:44 · Matt as informed peer 0/10 Astro Cloud IDE Product Demo Pete walks through a live demonstration of the Astro Cloud IDE, showing interactive SQL/Python DAG construction for non-engineering personas. The host does not speak.14:44–17:36 · Matt as informed peer 0/10 Automated Scheduling, No-Git Version Control & CI/CD Pete and Viraj highlight Astro's automated CI/CD pipeline and Git-less commit process, enabling version control without Git syntax knowledge. The host is inactive.17:36–20:00 · Matt as informed peer 0/10 Data Observability, Lineage & Company History Pete demonstrates lineage tracking and data observability powered by Datakin, before concluding with an event announcement. The host does not participate in this portion.20:00–23:01 · Matt as informed peer 2/10 Audience Q&A: Competitors & CI/CD Architecture Matt Turck steps in to prompt questions on company history and competitive landscape, guiding the Q&A session. The guests cooperatively explain their origin and commercial competitors like cloud managed services.0:10–3:18 · Guest teaching 1/10 Speaker Introductions and Audience Survey The guests open with introductions, conduct an audience poll, and present a slide illustrating data orchestration architecture. The host does not speak during this presentation block.3:18–6:27 · Guest teaching 2/10 Apache Airflow Overview and Ecosystem Growth Viraj outlines the evolution of Apache Airflow and details the distinct needs of data analysts, data engineers, and data scientists. The host remains silent throughout the presentation segment.6:27–9:33 · Guest teaching 2/10 Extending Airflow & Introducing the Astro Platform Pete and Viraj explain Astronomer's strategic vision, introducing the product pillars of Build, Run, and Observe. The host does not interject.9:33–14:44 · Guest teaching 3/10 Astro Cloud IDE Product Demo Pete walks through a live demonstration of the Astro Cloud IDE, showing interactive SQL/Python DAG construction for non-engineering personas. The host does not speak.14:44–17:36 · Guest teaching 2/10 Automated Scheduling, No-Git Version Control & CI/CD Pete and Viraj highlight Astro's automated CI/CD pipeline and Git-less commit process, enabling version control without Git syntax knowledge. The host is inactive.17:36–20:00 · Guest teaching 2/10 Data Observability, Lineage & Company History Pete demonstrates lineage tracking and data observability powered by Datakin, before concluding with an event announcement. The host does not participate in this portion.20:00–23:01 · Guest teaching 3/10 Audience Q&A: Competitors & CI/CD Architecture Matt Turck steps in to prompt questions on company history and competitive landscape, guiding the Q&A session. The guests cooperatively explain their origin and commercial competitors like cloud managed services.0:10–3:18 · Guest disagreement 0/10 Speaker Introductions and Audience Survey The guests open with introductions, conduct an audience poll, and present a slide illustrating data orchestration architecture. The host does not speak during this presentation block.3:18–6:27 · Guest disagreement 0/10 Apache Airflow Overview and Ecosystem Growth Viraj outlines the evolution of Apache Airflow and details the distinct needs of data analysts, data engineers, and data scientists. The host remains silent throughout the presentation segment.6:27–9:33 · Guest disagreement 0/10 Extending Airflow & Introducing the Astro Platform Pete and Viraj explain Astronomer's strategic vision, introducing the product pillars of Build, Run, and Observe. The host does not interject.9:33–14:44 · Guest disagreement 0/10 Astro Cloud IDE Product Demo Pete walks through a live demonstration of the Astro Cloud IDE, showing interactive SQL/Python DAG construction for non-engineering personas. The host does not speak.14:44–17:36 · Guest disagreement 0/10 Automated Scheduling, No-Git Version Control & CI/CD Pete and Viraj highlight Astro's automated CI/CD pipeline and Git-less commit process, enabling version control without Git syntax knowledge. The host is inactive.17:36–20:00 · Guest disagreement 0/10 Data Observability, Lineage & Company History Pete demonstrates lineage tracking and data observability powered by Datakin, before concluding with an event announcement. The host does not participate in this portion.20:00–23:01 · Guest disagreement 1/10 Audience Q&A: Competitors & CI/CD Architecture Matt Turck steps in to prompt questions on company history and competitive landscape, guiding the Q&A session. The guests cooperatively explain their origin and commercial competitors like cloud managed services.0:10–3:18 · Matt pushing back 0/10 Speaker Introductions and Audience Survey The guests open with introductions, conduct an audience poll, and present a slide illustrating data orchestration architecture. The host does not speak during this presentation block.3:18–6:27 · Matt pushing back 0/10 Apache Airflow Overview and Ecosystem Growth Viraj outlines the evolution of Apache Airflow and details the distinct needs of data analysts, data engineers, and data scientists. The host remains silent throughout the presentation segment.6:27–9:33 · Matt pushing back 0/10 Extending Airflow & Introducing the Astro Platform Pete and Viraj explain Astronomer's strategic vision, introducing the product pillars of Build, Run, and Observe. The host does not interject.9:33–14:44 · Matt pushing back 0/10 Astro Cloud IDE Product Demo Pete walks through a live demonstration of the Astro Cloud IDE, showing interactive SQL/Python DAG construction for non-engineering personas. The host does not speak.14:44–17:36 · Matt pushing back 0/10 Automated Scheduling, No-Git Version Control & CI/CD Pete and Viraj highlight Astro's automated CI/CD pipeline and Git-less commit process, enabling version control without Git syntax knowledge. The host is inactive.17:36–20:00 · Matt pushing back 0/10 Data Observability, Lineage & Company History Pete demonstrates lineage tracking and data observability powered by Datakin, before concluding with an event announcement. The host does not participate in this portion.20:00–23:01 · Matt pushing back 2/10 Audience Q&A: Competitors & CI/CD Architecture Matt Turck steps in to prompt questions on company history and competitive landscape, guiding the Q&A session. The guests cooperatively explain their origin and commercial competitors like cloud managed services.

speaking balance: gold is Matt, purple is the guest (3 minute bins)

0:00 · Matt 0% · guest 100%0:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%15:00 · Matt 0% · guest 100%15:00 · Matt 0% · guest 100%18:00 · Matt 6.9% · guest 93.1%18:00 · Matt 6.9% · guest 93.1%21:00 · Matt 5.2% · guest 94.8%21:00 · Matt 5.2% · guest 94.8%
Sharpest disagreement ▶ 21:10 Addressing DIY Airflow as competition

Viraj subtly pushes back on standard vendor competitive framing, arguing their true competition is internal data teams attempting to build and maintain custom Airflow infrastructure.

 Hardest push from Matt ▶ 20:00 Steering presentation to company metrics

Matt interrupts the product presentation flow to demand foundational business context regarding company founding, location, and headcount.

 Biggest teaching moment ▶ 15:05 Explaining non-technical Git abstraction

Pete and Viraj educate the audience on how Astro's Cloud IDE allows analysts and data scientists to maintain branch-based CI/CD workflows without needing Git command-line skills.

 Matt holds his own ▶ 21:07 Re-focusing competitive dynamics question

Matt steps in to sharpen an audience prompt, directly challenging the co-founders to name their biggest commercial rivals.
the scores for every segment, with the reasoning behind each
ChapterTopicMatt as informed peerGuest teachingGuest disagreementMatt pushing backWhy
Speaker Introductions and Audience Survey 0100 The guests open with introductions, conduct an audience poll, and present a slide illustrating data orchestration architecture. The host does not speak during this presentation block.
Apache Airflow Overview and Ecosystem Growth 0200 Viraj outlines the evolution of Apache Airflow and details the distinct needs of data analysts, data engineers, and data scientists. The host remains silent throughout the presentation segment.
Extending Airflow & Introducing the Astro Platform 0200 Pete and Viraj explain Astronomer's strategic vision, introducing the product pillars of Build, Run, and Observe. The host does not interject.
Astro Cloud IDE Product Demo 0300 Pete walks through a live demonstration of the Astro Cloud IDE, showing interactive SQL/Python DAG construction for non-engineering personas. The host does not speak.
Automated Scheduling, No-Git Version Control & CI/CD 0200 Pete and Viraj highlight Astro's automated CI/CD pipeline and Git-less commit process, enabling version control without Git syntax knowledge. The host is inactive.
Data Observability, Lineage & Company History 0200 Pete demonstrates lineage tracking and data observability powered by Datakin, before concluding with an event announcement. The host does not participate in this portion.
Audience Q&A: Competitors & CI/CD Architecture 2312 Matt Turck steps in to prompt questions on company history and competitive landscape, guiding the Q&A session. The guests cooperatively explain their origin and commercial competitors like cloud managed services.

Statements from this episode (10)

Assertion Not checkable as stated
Cloud data warehouses enabled modern multi-tool data architectures
“Architectures like this really have only become possible with the rise of folks like Snowflake, Databricks, Redshift, BigQuery, and this idea that as a data practitioner, I can have infinite compute at the kind of tips of my fingers without having to go outsid…”
Viraj Parekh Oct 10, 2022 ▶ 2:20
Assertion Supported
Apache Airflow was created in 2014 at Airbnb
“Airflow was founded in 2014 at Airbnb, and it's an absolutely viral open source project.”
Viraj Parekh Oct 10, 2022 ▶ 3:26
Opinion
Data scientists do the 'sexiest work' on data teams
“And lastly, you know, your data scientists who, in my opinion, are kind of doing the sexiest work when making business predictions based off of data, their needs are also different.”
Viraj Parekh Oct 10, 2022 ▶ 5:52
Assertion Supported
Astro Cloud IDE generates standard Airflow DAG files, avoiding proprietary formats
“We're not actually producing a YAML file or any proprietary file format at the end of the day. We're really just producing a best practice Airflow DAG file.”
Pete DeJoy Oct 10, 2022 ▶ 14:15
Insight
Bypassing Git simplifies source control for non-software data teams
“The fact that I can take my skill set and have a source controlled pipeline without actually having to learn Git makes all the downstream people on my team's lives so much easier because they know exactly what I did and what I changed and they still get that r…”
Viraj Parekh Oct 10, 2022 ▶ 15:18
Opinion
Netlify serves as inspiration for exceptional developer experience workflows
“I draw a lot of inspiration from companies like Netlify that have done this stuff very, very well.”
Pete DeJoy Oct 10, 2022 ▶ 16:26
Disclosure
Astronomer acquired data observability startup Datakin
“We actually acquired a company called Datakin not so long ago.”
Pete DeJoy Oct 10, 2022 ▶ 17:40
Insight
Seeing manipulated data directly increases data engineer productivity
“I just think that as a data engineer, if you have to actually go ahead and maintain a pipeline, being able to see the actual data manipulated really increases your productivity.”
Viraj Parekh Oct 10, 2022 ▶ 18:41
Assertion Not checkable as stated
Astronomer has grown to roughly 350 remote-first employees
“I think we're around 350 people now, give or take. We have folks in New York, London, San Francisco, India, and kind of all over remote first.”
Viraj Parekh Oct 10, 2022 ▶ 20:43
Opinion
AWS and Google Cloud managed Airflow services are Astronomer's primary threats
“I think, like, the cloud, it's worth also mentioning, like, the major public clouds that have managed air flow services as well. You know, we see quite a lot of adoption of Amazon's MWAA service and Google's cloud composer service, and I think, like, that real…”
Pete DeJoy Oct 10, 2022 ▶ 21:50
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 400 conversations transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more