Airflow

“Those two projects are, like, very interesting, but to some extent don't go far enough, because what you really want, at the end of the day, is the ability to capture, within the platform, everything related to, ah, to your data science project.”

“How, what time will this pipeline run? How will I know if it failed? We'll leave those tools to Dagster, Airflow, Prefect, Luigi, and many others to actually handle for you, because they do that really well.”

“Jeremiah was a primary contributor to Airflow, wanted to make some changes, wasn't able to do so, and went and started went and started Prefect.”

“Airflow was founded in 2014 at Airbnb, and it's an absolutely viral open source project.”

“One of the teams at Riot Games was like deploying their ML models and they were using Airflow.”

“At Facebook, you essentially have your internal airflow that a different team runs, and you can just push files to it and it picks up the data pipelines, you know, automatically. Whereas in most other companies, you might be the person that has to even manage,…”