Kedro leaves pipeline scheduling and failure monitoring to Airflow and Dagster
Kedro Product Manager · Introducing Kedro · Feb 17, 2021 · at 6:25
Yetunde Dada, Principal Product Manager at QuantumBlack, explains Kedro's architectural boundaries within the MLOps ecosystem relative to workflow orchestrators.
0:00 / 0:12exact quote · 12.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“How, what time will this pipeline run? How will I know if it failed? We'll leave those tools to Dagster, Airflow, Prefect, Luigi, and many others to actually handle for you, because they do that really well.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Kedro Product Manager
Dada argues building data pipelines is harder than machine learning itself
“And this is where we believe that machine learning is not really the hard part, but building and maintaining the data pipeline is”
Dada asserts notebooks are challenging for reproducible, production-ready data pipelines
“In terms of actually building your full pipeline and notebook that's where things get a little bit complicated because notebooks are a little bit challenging for reproducible and what we call production-ready code.”
Dada says data science experiment parameters must be separated from codebases
“So your experiment parameters for data science workflow should also be outside of your code base because we really believe it helps you make code that is generalizable and reusable by removing things that loading and load the code for loading and saving your d…”
Dada argues AWS SageMaker is a deployment target, not a code-structuring tool
“We use, we think of Kedra or SageMaker as a deployment target for Kedra. So still that whole process of like creating like, well document, well structured, Modular data science code, but still having the freedom to deploy it on SageMaker is how we see those ro…”