Netflix's Metaflow is not intended to be a workflow orchestrator
Savin Goyal · Fireside Chat: Savin Goyal (ML Infra team (Metaflow), Netflix) with Matt Turck (Partner, FirstMark) · Feb 17, 2021 · at 13:46
Savin Goyal is the Tech Lead for the ML Infrastructure team at Netflix, discussing the architectural boundaries and product scope of Metaflow.
0:00 / 0:02exact quote · 2.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Metaflow does not intend to be a workflow orchestrator.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Savin Goyal
Netflix's machine learning infrastructure team consists of six engineers
“My team has six engineers right now.”
Netflix uses S3, Spark, Presto, and Snowflake for data querying
“We use SG as so like the storage layer for our data warehouse, and we use Spark, Presto, Snowflake as our query engines.”
Netflix uses open-source Titus for container orchestration
“In terms of compute our container orchestration platform is called Titus, which is yet another open source project.”
Netflix drives ETL and ML pipelines using internal scheduler Meson
“We have a workflow scheduler called Mason. Mason, which is a program scheduler that's being used to drive all of our ETL as well as machine learning pipelines.”