OpenLineage will take a couple of years to achieve widespread adoption
Julien Le Dem · Data Observability and Pipelines: OpenLineage and Marquez · Feb 1, 2021 · at 27:17
Julien Le Dem, CTO of Datakin and creator of OpenLineage, compares the timeline of OpenLineage's ecosystem adoption to Apache Arrow during a Q&A session.
0:00 / 0:09exact quote · 9.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It will take a couple of years to get that but all the open source project are kind of the early adopters of the idea.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Julien Le Dem
The best time to collect data pipeline metadata is during runtime
“The best time to collect metadata is when the job is running and we can inspect it to figure out what the inputs were, what the outputs were, how long it took you know, what was the version of the code and things like that.”
Collecting massive amounts of metadata creates a signal-to-noise challenge
“I think the curse of collecting all that metadata that becomes really hard to figure out, to find the right information in the mass of data, and therefore that's what we strive to solve, right?”