90% of contacted open source projects want to contribute to OpenLineage
Julien Le Dem · Data Observability and Pipelines: OpenLineage and Marquez · Feb 1, 2021 · at 26:19
Julien Le Dem, CTO of Datakin and co-creator of OpenLineage, responds to an audience question regarding industry adoption and ecosystem support for the OpenLineage standard.
0:00 / 0:06exact quote · 6.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“90% of the project we reach out to see a lot of value in this and want to contribute”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Julien Le Dem
The best time to collect data pipeline metadata is during runtime
“The best time to collect metadata is when the job is running and we can inspect it to figure out what the inputs were, what the outputs were, how long it took you know, what was the version of the code and things like that.”
Collecting massive amounts of metadata creates a signal-to-noise challenge
“I think the curse of collecting all that metadata that becomes really hard to figure out, to find the right information in the mass of data, and therefore that's what we strive to solve, right?”