Uber transitioned from ETL into Vertica to EL into Hadoop
Praveen Murugesan · The Uber Big Data Story // Praveen Murugesan, Uber (Data Driven NYC / FirstMark) · Sep 30, 2016 · at 6:19
Praveen Murugesan, Ride Experience Lead at Uber, explains the architectural transition of Uber's data pipeline from traditional ETL to a Hadoop data lake.
0:00 / 0:14exact quote · 15.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We went from an ETL model, where we scraped from, like, the original source, transformed the data and loaded to Vertica, to, like, just an EL model, where we just, like, just copy the data as soon as possible into, like, Hadoop, and all the transformation can basically just happen as secondary steps on Hadoop.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Praveen Murugesan
Uber deployed trip similarity algorithms to fight incentive fraud in China
“Uber used to run this incentive program in China, and, ah, people are trying to game it, so they used to always have, like, similar trips simulated From their, ah, various different devices. So what we did was actually, like, try to find an algorithm where we …”
Murugesan: Uber practically had no data infrastructure when he joined in 2014
“We practically did not have an infrastructure is what the honest reality is.”
Schemaless JSON fails as companies scale beyond 50 employees
“It works well, like, if you're, like, a ten-person company, or, like, even to a fifty-person company, when you're, like, scaling to, like, thousands of people, like, you actually need a proper negotiation in between.”
Murugesan: City operations personnel constitute most of Uber's data consumers
“Most of Uber's data consumers are actually these operations people.”