HDFS

“What we really created was, like, using HDFS, like, a data lake, where we basically copied the whole data sets from, like whatever we get from, like, analytical logs or, like, all our business data sources, too, into HDFS.”

“HDFS, a single cluster, can do about a hundred million files.”

“If you look at the machine learning, it's, fundamentally, it's an iterative algorithm, and every iteration is turned into a Hadoop job. So between the iteration, you write the data and read the data from HDFS, so that's why it's very slow.”

“HDFS, which will scale forever for probably like 99.9% of the companies on the planet. They're just never going to run out of space, you know, to put data into it.”