Ben Johnson: No single storage engine could completely replace Elasticsearch
Ben Johnson · A New Kind of Logging System // Zach Sherman & Ben Johnson, Timber (FirstMark's Data Driven NYC) · Dec 5, 2018 · at 4:19
Ben Johnson, co-founder of Timber.io, discusses evaluating alternative data storage systems to replace Elasticsearch in Timber's logging architecture.
0:00 / 0:10exact quote · 10.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“What was interesting is that all of these storages have different strengths, but none of them really could just completely replace Elasticsearch.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ben Johnson
Ben Johnson: Real-time data processing was a major challenge with Elasticsearch
“So being being real-time and processing this data was really important, and that was a real challenge with Elasticsearch.”
Ben Johnson: Routing layer is the most valuable part of Timber's pipeline
“We want to talk to you about what we have found to be the most valuable aspect of our data pipeline. We call it the routing layer. It's really been the foundation of all the value we've extracted out of that.”
Ben Johnson: Timber struggled to scale Elasticsearch past multiple terabytes
“What we found though is that while it was good getting started, we encountered a lot of challenges with it, especially when we started to cross over into the multiple terabyte range. And we had a lot of trouble scaling it.”
Johnson: Timber intends to open-source its log routing technology
“Our intent is to open source this project.”