Alexis Le-Quoc, Founder and CTO of Datadog, explains the fundamental limits of software telemetry during a presentation on modern monitoring infrastructure.
Insight
Datadog CTO: False positives, not missed alerts, are the primary enemy in monitoring
“So that, that's why I would say for us to, it's really false positives. That's the enemy. Somehow you'll find out when something goes wrong.”
Assertion Not checkable as stated
Le-Quoc: Vast majority of industry still struggles with data visualization
“Not the sort of advanced customer of ours, you know, say, you know, Airbnb or Twitter or, Spotify, but the vast majority of the industry is, is still kind of struggling with that.”
Insight
Le-Quoc: Continuous dashboard monitoring fails due to human visual fatigue
“Once you can put it on a screen, it's great, but you're not gonna watch watch this for an entire day, or even a few hours, because A, you'll get tired of it, B, your brain will sort of things will become blurry, and you'll, you'll use your, you'll lose your ef…”
Insight
Datadog CTO: Software uniquely rewards maximizing the rate of deployment change
“In the industry of software, we pay a premium for rate of change. By that I mean, and it's very different from a physical production line, is the faster the production line changes, the better off we are. That means the faster you can go to market the more fea…”
Insight
Le-Quoc: Monitor statistical outliers rather than individual metric time series
“I don't want to monitor any particular of the time series, because there are too many of them, just the volume of data is too large. Rather, I want to find the ones that, that sort of deviates.”
Insight
Alexis Le-Quoc: Prioritize work metrics over resource metrics in system monitoring
“Not all metrics are created equal, and the ones you want to really care about it are the work metrics. So in that large volume of metrics, if you already classify in, in two classes of metrics, work and resource, you can reduce your search space by a lot.”