Jun 12, 2019 · 23m · mad
What's Next for Open-Source Time Series Data? // Paul Dix, Influx Data (FirstMark's Data Driven NYC)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Data Driven NYC keynote, InfluxData CTO and Founder Paul Dix explores the unique database requirements of time series data and introduces InfluxDB 2.0 alongside Flux, a novel functional scripting language designed to unify data collection, querying, and stream processing.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 1.4% of the talking time here. How this is scored →
Matt as informed peer 0.0 Guest teaching 0.3 Guest disagreement 0.3 Matt pushing back 0.0
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 21:50 Dismissing Lisp and JavaScript Subsets
Paul forcefully and humorously dismisses adopting existing languages like Lisp or JavaScript for Flux, arguing that if Paul Graham and Rich Hickey could not make Lisp popular, nobody will.
Hardest push from Matt ▶ 19:38 Host Caps Talk Duration for Q&AHost Matt Turck steps in to end the formal presentation due to time limits and redirects the remaining time to quick audience questions.
Biggest teaching moment ▶ 20:10 Explaining Dual Inverted Index ArchitecturePaul educates an audience member on how InfluxDB combines a columnar data store for time series values with an inverted index mapping metadata tag key-value pairs.
Matt holds his own ▶ 19:38 Host Contextualizes Company ProgressHost Matt Turck highlights the massive progress made by InfluxData since Paul last presented at the Data Driven NYC event.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Defining Time Series Data Across Key Use Cases | 0 | 0 | 0 | 0 | Paul Dix provides an introductory presentation defining time series data and distinguishing between regular metrics and irregular event streams. Because this is a solo keynote segment, all host-side dynamic scores are zero. | |
| Unique Database Workload Characteristics of Time Series | 0 | 0 | 0 | 0 | Paul explains database workload characteristics unique to time series, such as high write throughput, large range scans, and retention eviction. Host interaction scores remain zero for this monologue segment. | |
| The Origin and Evolution of the TICK Stack | 0 | 0 | 0 | 0 | Paul traces the origin of InfluxDB and the emergence of Telegraf, Kapacitor, and Chronograf to form the TICK stack. This is a solo technical talk with no host participation. | |
| InfluxDB Line Protocol Schema and Nanosecond Precision | 0 | 0 | 0 | 0 | Paul details the Line Protocol schema, support for varied field types, and nanosecond timestamp precision. Host interaction scores are zero during this monologue presentation. | |
| The Unified Architecture of InfluxDB 2.0 | 0 | 0 | 0 | 0 | Paul describes consolidating the separate TICK stack components into a unified InfluxDB 2.0 platform. Host scores remain zero as this is part of the main presentation. | |
| Client Libraries and Modular Visualization Tools | 0 | 0 | 0 | 0 | Paul covers official client libraries, UI JavaScript components, and the Flux engine pluggable parser architecture. Host scores are zero due to the solo monologue format. | |
| Flux Code Demonstration and Serverless Execution Platform | 0 | 0 | 0 | 0 | Paul demonstrates Flux syntax, functional pipe operators, and its capability as a serverless execution engine inside the database. Host-side scores are zero for this monologue. | |
| InfluxDB 2.0 Distribution Options and Keynote Conclusion | 0 | 2 | 2 | 0 | Host Matt Turck opens the floor to audience questions regarding indexing and language design rationale. Paul playfully dismisses Lisp and alternative options while answering the audience, maintaining full conversational authority. |
Statements from this episode (10)
Paul Dix: Regular time series data summarizes irregular event data
“The thing that's interesting about irregular time series data is that you can actually induce a regular time series from irregular time series data. Basically, a regular time series is just a summary of an irregular Series.”
Why traditional databases fail at time series workloads
“And actually no
Regular database is designed with this kind of workload in mind.
Databases, for the most part, assume that you want to keep your data around for a very long time.
So, these kind of unique aspects of time series make it kind of a degenerate case…”
Paul Dix: Telegraf is likely on tens to hundreds of millions of servers
“This project actually is our most popular open source project by far. We don't have any sort of tracking on it right now, but I would estimate that it's probably deployed on tens of millions of servers across the world, if not hundreds of millions at this poin…”
An InfluxData customer deployed Telegraf to 45,000 servers in a single day
“Our, we've had customers, like, deploy it to 45,000 servers in a single day, so.”
HFT firms use InfluxDB and atomic clocks for sub-300ns drift
“There are high frequency trading firms that use it to track latencies in their network infrastructure, and they actually have, like, atomic clocks deployed in their data centers, so they guarantee less than 300 nanoseconds of clock drift Worldwide.”
InfluxDB 2.0 unifies the TICK stack into a single database
“So, my idea was within FluxDB two dot O, we could collapse these things into one cohesive whole, and have one language that ties all of it together.”
Flux combines a query optimizer, VM, and Turing-complete scripting language
“Flux is basically a new language that we're creating for two dot O. It's a combination of a bunch of things. It's basically a query planner, it's a query optimizer, but it's also a Turing complete scripting language, which includes a virtual machine, And a que…”
InfluxData is building PromQL support directly into Flux
“And we also are in the middle of building in support for PromQL, the Prometheus query language.”
Flux turns InfluxDB into a serverless execution platform for time series
“It essentially turns the database into a serverless execution platform for time series data. The idea is you can define any sort of custom logic that you want, inject it into the database, and it will periodically run that logic over the data that you're writi…”