StarZero All shows Try the podcast clipper →

Dec 5, 2013 · 24m · mad

Hilary Mason, Bitly // Data Driven NYC #3 // Feb 2012

Hilary Mason · 18m spoken Matt Turck · 31s spoken
0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

At a Data Driven NYC event, Bitly Chief Scientist Hilary Mason explains how Bitly analyzes global internet traffic, builds real-time production data systems, and extracts actionable behavioral insights from billions of shortened URL interactions.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 2.5% of the talking time here. How this is scored →

Matt as informed peer 0.8 Guest teaching 3.4 Guest disagreement 1.2 Matt pushing back 1.0
05100:0010:0020:002:00–6:27 · Matt as informed peer 0/10 URL Shortening Mechanics and Global Data Reach Hilary presents Bitly's foundational role in handling 301 redirects and scaling global data collection. As this is a pure presentation monologue, host expertise and pushback are non-existent.6:27–9:48 · Matt as informed peer 0/10 Internet Behavior, Personal Experience, and Device Trends Hilary shares findings on device usage patterns, peak iPad times, and cross-device similarities. The host remains silent during this monologue segment.9:48–14:35 · Matt as informed peer 0/10 Link Half-Lives, Arab Spring Data, and Knowledge Mapping Hilary illustrates how link half-lives differ across networks and shows traffic patterns during the Arab Spring. The host is inactive throughout the presentation.14:35–20:09 · Matt as informed peer 1/10 Live Real-Time Data Demonstration Hilary performs a live demonstration of Bitly's real-time click engine until Matt gently interjects at 19:48 to keep the schedule on track for audience Q&A. Hilary politely agrees and wraps up.20:09–24:01 · Matt as informed peer 3/10 Audience Q&A and Event Conclusion Matt prompts Hilary to fulfill her promise on why Hadoop sucks and asks about her real-time architecture, while an audience member pushes back on her presentation showing just fancy graphs. Hilary defends her approach by reframing problem definition versus presentation style.2:00–6:27 · Guest teaching 3/10 URL Shortening Mechanics and Global Data Reach Hilary presents Bitly's foundational role in handling 301 redirects and scaling global data collection. As this is a pure presentation monologue, host expertise and pushback are non-existent.6:27–9:48 · Guest teaching 3/10 Internet Behavior, Personal Experience, and Device Trends Hilary shares findings on device usage patterns, peak iPad times, and cross-device similarities. The host remains silent during this monologue segment.9:48–14:35 · Guest teaching 4/10 Link Half-Lives, Arab Spring Data, and Knowledge Mapping Hilary illustrates how link half-lives differ across networks and shows traffic patterns during the Arab Spring. The host is inactive throughout the presentation.14:35–20:09 · Guest teaching 2/10 Live Real-Time Data Demonstration Hilary performs a live demonstration of Bitly's real-time click engine until Matt gently interjects at 19:48 to keep the schedule on track for audience Q&A. Hilary politely agrees and wraps up.20:09–24:01 · Guest teaching 5/10 Audience Q&A and Event Conclusion Matt prompts Hilary to fulfill her promise on why Hadoop sucks and asks about her real-time architecture, while an audience member pushes back on her presentation showing just fancy graphs. Hilary defends her approach by reframing problem definition versus presentation style.2:00–6:27 · Guest disagreement 1/10 URL Shortening Mechanics and Global Data Reach Hilary presents Bitly's foundational role in handling 301 redirects and scaling global data collection. As this is a pure presentation monologue, host expertise and pushback are non-existent.6:27–9:48 · Guest disagreement 0/10 Internet Behavior, Personal Experience, and Device Trends Hilary shares findings on device usage patterns, peak iPad times, and cross-device similarities. The host remains silent during this monologue segment.9:48–14:35 · Guest disagreement 0/10 Link Half-Lives, Arab Spring Data, and Knowledge Mapping Hilary illustrates how link half-lives differ across networks and shows traffic patterns during the Arab Spring. The host is inactive throughout the presentation.14:35–20:09 · Guest disagreement 1/10 Live Real-Time Data Demonstration Hilary performs a live demonstration of Bitly's real-time click engine until Matt gently interjects at 19:48 to keep the schedule on track for audience Q&A. Hilary politely agrees and wraps up.20:09–24:01 · Guest disagreement 4/10 Audience Q&A and Event Conclusion Matt prompts Hilary to fulfill her promise on why Hadoop sucks and asks about her real-time architecture, while an audience member pushes back on her presentation showing just fancy graphs. Hilary defends her approach by reframing problem definition versus presentation style.2:00–6:27 · Matt pushing back 0/10 URL Shortening Mechanics and Global Data Reach Hilary presents Bitly's foundational role in handling 301 redirects and scaling global data collection. As this is a pure presentation monologue, host expertise and pushback are non-existent.6:27–9:48 · Matt pushing back 0/10 Internet Behavior, Personal Experience, and Device Trends Hilary shares findings on device usage patterns, peak iPad times, and cross-device similarities. The host remains silent during this monologue segment.9:48–14:35 · Matt pushing back 0/10 Link Half-Lives, Arab Spring Data, and Knowledge Mapping Hilary illustrates how link half-lives differ across networks and shows traffic patterns during the Arab Spring. The host is inactive throughout the presentation.14:35–20:09 · Matt pushing back 3/10 Live Real-Time Data Demonstration Hilary performs a live demonstration of Bitly's real-time click engine until Matt gently interjects at 19:48 to keep the schedule on track for audience Q&A. Hilary politely agrees and wraps up.20:09–24:01 · Matt pushing back 2/10 Audience Q&A and Event Conclusion Matt prompts Hilary to fulfill her promise on why Hadoop sucks and asks about her real-time architecture, while an audience member pushes back on her presentation showing just fancy graphs. Hilary defends her approach by reframing problem definition versus presentation style.

speaking balance: gold is Matt, purple is the guest (3 minute bins)

0:00 · Matt 1.8% · guest 98.2%0:00 · Matt 1.8% · guest 98.2%3:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%15:00 · Matt 0% · guest 100%15:00 · Matt 0% · guest 100%18:00 · Matt 11.6% · guest 88.4%18:00 · Matt 11.6% · guest 88.4%21:00 · Matt 6.9% · guest 93.1%21:00 · Matt 6.9% · guest 93.1%24:00 · Matt 100% · guest 0%24:00 · Matt 100% · guest 0%
Sharpest disagreement ▶ 22:02 Reframing Audience Critique

When an audience member asserts that her presentation showed fancy graphs that do not help solve problems, Hilary asks for clarification and rejects the premise by breaking down data science into asking good questions.

 Hardest push from Matt ▶ 19:48 Steering Talk to Audience Q&A

Matt interrupts Hilary's live web demonstration to pivot the presentation to Q&A so attendees have time to ask questions.

 Biggest teaching moment ▶ 20:10 Why Hadoop Fails for Real-Time

Hilary breaks down the technical reality of Hadoop, explaining why parallel querying over redundant file systems makes it entirely impractical for real-time product features.

 Matt holds his own ▶ 20:47 Targeted Real-Time Architecture Inquiry

Matt demonstrates domain awareness by holding Hilary to her earlier teaser on Hadoop and asking specifically what alternate architecture she utilizes for real-time monitoring.
the scores for every segment, with the reasoning behind each
ChapterTopicMatt as informed peerGuest teachingGuest disagreementMatt pushing backWhy
URL Shortening Mechanics and Global Data Reach 0310 Hilary presents Bitly's foundational role in handling 301 redirects and scaling global data collection. As this is a pure presentation monologue, host expertise and pushback are non-existent.
Internet Behavior, Personal Experience, and Device Trends 0300 Hilary shares findings on device usage patterns, peak iPad times, and cross-device similarities. The host remains silent during this monologue segment.
Link Half-Lives, Arab Spring Data, and Knowledge Mapping 0400 Hilary illustrates how link half-lives differ across networks and shows traffic patterns during the Arab Spring. The host is inactive throughout the presentation.
Live Real-Time Data Demonstration 1213 Hilary performs a live demonstration of Bitly's real-time click engine until Matt gently interjects at 19:48 to keep the schedule on track for audience Q&A. Hilary politely agrees and wraps up.
Audience Q&A and Event Conclusion 3542 Matt prompts Hilary to fulfill her promise on why Hadoop sucks and asks about her real-time architecture, while an audience member pushes back on her presentation showing just fancy graphs. Hilary defends her approach by reframing problem definition versus presentation style.

Statements from this episode (9)

Assertion Not checkable as stated
Hilary Mason: Bitly processes tens of millions of unique URLs daily
“Tens of millions of unique URLs per day, hundreds of millions of clicks on those URLs from every imaginable social network.”
Hilary Mason Dec 5, 2013 ▶ 4:31
Prediction Not checkable as stated
Hilary Mason predicts commoditization of tools for processing Big Data
“I think we'll see a change in the vocabulary and a commoditization of the tools to make that happen.”
Hilary Mason Dec 5, 2013 ▶ 5:15
Insight
Hilary Mason: Data science sits at the intersection of math, engineering, and hacking
“I tend to think that data science exists in the middle of all of these things. So math, statistics, computer science, and the ability to write algorithms, engineering, and then hacking”
Hilary Mason Dec 5, 2013 ▶ 5:28
Assertion Partly supported
Mason: Link half-life is 2.8 hours on Twitter versus 3.1 on Facebook
“We found that for Twitter, it's short, it's 2.8 hours. For Facebook, it's a little longer, about 3.1 hours.”
Hilary Mason Dec 5, 2013 ▶ 10:06
Assertion Supported
Mason: 3% of all 2011 Bitly clicks went to top 100 celebrity pages
“Three percent of all clicks in 2011 went to pages about the top hundred celebrities.”
Hilary Mason Dec 5, 2013 ▶ 12:13
Assertion Not checkable as stated
Mason: Bitly data shows isolated consumption for adult, religion, and chemistry content
“People who look at adult content only look at adult content. People who look at religion only look at religion, and people who look at chemistry really only look at chemistry.”
Hilary Mason Dec 5, 2013 ▶ 14:01
Assertion Not checkable as stated
Hilary Mason: Bitly data shows zero overlap between NYT and Fox News readers
“Nobody who reads the New York Times reads Fox News.”
Hilary Mason Dec 5, 2013 ▶ 14:47
Assertion Not checkable as stated
Hilary Mason: Hadoop is entirely impractical for real-time products
“All it is is a structure for running queries in parallel against data that you store in a redundant file system, and so it is entirely impractical for doing a real-time product.”
Hilary Mason Dec 5, 2013 ▶ 20:21
Disclosure
Hilary Mason: Bitly's real-time stack relies on C, Python, and Redis
“A lot of our stuff is homegrown, unfortunately. A lot of it's written in C. The rest is in Python. We use Redis quite a lot as a data store.”
Hilary Mason Dec 5, 2013 ▶ 20:58
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 400 conversations transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more