Jul 30, 2023 · 1h 16m · lennys-podcast
How to measure and improve developer productivity | Nicole Forsgren (Microsoft Research, Github)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this in-depth interview, Dr. Nicole Forsgren unpacks the empirical research behind developer productivity, explaining how frameworks like DORA and SPACE enable engineering organizations to balance velocity with system stability. She provides actionable guidance on aligning leadership goals, combining qualitative developer sentiment with system telemetry, and adapting measurement practices for the generative AI era.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 19.1% of the talking time here. How this is scored →
Lenny as informed peer 3.1 Guest teaching 4.6 Guest disagreement 1.1 Lenny pushing back 0.4
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 52:05 Pushing back against cutting engineering headcount with AI
Nicole emphatically dismisses simplistic executive narratives that AI efficiency metrics justify laying off half the engineering workforce, insisting AI transforms cognitive focus rather than simple task duration.
Hardest push from Lenny ▶ 22:33 Lenny challenges DORA benchmarks as inapplicable to startupsLenny directly challenges the generalizability of DORA thresholds, arguing that startups already ship constantly and that the published benchmarks are primarily tailored for large enterprises.
Biggest teaching moment ▶ 22:55 Nicole disproves company size myths with empirical dataNicole dismantles Lenny's startup hypothesis with statistical findings showing that company size has zero predictive power over software delivery performance, explaining that small and large firms make identical excuses.
Lenny holds their own ▶ 1:03:28 Lenny connects framework to his own statistical analysesLenny demonstrates his technical domain knowledge by showing his own diagram of the Four Box framework and linking Nicole's metric modeling directly to his published resources on correlation and regression.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Nicole Forsgren's Career: From IBM Sysadmin to DORA and Microsoft Research | 2 | 2 | 1 | 0 | Lenny opens with an invitation for Nicole to walk through her unique career history. Nicole details her transition from an IBM sysadmin to academia, DORA, Chef, Google, and Microsoft Research in a cooperative storytelling dynamic. | |
| Defining Developer Productivity, Developer Experience, and DevOps | 3 | 4 | 2 | 1 | Lenny asks how to distinguish developer productivity, experience, and DevOps, assuming everyone naturally wants to move faster. Nicole gently reframes the premise by pointing out that leaders often resist speed due to ungrounded fears of system instability. | |
| The DORA Framework: Unpacking the Four Key Metrics and Speed vs. Stability | 2 | 6 | 1 | 0 | Lenny prompts Nicole to unpack DORA and its core metrics. Nicole provides an authoritative breakdown of the four key metrics and explains the counterintuitive finding that speed and stability move in tandem rather than in opposition. | |
| Industry Benchmarks and Performance Dynamics Across Company Sizes | 4 | 7 | 2 | 2 | Lenny pushes back by suggesting DORA benchmarks apply primarily to large enterprises rather than startups that ship multiple times daily. Nicole corrects this assumption, noting data showed no statistical difference across company sizes, with retail surviving as the sole outlier. | |
| Beyond the Four Metrics: Core Capabilities That Predict Delivery Performance | 3 | 5 | 1 | 0 | Lenny summarizes takeaways and asks about the underlying capabilities driving DORA metrics. Nicole explains that DevOps is not purchased tooling but a collection of technical, architectural, and cultural capabilities. | |
| The SPACE Framework: A Multidimensional Approach to Measuring Creative Work | 4 | 6 | 2 | 1 | Lenny asks about the SPACE framework and how it relates to DORA. Nicole educates on the five dimensions of creative work, warning against tracking reductive activity metrics like lines of code or raw pull request counts. | |
| Recommended Literature on Productivity Measurement and Data Triangulation | 2 | 4 | 0 | 0 | Lenny asks for recommended reading for practitioners seeking deeper methodology. Nicole points listeners to her ACM SPACE paper, her collaboration with Mick Kirsten on system and human data triangulation, and Douglas Hubbard's measurement literature. | |
| Sponsor Message: Combining Qualitative and Quantitative Insights with DX | 2 | 4 | 1 | 0 | Following the DX sponsor read, Lenny invites Nicole to preview her upcoming book. Nicole outlines common executive errors in defining DevEx goals and explains how to progress along a measurement maturity curve. | |
| Avoiding Common Pitfalls in Developer Experience Initiatives | 3 | 4 | 1 | 0 | Lenny inquires about common organizational pitfalls and the historical shift in engineering culture. Nicole highlights the necessity of combining top-down leadership alignment with bottom-up individual contributor buy-in. | |
| How Generative AI Tools Reshape Developer Cognitive Load and Metrics | 4 | 6 | 2 | 1 | Lenny brings up real-world developer sentiment around AI coding tools like Copilot. Nicole dissects the cognitive shift from writing code to reviewing code and pushes back on executive assumptions that AI tools simply halve engineering headcounts. | |
| Immediate Next Steps for Teams and the Google Measurement Model | 3 | 5 | 1 | 0 | Lenny asks for immediate action items and gold-standard company models. Nicole advises immediate problem-definition audits and points to Google's practice of prioritizing developer surveys whenever subjective data conflicts with automated instrumentation. | |
| The Four Box Framework for Structuring Hypotheses and Metrics | 5 | 4 | 1 | 1 | Lenny asks about Nicole's Four Box Framework, shows off his live drawing, and cites his own newsletter guide on regression and correlation analyses. Nicole guides through converting words into testable data proxies. | |
| Data-Informed Decision-Making and Strategic Prioritization | 4 | 4 | 1 | 0 | Lenny explores Nicole's approach to decision-making frameworks and suggests publishing her weighted spreadsheet as a public template. Nicole explains being data-informed rather than data-driven and framing strategy around what not to fund. | |
| Lightning Round: Books, Culture, Interview Questions, and Products | 3 | 3 | 0 | 0 | Lenny conducts the lightning round covering book recommendations, television, interview questions, and tactical tips. Nicole provides energetic, personal answers spanning Richard Rumelt to Korean skincare. |
Statements from this episode (16)
Forsgren: Two-week change approval wait for software stability is a myth
“When I first started like ITIL and ITSM, right, the old school kind of change management processes, the common knowledge was that you had to have at least a two week wait for change approvals in order to get that stability. Turns out that's not right, right? I…”
Forsgren: Automated testing and architecture enable speed without sacrificing stability
“We want to implement Good technical practices like automated testing, good architectural practices so that when you move fast, you are also more stable, right?”
Forsgren: Software delivery speed and stability move in tandem, not trade-offs
“When we started measuring these, we found that they move in tandem. Now, like, with very strong significance, right, from a statistical standpoint. Now, what this means is, now we say speed and stability move together.”
Forsgren: Internal engineering progress matters more than elite performer benchmarks
“What's most important is knowing where you are and the progress that you're making, right? It doesn't matter if, frankly, you're a high performer or you're an elite performer. It matters that you know where you are and you're making progress”
Forsgren: Elite engineering teams deploy on demand and recover under an hour
“The elite performance of deployment frequency. You can deploy on demand. Lead time for changes takes less than a day. Time to restore is less than an hour and your change fail rate is between zero and 15%.”
Forsgren: Software delivery performance does not statistically vary by company size
“We saw no statistical significance between Small companies and large companies, the only statistical significant, statistically significant difference was with retail.”
Forsgren: DevOps is a set of organizational capabilities, not a toolchain
“DevOps is a set of capabilities. They're technical capabilities. They're architectural capabilities. They're cultural capabilities. They are lean management practices. That predict speed and stability and then speed and stability gives you money, right?”
Forsgren: Developer satisfaction strongly correlates with overall productivity and system stability
“This actually matters because we find that satisfaction well-being ends up being incredibly highly correlated with all of the other dimensions of productivity and doing things well. And as soon as satisfaction well-being, things like sustainability, if you're …”
Forsgren: The SPACE framework requires measuring at least three dimensions simultaneously
“Now, to use space correctly, we want to use at least three dimensions at a time, because that helps us balance.”
Forsgren: Survey data from engineers is an essential complement to system telemetry
“Data from people and data from systems are really important complements because we can get certain insights from people that we'll never get from systems, right?”
Forsgren: Dev productivity measurement starts with surveys before shifting to telemetry
“Because when you first start off, you'll be relying much more on data from people, because you can get it relatively quickly. But as you kind of transition through this measurement journey, you'll get more and more data from your systems, because it's scalable…”
Forsgren: MSR Paper Finds AI Coding Shifts 50% of Time to Reviewing
“There's this wonderful paper out this is the CUPS model. I'll share it with you. A team at MSR did it. It finds that, you know, about 50% of your time now is spent reviewing versus writing.”
Forsgren: Research Shows AI Speeds Up Specific Tasks by 50%
“Some of my colleagues also did a paper that showed that, you know, you can do certain tasks like build an HTTP server, you know, 50% faster.”
Forsgren: AI Coding Tools Should Free Up Cognitive Space, Not Prompt Layoffs
“Anyone who's looking at that and like, dear CEOs or whoever who are like, now I can lay off half my workforce. That's not what this is about, right? It's not about taking a task and cutting your time in half because now what we've enabled is your ability to do…”