Sep 23, 2019 · 53m · capital-allocators
Gary Klein with Paul Sonkin and Paul Johnson – Conducting Pre-Mortem Analysis (Capital Allocators, EP.109)
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Cognitive psychologist Dr. Gary Klein joins investment authors Paul Sonkin and Paul Johnson to explain how naturalistic decision-making, tacit expertise, and properly executed pre-mortem analyses help investment teams overcome overconfidence and uncover critical risks.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Ted holds 22.9% of the talking time here. How this is scored →
Ted as informed peer 3.7 Guest teaching 6.0 Guest disagreement 1.2 Ted pushing back 1.0
speaking balance: gold is Ted, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 41:52 Dismissal of Devil's Advocates
Gary Klein forcefully rejects the conventional Devil's Advocate method, citing empirical evidence that it produces worse outcomes by outsourcing criticism and causing teams to discount genuine concerns.
Hardest push from Ted ▶ 39:31 Host questions what happens to overlooked risksTed presses the panel on a practical flaw in the framework: if a group brainstorms 27 failure points, how do leaders ensure the true fatal flaw isn't hidden in the unaddressed remainder.
Biggest teaching moment ▶ 10:45 Explaining the Recognition-Primed Decision ModelGary educates listeners and the host on how classical decision theory failed because it studied college sophomores in labs rather than true domain experts who rely on intuitive recognition and mental simulation.
Ted holds their own ▶ 9:04 Ted introduces Gawande's Checklist ManifestoTed demonstrates his own subject-matter knowledge by citing Atul Gawande's work on checklists to probe the exact boundary between structured procedure and tacit intuition.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Ted as informed peer | Guest teaching | Guest disagreement | Ted pushing back | Why |
|---|---|---|---|---|---|---|
| Gary Klein's Background: Air Force Flight Simulators and Tacit Knowledge | 5 | 6 | 1 | 2 | Ted probes Gary's origins in cognitive psychology and actively brings up Atul Gawande's Checklist Manifesto to ask how intuition blends with procedural rules. Gary clarifies that complex, ambiguous environments require tacit knowledge rather than mere checklists. | |
| The Recognition-Primed Decision (RPD) Model | 3 | 7 | 2 | 1 | Gary explains his Recognition-Primed Decision (RPD) model, contrasting real-world expert decision-making with traditional academic models tested on college sophomores. Ted asks open-ended clarifying questions about its effectiveness across domains. | |
| Practical Decision Tools: ShadowBox and Cognitive After-Action Reviews | 3 | 6 | 0 | 0 | Gary and Paul Sonkin explain practical training techniques like ShadowBox and Cognitive After-Action Reviews. The guests dominate the explanation while Ted acts as a curious facilitator asking where these frameworks get applied. | |
| The Origin and Mechanics of the Pre-Mortem | 4 | 7 | 1 | 1 | Gary lays out the origin story and strict step-by-step mechanics of the premortem method. Ted drills into the specific design choices, asking about the significance of the two-minute timer and having the team leader speak first. | |
| Common Pitfalls, Prospective Hindsight, and Psychological Safety | 4 | 7 | 2 | 1 | Gary, Paul Johnson, and Paul Sonkin describe why asking 'what could go wrong' ruins a premortem compared to framing failure as a certainty using prospective hindsight. Ted asks about fostering psychological safety and drawing out introverted team members. | |
| Sponsor Message: Ridgeline Investment Management Platform | 5 | 6 | 1 | 3 | Following the sponsor break, Ted challenges the process by asking what happens if the critical risks were overlooked among the low-priority ideas. The guests respond with the garden path study, cognitive diversity, and post-premortem replanning. | |
| Evaluating Alternative Risk Methods: Devil's Advocates, Red Teams, and Risk Lists | 4 | 7 | 3 | 1 | Gary and the Pauls critique popular alternatives like Devil's Advocates and generic risk matrices, pointing out research showing Devil's Advocates often backfire. Ted asks how red teams operate in contrast. | |
| The Core Value of Pre-Mortems: Reducing Overconfidence and Culture Shift | 3 | 6 | 1 | 0 | Gary and Paul Johnson outline the ultimate benefits of premortems: reducing overconfidence and shifting company culture toward psychological safety and candor. Ted facilitates smoothly without contest. | |
| Distributing the Pre-Mortem Paper and 'Pitch the Perfect Investment' | 2 | 2 | 0 | 0 | A collaborative wrap-up discussing the distribution of their co-authored paper, Columbia Business School classes, reception of 'Pitch the Perfect Investment', and concluding personal questions for Gary. |
Statements from this episode (15)
Klein: True expertise relies on tacit knowledge, not explicit rules
“For complex tasks and ambiguous situations with lots of uncertainty, there's no set of rules that are sufficient, and an expertise is It's not about just incorporating all the rules. Expertise is about building tacit knowledge, and that's the core.”
Klein: Effective simulators require rigorous training scenarios, not visual fidelity
“It doesn't have to look like the real thing, and you don't have to waste a lot of money on that. You have to have really good training scenarios that put people through their paces.”
Klein: Experts rely on situational recognition, not comparing options
“And the discovery we made is that people don't Generate lots of options and use lots of standard evaluation dimensions. They use their experience to size up situations very quickly and to identify what's going on, what's likely to happen next, what can I accom…”
Klein: After-action reviews fail by ignoring the decision-maker's cognition
“And frequently the after action review is you did X and you should have done Y. If you had done Y, things would have turned out better. But it doesn't get into the cognitive part. When you chose X, what were you seeing? What were you interpreting? In hindsight…”
Klein: Cap pre-mortem risk writing at exactly two minutes
“When we did this for a number of projects in my company, one minute was leaving people frustrated, because people were still writing when they said time is up. When we tried three minutes, a lot of people would be finished, and they'd be twiddling their thumbs…”
Klein: Pre-mortem leaders must share a serious risk first to set candor
“And the reason for that is people don't know how candid to be, and the leader is going to set an example. And if the leader picks something that's frivolous or trivial, that tells people not to take it seriously. So it's really important for the leader to come…”
Klein: Pre-mortems fail unless facilitators assert the plan has definitively failed
“We see premortems fail when the person who's facilitating will say something like, okay, we've seen the plan, let's do a premortem, what could go wrong with the plan? That's the wrong approach. What can go wrong with the plan? That's a typical critiquing appro…”
Klein: Pre-mortems succeed by turning risk identification into a competition
“On the other hand, with a pre-mortem, The contest is to see who can come up with the most clever and worrisome issues that haven't been discussed at the time, so the whole incentive system changes, and now people, people are competing with each other to come u…”
Klein: Teams in a garden path study failed to surface collective doubts
“And we once did a study where we had four-person teams going through an exercise, and it's called a garden path exercise, where we introduced something at the beginning that made it seem that certain thing was going to happen, and everybody was preparing for i…”
Johnson: A 27-person pre-mortem uncovered four path-critical risks I missed
“I ran a pre-mortem a year ago on a project we were going to launch. 27 people in the room. Wrote 27 things down on the first pass. Five of them were path critical. Of those five, I'd only thought of one.”
Klein: Research shows using a devil's advocate produces significantly worse outcomes
“I'm only aware of one study that tried to evaluate the effect of a devil's advocate to see if it had an effect, and they got a significant result. It was significantly worse than not using a devil's advocate.”
Johnson: External red teams are too time-prohibitive for daily investment decisions
“And the problem with red teams, particularly in the investment businesses, since we're making decisions almost daily or certainly fairly frequently, having a red team coming in and prepping them is probably prohibitive in terms of time and energy.”
Klein: Pre-mortems must be capped at 20 to 30 minutes
“Premortem should take no more than a half hour, Maybe 20 minutes. If it goes for an hour, that's too long. So there's a virtue of keeping it really brief, because that compresses the energy rather than dissipates it.”
Klein: Pre-mortems reduce overconfidence significantly more than listing pros and cons
“Listing pros and cons are just cons. Reduced confidence, but the premortem reduced confidence more than any of the other techniques we measured.”
Klein: Meet others' apparent mistakes with curiosity, not criticism
“When I see people make mistakes, instead of immediately becoming critical and impatient, to become curious. Why would somebody smart do something that looks stupid? Maybe there's a reason. Maybe I need to try to investigate further.”