Dec 8, 2015 · 45m · knowledge-project
#6 Philip Tetlock: How to See the Future
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The Knowledge Project, host Shane Parrish interviews Philip Tetlock to explore the empirical science of superforecasting and probabilistic judgment. Tetlock outlines the cognitive habits, debiasing methods, mathematical aggregation algorithms, and organizational strategies that allow individuals and teams to dramatically improve predictive accuracy in uncertain environments.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Shane holds 16.6% of the talking time here. How this is scored →
Shane as informed peer 3.4 Guest teaching 2.6 Guest disagreement 0.8 Shane pushing back 1.0
speaking balance: gold is Shane, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 17:35 Tetlock refutes David Leonhardt's binary prediction market critique
Tetlock sharply rejects the simplistic conclusion that a 75 percent probability estimate on a Supreme Court ruling was 'wrong' merely because the minority outcome occurred.
Hardest push from Shane ▶ 11:28 Parrish presses Tetlock on Kahneman's bias pessimismParrish questions the efficacy of a 50-minute debiasing module by pointing out that Daniel Kahneman himself claims a lifetime of studying cognitive biases did not make him better at avoiding them.
Biggest teaching moment ▶ 36:23 Tetlock nuances short versus long horizon predictabilityTetlock educates Parrish on why shorter timeframes are not universally easier to predict, using daily market noise versus ten-year market trends as a clear counterexample.
Shane holds their own ▶ 11:28 Parrish leverages behavioral economics literature against debiasing claimsParrish demonstrates subject mastery by immediately confronting Tetlock's debiasing claims with Kahneman's well-documented stance on cognitive bias permanence.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Shane as informed peer | Guest teaching | Guest disagreement | Shane pushing back | Why |
|---|---|---|---|---|---|---|
| Origins of the Good Judgment Project and Superforecasting | 3 | 2 | 1 | 1 | Parrish introduces Tetlock's book and asks foundational questions about the Good Judgment Project and the distinction between forecasting and predicting. Tetlock provides background on the IARPA tournament setup. | |
| Cognitive Attributes and Granularity of Elite Forecasters | 2 | 3 | 1 | 0 | Parrish asks why certain individuals excel at forecasting. Tetlock dismantles the pure luck hypothesis with statistical evidence on regression to the mean and explains granular probability estimation. | |
| Debiasing Techniques and the Outside View Advantage | 5 | 3 | 1 | 4 | Parrish challenges Tetlock by citing Daniel Kahneman's public pessimism about personal debiasing despite decades of study. Tetlock concedes Kahneman's skepticism while standing behind the 10 percent empirical improvement. | |
| Status Hierarchies, Vague Verbiage, and Accountability Resistance | 4 | 3 | 2 | 1 | Parrish and Tetlock examine why institutional status hierarchies and pundits rely on vague verbiage to evade accountability. Tetlock critiques media figures like Tom Friedman and David Leonhardt for misrepresenting probability. | |
| Fermi-Style Problem Decomposition and Managing Analytical Ignorance | 4 | 2 | 0 | 0 | Parrish connects Fermi estimation to corporate tech interview techniques and synthesizes the value of transparent ignorance mapping. Tetlock illustrates the technique using Drake equation style galactic estimates. | |
| Implementing Forecasting Tournaments within Organizational Structures | 3 | 3 | 1 | 1 | Parrish asks how to deploy elite forecasting teams inside standard corporate environments like IBM. Tetlock warns that transplanting tournament elitism directly into corporate cultures risks serious political friction. | |
| Mathematical Aggregation Algorithms and the Mechanics of Extremizing | 3 | 3 | 1 | 1 | Tetlock explains the mathematics behind crowd aggregation and the algorithmic technique of extremizing diverse independent estimates, using the Bin Laden raid scenario as an illustration. | |
| Time Horizons, Roulette Problems, and Prediction Tractability | 4 | 3 | 1 | 2 | Parrish inquires about tractable question types and time horizons. Tetlock reframes common assumptions by showing how 10-year stock forecasts can be more tractable than next-day stock predictions. | |
| Bayesian Belief Updating and Analyzing Forecasting Successes | 3 | 2 | 0 | 0 | Parrish asks about fostering open-mindedness and team learning dynamics. Tetlock explains Bayesian updating limits when ideological priors exist and highlights how superforecasters conduct post-mortems on their successes. | |
| Intuitive Thinking versus Deliberate Analysis in History | 3 | 2 | 0 | 0 | Parrish raises the debate between intuitive and deliberate thinking. Tetlock emphasizes deliberate analysis over raw intuition for geopolitical forecasting and reflects on influential books. |
Statements from this episode (18)
Tetlock: Making explicit probabilistic judgments improves forecasting accuracy
“And I think one of the major takeaways from the forecasting tournaments we've been running is that when people make explicit judgments and they're fully self-conscious about what they're doing, they can learn to do it better.”
Tetlock: Good Judgment Project won IARPA's 2011–2015 forecasting tournaments
“The tournaments ran from 2011 to 2015. They ended in June of this year. And the Good Judgment Project I am proud to say, was the winner of those forecasting tournaments.”
Tetlock: Geopolitical forecasting relies on 70% skill and 30% luck
“Well, what we find in the ARPA tournament is, is that there certainly is an element of chance in predicting geopolitical and geoeconomic outcomes but the skill luck ratio seems to be about 70 30.”
Tetlock: Superforecasters uniquely believe probability estimation is a cultivatable skill
“They tend to score higher on measures of fluid intelligence. They tend to score higher on measures of active open-mindedness. But if I had to identify one factor that I think best distinguishes super forecasters from other forecasters who are equally intellige…”
Tetlock: 50 minutes of debiasing training improves forecasting accuracy by 10%
“We got about, for average forecasters who are randomly assigned to an experimental condition in which they get Kahneman style debiasing exercises the improvement is in the vicinity of 10%. And that's a big effect when you consider that, you know, we're talking…”
Tetlock: Effective forecasters anchor on statistical base rates before adjusting
“Start with the outside and work inside. That, that, that's a, that's, it's one of our mantras.”
Tetlock: Vague terms like 'could' allow pundits to evade accountability
“When you say something could or might happen, that could mean anything from .1 to low .9 in probability terms. And, you know, if it happens, I can say, well, I told you it could. And if it doesn't happen, I can say look, I merely said it could.”
Tetlock: Executives resist forecasting tournaments to protect their high-status judgment
“Even though I think we have shown that forecasting tournaments can appreciably improve probability estimates, there are a lot of reasons why organizations don't adopt them. One is the people at the top of the status hierarchy. They're not very enthusiastic. Bo…”
Tetlock: Prediction markets have proven well-calibrated across hundreds of forecasts
“Prediction markets have generated hundreds of forecasts over many years, and they've proven to be pretty darn well calibrated, which is another way of saying when they say 75% probability of something happening, things happen about 75% of the time. And they do…”
Tetlock: Superforecasters break intractable problems into exposed subcomponent estimates
“And that's what super forecasters are pretty good at doing. Breaking down seemingly attractable problems into semi-attractable components and then just pushing. They're not afraid of looking stupid and making estimates that observers can see and look at and sa…”
Tetlock: Corporate forecasting compromises accuracy with vague verbiage to avoid embarrassment
“Well, I think it's something you want to consider seriously that when people make forecasts inside organizations, most organizations today, accuracy is only one of the goals that they're pursuing. They're also interested in making forecasts that are going to b…”
Tetlock: Crowd average forecasts beat most individual forecasters
“The average of a group of forecasters, the average forecast from those forecasters is going to be more accurate than most of the individuals from whom the average was derived.”
Tetlock: Weighted averages outperform unweighted crowd forecasts
“Weighted averages beat the average.”
Tetlock: Algorithmically extremizing independent forecasts produces more accurate predictions
“Each of you has very different reasons for believing .7. This leaves me to suppose that the answer is probably more extreme than .7, because if each of you knew the reasons the others had, you would probably become more extreme. And that's exactly what the bes…”
Tetlock: Modeling randomness is a good way to fail as a forecaster
“If you want to be a good forecaster, you don't spend very much time working on roulette wheel type problems. I mean, if, I mean, if you go to, if you visit casinos, you'll find lots of people who think they can detect patterns in roulette wheel spins. And they…”
Tetlock: Predicting 10-year stock market direction is easier than tomorrow's
“It's very hard to say whether the stock market's going to up or, going to go up or down tomorrow. So that's a short range question. In some ways, it's easier to predict where the stock market's going to be up or down 10 years from now relative to now that it i…”
Tetlock: Belief updating works without strong priors, but fails with ideological convictions
“I think we can make people better belief updaters on problems where they don't have very strong ideological priors or preconceptions. But when people have really strong emotions and ideological convictions about presidential candidates or economic policy or wh…”
Tetlock: Superforecasters actively second-guess their successes rather than just taking credit
“And supers do that too. But they also second guess their successes. They say, well, were we lucky just where we really nailed this question, but were we lucky? Could it have gone otherwise? Were we almost wrong? That's an unusual question for people to ask the…”