Superforecasting: The Art and Science of Prediction

By Philip E Tetlock, Dan Gardner (2015)

… the more famous an expert was, the less accurate he was. That’s not because editors, producers, and the public go looking for bad forecasters. They go looking for hedgehogs, who just happen to be bad forecasters.

Foxes don’t fare so well in the media. They’re less confident, less likely to say something is “certain” or “impossible,” and are likelier to settle on shades of “maybe.” And their stories are complex, full of “howevers” and “on the other hands,” because they look at problems one way, then another, and another. This aggregation of many perspectives is bad TV.

My Notes

• Paul Meehl: simple algorithms performed better than experts

Meehl’s claim upset many experts, but subsequent research—now more than two hundred studies—has shown that in most cases statistical algorithms beat subjective judgment, and in the handful of studies where they don’t, they usually tie.

• Numbers represent estimates only
• People don’t like uncertainty: hedgehogs calm nerves
• Consulting other perspectives is hard work
• Inverse correlation between fame and accuracy

Hedgehogs 🦔

• Intuitive
• Overconfident
• Focus on big ideas

… hedgehogs tell tight, simple, clear stories that grab and hold audiences.

Foxes 🦊

• Pragmatic
• Use many tools
• Gather a lot of information
• Use probabilities and use “however,” “although” and “on the other hand” etc

Wisdom of the crowds:

• Independent thought is maintained (no collaboration)
• Errors cancel each other out

Groupthink:

• Independent thought/opinions/knowledge are lost through collaboration
• Errors are amplified

Beyond the forecasting horizon, plan for surprise

Obviously, a forecast without a time frame is absurd.

More or less:

• Individual forecasters
• Wisdom of crowds, 15% better
• Teams of regular individuals, 10% better
• Prediction markets, 20% better
• Super teams, 15-30% better

Superforecasting:

• You need continual quick feedback
• You need the team to feel safe and be open to criticism, “disagree without being disagreeable,” thanking constructive criticism, admit ignorance
• Growth mindset
• Perpetual beta is the strongest predictor. “Progressive improvement is attainable. Perfection is not.”
• Good with numbers
• Humility in the face of the game, not opponents

Tips:

• Unpack the question into components
• Distinguish the known (and knowable) from the unknown (and unknowable)
• Start with an objective base-rate: downplay its uniqueness
• Then look at the uniqueness of this problem
• Explore the views of others
• Synthesise and express as precisely as you can
• Update your forecasts with new information (careful not to over- or under-react)
• In teams, diversity of knowledge is helpful (to be able to combine different perspectives and be more certain about final forecasts—in teams where everyone shares everything though, maybe a team of superforecasters, this won’t be the case)