The post Overconfidence discussed and disputed Bryan Caplans reactions to Philip Tetlock's new book - Expert Political Judgment: How Good Is It? How Can We Know?, and drew heavily on J.D. Trout & Michael Bishop, especially their essay 50 Years of Successful Predictive Modeling Should be Enough: Lessons for Philosophy of Science.

Caplan's post has some interesting back and forth in the follow on comments, and he has posted again on another finding of the book. In Mind Wide Open he discusses Tetlock's findings about scenario consultants.

Tetlock tried out this technique on his political experts. He made them imagine all sorts of possible scenarios about (a) Canadian disintegration, and (b) the Japanese economy. His findings:

1. Before the exercise, people's probabilities at least added to about 100%. The exercise led participants to violate this elementary logical principle: after thinking about different scenarios, every outcome seemed more likely. For the Canadian problem, the average expert gave all the possibilities a 158% chance of happening. The error, in essence, was that merely imaginining possibilities, however outlandish, made people take them seriously. As Tetlock explains:

One takes a vague abstraction, all possible paths to Canada's disintegration, and explores increasingly specific contingencies. Quebec secedes and the rest of Canada fragments: the Maritimes - geographically isolated - clings to Ontario, but Alberta flirts with the United States... [T]he images are vivid, the plotlines plausible, and it becomes increasingly taxing to keep all the other logical possibilities in focus.

2. The exercise led people to raise their probabilities for events that didn't happen. In other words, more openness led to less accuracy.

3. The people most subject to this error were foxes who specialized in the topic at hand! Hedgehogs' "dogmatic" scoffing at odd hypotheticals saved them from a serious intellectual embarassment.

Score one for Caplan, who is an adamant though not dogmatic hedgehog much of the time, worth our attention precisely because he knows his own mind and so isn't as vulnerable to fanciful enthusiasms as those who are more imaginative and credulous.

More importantly, we should think deeply about Tetlock's findings since the scenario hustle is big business these days, heavily and deceptively marketed to both businesses and governments, often for instrumental reasons. Apparently it works on Tetlock's "foxes", confusing them thoroughly and in effect selling them on policies and processes they would otherwise reject, though there is little or no basis and is obviously fanciful to "hedgehogs".

It's another way to do politics in other words, which seems obvious in retrospect since scenario hustlers are often (usually?) massively politicized activists. My main complaint about scenarios has always been that the ones developed and considered seemed nonsensical, and so the whole exercise was a waste. It is clear now that this is intentional, not just incompetence, since the objective is to sell nonsensical views to gullible "foxes" who can be befuddled by their own active imaginations.

When scenario hustlers come around be sure to wear a clump of garlic on a string around your neck, and bring along your pit hedgehog because he can smell the grift and is immune to the mind drugs.