Muck and Mystery
   Loitering With Intent
blog - at -
December 26, 2005

I've been wondering how those who were confident of their expertise would respond to the recent discussion of Philip Tetlock's new book - Expert Political Judgment: How Good Is It? How Can We Know? - mentioned in Science Class. Bryan Caplan's response:

Is my confidence in experts completely misplaced? I think not. Tetlock's sample suffers from severe selection bias. He deliberately asked relatively difficult and controversial questions. As his methodological appendix explains, questions had to "Pass the 'don't bother me too often with dumb questions' test." Dumb according to who? The implicit answer is "Dumb according to the typical expert in the field." What Tetlock really shows is that experts are overconfident if you exclude the questions where they have reached a solid consensus.

This is still an important finding. Experts really do make overconfident predictions about controversial questions. We have to stop doing that! However, this does not show that experts are overconfident about their core findings.

It's particularly important to make this distinction because Tetlock's work is so good that a lot of crackpots will want to highjack it: "Experts are scarcely better than chimps, so why not give intelligent design and protectionism equal time?" But what Tetlock really shows is that experts can raise their credibility if they stop overreaching.

I think this is wrong. As discussed in this Louis Menand review Tetlock's finding is:
. . .just one of more than a hundred studies that have pitted experts against statistical or actuarial formulas, and in almost all of those studies the people either do no better than the formulas or do worse.

There are also many studies showing that expertise and experience do not make someone a better reader of the evidence. In one, data from a test used to diagnose brain damage were given to a group of clinical psychologists and their secretaries. The psychologists’ diagnoses were no better than the secretaries’.

Randall Parker at FuturePundit had some interesting points to make on this a couple of years ago.
J.D. Trout & Michael Bishop, writing in an essay entitled "50 Years of Successful Predictive Modeling Should be Enough: Lessons for Philosophy of Science" argue that we continue to rely too much on the individual judgements of experts to make important decisions on subject matters where automated computer implementation of Statistical Prediction Rules would yield more accurate results.
In 1954, Paul Meehl wrote a classic book entitled, Clinical Versus Statistical Prediction: A Theoretical Analysis and Review of the Literature. Meehl asked a simple question: Are the predictions of human experts more reliable than the predictions of actuarial models? To be a fair comparison, both the experts and the models had to make their predictions on the basis of the same evidence (i.e., the same cues). Meehl reported on 20 such experiments. Since 1954, every non-ambiguous study that has compared the reliability of clinical and actuarial predictions (i.e., Statistical Prediction Rules, or SPRs) has supported Meehl’s conclusion. So robust is this finding that we might call it The Golden Rule of Predictive Modeling: When based on the same evidence, the predictions of SPRs are more reliable than the predictions of human experts. . .

SPRs will be right more often than human experts. But their (over)confidence in subjective powers of reflection leads them to deny that we should believe the SPR’s prediction in some particular case.

Training of large numbers of experts by universities has probably had the perverse effect of increasing the number of people running around making highly confident but wrong judgements. But the tendency to not notice our errors and to place excessive confidence in our subjective judgements is something that all humans suffer from to varying degrees.
To repeat, Caplan concludes that "what Tetlock really shows is that experts can raise their credibility if they stop overreaching." I think Tetlock shows that experts need help and should just get over their hang ups about credibility. It has been over 50 years since Meehl showed that experts really aren't reliable interpreters of evidence, a finding that has been repeatedly confirmed. The reason is simple and human, the arguably necessary confidence in subjective powers of reflection that promises reward for effort. Without that confidence there is less motivation to make the effort. Why bother if you're likely to be wrong. . . again?

It's the lone beaver syndrome, the heroic expert single handedly tackling tough problems. A more mature, and seemingly more reliable approach, would be to use the tools available and get a second and third opinion. Ask the janitor or the gardener too just for a reality check. Also see Tag Teams which discusses Groups of diverse problem solvers can outperform groups of high-ability problem solvers.

Posted by back40 at 11:48 PM | Tools

TrackBack URL for Overconfidence -