Is in.

Risk Management Solutions (RMS) is a leading company that provides catastrophe models which are used to asses risk in the insurance and reinsurance industries. . .

A key part of RMS short-term prediction of hurricane activity is an expert elicitation. . .

The RMS elicitation has been controversial because it has resulted in a short-term prediction of activity higher than the climatological average, which means higher risk, and for coastal property owners the result is higher insurance rates. In 2006 for example the elicitation by the RMS panel of experts resulted in a prediction of 0.92 Category 3-4-5 storms making landfall every year for the period 2007 to 2011, which is considerably higher than the 1950-2006 average of 0.64. At the time, loss estimates increased by about 40%, with consequences for insurance and reinsurance rates. . .

I created a panel of 5 “monkeys” by allocating weights randomly across the 20 models for each of my participating monkeys. My panel of 5 monkeys came up with an estimated 0.91 storms per year for 2007 to 2011. I did this three more times, and these three different panels of “monkeys” projected 0.90, 0.93, 0.92 landfalling Category 3-4-5 storms per year. The RMS experts came up with 0.92.

In short, my panels of “monkeys” produced the exact same results as those produced by the RMS expert panels comprised of world leading scientists on hurricanes and climate. How can this be? The reason for this outcome is that the 20 models used in the expert elicitation (i.e., from Jewsen et al.) provide results that range from 0.63 on the low side to 1.21 at the high end, with an average of 0.90. Thus, if the elicitiors spread their weights across models the process will always result in values considerably higher than the historical average. The more spreading of weights and the more participants will necessarily mean that the results will gravitate to the average across the models. So with apologies to my colleagues, we seem to be of no greater intellectual value to RMS than a bunch of monkeys. Though we do have credentials, which monkeys do not. . .

What does this mean? It means that the RMS elicitation process is biased to give results above the historical average. Of course, it may in fact be the case (and probably is) that many of the elicited experts actually do believe that storm activity will be enhanced during this period. However the fact that the RMS elicitation methodology does not distinguish between the views of these experts and a bunch of monkeys should give some concern about the fidelity of the process.

It's not just GIGO (garbage in, garbage out), it's whatever comes in garbage comes out. It's a variant of the hockey stick methodology where the process always produces the desired result but is sufficiently opaque to casual examination that many accept the outputs as expert analysis and sophisticated calculation.

This isn't helpful. Why do we pay these people for such shoddy work?