Richard Hamming: on how to do great research. [via Structure+Strangeness]

Great scientists tolerate ambiguity very well. They believe the theory enough to go ahead; they doubt it enough to notice the errors and faults so they can step forward and create the new replacement theory. If you believe too much you'll never notice the flaws; if you doubt too much you won't get started. It requires a lovely balance. But most great scientists are well aware of why their theories are true and they are also well aware of some slight misfits which don't quite fit and they don't forget it.

But, great scientists are very rare. It seems to me that great thinkers of all sorts share these mental habits, and that they are also very rare. Part of the problem may be that there are often great rewards for being "sooners", for jumping the starting gun and making premature commitment to dodgy theses. Becoming a believer and advocate for a body of thought can be profitable: it can attract grant money, sell books, lead to fame and get you elected. But it's hard to live long and continuously prosper when riding such tigers, as many have discovered. If you choose to be a tiger wrangler it is well to also be skilled in reinvention, to have a bit of Phoenix in you so that you can reemerge after crashing and burning.

Prometheus noticed a few current examples in the op-ed pages on Easter Sunday, which seems an appropriate day for such stories. Francis Fukuyama, who has been called the Paul Ehrlich of politics since he has been so frequently and publicly wrong, writes about the perils of public intellectualism. What if you change your mind?

. . . apparently this kind of honest acknowledgment is verboten. In the weeks since my book came out, I've been challenged, attacked and vilified from both ends of the ideological spectrum. . .

Political debate has become a spectator sport in which people root for their team and cheer when it scores points, without asking whether they chose the right side. Instead of trying to defend sharply polarized positions taken more than three years ago, it would be far better if people could actually take aboard new information and think about how their earlier commitments, honestly undertaken, actually jibe with reality — even if this does on occasion require changing your mind.

No, it isn't a crime to change your mind, but when you repeatedly do so after having hyped a view you get a reputation for being a wanker, and so each new view is greeted with derision as the next instance of your defective intelligence, an opportunity for someone else to make a name for themselves. They assume that you are wrong, since you are always wrong, and flail away at you with abandon in full confidence that they are correct at least in part and will not face similar humiliation.

Prometheus took it in a different direction after having read another article on confirmation bias.

Much of what happens in the brain is not evident to the brain itself, and thus people are better at playing these sorts of tricks on themselves than at catching themselves in the act. People realize that humans deceive themselves, of course, but they don't seem to realize that they too are human. . .

A Princeton University research team asked people to estimate how susceptible they and "the average person" were to a long list of judgmental biases; the majority of people claimed to be less biased than the majority of people. A 2001 study of medical residents found that 84 percent thought that their colleagues were influenced by gifts from pharmaceutical companies, but only 16 percent thought that they were similarly influenced. Dozens of studies have shown that when people try to overcome their judgmental biases — for example, when they are given information and told not to let it influence their judgment — they simply can't comply, even when money is at stake.

And yet, if decision-makers are more biased than they realize, they are less biased than the rest of us suspect. Research shows that while people underestimate the influence of self-interest on their own judgments and decisions, they overestimate its influence on others. . .

In short, I'm O.K., You're Biased. Not.

The significance of these two articles for Prometheus is that the substance of Fukuyama's new views will be variably interpreted depending on the bias of the reader, and used as a blunt instrument to bludgeon opponents for their biases and unwillingness to reason in good faith from evidence. He cites two more articles, one by Greenpeace co-founder Patricke Moore about nuclear energy, and another about the recent kerfuffle with hybrid vehicles and their failure to deliver imagined savings, and speculates that the substance of these views will be subjected to much twirling of the cognitive kaleidoscope.

Each is fairly nuanced and raises complicated points, which, if you agree with Fukuyama and Gilbert, may be more likely to be spun as wedge devices in ideological battles among people whose views are hardened irrespective of data or argument, rather than considered on their intellectual merits. For my part, I do think that argumentation matters and that many people are open to new information, analysis, and the related evolution of their thinking on policy issues. But this probably does not fully extend to many of the loudest, most certain, and strident commentators that sometimes seem to dominate public debates.

This is a subject dealt with so many times here that I've taken to using the turn of phrase "cognitive kaleidoscope" as if everyone knows what it refers to. You may not if you haven't been following the extended discussion, but you can get the gist of it just from the words. But I think that Prometheus has missed the mark a bit by not giving sufficient weight to the fact that the apostates he cites weren't careful thinkers who formed pragmatic and provisional conclusions based on the evidence available, and then altered those conclusions in response to new and better evidence. They are dark-siders who took up arms to fight for a cause based on skimpy evidence floating in a lake of ideology. They are doubly defective in that the quality of their intelligence is questionable, and they launched crusades based on these defective analyses. They lacked the quality of great scientists for tolerating ambiguity, of being able to proceed and hold the mark while continuing to question even their own acts.

That's also a subject dealt with many times here. The Bernard DeVoto essay The Ex-Communists, though many years old, speaks to this issue.

Embracing communism, like religious conversion, is an act of the total personality. It is packed with private and even unconscious as well as rational and objective reasons, with emotion as well as intelligence. What the apostates have been saying shows that frequently intelligence played only a small part in it. Yet it played some part and they are eager to show that it was decisive in their apostasy, their repudiation of communism. I propose to discuss only their intelligence. We will agree that the American intellectual who became a Communist was, typically, a generous, warmhearted man, an idealist deeply disturbed by the catastrophe of the modern world and deeply concerned for the betterment of mankind. But how good was his thinking?

Moore's Greenpeace activities fit this pattern of emotion ascendant over intelligence as do Fukuyama's serial conversions. Earlier posts discussed former Earth First! activist Dan Dagget, now reformed and an advocate for human management of environments rather than human exclusion from them, and founder and chairman of the Revolutionary Communist Party (RCP) of Great Britain Frank Furedi, using the name Frank Richards, now considered to be on the libertarian right wing.

While tolerating ambiguity may be a necessary attribute of great scientists, or great thinkers in general, it isn't sufficient, it won't make a mediocre intellect great. But it may well help prevent the type of problems that people like Fukuyama (and Ehrlich) and Moore have had.

This isn't only self-help advice, it is relevant to the polarization that bothers Prometheus. There are many posts there that deal with the politicization of science by all sides to policy disputes, and persistent advocacy for being more direct, for openly disputing policy prescriptions without hiding behind and skewing the science involved in the attempt to favor one policy above another.

While true I think this is not the only or best lesson. There is too much politicization because there is too much policy. The mistake is in assuming that large scale problems always or often require large scale political policies enacted at high levels of government. Many problems are ill suited to that sort of approach which leaves partisans bickering about which bad policy to enact. There may be those who fail to grasp that all of the options are bad, but I suspect that this is obvious to many of those who lobby for and in the end define policies. They know they are making junk policy but it advances their political agenda and that is a higher good to them. The issue is merely a wedge and their advocacy is instrumental. There may also be those who simply can't see better alternatives since they can't grasp policies that are not initiated and conducted by controlling authorities.

There is always a role for every level of government in large scale problems but it isn't often leadership or control, and the higher the level of government the more this is true. In my view official histories often mislead in that they attribute events and their resolutions to some authority or great man when the truth is less certain, more of a standing on the backs of an army of anonymous practitioners than the shoulders of famous elder giants. Those who get credit and blame are usually undeserving of so great a portion. We would do well to focus more intently on this truth and become more adept at formulating policies that seek to enable and benefit from this type of boots-on-the-ground task management. Such policies would seek to enable problem solvers and remove impediments to emergent approaches rather than formulate detailed responses.

Prometheus may be right that ". . . argumentation matters and . . . many people are open to new information, analysis, and the related evolution of their thinking on policy issues", but it doesn't matter much except for political contests. Most of the effort expended on persuasion is wasted in the sense that it doesn't contribute much to problem solving. It's a pass time, entertainment for those charmed by politics, debate and pub talk. Cognition of any useful scope and scale is distributed, and in that sense policies are only recognizable in hindsight, and even then only dimly. We not only don't know what is happening, we don't know what happened, much less what will happen.

We are left with little more to guide our acts than homely virtues thought to be commonly useful such as honesty and mindful work. We have no choice but to trust each to do his bit. Wisdom mostly consists of avoiding behaviors that interfere with the efforts of our neighbors, so that they can do their best. We might each do well to emulate those great scientists and their tolerance for ambiguity too. "It requires a lovely balance."