“It is better to fail conventionally than to succeed unconventionally.” – Keynes.

"It is easy in the world to live after the world's opinion; it is easy in solitude to live after our own; but the great man is he who in the midst of the crowd keeps with perfect sweetness the independence of solitude."

Self-Reliance - Ralph Waldo Emerson

Keynes was wrong in this as in so much else. Though it may be better for an individual to go along and get along, even if it means failure, the group does poorly. The problems noted in the earlier post Unanimous Fallacies that resulted in social collapse, and the problems noted in Situation Normal that lead to bad governance, result from this very human but very mistaken fallacy.

In addition to the mistakes individuals make when they fail to heed Emerson's advice and choose the more cowardly Keynesian approach instead, groups composed of such degraded individuals seek to dominate others, as noted in Agree To Disagree. Those posts reference other posts, each dealing with some aspect of this large and multi-faceted issue, and in a few places reference James Surowiecki's The Wisdom Of Crowds: Why The Many Are Smarter Than The Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations. That book has been misunderstood by many it seems, confused by the use of the word "collective" which has so many associations that are antithetical to Surowiecki's point. He's speaking out about this, as noted in this transcript like set of notes taken at the O'Reilly Emerging Technologies conference in San Diego of a talk given by Surowiecki called Independent Individuals and Wise Crowds, or Is It Possible to Be Too Connected? [via Travels With Rhody]

Selected grafs from that transcribed talk:

Not all forms of collective action are created equal. If you use the wrong kind of collective action, you can end up with worse problems than the ones you set out to solve. There’s been a lot of fuzzy thinking about what we mean when we talk about collective intelligence, network, and interaction. I want to parse these distinctions.

In The Wisdom of Crowds, I wrote about the power of groups under certain circumstances to be remarkably intelligent. A model of collective intelligence: a large group of people reflecting diverse opinions offering judgments independently with some mechanism to aggregate the judgments, collectively ending up with an intelligent outcome. . .

The wisdom of crowds works well when there is a true answer, and as long as some choices are better than others. The key is that people are mostly working on their private information, which may not be good, may be fragmented, but it is diverse. Collective wisdom does not emerge out of consensus. The goal is not to get everyone to agree – it’s to tap into people who disagree, into the diverse information everybody has. It works best when people are not paying too much attention to what everyone else is doing. They have some sense – like feedback in the form of odds at the racetrack – but there isn’t a lot of personal interaction. . .

Human beings are not ants. We don’t have the biological programming or tools that ants have. The way ants find food has to do with their formic acid secretions; the more trails the more signals; the entire colony can find its way to the food source. We have no equivalent to this. For us, interaction is incredibly problematic, especially when it comes to group behavior. If there is too much interaction among human beings, groups end up being less intelligent than they would otherwise be. The more we talk to each other the dumber it is possible for us to become. The book has quite a bit about small groups. Put a bunch of smart people into a room and they emerge dumber than when they went in. . . The question for all of us is, how can you have interaction without information cascades, without losing the independence that’s such a key factor in group intelligence? I’m not going to come to a final answer. But there are a few things worth thinking about. First one: the best thing to do is to keep your ties loose. You’re better off, and the group is better off, if the ties are looser, because loose ties minimize the influence of those around you. I don’t think Duncan Watts’ model of the information cascade is quite true. I don’t think people are as subject to the influences around them as Duncan thinks. But we are clearly shaped by those influences. One way around that: limit the power of the influences.

Second, keep yourself exposed to as much information as possible. Injecting some level of randomness into the system is a good thing. Diversity is a good thing. In computer science experiments at the University of Michigan, a researcher, Scott Page, had his agents compete until they differentiated into three groups, Dumb, Intelligent, and Random. Then he had them solve problem as groups. The Intelligent group outperforms the Dumb group, but not by very much. But the Random group almost always outperforms the Intelligent group. Page’s theory is that the reason for this is that even if the less intelligent groups know less, what they know is different.

This has important implications for the way decision making works inside organizations. Make groups that range across hierarchies. The conclusion is that you actually can be too connected, if the connections are of the wrong kind and if they’re reinforcing your existing prejudices rather than altering them. You can pay to much attention to those around you, even if they’re really smart. The flip side of Pascal’s isolation is the cacophony you find on the net; it bombards you with many voices. Isolation and cacophony, interestingly, allow you to arrive at the same place: independence.

Good stuff. Many are either afraid of, or angered by, the freedom and anarchy of the internet and want to control, civilize, regulate and organize it. Many want to use it as a tool to increase herd effects and information cascades. Their fears and anger can be understood since so many of their unexamined certainties and unreasoning faiths are challenged by the internet, but it would be very bad for the world if they are allowed to succeed. Speaking against such people and such ideas is a kind of social duty once you grasp the implications.

Update:

Cosma points out that the quote from above, attributed to Keynes - "It is better to fail conventionally than to succeed unconventionally" - was not complete or intended by Keynes as a recommendation for the behavior. When the full sentence as written is considered this is a bit clearer - "Worldly wisdom teaches that it is better for reputation to fail conventionally than to succeed unconventionally". This is often, sadly, true. And for many reputation is more important than actual success. But it wasn't so for Keynes himself or the intellectual circles he inhabited where members consciously cultivated their own eccentricities and valued success far above reputation.

Cosma also alerted me that there are some interesting papers in the pipe that deal with aspects of the issues discussed above. I can't wait, though I must.

Update:

See Heuristic Diversity, Your Key to Knowledge, Wealth and Power (Dept. of "Yay Team!") where it is revealed that Groups of diverse problem solvers can outperform groups of high-ability problem solvers, and that there is more to come.

» Tough Calls from Crumb Trail
A series of posts at M&M grapple with the scope, scale and methods of decision making appropriate and effective for current issues. See Disordered by Design, Independent Cacophony and Shapeless Futures if you have some time to kill. Follow......[read more]
Tracked: March 30, 2005 01:33 PM
» GIGO from Crumb Trail
A post over at M&M used a quote attributed to Keynes that came from a talk given by James Suroweicki at the O'Reilly Emerging Technologies conference. The quote - "It is better to fail conventionally than to succeed unconventionally"......[read more]
Tracked: March 30, 2005 07:26 PM