Be Careful Which Crowd You Ask

James Surowiecki's book The Wisdom of Crowds is brilliant. The concept is that groups are often smarter than any individual in the group. He makes the point with amazing examples - my favorite is the case of the missing US submarine Scorpion in 1968. The sub disappeared off the eastern coast after a tour of duty. The navy knew its last reported location, but had no idea how far it travelled after last contact. The search area was impossibly large and a rescue mission seemingly hopeless. All the same, a naval officer named John Craven asked a group of mathematicians, submarine specialists and salvage experts to individually estimate why the sub came into trouble and what its speed and heading were at the time it did. Craven then combined the predictions from each expert and came up with a collective estimate of where the sub was. That collective estimate was, in fact, 220 yards off from the actual location. Now that's wisdom - and a powerful case of how many minds are better than one.

Which brings me to the idea of individually polling people to get a sense of where a market is going. I was just reading a back issue of Canadian Business that had a CEO Poll on rising gas prices (Cdn Bus, April 14). As of the poll date, the price of oil was $109 a barrel and the 133 Canadian CEOS who participated in the poll thought prices wouldn't rise above $113.

It's $135 now. Apparently CEOs are good at running companies, but don't know much about the market for oil.

There's an important point here: polls and surveys are only as good as the people who participate in them. When we're formulating research for clients, they often focus on the sample size - and yes, there needs to be a representative sample. But just as important is the ‘who' in the research. Make sure the people in your research are experts in the field, or the decision makers you're trying to reach. Otherwise your results are about as good as asking your barber what's going to happen with the exchange rate.