The WSJ has printed one of the best "fooled by randomness" pieces I've seen in quite a while, titled "The Triumph of the Random." This one uses streaks in sports as a central metaphor, with DiMaggio's 56-game hitting streak as exhibit A. It presents an immediate disclaimer:
Recent academic studies have questioned whether DiMaggio’s streak is unambiguous evidence of a spurt of ability that exceeded his everyday talent, rather than an anomaly to be expected from some highly talented player, in some year, by chance, something like the occasional 150-yard drive in golf that culminates in a hole in one. No one is saying that talent doesn’t matter. They are just asking whether a similar streak would have happened sometime in the history of baseball even if each player hit with the unheroic and unmiraculous—but steady—ability of an emotionless robot.
The lengthy article then deals with the mathematics of streaks, demonstrating that they are far more probable than we would otherwise think:
A few years ago Bill Miller of the Legg Mason Value Trust Fund was the most celebrated fund manager on Wall Street because his fund outperformed the broad market for 15 years straight. It was a feat compared regularly to DiMaggio’s, but if all the comparable fund managers over the past 40 years had been doing nothing but flipping coins, the chances are 75% that one of them would have matched or exceeded Mr. Miller’s streak.
Next, it moves to psychology and describes the way in which humans seek patterns in randomness as a grounding mechanism with a nice segway by way of my favorites, Kahneman and Tversky, who authored a seminal paper on hot hands in basketball:
If a person tossing a coin weighted to land on heads 80% of the time produces a streak of 10 heads in a row, few people would see that as a sign of increased skill. Yet when an 80% free throw shooter in the NBA has that level of success people have a hard time accepting that it isn’t. The Cognitive Psychology paper, and the many that followed, showed that despite appearances, the “hot hand” is a mirage. Such hot and cold streaks are identical to those you would obtain from a properly weighted coin.
Finally, it deals with the perception of random events:
Why do people have a hard time accepting the slings and arrows of outrageous fortune? One reason is that we expect the outcomes of a process to reflect the underlying qualities of the process itself. For example, if an initiative has a 60% chance of success, we expect that six out of every 10 times such an initiative is undertaken, it will succeed. That, however, is false.
A critical conclusion is laid out:
We find false meaning in the patterns of randomness for good reason: we are animals built to do just that... Many studies illustrate how this basic aspect of human nature translates to a misperception of chance.
Truly an excellent read and I can't recommend it more.