I’ve covered Benford’s method for first-digit fraud analysis before, and now Nate Silver has applied a similar method to polling results. He looked at the last digit of various polls (i.e. a 48% McCain, 49% Obama, 3% undecided poll would be recorded as an 8 and a 9) and compiled histograms of their frequencies. Following up on his suspicions that all was not right with one polling firm in particular, Nate noticed that their results did not conform to the expected random distribution:
This data is not random at all. For instance, the trailing digit was ’8′ on 676 occasions, almost 60 percent more often than the 431 times that it was ’1′. Over a sample of more than 5,000 data points, such an outcome occurring by chance alone would be an incredible fluke — millions to one against. Bad luck can essentially be ruled out as an explanation.
One of two things seems to have happened, then.
One possibility is that there is some intrinsic, mathematical reason that certain trailing digits are more likely to come up than others. This is certainly possible — and in fact, it would be somewhat likely if the polling data that we were looking at were homogeneous — McCain versus Obama polls in Ohio, for instance.
But Strategic Vision’s polls cover a wide array of topics: Presidential horse race numbers in any of a dozen or so states, senate and gubernatorial polling, primary polling, approval ratings of various kinds, polling on issues like the war in Iraq, and more abstract questions such as whether voters think that ‘experience’ or ‘change’ is the more important quality in a Presidential candidate. No one type of question, in no one state, represents more than a relatively small fraction of the sample. Under those circumstances, I can’t think of any reason why the trailing digit wouldn’t approach being random — although there absolutely might be reasons that I haven’t thought of.
But this data is not random. It’s not close to random. It’s not close to close. Which brings up the other possibility: Strategic Vision is cooking the books. And whoever is doing so is doing a pretty sloppy job. They’d seem to have a strong, unconscious preference for numbers ending in ’7′, for instance, as opposed to those ending in ’6′. They tend to go with round numbers that end in ’5′ or ’0′ slightly too often. And they much prefer numbers with high trailing digits like 49 and 38 to those with low ones like 51 and 42.
I haven’t really seen anyone approach polling data like this before, and I certainly haven’t done so myself. So, we cannot rule out the possibility that there is some mathematical rationale for this that I haven’t thought of. But it looks really, really bad. There is a substantial possibility — far from a certainty — that much of Strategic Vision’s polling over the past several years has been forged.
Is there a mathematical reason for such a discrepency in poll results? I can think of only one possibility – there is a weak dependence structure in the data. One last digit exerts some influence over the other. With no one undecided, the dependence is perfect: a 4 on one sample requires a 6 on the other (i.e. 44% and 56%; 34% and 66%). With a fixed level of undecided people, the dependence remains perfect. If the undecided level is stochastic, then the dependence becomes more weak. However, it’s unclear to me why this would skew the results in the manner this firm exhibits; this could require high numbers to be paired with low numbers, or high numbers to be paired with high numbers (or vice versa), but wouldn’t lead to more high numbers in and of itself.
I’m very curious (and not just as a statistician) to see what comes of this… as much as we’d like statistics to give us firm answers, often the limit of its ability is to reveal probable courses of investigation, or lend strong – but at some level uncertain – backing to an argument.
In response to my post on the WSJ’s recent randomness article, B emailed me the following (reproduced here with permission):
The quoted WSJ article writes
“We find false meaning in the patterns of randomness for good reason: we are animals built to do just that… Many studies illustrate how this basic aspect of human nature translates to a misperception of chance.”
This cannot be right. It cannot even be meaningfully wrong. Animals are not built; we are constantly re-encoded at every generation by a process that selects only for whatever helps the survival of offspring that will of course carry that novel encoded information. Over the past ~ 10 million years, this positive natural selection on our ancestral hominid DNA has selected our species for delayed brain development, with the longer period of vulnerability allowing – requiring – love and attention.
From that infant socialization unique to our species, comes the emergence of subjectivity, free will and self-consciousness, by imitation of the care-giving adult’s constant attentions. So if we are built for anything, we are built for reproducibility of emotional states: imitation in other words is a necessity, and imitation is by definition a pattern.
In short, love is a pattern, and a random sequence of emotional states is a torture; to make this a matter of intellectual habit, misses the point.
The WSJ has printed one of the best “fooled by randomness” pieces I’ve seen in quite a while, titled “The Triumph of the Random.” This one uses streaks in sports as a central metaphor, with DiMaggio’s 56-game hitting streak as exhibit A. It presents an immediate disclaimer:
Recent academic studies have questioned whether DiMaggio’s streak is unambiguous evidence of a spurt of ability that exceeded his everyday talent, rather than an anomaly to be expected from some highly talented player, in some year, by chance, something like the occasional 150-yard drive in golf that culminates in a hole in one. No one is saying that talent doesn’t matter. They are just asking whether a similar streak would have happened sometime in the history of baseball even if each player hit with the unheroic and unmiraculous—but steady—ability of an emotionless robot.
The lengthy article then deals with the mathematics of streaks, demonstrating that they are far more probable than we would otherwise think:
A few years ago Bill Miller of the Legg Mason Value Trust Fund was the most celebrated fund manager on Wall Street because his fund outperformed the broad market for 15 years straight. It was a feat compared regularly to DiMaggio’s, but if all the comparable fund managers over the past 40 years had been doing nothing but flipping coins, the chances are 75% that one of them would have matched or exceeded Mr. Miller’s streak.
Next, it moves to psychology and describes the way in which humans seek patterns in randomness as a grounding mechanism with a nice segway by way of my favorites, Kahneman and Tversky, who authored a seminal paper on hot hands in basketball:
If a person tossing a coin weighted to land on heads 80% of the time produces a streak of 10 heads in a row, few people would see that as a sign of increased skill. Yet when an 80% free throw shooter in the NBA has that level of success people have a hard time accepting that it isn’t. The Cognitive Psychology paper, and the many that followed, showed that despite appearances, the “hot hand” is a mirage. Such hot and cold streaks are identical to those you would obtain from a properly weighted coin.
Finally, it deals with the perception of random events:
Why do people have a hard time accepting the slings and arrows of outrageous fortune? One reason is that we expect the outcomes of a process to reflect the underlying qualities of the process itself. For example, if an initiative has a 60% chance of success, we expect that six out of every 10 times such an initiative is undertaken, it will succeed. That, however, is false.
A critical conclusion is laid out:
We find false meaning in the patterns of randomness for good reason: we are animals built to do just that… Many studies illustrate how this basic aspect of human nature translates to a misperception of chance.
Truly an excellent read and I can’t recommend it more.
Daniel Becker’s diploma dissertation was on the visualization of randomness – finding concrete ways to map the highly abstract idea of random behaviors and patterns. The resulting portfolio is fascinating, even for someone without a statistical background, in particular for the way in which it lends a semblance of order to these inherently chaotic processes.
The first one that caught my eye was a clean visualization of Benford’s law (I recently rambled on the subject). It isn’t a derivation of the law, but rather an illustration of its presence in a dataset of countries’ areas in square kilometers (a zoomable version is on the diploma website):
However, my favorite visualization is one of a quincunx. A quincunx (also called a Galton box, but that’s not as fun to say) is a device consisting of a vertical space filled by evenly spaced horizontal pins. A ball is dropped from the top of the box and allowed to bounce off the pins on its way down. At each pin, it is impossible to say whether the ball will bounce left or right. However, the box is used to illustrate the normal distribution (strictly speaking, it really illustrates a binomial distribution, which in the limit approaches a normal distribution) because the ball’s ultimate location will take the shape of a familiar bell curve – it is likely to land directly under the point at which it was dropped, and less likely to land far away.
My dad used to have a quincunx in his office when I was little. It took the form of a game (circa 1900) – there were bins along the bottom, and you dropped a penny in the top. If the penny landed in one of the extreme bins, you won — but you lost if it landed in one of the central bins. In retrospect, the game was obviously mispriced, since many of the bins had equal payouts despite their chances of receiving the penny. But I’m sure the mispricing was in the house’s favor.
Anyway, here is Becker’s visuaization of the process:
Finally, I also liked this visualization of the output of various random number generators. Needless to say, there should be no apparent pattern in uniform numbers generated by an algorithm – the presence of such a pattern indicates faulty number generators. (It is well documented that Excel’s random number generator is particularly disappointing). The box in the lower right represents the “iPhone” of random number generators – the Mersenne Twister. It is the closest to being pattern-free, though it looks a bit like a Clayton copula to me!
