Recently, there have been countless ads for auto insurance all making a similar claim: drivers who switch to that firm save significant amounts of money. How can every major insurance company make a similar statement? They can't all be cheaper than every other company, on average.
As a particularly egregious example, Allstate's website declares it via an improperly constructed sentence:
Drivers who switched saved with Allstate saved an average of $396 a year. [sic]
Does that mean that Allstate is $396 cheaper than its competitors? Of course not.
The answer lies unsurprisingly in sample selection bias (not to mention highly misleading advertising). The statement is true: people who changed insurance companies saved money. But extrapolating that fact into savings for the average customer is a mistake, because the sub-population of switchers is very different from the full customer base. These switchers are a relatively small group of people who, thanks to quirks in different companies' risk profiles, would pay much less at a different company. The average person is not in the switcher camp - that is simply a competitive impossibility.
This leads us to an obvious, forehead-slapping result: of course people who switch save money, why else would they be switching? When Allstate claims that they save switchers $400 a year, they aren't signaling to other customers that their policies are $400 cheaper on average; they are simply stating that for some people, they are cheaper. They could as easily run an ad that says "people who left us for another firm saved $400 a year" with equally little bearing on their average customers.
We're left with a stunning, real-world example of sample selection bias: a phenomenon in which (for reasons accidental, intentional or malicious) a chosen group of people is not representative of the population they are meant to represent. It is an important consideration for any experiment. In this case, the switchers are the sample which we are meant to see as representative of the whole population of insurance consumers, even though this is obviously not correct. The insurance companies are all very careful to state that the savings apply to "drivers who switched" as opposed to saying "our policy is cheaper" because it avoids accusations of misleading advertising in a legal sense, but people aren't really so perceptive that they distinguish the subtle difference. The aforementioned Allstate quote is meant to communicate that "if you switch, you will save $400," despite its actual - and very different - meaning.
Don Draper would be proud.