Customers who switched from [...], saved...

"We can't solve problems by using the same kind of thinking we used when we created them" -- Albert Einstein

We all saw them, the ads that promise you to save money on your car insurance. Geico does it, Allstate does it, 21st Century does it. We look at them and think that maybe the better deal is around the corner. OK, maybe we are not that gullible, so let's dig into the numbers.

The claim is that customers, who switched from [another insurance company] on average have saved $X, and those who switched from [the other insurance company] saved even more, $Y. Sounds like a good deal; sounds like everybody is saving. But is it really everybody? Those who switched by definition must have had a lower rate with the new company, otherwise, they... would not have switched. This is typical case of not looking at the total picture, but using a qualifying condition to isolate the part of picture that we will be looking at. In this particular case, it is caused by self-selection, since the customers self select to switch.

So, basically, if we have an insurance company A that for 90% of the insurance seeking population on charges more than insurance company B, but for 10% of the population charges less than A, on average, A will be a higher priced alternative. However, it still will be able to make a low price price against company B because, yes, it is correct, when people in the 10% group switch from B to A, they do indeed save money.

Now, this was kind of a silly case. We all know advertisers will tell everything and anything to get prospects interested. However, this type of self fulfilling prophecy is being used every day at the workplace to justify programs - and justify them with what appears to an untrained eye to be solid quantitative analysis. The most upsetting case of selection bias that I have seen was a program where customers "competed" for a prize from a company. Those who have increased their purchases most during a qualifying period of time were declared winners, and then their purchasing lift during that period of time was used to justify the program. Basically, it's like the race was judged based on speed, and then the winners were compared to everyone else and declared that they... had the highest speed. Obviously, the program has always "delivered".