Want instant results? Concentrate on improving the worst performers.

"He who joyfully marches to music in rank and file has already earned my contempt. He has been given a large brain by mistake, since for him the spinal cord would suffice." -- Albert Einstein

Concentrating on the worst performers can be particularly fruitful in a well-managed organization, where performance is distributed in a close to random manner.

Mind you, we are not talking real improvements here. We are talking about putting yourself in a situation where there is nothing, but the upside.

Here is how you can spot the right type of the "worst" performers. They are usually smaller in size compared to an average performer. Whether it is smaller regions or smaller segments, groups of smaller sample size tend to have a higher variance, and thus, a higher probability to be an outlier, including the "worst" kind. You also need to make sure there are no systemic factors that drive poor performance, or otherwise simple random nature of the world may not be able to compensate for the systemic factor. That means you will have to actually do something to improve performance of your "worst" group. So, check your averages over several time periods to confirm that your worst performers had a particularly hard time when you pick them.

After you have validated the random (or even semi-random) nature of your "worst" performance group, you can crate a project to improve how it is doing. You would not expect your "worst" performer group to stay exactly the same every period, would you? If so, there are pretty good chances that next period you will "pull" a good number of the worst performers closer to the average, or out of the "worst" group. Congratulation, you just got yourself a "real" quantifiable result!

Remember, there is always a bottom 5% to work on!

When comparing, make sure the groups are representative

Sometimes people call them "matched", which is a layman's term for representative. So, why do they have to be matched? Because having non-representative groups may be so misleading, your analysis result may be the opposite of what they should be.

Here is a quick example. Let's say you have two groups of customers, each one of them consists of customers of two types/segments. Sometimes you may not even be aware that there two types of customers in your groups. Let's assume those segments exhibit different behavior. The sample behavior I chose was Churn, but it may be anything. Let's say we applied some sort of treatment to Group #2, and their churn went down by 1% in both segments. We are trying to use Group #1 to establish a baseline (or, what would have happened to the best of our knowledge) to Group #2 if we had not had the treatment. However, because the composition of groups is not representative of each other, we get exactly opposite result for the total - Group #2 appears to have a higher, not lower churn. See table below.

												Group 1	Group 2	Difference
					Segment #1							1,000	5,000
					Seg #1 Churn							5.0%	4.0%	-1.0%
					Segment #2							5,000	1,000
					Seg #2 Churn							2.0%	1.0%	-1.0%
					Total							6,000	6,000
					Total Churn							2.5%	3.5%	+1.0%

Pretty picture = bad methodology?

"Information is not knowledge" -- Albert Einstein

A couple of jobs ago I was working for a large, publicly traded retail company that spent a lot of money on being able to track transactions to particular customers, and got lots of insight from the analysis of their purchasing behavior. Most of the analysis was done very well, however, one particularly important piece of analytics that went through the executive levels like a firestorm as soon as it appeared in a presentation prepared by one very respected management consulting company. The piece of information was so popular, I have heard it reiterated numerous times at investor calls, so I have no conciseness spasms dragging it out to the blog, as this is no longer proprietary information.

Here is how it went: "Our customers who purchase from just one channel spend $X with us, while those who purchase from two channels spend $2X, and those who purchase from all three channels spend $3X with us". By "channels" the company meant physical stores, catalogue, and web-site. The conclusion always was that you want to get your customer to purchase from as many channels as possible, thus increasing her involvement with the brand. What got people to love this particular piece of data the most was the pretty chart - nice bar graph, growing from left to right, showing how the amounts of "spent" money spent nicely stack up, as we see increased "involvement".

I have stated that before and I am going to do it now - beware of the pretty numbers. The ones nicely confirming your hypothesis, and almost magically putting that conclusion into your mind. The numbers may not necessarily be misleading, but as analysts we need to understand the nature of the beast first.

Going back to the beast... namely, the database. I had been working with that database for a while, dutifully slicing and dicing the data, and at some point I got acquainted with the beast enough to project the kind of outcome we get from every dice. For example, if you pick up customers who have had a transaction in Large Category B, their average dollars spent and even the number of transactions is likely to be more than average for the database. Same went for Large Category V, and even not so large Category N. It always looked pretty - we get customers to buy something, and boom they spend more money with us. But do they? Is it a self-fulfilling prophecy again?

Turns out, it was. Around 60% of customers in the database were single-purchase customers, and they probably bought 2 items on average, thus limiting their possible exposure to the product categories. Naturally, the 40% of customers who had multiple transactions were vastly more likely to buy items from any given category. Thus, the comparison was always between average customer and more than an average customer.

Now let's go back to our beautiful channel chart. My careful investigation of the analysis methodology revealed that there was no adjustment for the number of transactions the customers had made. Basically, those who have purchased from three channels must have had at least three transactions (average for the database at the time 1.5), and those who have purchased from two channels must have made at least two purchases, and the single channel purchasers were mostly comprised of the single-purchase customers who by definition could not have have made it to the other groupings - they just did not spend enough with the company! Now, it is my best knowledge that the comparison of those who have spent more with the company to those who have spent less always confirms that the first group spent more, and the second spent less. I believe that's an imprtant finding that we need to keep in mind whenever the numbers are a little too pretty.

There is nothing more misleading than facts with no context

I have read an excellent article the other day about the value of good analysis, and the pitfalls of the bad. It researches WSJ's claim that increasing taxes on the people in the highest income bracket in Maryland led to their flight to other places within a year. It did not take into account any national trends of wealth and income, or any other measures "the rich" may take to lower their tax base (munis anyone?), and simple attributed the 50% decline in returns filed in the corresponding bucket to the fictional flight. However, a simple national trend analysis showed that Maryland was sitting close to the middle in terms of dynamics of wealthy households.

P.S. I put quotes around "the rich" because I believe the whole the rich vs the poor battle is pretty much made up, especially, on taxation. We all heard Warren Buffet advocating higher taxes for "the rich", and $40K-a-year Joe the Plumber being outraged by out of control taxes Democrats were allegedly ready to put into the law. My take, saying things like that makes Joes feel like they are one of the rich. Much like buying a Louis Vuitton bag with credit - not a particularly rational economic behavior, but what the heck, there was a time I was working for a company that successfully translated this behavior into a nice chunk of change, and studying it made a fascinating subject of research. But I digress.

Customers who switched from [...], saved...

"We can't solve problems by using the same kind of thinking we used when we created them" -- Albert Einstein

We all saw them, the ads that promise you to save money on your car insurance. Geico does it, Allstate does it, 21st Century does it. We look at them and think that maybe the better deal is around the corner. OK, maybe we are not that gullible, so let's dig into the numbers.

The claim is that customers, who switched from [another insurance company] on average have saved $X, and those who switched from [the other insurance company] saved even more, $Y. Sounds like a good deal; sounds like everybody is saving. But is it really everybody? Those who switched by definition must have had a lower rate with the new company, otherwise, they... would not have switched. This is typical case of not looking at the total picture, but using a qualifying condition to isolate the part of picture that we will be looking at. In this particular case, it is caused by self-selection, since the customers self select to switch.

So, basically, if we have an insurance company A that for 90% of the insurance seeking population on charges more than insurance company B, but for 10% of the population charges less than A, on average, A will be a higher priced alternative. However, it still will be able to make a low price price against company B because, yes, it is correct, when people in the 10% group switch from B to A, they do indeed save money.

Now, this was kind of a silly case. We all know advertisers will tell everything and anything to get prospects interested. However, this type of self fulfilling prophecy is being used every day at the workplace to justify programs - and justify them with what appears to an untrained eye to be solid quantitative analysis. The most upsetting case of selection bias that I have seen was a program where customers "competed" for a prize from a company. Those who have increased their purchases most during a qualifying period of time were declared winners, and then their purchasing lift during that period of time was used to justify the program. Basically, it's like the race was judged based on speed, and then the winners were compared to everyone else and declared that they... had the highest speed. Obviously, the program has always "delivered".