When comparing, make sure the groups are representative

Sometimes people call them "matched", which is a layman's term for representative. So, why do they have to be matched? Because having non-representative groups may be so misleading, your analysis result may be the opposite of what they should be.

Here is a quick example. Let's say you have two groups of customers, each one of them consists of customers of two types/segments. Sometimes you may not even be aware that there two types of customers in your groups. Let's assume those segments exhibit different behavior. The sample behavior I chose was Churn, but it may be anything. Let's say we applied some sort of treatment to Group #2, and their churn went down by 1% in both segments. We are trying to use Group #1 to establish a baseline (or, what would have happened to the best of our knowledge) to Group #2 if we had not had the treatment. However, because the composition of groups is not representative of each other, we get exactly opposite result for the total - Group #2 appears to have a higher, not lower churn. See table below.

												Group 1	Group 2	Difference
					Segment #1							1,000	5,000
					Seg #1 Churn							5.0%	4.0%	-1.0%
					Segment #2							5,000	1,000
					Seg #2 Churn							2.0%	1.0%	-1.0%
					Total							6,000	6,000
					Total Churn							2.5%	3.5%	+1.0%

It's easy to calculate a number, it is much harder to tell what it means

We all know, that's how it usually starts - we need to know X. So, let's say X=25%. Is it bad? Is it good? Too high? Too low? In my practice, when I bring the number within context with all the other numbers that shed light on what it means... people think it is too much, that they do not need "all of that". That just need one number. So, when they get the number, they start asking questions about "all of that". The circle is now complete. If anyone knows a way out of this conundrum, please let me know.

My take on the classic

"Every truth passes through three stages before it is recognized. In the first it is ridiculed, in the second it is opposed, in the third it is regarded as self-evident" - Arthur Schopenhauer

Here is my version: "Every truth passes through three stages before it is recognized. In the first it is ignored, in the second it becomes a fad, in the third it is forgotten".

Every business is seasonal

At least, I have yet to see one that is not. Out of all basic analytical concepts seasonality is the one that I find underestimated the most. On more than a few occasions I have heard that "our business is not that seasonal" while in reality, seasonal swings may explain up to 80% of sales deviation. Always check for seasonality.

Scientific Principles of Direct Marketing Optimization

Here is the list of my principles so far:

1. Always use a control group. Preferably randomized, representative (of your treatment) control group takes care of other things going on in the marketplace, including your own campaigns, and also acts as the “great equalizer”, fixing even the worst metrics, response models.
2. Maximize lift, not response. Lift is the difference between treated and control groups. That’s what you are trying to impact.
3. Optimal frequency is often more powerful/important than optimal segmentation. Ideally, you want to optimize (i.e. maximize lift) frequency by segment, but if you are confused where to start testing, you should start with frequency. None of the segmentation work will be insightful unless your frequency is within the shooting range of the optimal.
4. Test, test, test. It’s one of the easiest and simplest ways to learn.
5. When testing, have a hypothesis, then design the test around it. Sending a customized piece to a segment is a great idea, until you realize that you did not send your regular piece to the same audience at the same time, and thus can’t tell whether customized piece would have done better than a regular one.
6. Track your treated group against control group for a while to understand how long the impact of your mailing lasts. Some people want to use LTV. That’s because they want a higher ROI. True measurable difference traceable to the impact of a direct mail piece rarely lasts more than a few months, even though average lifetime maybe measured in years.
7. When choosing the size of the control group, you first need to understand what kind of a difference will justify the effort (i.e. break even lift), and then determine a sample size that will make this difference statistically significant. If you’re measuring with a yardstick, it’s hard to determine a half-inch of a difference.

Marketing analytics case study - Direct Mail list cleanup

"I think and think for months and years. Ninety-nine times, the conclusion is false. The hundredth time I am right." -- Albert Einstein

Just in time for my posts on measurement against a control group, I got a perfect real-life case at work. The situation is pretty typical for many people who run large direct mail lists out of a corporate system. The system has addresses of your current customers as well as prospects, and after you apply your targeting criteria, you can use a random selection procedure to identify your control, and make a record of both mail and control addresses. In the last step, the system produces your mail list to be sent to the mail house. For the measurement, customer purchases are tracked back to the addresses that were recorded in the mail and control groups, and the count and revenue of mail group and control groups are compared to determine incremental purchases and revenue.

The mail house does all sorts of address hygiene and cleaning, like removing duplicate addresses, taking out vacancies, running the addresses against known address database by USPS, which both cleans out non-compliant and nonexistent addresses. While current customer lists usually yield a very high percentage of mailable addresses, prospect lists lose around 20%-25% of the addresses in the hygiene process. This presents an issue for tracking, because we are tracking the purchases back to the lists that do not accurately reflect the addresses that were actually mailed. To improve measurement of the direct mail performance, the IT system proposes a solution that can take the post clean-up mail list (to be received from the mailhouse), and use it to clean up the original mail group list.

Will this solution improve quality of measurement? What are the advantages and shortcomings of this solution?

(I will pulish my opinion as a comment to the post)

Maximizing response often leads to poor campaign performance

I did some research for presentation at work today, and found this very nice white paper on the use of lift (as they call it, "uplift") modeling in driving true incremental sales. It correctly highlights the difference between correlation and causation, as response models simply correlate to propensity to buy, and incremental lift models track impact of marketing communication. Then it goes to explain the impact of segmentation on response and incremental sales, and I just loved those two charts showing how response is correlated to lift. Negatively, in their particular case. The higher the response, the lower the incremental sales. Actually, that's the conclusion that I often found in my own job. I am not saying it always happens, but it does happen often when we maximize response and focus solely on those who are likely to buy from us anyway. You should also watch the opposite end of the curve, where you have prospects so unlikely to purchase, that even though you do get high incrementality, it may still not be enough to pay for the program. Thus, your most profitable targets are usually in the sweet spot somewhere in the middle of the response curve. This is kind of article every direct marketer needs to read.
Generating Incremental Sales