"I think and think for months and years. Ninety-nine times, the conclusion is false. The hundredth time I am right." -- Albert Einstein
Just in time for my posts on measurement against a control group, I got a perfect real-life case at work. The situation is pretty typical for many people who run large direct mail lists out of a corporate system. The system has addresses of your current customers as well as prospects, and after you apply your targeting criteria, you can use a random selection procedure to identify your control, and make a record of both mail and control addresses. In the last step, the system produces your mail list to be sent to the mail house. For the measurement, customer purchases are tracked back to the addresses that were recorded in the mail and control groups, and the count and revenue of mail group and control groups are compared to determine incremental purchases and revenue.
The mail house does all sorts of address hygiene and cleaning, like removing duplicate addresses, taking out vacancies, running the addresses against known address database by USPS, which both cleans out non-compliant and nonexistent addresses. While current customer lists usually yield a very high percentage of mailable addresses, prospect lists lose around 20%-25% of the addresses in the hygiene process. This presents an issue for tracking, because we are tracking the purchases back to the lists that do not accurately reflect the addresses that were actually mailed. To improve measurement of the direct mail performance, the IT system proposes a solution that can take the post clean-up mail list (to be received from the mailhouse), and use it to clean up the original mail group list.
Will this solution improve quality of measurement? What are the advantages and shortcomings of this solution?
(I will pulish my opinion as a comment to the post)
Posts
1 comment:
This shortcut will only improve the quality of measurement if both mail and control groups are sent through the hygiene process and re-uploaded for measurement. Cleaning out the mail group alone will result in mail and control groups no longer being representative of each other, and thus not comparable in terms of purchases.
No matter how inconvenient is the process where less than the whole mail group is actually mailed, it is still analytically sound to measure the results against a matched control. This is happening because the control group has the same number of bad addresses, and thus we should expect the same number of purchases under the same conditions (i.e. there is no bias). If you have this situation, you will need to adjust mailing costs and your break even lift for the larger denominator.
Post a Comment