Multiple models in Annual Fund: Worth the trouble?

In planning which predictive models I was going to create for this fall’s Annual Giving appeals, I had one main idea in mind: Donors are not equally receptive to both mail and phone solicitation. I knew from previous experience that I could build a good model trained on all Annual Fund giving regardless of source, but that it would not be optimal because it would fail to take “preferred mode of solicitation” into account. Great donors with high propensity-to-give scores will hang up on your Phonathon callers if their preferred mode of giving is by mail! (See Preventing hangups and rudeness in your Phonathon program.)

The question is: Is it realistic to think that you can predict which mode is the preferred? If you can’t, why bother trying? Let’s explore that question.

I created two separate models, to answer separate questions:

• The Mail Model answers the question, “Who is most likely to give in response to a mailed solicitation?” The outcome variable was defined as each individual’s total giving in response to mail solicitation.
• The Phone Model answers the question, “Who is most likely to give in response to phone solicitation?” The outcome variable was defined as each individual’s total giving in response to phone solicitation.

The question is: Is this really worth the bother? It’s difficult to say what, if anything, separates mail-receptive donors from phone-receptive donors, although it’s always been my sense that people who prefer mail over phone tend to be older. A look at my data shows that there is indeed an age difference:

• Alumni who have given by phone but not by mail have a median age of 47.
• Alumni who have given by mail but not by phone have a median age of 55.
• Alumni who have given via both channels are closer in median age to the by-mail-only group, at 56.

The giving data is potentially biased: Perhaps young alumni tended to be solicited by phone rather than by mail, therefore that’s how they’ve given. However, it’s my understanding that in this case, no special consideration was given in the past to emphasizing one mode of solicitation over another based on any criteria such as age. Therefore, I’ve assumed that the age difference at least is real.

But what besides age could possibly differentiate between the two? Nothing intuitive comes to mind. So it’s fair to ask whether the two models are different enough from each other to warrant the extra work. I can think of only one way to test it: Compare how each individual is scored in both models. If most people hold the same rank from one to the other, then both models essentially predict the same thing, i.e. giving of any sort.

In both models, the same 87,000 alumni were ranked in deciles according to their score, with Decile 10 being the highest possible ranking. The table below shows the results when I subtract each alum’s Phone Model decile from their Mail Model decile. When I compared Phone scores with Mail scores, I found that a little more than 20% of alumni had the same decile score in both models (the zero bar, in the centre):

Another 27% of alumni have scores that differ by one decile (plus 1 or minus 1). Because it’s easy for an individual to get bumped from one rank to another, I will discount that difference as insignificant. Added to the alumni who didn’t move at all, a total of 48% of alumni have basically the same score from model to model.

That leaves a slight majority (52%) who differ significantly from model to model. About one-quarter of alumni are predicted to have a relative preference for mail, and one quarter have a relative preference for phone. Curiously, these two quarters do NOT differ significantly in their median age — so how the two groups DO differ from each other is still rather mysterious.

This exercise does not prove that two models are better than one — but I think it DOES show that they are predicting different things. I will have to be content with that for now. If it sounds as though I’m not certain about the benefits of modeling for channel preference, well — I’m not. I think more work needs to be done. As usual, your thoughts are welcome.