Data 1, Gut Instinct 0

Sometimes I employ a practice in our Phonathon program simply because my gut says it’s gotta work. Some things just seem so obvious that it doesn’t seem worthwhile testing them to prove my intuition is valid. And like a lot of people who work in Annual Giving, I like to flatter myself that I can make a non-engaged alum give just by making shrewd tweaks to the program.

It turns out that I am quite wrong. I am thinking about a practice that seems to be part of the Phonathon gospel of best practices. I firmly believed in it, and I got serious about using it this fall. As the song says, though, it ain’t necessarily so.

When possible, I am pairing up student callers with alumni whose degree is in the same faculty of study. If I have business students in the calling room, for example, I’ll assign them alumni with degrees associated with the Faculty of Management. A grad with a BSc majoring in chemistry, meanwhile, will get a call from a student majoring in one of the sciences, rather than arts or business. It’s not perfect: There are too many degree programs, current and historic, for me to get any more specific than the overall faculty, but at least it increases the chance that student and alum will have something in common to talk about.

It’s easy to see why this ought to work. When speaking with young alumni, callers are somewhat more likely to have had certain professors or classes in common, and their interests may be aligned — for example, the alum might be able to provide the student with a glimpse into the job market that awaits. With older alumni, the callers might at least know the campus and buildings that alumni of the past inhabited just as they do today. If alumni feel so inclined, the conversation might even lead to a discussion about life and career.

These would be meaningful conversations, the kind of connection we hope to achieve on the phone. Just that much, even without a gift (this year), would be a desirable result.

On the other hand … if faculty pairings really lead to longer, better-quality conversations, would we not expect that faculty-paired conversations would, on average, result in more gifts than non-paired conversations? In the long run, is that not our goal? If it makes no difference who asks whom, then why complicate things?

First let me say that I embarked on this analysis fully expecting that the data would demonstrate the effectiveness of faculty-paired conversations. I might be a data guy, but I am not unbiased! I really hoped that my intervention would actually produce results. Allow me to admit that I was quite disappointed by what I found.

Here’s what I did.

Last year, I did not employ faculty pairings. We made caller assignments based on prospects’ donor status (LYBUNT, SYBUNT, etc.), but not faculty. I don’t know how our automated software distributes prospects to callers, but I am comfortable saying that, with regards to the faculty of preferred degree, the distribution to callers was random. This more or less random assignment by faculty allowed me to compare “paired” conversations with “unpaired” conversations, to see whether one was better than the other with regards to length of call, participation rate, and average pledge.

I dug into the database of our automated calling application and I pulled a big file with every single call result for last year. The file included the caller’s ID, the length of the call in seconds, the last result (Yes Pledge, No Pledge, No Answer, Answering Machine, etc. etc.), and the pledge amount (if applicable).

Then I removed all the records that did not result in an actual conversation. If the caller didn’t get to speak to the prospect, faculty pairing is irrelevant. I kept any record that ended in Yes Pledge (specified-dollar pledge or credit card gift), Maybe (unspecified pledge), No Pledge, or a request to be put on the Do Not Call list.

I added two more columns (variables) of data: The faculty of the caller’s area of study, and the faculty of the prospect’s preferred degree. Because not all of our dozen or so faculties is represented in our calling room, I then removed all the records for which no pairing was possible. For example, because I employed no Law or Medicine students, 100% of our Law and Medicine alumni would end up on the “non-paired” side, which would skew the results.

As well, I excluded calls with call lengths of five seconds or less. It is doubtful callers would have had enough time to identify themselves in less than five seconds — therefore those calls do not qualify as conversations.

In the end, my data file for analysis contained the results of 6,500 conversations for which a pairing was possible. Each prospect record, each conversation, could have one of two states: ‘Paired’ or ‘Unpaired’. About 1,500 conversations (almost 22%) were paired, as assigned at random by the calling software.

I then compared the Paired and Unpaired groups by talk time (length of call in seconds), participation, and size of pledge.

1. Talk time

Better rapport-building on the phone implies longer, chattier calls. According to the table, “paired” calls are indeed longer on average, but not by much. A few seconds maybe.

2. Participation rate

The donor rates you see here are affected by all the exclusions, especially that of some key faculties. However, it’s the comparison we’re interested in, and the results are counter-intuitive. Non-paired conversations resulted in a slightly higher rate of participation (number of Yes Pledges divided by total conversations).

3. Average and median pledge

This table is also affected by the exclusion of a lot of alumni who tend to make larger pledges. Again, though, the point is that there is very little difference between the groups in terms of the amount given per Yes pledge.

The differences between the groups are not significant. Think about the range of values your callers get for common performance metrics (pledge rate, credit card rate, talk time, and so on). There are huge differences! If you want to move the yardsticks in your program, hire mature, friendly, chatty students who love your school and want to talk about it. Train them well. Keep them happy. Reward them appropriately. Retain them year over year so they develop confidence. These are the interventions that matter. Whom they are assigned to call doesn’t matter nearly as much.

Over and above that, pay attention to what matters even more than caller skills: The varying level of engagement of individual alumni. Call alumni who will answer the phone. Call alumni who will give a gift. Stop fussing over the small stuff.

You know what, though? Even faced with this evidence, I will probably continue to pair up students and alumni by faculty. First of all, the callers love it. They say they’re having better conversations, and I’m inclined to believe them. It’s not technically difficult to match up by faculty, so why not? As well, there might be nuances that I overlooked in my study of last year’s data. Maybe the faculty pairings are too broad. (Anytime you find economics in the same faculty as physics, you have to wonder how some people define Science. A discussion for someone else’s blog, perhaps.)

But my study has cast doubt on the usefulness of going to any great length to target alumni by faculty. For example, should I try hard to recruit a student caller from Law or Medicine to maximize on alumni from those faculties? Probably not.

Finally, I caution readers not to interpret my results as being generally applicable. I’m not saying that faculty pairing as a best practice is invalid. You need to determine for yourself whether a practice is part of your core strategy, or just a refinement, or completely useless. As I opined in my previous post (Are we too focused on trivia?), I suspect a lot of Annual Fund professionals aren’t making these distinctions.

The answers are in the data. Go find them.