There’s been some back-and-forth on one of the listservs about the “correct” way to measure and score alumni engagement. An emphasis on scientific rigor is being pressed for by one vendor who claims to specialize in rigor. The emphasis is misplaced.
No doubt there are sophisticated ways of measuring engagement that I know nothing about, but the question I can’t get beyond is, how do you define “engagement”? How do you make it measurable so that one method applies everywhere? I think that’s a challenging proposition, one that limits any claim to “correctness” of method. This is the main reason that I avoid writing about measuring engagement — it sounds analytical, but inevitably it rests on some messy, intuitive assumptions.
The closest I’ve ever seen anyone come is Engagement Analysis Inc., a firm based here in Canada. They have a carefully chosen set of engagement-related survey questions which are held constant from school to school. The questions are grouped in various categories or “drivers” of engagement according to how closely related (statistically) the responses tend to be to each other. Although I have issues with alumni surveys and the dangers involved in interpreting the results, I found EA’s approach fascinating in terms of gathering and comparing data on alumni attitudes.
(Disclaimer: My former employer was once a client of this firm’s but I have no other association with them. Other vendors do similar and very fine work, of course. I can think of a few, but haven’t actually worked with them, so I will not offer an opinion.)
Some vendors may make claims of being scientific or analytically correct, but the only requirement of quantifying engagement is that it be reasonable, and (if you are benchmarking against other schools) consistent from school to school. In general, if you want to benchmark, then engage a vendor if you want to do it right, because it’s not easily done.
But if you want to benchmark against yourself (that is, over time), don’t be intimidated by anyone telling you your method isn’t good enough. Just do your own thing. Survey if you like, but call first upon the real, measurable activities that your alumni participate in. There is no single right way, so find out what others have done. One institution will give more weight to reunion attendance than to showing up for a pub night, while another will weigh all event attendance equally. Another will ditch event attendance altogether in favour of volunteer activity, or some other indicator.
Can anyone say definitively that any of these approaches are wrong? I don’t think so — they may be just right for the school doing the measuring. Many schools (mine included) assign fairly arbitrary weights to engagement indicators based on intuition and experience. I can’t find fault with that, simply because “engagement” is not a quantity. It’s not directly measurable, so we have to use proxies which ARE measurable. Other schools measure the degree of association (correlation) between certain activities and alumni giving, and base their weights on that, which is smart. But it’s all the same to me in the end, because ‘giving’ is just another proxy for the freely interpretable quality of “engagement.”
Think of devising a “love score” to rank people’s marriages in terms of the strength of the pair bond. A hundred analysts would head off in a hundred different directions at Step 1: Defining “love”. That doesn’t mean the exercise is useless or uninteresting, it just means that certain claims have to be taken with a grain of salt.
We all have plenty of leeway to chose the proxies that work for us, and I’ve seen a number of good examples from various schools. I can’t say one is better than another. If you do a good job measuring the proxies from one year to the next, you should be able to learn something from the relative rises and falls in engagement scores over time and compared between different groups of alumni.
Are there more rigorous approaches? Yes, probably. Should that stop you from doing your own thing? Never!
CoolData blog 
I agree with you on several fronts. Per my reply on the listerv yesterday following this thread, I believe firmly that it is erroneous to try to imply there is any one single definition for engagement or to say that there is one definitive measuring/scoring system to measure alumni engagement. I am familiar with the firm you mention, and we do similar work and have build comparable indexes and questions to allow schools to benchmark with themselves over time and with other institutions (perhaps peer or aspirant). However, our philosophy is not to try to tell a particular school that we have the right proxies (as you refer to it) to measure – although we too have developed a core set of engagement-related questions that we hold constant over time, but rather allow the feedback from their alumni and from the alumni/advancement staff to help facilitate our measurements on those areas that show up as being the most important and most popular with alumni for that respective institution. Some schools already have set measuring systems in place – like the ones I mentioned on the listserv yesterday – and for those we build that criteria into their individual projects to give us the direction we need to provide proper analysis and action planning for increasing alumni engagement in those areas. Something else to consider, which is something we often discuss within our firm, is that perhaps it is less important to worry about being “valid” and more about engaging the masses. I think as more schools look at ways to effectively “measure/score engagement” (however they define engagement), they tend to over think this process and end up being too granular. At least to me, the true value is not to develop a scoring system so that we can identify the top 5 or even 50 “most engaged” alumni. After all, for most schools that is a tiny percentage of their overall population. Rather, I think the true value is to measure on a much larger scale to identify (in aggregate, then segmenting on particular demographics that best fit that institution) what items have the most impact on alumni participation in programs, communication, giving, etc – all proxies that most schools would confirm as being areas of engagement – and then benchmarking that over time both internally and against other schools. Just my two cents further on this topic. Good discussion.
Comment by Aaron — 6 June 2012 @ 12:36 pm
I agree with the comments about no single way to measure engagement. It also depends largely on your motives. We’re looking at this more or less as a two-fold proposition: 1) to track engagement as an overall performance indicator for alumni operations (this as an annual benchmark puts “teeth” into our operations; it’s critical to do so because in order to be on a level playing field with our fundraising colleagues whose metrics are quite clear); and 2) to provide empirical evidence regarding the impact of engaged alumni on fund raising. If we can demonstrate in concrete terms (and we have) that engaged alumni are more likely to give and to give at far greater amounts than non-engaged graduates, then this legitimizes our role on the advancement team beyond being viewed as “friend-raisers” (ick!) or party planners.
Comment by Christopher J. Vlahos — 21 June 2012 @ 8:56 am