CoolData blog

20 February 2013

The ‘analytic’ investment

Filed under: Analytics, Data — Tags: , — kevinmacdonell @ 10:49 am

Everyone’s talking about predictive analytics, Big Data, yadda yadda. The good news is, many institutions and organizations in our sector are indeed making investments in analytics and inching towards becoming data-driven. I have to wonder, though, how much of current investment is based on hype, and how much is going to fall away when data is no longer a hot thing.

Becoming a data-driven organization is a journey, not a destination. Forward progress is not inevitable, and it is possible for an office, a department or an institution to slip backward on the path, even when it seems they’ve “arrived”. In order for analytics to mature from a cutting-edge “nice-to-have” into a regular part of operations, the enterprise needs to be aware of its returns to the bottom line.

In my view, current investments in analytics are often done for reasons that are well-intentioned but vague: It seems to be the right thing to do these days … we see others doing it, so we feel we need to as well … we have an agenda for innovation and this fits the bill … and so on. I’m glad to see the investment, but not every promising innovation gets to stick around. Demonstrating ability to generate revenue — either through savings or through identifying new sources of revenue — will carry the day in the long run.

As I write this, I hear the jangle of railway bells at the level crossing in the early-morning dark outside my hotel room on the city’s downtown waterfront. I’m in Seattle today to attend the DRIVE 2013 conference, hosted by the University of Washington. I’ll be speaking on this topic — the “analytic” investment — later today. I have to admit to having struggled with making the session relevant for this group. For one, they don’t need convincing that making the investment is worth it. And second, if they think that I and my employer have figured out how to calculate the return on investment for analytics programs, they may be in for a disappointment. We have not.

In fact, when it comes right down to it, I like to spend my day working on cool things, interesting problems that face our department, and not so much on stuff that sounds like accounting (“ROI”). I’m betting many of the attendees of my session feel the same way. So I’ll be asking them to stop thinking about how they can get their managers, directors and vice presidents to understand the language of data and analytics. They’ll be far more successful if they try to speak the language their bosses respond to: Return on investment.

I may be a little short on answers for you, but I do have some pretty good questions.

30 November 2012

Analytics conferences: Two problems, two antidotes

A significant issue for gaining data-related skills is finding the right method of sharing knowledge. No doubt conferences are part of the answer. They attract a lot of people with an interest in analytics, whose full-time job is currently non-analytical. That’s great. But I’m afraid that a lot of these people assume that attending a conference is about passively absorbing knowledge doled out by expert speakers. If that’s what you think, then you’re wasting your money, or somebody’s money.

There are two problems here. One is the passive-absorption thing. The other is a certain attitude towards the “expert”. Today I want to describe both problems, and prescribe a couple of conferences related to data and analytics which offer antidotes.

Problem One: “Just Tell Me What To Do”

You know the answer already: Knowledge can’t be passively absorbed. It is created, built up inside you, through engagement with an other (a teacher, a mentor, a book, whatever). We don’t get good ideas from other people like we catch a cold. We actively recognize an idea as good and re-create it for ourselves. This is work, and work creates friction — this is why good ideas don’t spread as quickly as mere viral entertainment, which passes through our hands quickly and leaves us unchanged. Sure, this can be exciting or pleasant work, but it requires active involvement. That’s pretty much true for anything you’d call education.

Antidote One: DRIVE

Ever wish you could attend a live TED event? Well, the DRIVE conference (Feb. 20-21 in Seattle — click for details) captures a bit of that flavour: Ideas are front and centre, not professions. Let me explain … Many or most conferences are of the “birds of a feather” variety — fundraisers talking to fundraisers, analysts talking to analysts, researchers talking to researchers, IT talking to IT. The DRIVE conference (which I have written about recently) is a diverse mix of people from all of those fields, but adds in speakers from whole other professional universes, such a developmental molecular biologist and a major-league baseball scout.

Cool, right? But if you’re going to attend, then do the work: Listen and take notes, re-read your notes later, talk to people outside your own area of expertise, write and reflect during the plane ride home, spin off tangential ideas. Dream. Better: dream with a pencil and paper at the ready.

Problem Two: “You’re the Expert, So Teach Me Already”

People may assume the person at the podium is an expert. The presenter has got something that the audience doesn’t, and that if it isn’t magically communicated in those 90 minutes then the session hasn’t lived up to its billing. Naturally, those people are going to leave dissatisfied, because that’s not how communicating about analytics works. If you’re setting up an artificial “me/expert” divide every time you sit down, you’re impeding your ability to be engaged as a conference participant.

Antidote Two: APRA Analytics Symposium

Every year, the Association of Professional Researchers for Advancement runs its Data Analytics Symposium in concert with its international conference. (This year it’s Aug 7-8 in Baltimore.) The Symposium is a great learning opportunity for all sorts of reasons, and yes, you’ll get to hear and meet experts in the field. One thing I really like about the Symposium is the  case-study “blitz” that offers the opportunity for colleagues to describe projects they are working on at their institutions. Presenters have just 20 or so minutes to present a project of their choice and take a few questions. Some experienced presenters have done these, but it’s also a super opportunity for people who have some analytics experience but are novice presenters. It’s a way to break through that artificial barrier without having to be up there for 90 minutes. If you have an idea, or would just like more information on the case studies, get in touch with me at kevin.macdonell@gmail.com, or with conference chair Audrey Geoffroy: ageoffroy@uff.ufl.edu. Slots are limited, so you must act quickly.

I present at conferences, but I assure you, I have never referred to myself as an “expert”. When I write a blog post, it’s just me sweating through a problem nearly in real time. If sometimes I sound like I knew my way through the terrain all along, you should know that my knowledge of the lay of the land came long after the first draft. I like to think the outlook of a beginner or an avid amateur might be an advantage when it comes to taking readers through an idea or analysis. It’s a voyage of discovery, not a to-do list. Experts have written for this blog, but they’re good because although they know their way around, every new topic or study or analysis is like starting out anew, even for them. The mind goes blank for a bit while one ponders the best way to explore the data — some of the most interesting explorations begin in confusion and uncertainty. When Peter Wylie calls me about an idea he has for a blog post, he doesn’t say, “Yeah, let’s pull out Regression Trick #47. You know the one. I’ll find some data to fit.” No — it’s always something fresh, and his deep curiosity is always evident.

So whichever way you’re facing when you’re in that conference room, remember that we are all on this road together. We’re at different places on the road, but we’re all traveling in the same direction.

6 June 2012

How you measure alumni engagement is up to you

Filed under: Alumni, Best practices, Vendors — Tags: , , , — kevinmacdonell @ 8:02 am

There’s been some back-and-forth on one of the listservs about the “correct” way to measure and score alumni engagement. An emphasis on scientific rigor is being pressed for by one vendor who claims to specialize in rigor. The emphasis is misplaced.

No doubt there are sophisticated ways of measuring engagement that I know nothing about, but the question I can’t get beyond is, how do you define “engagement”? How do you make it measurable so that one method applies everywhere? I think that’s a challenging proposition, one that limits any claim to “correctness” of method. This is the main reason that I avoid writing about measuring engagement — it sounds analytical, but inevitably it rests on some messy, intuitive assumptions.

The closest I’ve ever seen anyone come is Engagement Analysis Inc., a firm based here in Canada. They have a carefully chosen set of engagement-related survey questions which are held constant from school to school. The questions are grouped in various categories or “drivers” of engagement according to how closely related (statistically) the responses tend to be to each other. Although I have issues with alumni surveys and the dangers involved in interpreting the results, I found EA’s approach fascinating in terms of gathering and comparing data on alumni attitudes.

(Disclaimer: My former employer was once a client of this firm’s but I have no other association with them. Other vendors do similar and very fine work, of course. I can think of a few, but haven’t actually worked with them, so I will not offer an opinion.)

Some vendors may make claims of being scientific or analytically correct, but the only requirement of quantifying engagement is that it be reasonable, and (if you are benchmarking against other schools) consistent from school to school. In general, if you want to benchmark, then engage a vendor if you want to do it right, because it’s not easily done.

But if you want to benchmark against yourself (that is, over time), don’t be intimidated by anyone telling you your method isn’t good enough. Just do your own thing. Survey if you like, but call first upon the real, measurable activities that your alumni participate in. There is no single right way, so find out what others have done. One institution will give more weight to reunion attendance than to showing up for a pub night, while another will weigh all event attendance equally. Another will ditch event attendance altogether in favour of volunteer activity, or some other indicator.

Can anyone say definitively that any of these approaches are wrong? I don’t think so — they may be just right for the school doing the measuring. Many schools (mine included) assign fairly arbitrary weights to engagement indicators based on intuition and experience. I can’t find fault with that, simply because “engagement” is not a quantity. It’s not directly measurable, so we have to use proxies which ARE measurable. Other schools measure the degree of association (correlation) between certain activities and alumni giving, and base their weights on that, which is smart. But it’s all the same to me in the end, because ‘giving’ is just another proxy for the freely interpretable quality of “engagement.”

Think of devising a “love score” to rank people’s marriages in terms of the strength of the pair bond. A hundred analysts would head off in a hundred different directions at Step 1: Defining “love”. That doesn’t mean the exercise is useless or uninteresting, it just means that certain claims have to be taken with a grain of salt.

We all have plenty of leeway to chose the proxies that work for us, and I’ve seen a number of good examples from various schools. I can’t say one is better than another. If you do a good job measuring the proxies from one year to the next, you should be able to learn something from the relative rises and falls in engagement scores over time and compared between different groups of alumni.

Are there more rigorous approaches? Yes, probably. Should that stop you from doing your own thing? Never!

26 April 2012

For agile data mining, start with the basics

Filed under: Analytics, Pitfalls, Training / Professional Development — Tags: , , , — kevinmacdonell @ 8:56 am

Lately I’ve been telling people that one of the big hurdles to implementing predictive analytics in higher education advancement is the “project mentality.” We too often think of each data mining initiative as a project, something with a beginning and end. We’d be far better off to think in terms of “process” — something iterative, always improving, and never-ending. We also need to think of it as a process with a fairly tight cycle: Deploy it, let it work for a bit, then quickly evaluate, and tweak, or scrap it completely and start over. The whole cycle works over the course of weeks, not months or years.

Here’s how it sometimes goes wrong, in five steps:

  1. Someone has the bright idea to launch a “major donor predictive modelling project.” Fantastic! A committee is struck. They put their heads together and agree on a list of variables that they believe are most likely to be predictive of major giving.
  2. They submit a request to their information management people, or whomever toils in extracting stuff from the database. Emails and phone calls fly back and forth over what EXACTLY THE HECK the data mining team is looking for.
  3. Finally, a massive Excel file is delivered, a thing the likes of which would never exist in nature — like the unstable, man-made elements on the nether fringes of the Periodic Table. More meetings are held to come to agreement about what to do about multiple duplicate rows in the data, and what to do about empty cells. The committee thinks maybe the IT people need to fix the file. Ummm — no!
  4. Half of the data mining team then spends considerable time in pursuit of a data file that gleams in its cleanliness and perfection. The other half is no longer sure what the goal of the project was.
  5. Somehow, a model is created and the records are scored by the one team member left standing. Unfortunately, a year has passed and the person for whom the model was built has left for a new job in California. Her replacement refers to the model as “astrology.”

Allow me a few observations that follow from these five stages:

  1. Successful models are rarely produced by committee, and variables cannot be pre-selected by popular agreement and intuition — although certainly experience is a valuable source of clues.
  2. Submitting requests to someone else for data, having to define exactly what it is you want, and then waiting for the request to be fulfilled — all of that is DEATH to creative data exploration.
  3. A massive, one-time, all-or-nothing data suction job is probably not the ideal starting point. Neither is handling an Excel file with 200,000 rows and a hundred columns.
  4. Perfect data is not a realistic goal, and is not a prerequisite for fruitful data mining.
  5. A year is too long. The cycle has to be much, much tighter than that.

And finally, here are some concrete steps, based on the observations, again point-for-point:

  1. If you’re interested in data mining, try going it alone. Ask for help when you need it, but you’ll make faster progress if you explore on your own or in a team of no more than two or three like-minded people. Don’t tell anyone you’re launching a “project,” and don’t promise deliverables unless you know what you’re doing.
  2. Learn how to build simple queries to pull data from your database. Get IT to set you up. Figure out how to pull a file of IDs along with sum of all their hard-credit giving. Then, pull that AND something else — anything else. Email address, class year, marital status, whatever. Practice, get comfortable with how your data is stored and how to limit it to what you want.
  3. Look into stats software, and learn some of the most common stats terms. Read up on correlation in particular. Build larger files for analysis in the stats software rather than in Excel. Read, read, read. Play, play, play.
  4. Think in terms of pattern detection, and don’t get hung up on the validity of individual data points.
  5. If you’ve done steps 1 to 4, you have the foundations in place for being an agile data miner.

Mind you, it could take considerable time — months, maybe even years — to get really comfortable with the basics, especially if data mining is a sideline to your “real” job.  But success and agility does depend on being able to work independently, being able to snag data on a whim, being able to understand a bit of what is going on in your software, having the freedom to play and explore, and losing notions about data that come from the business analysis and reporting side. In other words, the basics.

19 March 2012

Symposium on Data Analytics is a must-attend

If you’re interested in working with data for the benefit of a non-profit organization or for education institutional advancement, then you must make room in your calendar for the APRA Symposium on Data Analytics.

Kate Chamberlin of Memorial Sloan-Kettering Cancer Center recently posted the listserv message below which I am quoting in its entirety, with her blessing. Kate is Chair of this year’s Symposium, being held this summer in Minneapolis. I’ve attended a few of these symposiums (and presented at one), and I can tell you that they’re great. This is a conference where you can really learn, and meet the people who are doing cool stuff with data for their institutions and organizations.

Of particular interest are the Case Study sessions, which are brief (20 minutes) presentations of analytics projects that your colleagues at other institutions have carried out. If you’ve worked on a such a project, consider sharing! Contact information is included below.

Here’s Kate’s message:

Hello everyone!

Many of you may have noticed the fifth annual APRA Symposium on Data Analytics is definitely happening again this summer in conjunction with APRA’s International Conference in Minneapolis!  The dates are Wednesday and Thursday, August 1st and 2nd — some additional information is available here: http://www.aprahome.org/p/cm/ld/fid=72.

We don’t have the full schedule yet, but hopefully will within a week or so.  In the meantime, let me give you some preliminary details:

Wednesday morning the conference will open with a keynote from Rob Scott at MIT, who was instrumental in founding the Symposium, and has a bird’s-eye view of the history of analytics in fundraising, from the perspective of research, IT, front-line fundraising, and fundraising management.  Thursday morning, we will have the opportunity to join the larger conference to hear Penelope Burke, President of Cygnus Applied Research Inc., on Donor-Centered Fundraising.  http://www.aprahome.org/p/cm/ld/&fid=73

The fundamental track is intended as a two day introduction to analytics in fundraising, with the goal of giving participants a solid road map to approach their first project.  Topics will include: Various Variables: Data Preparation and Management for Successful Analytics, Walkthrough: Understanding the Problem and the Resources, Key Questions in Project Management, and Implementation.  Presenters will include Chuck McClenon at the University of Texas, James Cheng at Dana Farber Cancer Institute, Audrey Geoffroy at the University of Florida, and myself.  In addition, six short case studies from a variety of nonprofits will be presented in the fundamental track.

In the intermediate/advanced track, we will continue the focus on case study with nine short project presentations.  We will also have a presentation from Jeff Shuck of Event 360, who applies predictive modeling and segmentation to fundraising events and peer-to-peer fundraising programs.  Marianne Pelletier of Cornell University and Josh Birkholz of Bentz Whaley Flessner will present on constituent engagement.  Chuck McClenon of the University of Texas will lead a panel of practitioners to discuss the intricacies of collaborating with development IT.

Finally, we will have our usual faculty/committee panel to close the Symposium.  We will be asking our faculty, committee members, and a few guests to tell us about the one best idea they’ve heard recently in the area of development analytics, and follow up with a free-wheeling conversation including these ideas and any and all questions from the floor.

Last year we experimented with a case study format that gave us the opportunity to hear many of our colleagues present on projects they are working on at their institutions.  As you see above, with a few tweaks, we are continuing to set aside some time for case study this year.  If you’re planning to attend, I’m hoping some of you might have a project you’d be interested in presenting?  You will have 20 minutes to present a project of your choice and take a few questions.  Emma Hinke at Johns Hopkins has kindly agreed to handle the logistics of case studies for me, so if you have an idea, or would just like more information on the case studies, please be in touch with Emma at ehinke2@jhu.edu.  If we have a great flood of ideas, we may not be able to pack them all in, but wouldn’t that be a great problem to have?  Please send us your thoughts, and if we can’t manage them all this year, we’ll start a list for next year.

I do hope you will consider joining us — it’s the variety of attendees that makes the Symposium great.  I’ll let you know when we have the full schedule up on the Symposium web site.

Many thanks,

Kate Chamberlin
Chair, APRA Symposium on Data Analytics
Campaign Strategic Research Director, Memorial Sloan-Kettering Cancer Center

7 February 2012

Figuring out what those poll results mean

Filed under: Analytics, predictive analytics — Tags: , — kevinmacdonell @ 1:04 pm

Last week’s poll (Where’s your institution on the Culture of Analytics Ladder?) had all the things I dislike about quickie online polls, including the fact that the respondents were self-selected, and it was open to anyone who stumbled upon it, including people with no connection to the field.

I don’t bother with polls very often, but hey, it’s my blog and I’ll poll if I want to. People DO like polls, even if we should take the results with a grain of salt.

The poll is still open, so you can go back to view up-to-date results anytime, but here’s the breakdown of the 81 responses received as of today. My thoughts about the results continue below …

These results do not distinguish between non-profits and higher-education institutions, nor between small and large organizations. We don’t know if respondents are managers or staff, or even if they understand the question. So … take this for what it’s worth.

Less than a fifth of people who viewed last week’s post opted to cast a vote. Of those who did, it does seem they were honest about their answers. I am not surprised that almost a third of respondents admitted that although they have data, no one in their organization knows how to analyze it. That is deeply unfortunate but it’s a fact.

Coming a near second are people who said that their analyst’s data insights support decisions only on an ad hoc basis. (Answer 6 includes organizations who may have contracted out for predictive model score sets, so these organizations may not even have analytics talent on staff.) And in third place are the people who said data analysis is a regular process, not ad hoc, but that the benefits are limited to just part of the organization.

What might be encouraging here is that almost three-quarters of respondents have the data they need and are somewhere along the path of becoming data-driven, even if they aren’t quite there yet. On the other hand, these are CoolData readers, so keep the bias in mind.

Then there are the eight people who claim that analytics has been wholly embraced by their organizations, from top to bottom and in all operational areas. I’d love for those people to come forward and identify themselves, because we could all learn from them. Email me!

Older Posts »

The Silver is the New Black Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 1,044 other followers