CoolData blog

18 April 2011

Do predictive scores get stale?

Filed under: Best practices, Model building, predictive modeling — Tags: — kevinmacdonell @ 11:46 am

This question was posed recently on a listserv I subscribe to. Wealth screening data does become outdated, the questioner noted, but what about modeling scores? Constituent giving and age change over time, but does that matter? Maybe it depends on what model you use and the variables that go into it?

The general answer is yes, scores from a predictive model do get “stale,” although they may not become merely outdated in the same way that wealth screening data does. They may become less relevant over time. Or, rather, they may not be as effective as they might be, lacking the benefit of recent data.

Given the volume of interactions and transactions that might occur over a year (gifts, events attended, surveys responded to, etc.), it’s probably a good idea to have fresh scores every year, or two years at most. Changes in the age of constituents is not as much a factor, because all constituents age at the same rate — which is not true of changes in behaviours such as giving.

I think the questioner is right in thinking the type of model is a factor. Certain models could be more sensitive to the passage of time, such as any model trained on relatively sparse data — a major gift propensity model, for example. A year could make a difference, and that’s important if you’re needing new names to feed a major giving or planned giving pipeline. If you require scores for your most recent constituents (say, young alumni, in an Annual Fund donor-acquisition model), that too would be a reason to build a model yearly — unless new constituents are automatically scored using an algorithm developed in a previous model.

Institutions with in-house capability are at an advantage because they can build models continuously — each iteration of the model will lead to improvements (one hopes) — at an attractive cost per model. As the saying goes, every model is wrong, but some are useful. Re-scoring constituents is as much about getting it “less wrong” as it is about being up to date.

For me, summertime has always been model-building season. The latest graduates have been loaded into the alumni database, the scores will be needed for Annual Giving appeals in September, and it’s worked out that summers have been less hectic than other times of year.

In general, though, for institutions with in-house capability, there is only one sure time to create a new model: Whenever there’s a business problem that requires one.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: