26 Nov 08 Engagement R&R

People argue for complex, customizable engagement metrics. Unfortunately, they often fail to specify what kind of engagement they are measuring. Brand engagement? Web site engagement? Social media engagement? These are different, but people unfortunately tend to lump them together, robbing metrics of significance. Almost everybody I see talking about engagement also fails to distinguish positive and negative engagement, which results in even greater vagueness. These problems are sometimes rendered invisible through complexity – an engagement formula that has lots of variables to play with surely it must be better than one that relies on a few.

I approach engagement metrics differently, partly because I constantly remind myself of two things:

• Metrics are only a proxy for the energy that goes into visitors’ behavior. What comes out of the analytics system is not necessarily what went it.
• Simpler is generally better because many metrics correlate very well to each other; greater complexity yields greater ambiguity.

The research I’ve done on community engagement suggests that the most important types of metrics are retention and responsiveness – my R&R of engagment. I look at those metrics by segment, but that’s about as complex as it gets. When I examine formulas with greater complexity, I always find that some of the variables are essentially measuring the same thing (page views and time on site, for example) or they reflect poor thinking about what is really being measured. At the very least, it is worthwhile to measure each variable’s contribution, via principle components analysis or a similar multivariate analytic method. If they don’t contribute significantly, you’re just wasting time and resources on them.

Retention measures how often people come back. Responsiveness measures how much more they do than just passive page viewing. In the context of social media, retention metrics look at how often people return to a site, a category, a forum or even a single discussion thread. Responsiveness refers to behaviors like posting messages or comments, voting in a poll, updating a profile, uploading content and so forth. There is a great temptation to weight those actions. E.g., posting a message is worth five points, but voting in a poll is worth only one. I have generally resisted this, mostly because I haven’t see a significant difference in the results, but also because of the way that I think about what’s being measured. It’s sort of a physics problem.

What we really want to know is how much energy visitors are putting into their participation. For example, writing a one-word reply to a message doesn’t take much energy. Finding a relevant web page and pasting its URL into a posting with some original comments takes a lot more. Sometimes a lot of energy is invested in a very small action. The ambiguity that is most important to recognize is valence – the fact that the energy people invest in social media can be positive or negative.

Valence is especially important in neutral venues (versus fan clubs) and when opinions run high. A simple example: during the last U.S. presidential election, Obama supporters may have been highly engaged in McCain venues and vice versa. Without a measure of valence, all you could say is that those people were engaged in the social media venue and the election, not in specific candidates. In other words, it is fine that we don’t measure valence as long as we remember what that means about the numbers we produce.

The same issue is present in influence measurement (which is closely related – you can’t be influential unless you are engaged). To an Obama supporter, McCain was highly influential, but with a negative valence. It is a well-known fact of marketing that people engage with things that they dislike and disagree with – frequently! We all have a bit of the talk radio host inside.

In short, it is critical to keep in mind that when we measure engagement, the action we measure may not accurately reflect the energy that went into it and that we almost always include visitors who are engaged as cheerleaders and critics. Keeping those two ideas in mind will lead to better metrics and interpretation.

Tags: engagement, influence, pca, valence

Tags: Engagement, responsiveness, retention