Possibly by coincidence, this week’s Social Media Club question of the week is about measuring influence in social networking… and I just wrote a bit about that topic in the Web Analytics group:
On Mon, Jun 29, 2009 at 8:07 AM, Peter Kristof wrote:
Can anyone point me to some good resources (articles, blogs, tools, vendors,etc.) for research on measurement of social media / Web 2.0?
I invented some of the original buzz measurement stuff (now owned by Nielsen/Buzzmetrics) …
Analytics progress in this field is slow – it depends very much on understanding language, which is fundamentally lousy and not progressing very fast. There is enormous ambiguity in the behavior and text it measures, which shouldn’t come as a surprise to anyone in web analytics. Despite all the talk around sentiment and such, I’m still convinced that the most important metric is how many people are talking about a topic; any system that doesn’t focus on that is probably off the mark. No. 2. is how influential those people could be. I say “could be” because generally speaking, we only can identify influencers by their potential to influence (because they participate in a lot of discussions, across venues) than their actual influence. Finally, I always pay attention to how such systems summarize what’s going on in social networks. Two million postings and here are the 10 that are best representative – how did you pick those?
Beware of cool visualizations… any sort of self-organizing mapping of social media space usually will not to scale well. There are some very hard graph problems behind them. I think most vendors will admit that the eye candy is more useful for selling services than for delivering intelligence.
In terms of where things are going, I think we’re seeing more innovations in packaging and pricing, away from the big expensive solutions to smaller, lower-cost tools, rather than breakthroughs in the technology of measurement. I suspect that will remain true for a while, if only because social media itself is evolving so fast that what works today is likely to be obsolete soon.
Analytics progress in this field is slow – it depends very much on understanding language, which is fundamentally lousy and not progressing very fast. There is enormous ambiguity in the behavior and text it measures, which shouldn’t come as a surprise to anyone in web analytics. Despite all the talk around sentiment and such, I’m still convinced that the most important metric is how many people are talking about a topic; any system that doesn’t focus on that is probably off the mark. No. 2. is how influential those people could be. I say “could be” because generally speaking, we only can identify influencers by their potential to influence (because they participate in a lot of discussions, across venues) than their actual influence. Finally, I always pay attention to how such systems summarize what’s going on in social networks. Two million postings and here are the 10 that are best representative – how did you pick those?Beware of cool visualizations… any sort of self-organizing mapping of social media space usually will not to scale well. There are some very hard graph problems behind them. I think most vendors will admit that the eye candy is more useful for selling services than for delivering intelligence.In terms of where things are going, I think we’re seeing more innovations in packaging and pricing, away from the big expensive solutions to smaller, lower-cost tools, rather than breakthroughs in the technology of measurement. I suspect that will remain true for a while, if only because social media itself is evolving so fast that what works today is likely to be obsolete soon.Nick
The idea of portable reputations in digital media has been around for many years, but not much has happened. The idea is fairly simple – create a means of taking credibility (or the lack of it) from one digital community to another, so that you’re not starting from zero each time you join a new one. When Usenet and CompuServe were the only games in town, this didn’t really matter, but with the explosion of social media in the last few years, it seems like reputation portability is becoming more attractive and practical. There are big obstacles, starting with the definition of terms.
“Reputation” is in the family of vague terms like “engagement,” “influence” and “community.” They sound very impressive. “People ith high positive reputation scores generate community engagement” sounds like one of those phrases that expensive consultants toss around at industry conferences, loaded with implications and empty of specific meaning. Nevertheless, reputation means something, even if we can’t agree what it is. It has at least two components – a value and a valence. For example, honesty is part of reputation; when its valence is negative, we call it dishonesty. Of course, personal reputation in digital communities has more to do with accuracy, interesting-ness and opinion leadership than black-and-white issues like honesty.
Even if everybody agrees on what reputation can mean, the way it happens varies from one community to another. Take “following” relationships, for example. Following somebody on Facebook requires their permission, but not on Twitter. Similarly, the way Twitter is dominated by “open” accounts means that being quoted (retweeted) on Twitter is more likely to happen and therefore possibly less significant than elsewhere. Some communities have explicit voting and scoring systems that rank people, posts, pictures and so forth. Few mean the same things, especially when they are based on raw scores from communities of vastly different sizes. If nothing else, this means that nobody is going to come up with a definitive reputation ranking system, which is probably just fine. There are many possible dimensions to reputation and various purposes for it, so I would expect that it’s a good thing if many systems arise. I’m wondering if they will start to arise by way of the rising number of social media APIs.
Open APIs not only allow third parties to experiment with reputation scoring systems for each social network – for example, all the Twitter influence scoring systems – they allow third parties to try out reputation mashups, which implies some sort of reputation portability. FriendFeed, being a social network mashup itself, is the kind of service that enables this. Anybody who has claimed more than one social network identity on FriendFeed potentially could bring their reputation from one to the other, since the links on FriendFeed tell third parties that the two identities (a/k/a accounts or profiles) belong to the same person. For example, if I have a large following on Twitter, then sign up for Facebook, a third party could inform Facebook users that I’m worth following, even though I haven’t done a thing yet on Facebook. That capability becomes particularly interesting in terms of competition between social networking sites. When the next Twitter comes along, whatever that might be, people may be able to get deeply engaged in it faster because of Twitter’s open API… assuming the new guys also have an open API, of course. Some of this is already happening between partnering social media sites, where you are invited to bring your friends along. That’s only mildly interesting because it doesn’t happen between direct competitors. When the APIs generally support reputation-related data and third parties are the ones who make the marriages, so to speak, the world becomes very interesting. That’s not just because of what you can do, but also because of what it makes harder – spamming.
Spammers will have a much harder time in a system of shared reputation data precisely because the social media networks are somewhat different from each other. Spamming is harder on some, easier on others, so if somebody shows up only on the “easy” ones, that’s a strong clue that they are not legitimate. If my follower relationships on Twitter are people with whom I have some sort of genuine relationship, I would expect a high percentage of them to be present on other social networks. Open APIs let third parties measure the differences and make some estimates of the likelihood of legitimacy. This feels to me something like the kind of robustness that arises from genetic diversity – yes, you might be able to conquer an individual or two, but you probably can’t beat the whole ecosystem at its own game.
One thing that becomes harder in this environment is creation of multiple, distinct identities. Some will argue that people want to keep their personal, work and perhaps entertainment identities independent of one another. The idea is that if you get your jollies in some socially embarrassing manner, you don’t want your boss or potential employer to find out. Or, more legitimately, perhaps you are part of a 12-step program and you want to participate anonymously, or more correctly, pseudonymously, since you’ll want to create and use a pseudonym for that purpose. I’ll respond to the first idea with a motto that is used in recovery – “you’re only as sick as your secrets.” The more I consider the idea of having distinct, private identities for work, personal life and whatever else you think you need them for, the more I think it is rubbish. Unless you’re spying for the CIA, there really isn’t much need to compartmentalize your life that way. And I mean “compartmentalize” in the bad way, really.
I rarely truly believe anything I write like this until I’ve seen some data, so my next step will be to explore some of the APIs to see what I can tell about myself and others through multiple APIs. I’m hoping that will be the subject of a blog post in a few days.
One of the most common naive errors in statistics is to confuse correlation with causality. Common sense tries to tell us that when two events co-occur, the first one is causing the second to happen (which often is the case). Red traffic lights correlate to cars stopping and sure enough, we know that red lights cause (most) people to stop. But sometimes things correlate because a third, external mechanism is influencing them. Drownings increase as ice cream sales rise, but ice cream isn’t causing drownings. The external factor is summer, of course.
This is on my mind because over the last few days, when cold medicine hasn’t fogged my brain up so much that I couldn’t think, or at least couldn’t think logically, I’ve been working with the Twitter APIs to see what I could come up with in terms of tracking topics as they move around on Twitter. I’m attracted to Twitter because its immediacy and brevity make it relatively easy to analyze.
Eventually, what I hope to do is find useful patterns in the interplay of words, Twitter screen names, URLs cited and hashtags (and any other entities that could be extracted). I’m focusing first on URLs, since they are sort of the “stories behind the headlines” on Twitter. My friend Dave Land this morning mentioned that writing a tweet is like writing a headline. If so, then the cited URLs in those tweets are like the stories behind the headlines.
I’ve put Python and SQL to work scraping statuses from Twitter, pulling out word pairs (I’m planning to analyze them with the other entities via LSA), screen names and URLs. I’m resolving all the little URLs to the pages they actually point to, since Twitter users, limited to 140 characters, frequently use services like TinyURL to shorten them, but I want to see when people are citing the same URLs even if the shrunken URLs are different. In fact, looking at the ratio of shrunken URLs to actual URLs is interesting – if it is high, that means that a lot of people are finding the cited page independently, rather than retweeting it or getting it from the same source external to Twitter.
As I find cited URLs, I’m using Twitter’s search API to get the most recent mentions of them, then storing the identities of the users who also cited them and when they did so. That gives me a timeline of URL citations. I’m not tracking explicit retweets, so I don’t know if the first people to cite a URL first are more influential or not.
I haven’t asked Twitter to white-list me yet, so I’m working within the limitations of their API – 100 requests per hour. That forces me to be as smart as possible about how my code explores the data. I started by choosing somebody who has a decent number of followers, but not too many, so that it wouldn’t take too long to scrape the person’s followers’ tweets. I chose Tim O’Reilly because I suspect he is fairly influential on Twitter and we’ve had some conversations that go back to the mid-90s about how to figure out “what the Internet is thinking today.”
O’Reilly’s company was one of the first, if not the very first, to measure social media for market research. Many years ago, they were scraping Usenet to help decide which technologies would make good topics for books. I recall that one of the first decisions they made from that data was to choose between two open-source database projects, MySQL and mSQL. They chose MySQL… and therein lies another reminder of causality v. correlation. Did MySQL succeed because O’Reilly chose to focus on it, or did O’Reilly succeed because it chose the right books to publish? There is no way of knowing, but I have personal evidence that O’Reilly doesn’t always choose the right topics… or perhaps the right authors. That’s a story for another day.
After a lot of wrangling with Python, MySQL and technical issues having to do with Unicode and my inability to write a correlated subquery under the influence of Sudafed, I have something working. It started by scraping Tim’s recent tweets and then searched for people who also cited the same URLs. Then it explores those who it has found cite the greatest number of URLs overall. It adds 10 people at a time and then re-ranks to see who it should explore next.
I’ve been running this for a couple of days now in various forms. So far I have found about 7,000 unique URLs. Only about 300 of them are duplicates – different shrunken URLs for the same page. The URLs have been mentioned about 20,000 times by 7,500 users. I have found about 40,000 two-word phrases (stop words, URLs, screen names and hashtags are excluded) and 52,000 mentions of those phrases, which means that a number of phrases are being used by multiple people.
What the heck, here are the top phrases and the number of Twitters (Twitterers? Twits?) who used them over the last few days (remember, this is far from comprehensive):
I suspect that the words “check out” on Twitter are much like the words “click here” were in the early days of the web. ”Emergency generator” intrigued me, so I linked it above to Twitter search. Hint: it has to do with the Toyota Prius. I suspect those same people included a link to a New York Times article about it. Interestingly, a number of the people who cited it were not retweeting (at least not explicitly)… but many of them were using a shrunken URL cited by – guess who – Tim O’Reilly. The interesting thing about the popularity of the phrase is that it gives my code a way to discover the other shrunken URLs in a single search, instead of having to scrape everything and resolve every shrunken URL to the actual page. Tim may or may not have influenced those people to look at the article, but it is clear that he is in tune with a topic that people are interested in, which makes him interesting, whether he is an influencer or just well-influenced, so to speak.
Time to publish this post, I guess, even though I’m tempted to wait until today’s cold medicine has worn off to proofread it one more time.
More results here as I come up with them.
Tags: causation, correlation, Influence, twitter
Social media proponents (a/k/a people who want your money for social media products and services) urge us many ways, but it all boils down to “People are talking about you on the Internet, so you’d better pay attention.” Tapping into the swelling ground, grabbing a long tail or otherwise engaging in social media is supposed to help you make better products, faster, leading to happier customers and more money. But how do you decide who to listen to?
Success generates noise. Millions of customers means millions of comments. The first and easiest answer to this dilemma, which may be good enough – for now – is to figure out which comments are most popular. See which ones are getting the most page views, the most Diggs, Tweets or other indicators that somebody cares.
The problem with that approach is that by the time something is popular, it is often too late do so anything about it. This is particularly true in two situations:
If you have a hit on your hands and you find out too late to create more inventory before the crowd has moved on, you missed an opportunity. There’s a corollary to this: the sooner you find out you have a dud, the faster you can stop wasting resources on it.
The really unfortunate thing about angry and unhappy people is that they consistently have more energy to invest in bad-mouthing than happy people have for paying compliments. That’s a well-known fact of marketing. And life. By the time grumpiness about you and your stuff becomes popular, a lot of damage has been done, obviously. Knowing who is influential can help you prevent grumpiness in the first place or do better at quelling it before it becomes popular opinion.
In mass media, the important metrics focus on popularity. Although popularity still matters, digital social networking allows us to measure influence, at least indirectly. The difference between being popular and being influential is very simple to understand in principle.
If I have 5,000 followers on Twitter, I’m obviously fairly popular. I probably am also influential. There are ways to figure that out, such as by measuring how much interaction I engage in or how many times my tweets are “re-tweeted.”
If I have one follower, does that mean I am not influential? Not if that one follower is a fellow named Barack Obama, who has more than 150,000 followers, according to Twitterholic. That is, if that Obama fellow really is following me. I mean really following me, the way we mean “follow” in the real world.
In other words, a few influential followers can be far more significant than thousands with limited influence.
I’m using Twitter as an example because I’ve been working with the Twitter APIs to see how hard it would be to come up with measures of influence. Twitter is growing on me and I think that part of the reason is the language it uses.
From a social network analysis standpoint, Twitter is much easier to deal with. Mostly.
Popularity is a first-order measurement. My popularity on Twitter is the number of followers I have. Influence is a second-order or greater measurement. The simplest measure of potential influence is to see how many followers my followers have. The API makes this very easy to measure (for people who aren’t so popular that the API limits become an obstacle).
My Twitter followers are followed by a bit more than 10,000 people. Pretty good, I think, since I only have 28 followers (I haven’t been on Twitter long).
My friend Dave Land has 95 followers and those people are followed by almost 175,000 others. Wow. Dave’s followers are followed by a lot more people than mine are.
Some of my followers are people I believe are influential in the world of web analytics. Let’s see how they do (a third-order measurement of my potential influence, if I did it for all of them). In no particular order:
Avinash Kaushik, Google’s web analytics evangelist, isn’t following me (hey, bub!), but anybody whose title is “evangelist” is supposed to be influential. At the risk of exceeding the Twitter API limits, I ran my gizmo to get his stats. Avinash has about 2,000 followers, who are followed by almost 600,000 others.
If you rank these people by popularity (followers), Avinash is No. 1, hands-down. But if you rank by potential influence, Marshall Sponder’s followers are followed by the most people, which is especially surprising given that Avinash appears to be more than twice as popular.
Dave Land comes in at No. 1 when this group is ranked by the ratio of second-order followers to followers. That means he is doing the best job of attracting followers who attract followers, which is what you need to do if you want your influence to scale beyond immediate popularity. But I should note that having a lot of followers will inevitably dilute your second-order influence, which should comfort Avinash, who came in last on that measurement (thereby saving me from last place in all three rankings).
I should note a messy bit of this measurement – sites like Woot, Twemes, hashtags.org and others that automatically follow you when you follow them. Ugh. I haven’t figured out a good way to exclude them, so I’m just doing it manually… and I haven’t thoroughly made sure I caught all of them. So there’s hope, Avinash – maybe Marshall is just signed up for more of those. In any case, don’t take these numbers too seriously. I’m going to work on some additional data points – number of replies and such, to strengthen the results.
Or can somebody save me this work and point to a site that has already done this sort of analysis? I searched but didn’t see anybody looking at second-order popularity.