“Unfortunately, we have had some bad experiences with some who believed they were here to teach or lecture. I would like to make it clear that while we welcome you and your medical expertise as well as your input, we are not interested in a forum where questions are asked and the medical visitors are quick with a medical answer. Such a forum has a tendency to stifle input from those who need to express themselves the most. “

Fascinating – people who are quick to provide medical answers tend to stifle the conversation.  This makes no sense in the context of technical support, but it makes perfect sense in crisis intervention. When people go through a trauma, it can be counter-productive to try to make things better.  Okay, that probably sounds crazy, but the reality is that trying to “fix” what’s wrong often leaves people feeling unsupported even when it is the “right” answer.  What people in crisis also need is a safe place to tell their story, which is very much what crisis intervention is about.  ”Safe,” in this context, means that their story will be accepted as it is, with no criticism or quick fix.  Trauma changes people; medicine doesn’t change them back.  Instead, they need to figure out how to live with their illness, which quick fixes ignore.

So yes, it makes sense that in a support community whose primary purpose is human, not technical, support, the kind of response that is totally appropriate in a technical support community is wrong. Here’s the short version of what I believe this implies: “new” social media – Facebook, Twitter, blog comments – are lousy platforms for emotional support, because they are really not good for story-sharing, at least not the kind of sharing that people do in support communities for serious illnesses and other trauma.  Web forums, mailing lists and Usenet (is Usenet still working?  haven’t looked in a while) are much more appropriate.  Twitter and Facebook are fine for sharing factual tidbits, pictures and videos and so forth, but not for holding conversations that really help people get to know each other in a way that is meaningful to the challenges of illness, etc.  Facebook, Twitter, etc., are much more about digitally enabling your existing relationships, not creating new and deep ones.

The new social media also fail to offer the kind of desirable compartmentalization that a support community does.  The default behavior is to post items in public.  That doesn’t create a safe environment for telling your story, if only because many of those friends and acquaintances just can’t relate to your need for a supportive community.  If the topic is my cancer (hypothetically, I’m not sick, thank heaven), I want to talk to other cancer victims and survivors, perhaps some medical professionals and allied health people… but not everybody I know.  I may not want my illness to be a complete secret, but there are likely to be people who I’d rather not share it with, even though they are “friends” in social media.

The next time somebody talks to me about a “support” community, I’m going to be thinking, is this a community where people want to tell their stories to get a quick answer, or is is it a one where they mainly want be in a place where the others know what they’re experiencing.

There are a bunch of companies that offer social media monitoring services that claim to “monitor the conversation” that social media has begun to enable among organizations and their clients, customers and other constituents.  A huge shortcoming of them, as I see it, is that they really are little more than digital clipping services.  All they find is what is indexed by search engines and when they find it, they are looking at web pages, not conversations.  They miss a lot.

They completely miss the conversations that take place on mailing lists and in web forums that aren’t friendly to search engine crawlers.  When they do find digital conversations, they don’t analyze them structurally to identify individual posts, replies, timestamps and so forth.  This means that they aren’t really getting a valid picture of how many people are talking (perhaps the most important metric), how many topics there are, how many people are engaged in each topic and other metrics that are only possible when analysis looks beyond the single web page.  Of course, they don’t do this because it is hard!

It is probably an impossible task to keep up with all of the digital conversation platforms so that this type of analysis could be as thorough as Google’s web page indexing.  There are dozens of open-source and commercial web community forum platforms.  Some are easier to analyze than others; some are probably totally impractical to automatically analyze.  Some are deliberately quite private and invisible.  Still, I have no doubt that conversation analysis is worthwhile, if only to solve the fundamental problem of listening to an enormous group of people whose concerns and viewpoints matter to you – your customers, if no one else.

Most of my work over the last nine years, aside from detours into necessary infrastructure building, has been focused on this challenge – how do you summarize millions of postings in digital communities?  How do you summarize the topics under discussion, how those topics are changing over time, how many people care about those topics and so forth.  Automatic summarization has never made an intuitive leap, so part of the summarization challenge is to automatically select a tractable number of typical phrases, sentences or postings for a human being to read, a selection that should enable the kind of leap of recognition that a computer cannot.

Anything “social” is about relationships.  My goal continues to be to use technology to discover what I can about the relationships among people, brands, companies, events and so forth.   Along those lines, I’ll be writing some more blog posts in the next few days.  One will be on the difference between customer support and technical support.  Another will take a look at how a conversation between a large organization and its constituents might be structured – how today’s (and yesterday’s) social media fit together.

The cool thing about Twitter is how open it is, providing plenty of data for those of us who imagine we can discover value in the ways that people are interacting.  The less cool thing about Twitter is that although it is popular, it is far from the only fish in the sea.  Personally, I get a lot of use out of Facebook, LinkedIn and sometimes Plaxo.  And I’m fairly certain that MySpace is doing okay, too.  Let’s not forget Digg, either.  And, I have realized for the umpteenth time, the web itself.  That last source of social network data – the web – came back to my attention as I started taking a look at a long list of social networking APIs on Programmable Web.  Wow.  I had no idea.  Some are definitely more interesting than others.

The one that really caught my attention was Google’s Social Graph API, which returns data about publicly identified connections among people on the Web.   It seems particularly useful when querying sites that people use for aggregating their web presence, such as FriendFeed.  For example, query my FriendFeed page and you’ll get links to this blog and other places I write and publish.  A query on my Facebook page doesn’t yield anything interesting other than the link to my FriendFeed page, which shows the difference between a quite closed system and an open one.

These APIs are itneresting because I’m always curious to see if I can quickly create and maintain smart data exploration robots, who can find useful and interesting information without trying to boil the ocean.  Twitter should be just one source in a generalized framework that leverages other peoples’ code and data as much as possible.  In fact, I think I’ll probably slow down directly querying the Twitter API in favor of relying on several of the URL tracking sites, such as Tweetmeme and Twitturly.  My main concern about that approach is that I don’t know how to rate the risk that such services will continue to exist… and accessing them via their API doesn’t seem to do anything to help them stay in business.  That’s a bit of a conundrum, but the only answer I’ve come up with is to avoid reliance on any one of them, just as I don’t think it would be a good idea to focus entirely on just one social network.

As I take a look at the various APIs, I’m trying to block out a software framework that will take advantage of these and future data sources to help me spot interesting themes, trends and so forth.  That’s been my goal all along… without boiling the ocean in the process.

The fact that for the most part, anybody can follow anybody, along with all the auto-followbacks, makes for a very densely connected graph.  For example, I follow somewhere over 300 people.  They follow several hundred thousand.  The graph that shows the follower relationships of me and all the people I follow has about 1.4 million edges.  That’s a lot to manipulate.  I’ve experimented with removing the people who follow more than 500 or more than 1,000 people, which reduces the graph size considerably, but it is still challenging to analyze on a desktop system.

I think my next blog post will focus on the social graph, why it matters and what direction Twitter might take it.  Feel free to ping me with your thoughts; I’m eager to hear them.

I should mention that even as I’m doing all this, I’m still looking for the right “real” job.  I’m focusing mainly on product management related to social media and analytics.

I think I finally found the right phrase when I updated Tweetsnet’s “About” page to say that it looks for people who are “in tune” with what becomes popular.  I see Twitter as a platform where people constantly organize themselves into choruses, amplifying the most pleasing melodies, generating and discovering harmonious ideas.  As with flocking behavior, these choruses have no single leader, but unlike a flock (as far as I know), some people are clearly more “in tune” than others.  Those are the people Tweetsnet seeks to identify – those who most frequently cite web pages that become popular.

I suspect there is a great opportunity in reporting what the choruses, known and discovered, are singing about.  In other words, monitoring the buzz in sections of an ecosystem of interacting, overlapping shared-interest communities.  This is where I want to take Tweetsnet, generating verticals, starting with known popular subject areas such as social media.  I’m sure there’s a lot of thinking and experimentation to be done about the ways we could define the intertwined borders of such choruses.  One thing I’m sure about – we need to change the way we tend think about redundancy.

Our left brains tend to think that duplicated effort is inherently wasteful, but the fact is that we are creatures of community.   But here’s the most important idea to take away from the “chorus” metaphor: when a bunch of people act similarly in social media (e.g., post the same URL), it is not redundant, it usually adds value.  That is deeply contrary to the one-to-many 20th century idea of information distribution, in which achieving stardom, not harmony, was the goal.  We still have room for stars, but some of them will be choruses.

Here are some thoughts on features that contribute to Twitter’s “choral value.”

• Retweeting has very choral  high value, as it strengthens the “melody” – people’s deliberate arrangement of information into tweets – and the “harmony” – the commonality among Twitter users that goes beyond simply posting the same information.
• UI design that makes retweeting easy is good, as long as it doesn’t encourage people to spew everything.
• Excessive tweeting and retweeting becomes noise – witness the efforts I’ve had to make to remove aggregators from Tweetsnet.  The greatest value is added by the “jazz” tweeters, who have a melody and know how to harmonize, but aren’t afraid to improvise.  In other words, have a focus, but don’t be a robot about it.
• Anything that shows how much human energy and thought went into a tweet adds value.  Anything that makes it easy to tweet will eventually diminish the value.  This is why the 140-character limit has added value – even headlines often are bigger, forcing people to think about how to squeeze information.
• Hashtags reduce Twitter’s choral value as  ”solos” they do more to discourage than encourage retweeting.  If a tweet is already tagged, I think people tend to assume there’s no need to retweet because interested people should be monitoring the hashtag.
• Followering somebody only matters if you take action; the main visible action is retweeting.  Blogging about something you found on Twitter would add “choral” value if there were an easy way to discover it.
• Twitter’s APIs make it fairly easy to track user, URL and word usage, which is good data not just for Twitter’s basic features, but for discovering things we didn’t know to look for.  It’s great that everything is open by default, unlike most other social networking platforms.

What’s the “choral value” you see in Twitter?  What could the company do to further encourage it?

P.S. I’m going to change Tweetsnet’s name to TwURLedNews.

I’m happy that there’s increasing evidence that Tweetsnet is useful.  On the other hand, what a strange world this is, in which I can create an automated information source that seems, by one metric, to be more popular than I am.  It seems impersonal and perhaps just plain silly… until I consider that we are creating a world in which increasingly intelligent robots will interact not just with us, but with each other, which will make them (a) stupider, because they will have to deal with rapidly increasing amounts of data and (b) smarter, because we will figure out how to make them take advantage of all that data.

If you’ve been following Tweetsnet or this blog for the last few days, you know that my No. 1 strategic problem (as opposed to various little bugs) is the fact that aggregators – other robots – tend to score quite high in the rankings.  An idealistic part of me wants every Twitter account to self-identify as robot or human… but I know that there’s no hope of compliance with anything like that.  I’m actually more intrigued by the notion that value will arise from writing code that guesses whether or not a user is a robot.  Web analytics has the same problem because some web robots and spiders masquerade as ordinary web browsers.  I spent a lot of time on this problem at LiveWorld, where some of our customers were not too eager to pay for robot page views at the same rate as human page views.

The cool thing about the challenge of distinguishing bots from humans is that we’re essentially collaborating and competing on Turing tests.  People are designing bots to gain influence in the Internet’s social networks, in competition with people who want to filter them out.  As long as bots are dumber than people (and they will be for a long time), this competition will persist and it will drive collaborations that make software smarter.  When we reach the singularity, it will stop mattering… or perhaps it will completely flip, so that the people who were trying to decrease the influence of stupid bots will focus on decreasing the influence of those stupid humans.  Or perhaps it will be a happy collaboration.

Tweetsnet gained its first bunch of followers by following everybody who cited a URL that made it into the feed.  A lot of those people automatically followed it in return.  The recent big spike appears to be driven by the fact that a few Twitter users are now retweeting Tweetsnet items.   That’s a kindness, really, because there’s no reason for them to do so.  They could retweet one of the original tweets.  

I imagine that one reason they give Tweetsnet the credit, so to speak, is that Tweetsnet doesn’t try to drive traffic to itself.  When it posts a tweet, the links in that tweet point directly to the original site, not back to the posting on Tweetsnet.  I get annoyed by tweets that point me to somebody’s site that does nothing more (for me) than provide a link to the site the tweet was really about.  

Meanwhile, today’s project is to keep other peoples’ robots out of the Tweetsnet scoring – because they are stupid.  The robots, I mean, the robots.

Each post shows the page title, summary and keywords (as tags) if available, and frequent two-word phrases that appear in conjunction with the page citations.

It’s still beta and I’m still deciding where to go with it. Your thoughts, etc., are more than welcome.

I’m considering similar feeds with a vertical focus. I’m also thinking of splitting out the pages that are cited by the big, popular aggregators, since they’re already well-known.

A lot of what is showing up now is news, so I’m also wondering if I can automate a comparison to something like Google News to see what the differences are.

Today, I’m looking at “centrality,” a typical social network metric.   I am interested in degree centrality, which looks at how many connections a person has, which shows who the key players are.  I’m considering two people to be connected if they cited the same URL in the same time frame, regardless of whether or not one was an explicit retweet of the other.  Later, I’ll probably weight the connections with explicit retweet and other data.  For now, I want to see if follower count, a far simpler metric than centrality, would work just as well.  Here is a log-log scatterplot of degree centrality v.  follower count.

Follower count v. degree centrality

The data points are scattered all over the place, which means that follower count does not correlate to the connections revealed by citing the same URLs.  I’m not surprised, given all the games people play to get followers, the robots and such that have little or any human thought behind them. 

As a reality check, let’s look at a similar plot that compares follower count to user mentions.  I would expect that people who have a lot of followers will be mentioned (in the form of @screen name, in a reply, retweet or any other context) more often.  Here’s the graph. 

Followers v. mentions

Bear in mind that my data gatherer is biased toward people who cite a lot of URLs, so when I say count mentions, those are mentions by people who tend to cite a lot of URLs in their posts.   As you can see, although there are many outliers, there is an obvious trend upward and to the right, which indicates a positive correlation – people with a lot of followers indeed do tend to be mentioned a lot.  The upper left area is almost empty because it is hard to get any mentions when you don’t have any followers.  On the other hand, you can have lots of followers and few mentions, which is why the there are more points toward the lower right.

Outliers are often interesting and I find myself wondering who is getting a lot of mentions even though they have very few followers.  The dot closest to the upper left corner is MsTweet, who is a “customer service evangelist for Mr.Tweet” and therefore doesn’t follow much of anyone, but gets mentioned a lot.  In the upper right border area, with lots of followers and mentions, are Shorty Awards, Chris Brogan, Guy Kawasaki, and ReTweetTrends (in the center of the top, not following nearly as many as the others).  The lower right corner outliers are people who are heavily followed, but rarely mentioned by people who cite URLs.  They include Kevin Rose, Jason Calacanis, Veronica and iJustine.  I’m surprised, actually, that these folks’ huge followings apparently either aren’t mentioning them often or aren’t often citing URLs.  Let’s reality-check that with Twitter search.

I’ll search on each of their user names, then repeat the search with their name and “http,” which will give a rough comparison of all mentions v. mentions with URLs in them.  Twitter’s search doesn’t give a result count, so it’s pretty hard to tell.  All I can go by is the frequency of recent tweets.  Let’s compare it to somebody who is mentioned a lot – Chris Brogan.  He is definitely getting a lot more frequent mentions in conjunction with URLs, so at first glance, the data seems believable.

Perhaps this indicates that the people with big followings yet few mentions have a different kind of influence.  People like Chris and Guy seem to be leading others to look outside of Twitter, while Kevin, Jason, Veronica and Justine have some other, perhaps more Twitter-centric influence.  Is it safe to say that the latter group is more engaged with Twitter for its own sake?  

It seems that some of the popular Twitterers are leading their followers mostly into Twitter navel-gazing, while others are leading people beyond what Twitter itself has to offer.  I find myself wondering how this might change as Twitter matures… and wondering if perhaps the navel-gazers are newer to Twitter and will get bored faster.  I’m gathering more of the user information now, so I should be able to compare the average number of days they have been using it.  In any event, from a business standpoint, I think I know which kind of leader I’d be more interested in.

Who is it that says that web analytics data is always messy? Whoever it is, right you are! Since a fundamental goal of the work I’m doing is to uncover interesting points of view, I need to downgrade sources that aren’t behaving as though they really have a point of view (or at least an intelligent one). I can tell instantly that I’m almost certainly looking at an automated system when I see that the “user” in question follows zero or very few people. That’s grounds for immediately downgrading. I’m not sure if I want to downgrade based on the volume of postings. Certainly beyond a believable number… and perhaps if every single post contains a URL.

Here are the top 20 sources from the last week or so, based on the criteria I described above.

1. Net2 (878)
2. techupdates (706)
3. OriginalSignal (587)
4. radi8 (565)
5. Dakshinamurti (542)
6. GaryTheGeek (453)
7. techupdate (449)
8. haripakorss (436)
9. readmashcrunch (392)
10. twittfeed (379)
11. TwitLinksRSS (359)
12. top_post (342)
13. tclauss (329)
14. TechFeed (303)
15. tc2tw (300)
16. vcsangels (295)
17. dlbrown06 (287)
18. davidsim (279)
19. mashable (272)
20. ReTweetTrends (268)
21. balduaashish (268)
22. wiredgnome (264)
23. julieti (259)
24. TechRSS (248)
25. davekresta_rss (246)

Tip 4: Follow everyone who follows you. When I first started on Twitter, Robert Scoble told me to follow everyone who followed me. “But why, Robert, would I follow everyone like that?” The answer is that it’s courteous to do so and because when you do, some people will respond to you and eveyone who follows them will see this—which is more exposure for you.

Having said this, when you get to more than fifty or so followers, it’s impossible to read what all your followers tweet. At that point, you have to focus on direct private messages (“Ds”) and direct public messages (“@s””).

Yipes. 

The first analysis I did on Twitter was to count the followers of followers, as an illustration of how influence is a more-than-first-order phenomenon.  The number of followers your followers have almost surely correlates to your potential influence.  People like Guy are just about impossible to measure that way – they have so many followers that it is impractical to count their followers’ followers.

When I first looked at Twitterholic and saw how many followers Guy has, I thought “How the heck does anybody follow that many people?”   That question is answered – he doesn’t.  It is possible that Guy decided to do me a favor (along with everybody else who looks at implicit, in addition to explicit, social networks).  Guy knows that this is where I’ve focused for years… but, okay, that’s probably not why he advises people (who want a lot of followers) to follow everybody who follows them.  Still, it is good news for me.

The pattern of followers is Twitter’s explicit social network.  As soon as I started analyzing it I was stymied a bit by robotic auto-followers.  They play havoc with metrics that depend on the follower-followee relationship.  My friend Dave suggested that Twitter might need user-agent meta data, which would reveal whether or not a given Twitter user is a real person or not.  This would allow software to omit users like hashtags, twemes, AmazonGoldDeals, mrtweet and so forth, all of which automatically follow you if you follow them.  But it wouldn’t solve the problem as long as there are people like Guy with robot-like behavior, automatically following everybody who follows them.  I’m fairly certain that  Guy really is not a robot – even though, in addition to tip No. 4, he advocates somewhat mindless direct replies:

Tip 2: Send @ messages to the smores. They probably won’t answer you, but that’s okay. All you want to do is appear like you have a relationship with them to enhance your credibility. The theory is, “If she is tweeting with @scobleizeer, she must be worth following.” Bull shiitake logic, admittedly, but it helps. To bastardize what a famous PR person once told me, “It’s not who you know. It’s who appears to know you.”

That tip guarantees that I’ll be sending Guy an @ message about the tweet that points to this post.  But I actually do know him, unlike the 10 or 15 other people on Twitter who don’t, who will @ message him anyway, further confounding those who naively analyze the explicit social network by looking at @ message relationships.

There’s nothing wrong with analyzing the explicit social network, but it is a big mistake to trust the results by themselves.  So I focus on the real meat in Twitter, figuring out as much as possible which items are getting real human energy into them and what they imply about relationships.  When a hundred people post links to the same URL, odds are that the page they’re tweeting really is meaningful and those people are likely to influence each other in the Twitterverse, especially if they used the same “shrunken” URL.  Throw in screen names, hash tags and language patterns and perhaps something truly useful and meaningful will come out.  I hope so.

As long as Guy doesn’t start advocating retweeting the “smores” tweets, I’m probably okay.  So far, all he’s done in that direction is to tell people to repeat their own tweets.

If Guy is right that it is a good idea to follow everybody who follows you, then I must be right to insist that any analysis that depends only on the explicit social network is inherently flawed.  Now I just have to decide if I really want to follow, not just Guy’s advice, but everybody who follows me. A voice in my head is saying, “If everybody jumped off a cliff…”