04 Dec 08 The social media data warehouse

I promised an overview of how data moves into and out of a data warehouse, so here goes.  The short version of “data in” is that there are ETL (extract, transform and load) processes that get data from various sources, change into a structure suitable for the data warehouse, then load it into tables in the database.  The way data comes out is via SQL queries.  Nothing really unusual there, so let me explain how these are different from typical databases.

In a dimensional data warehouse, the data is extremely “denormalized,” meaning that instead of designing tables to eliminate redundancy and maximize integrity, they are designed to be extremely simple.  Ideally, there are only two types of tables – facts and dimensions – and every query joins fact tables to dimensions.  This is called a “star” schema.  Imagine a fact table at the center with dimensions as points of the star.  A typical web analytics fact table is a clickstream log; associated dimensions might be days, users, visitors, ip addresses and so forth.  That’s it in a nutshell.  Grab a book about data warehousing if you want details (and there are plenty) but I’m going to focus on some of the ways that a data warehouse for social media might be different.  There’s a lot to say about this, so this just a first post in a series.

If I were starting today, I think I would completely violate one of the principles of data warehousing and plan for the social media data warehouse to be on-line, a production system, rather than keeping it isolated.  The normal model is that the data warehouse only serves reporting and analytics needs of managers, clients, etc..  However, “analytics needs” are becoming part of the user experience.

We are so accustomed to thinking of analytics and reporting as a management tool, a way to keep clients and advertisers happy, that we forget that our communities can benefit from analytics, too.  Increasingly, when we discover something interesting in the data, there is value in exposing it to the community.  Unfortunately, that tends to be very slow to happen because analytics is usually a step-child in the engineering family.  When resources are tight, the production system gets priority, as it should.  So make the data warehouse a production system, not for analytics job security, but because analytics can add real value to social media.

Picture a community where anybody can blog.  Typically, the only feedback is how many comments a posting gets.  Imagine if each blogger could see how many visitors and page views each post gets.  Imagine if they could see which of their words are generating search engine hits.  In other words, empower each user to do their own SEO by giving them analytics-based feedback.  The vast majority probably won’t, but those few who do may have a great impact.  In social media, empowering the super-users may be the key to success.  Analytics is how you identify them, but don’t stop there.

The possibilities are enormous.  Social media analytics can tell us which people or groups have the greatest influence.  Feed that data back to the community.  Analytics can tell us which topics are heating up – feed that back.  Tell the community where the new visitors are coming from; there’s probably something interesting out there when it changes.

The architectural challenges are not simple, given that data warehousing isn’t intended to give real-time results the way that live production systems are.  I suspect that the path to this kind of capability is to mirror an aggregated version of the data warehouse in near real-time and let the production system query the mirror.  In any event, I do believe it is the way things are headed.  Analytics isn’t just a management tool – or perhaps it is, but we forget that every visitor can be a manager.

Tags: analytics, data warehouse

03 Dec 08 Analytics data warehousing

For many analytics practitioners, how to store data is somebody else’s problem – Google, Omniture, Yahoo/IndexTools or another third-party provider.  As Gary Angel points out, “Sophisticated organizations are increasingly finding good reasons to move data from their web analytics tools to other data processing and analysis platforms.”  Indeed.

Having spent the last few years designing, building and using a terabyte-scale analytics data warehouse, I have to agree with Gary.  He calls it “moving” or “transferring” the data, but I see it as an “also,” rather than “instead of.”  At LiveWorld, our data warehouse was complementary to various third-party analytics systems that our clients used.  Name just about any tag-based solution and we were supporting it.  We were even starting to test Google Analytics in “hybrid” mode, where the data goes to Google and to the local server, potentially allowing tag-based data to go into the data warehouse, which would provide a cross-check, at the very least.  Some of our customers, particularly bankers, banned third-party JavaScript for security reasons, so they relied entirely on the data warehouse.  (We could have hosted the tag scripts locally, but decided not to deal with the resulting maintenance issues.)

One of our clients did something that I suspect many other large businesses will opt for – they asked us to create a daily feed for their data warehouse.  The feed didn’t include full detail, it was aggregated data, relatively easy to customize because the source was a set of queries against our warehouse.  This approach offers the advantage of allowing deep data integration on the client’s systems.  They were able to out-source social media to us, yet run integrated reports while preserving customer privacy.

With a data warehouse, you can go beyond the kind of data you can get from tag- or log-based systems.  Social media is virtually always based on application servers.  There’s a database underlying the app server, which means that you can periodically query the database and stick the results into the data warehouse.  In some cases, you’ll just want a snapshot of the state of the app server.  Much of the time, you’ll want the app server to record events with a timestamp, so you can query for the full detail of what happened and when.

Once you have a data warehouse up and running, the big advantage is reporting flexibility.  The whole point of a data warehouse is to store information in a way that allows fast-running queries to be written easily.  At the most basic level of a typical data warehouse, there are no pre-suppositions about what queries will be run.  However, to gain acceptable performance, aggregates often need to be created (ideally, invisibly to users) for commonly run queries.

Next, I’ll given a quick overview of how data typically gets into and out of a data warehouse.

Tags: analytics, data warehouse