Consumer Generated Data in Business Intelligence

The latest issue of the Economist has done a thorough job (14 pages) in its Special Report of looking at the implications of the massive amounts of data floating around the internet and data warhouse around the globe, and how we’re all going to be able to handle it.  Given the fact that Cisco estimates the amount of traffic to increase to 667 exabytes (equivalent to 10M copies of the Economist)  by 2013, it’s a pretty good bet that business professionals involved with consumer generated media should be paying attention.

“In recent years, Oracle, IBM, Microsoft and SAP between them have spent more than $15B on buying software firms specializing in data management and analytics.”  While not all organizations need to be making this level of investment, I can clearly see with our clients that dedicated and effective analysis of Social Media data is necessary to get the most out of listening and engagement tools.  When you don’t take the time to dig deep into the data they are receiving, there is a tendency to more confused by the additional information.  As one of the authors states, instead of finding a needle in a haystack, you’re just creating more hay.

While business intelligence is still a developing field, I think the next step is really going to be merging data from sales touch points with the data that is coming from Social Media.  “Wal-Mart handles more than 1M customer transactions every hour, feeding databases estimated at more than 2.5 petabytes”.  Using that data, Wal-Mart was able to make a correlation between approaching hurricanes and Pop-Tart buying (while people obviously buy flashlights and batteries, it also turns out that people buy Pop-Tarts as a storm approaches and they buckle down in their homes).  Imagine then, if Wal-Mart could take the large amounts of consumer generated media to make further correlations on what items sell under which market conditions.

In addition, “the IT industry is piling into business intelligence.  Accenture, PricewaterhouseCoopers, IBM and SAP are investing heavily in their consulting practices.  IBM has invested $12B in the past four years and is opening six analytics centers with 4,000 employees worldwide.”  We are seeing both management and technology consulting firms moving into Social Media mainly because Social Media is now becoming a bigger bear in terms of data overload than most large consumer facing organizations can handle and effectively analyze on their own.

The report also references a Swiss telecom operator (Cablecom) who reduced churn from 20% to 5% by looking at the volume of customer service calls.  If companies can make such significant achievements by evaluating such calls, there is an even greater opportuntity to look into the requests, complaints and observations being made by customers on microblogs and blogs on a constant cycle. 

When Best Buy identified the 7% of customers who made up the majority of buying (43%) through business intelligence analysis, they tailored their stores to meet the needs of those specific customers.  Tapping into the most influential content generators across the web related to your brand or that of your competitor has the potential to uncover even greater insights.

Through word clouds and theme graphs, many Social Media tools are already making insights easier to view.  Drilling down into these visual devices can help make the job of discovering and isolating certain areas of more interest easier.  As I talked about in my previous post about wikis and their role in gaining more understanding of the global view on products or organization, the report provides an example of a visualization across the edits made on Wikipedia.  The picture below is a visualization of the entry for “chocolate” and assigns “different colors to different users and shows how much of their contribution remains by the thickness of the line that represents it.”  While not all visualizations provide business critical information, we can start to use these techniques to understand better what a consumer may want or need from an organization.

As more and more Social Media is generated across various platforms, I think there is a valid business case to be made to evaluate what is being said on a much deeper level than simple engagement.  A thorough evaluation of this data can help your organization do two things.  One, to become better at evaluating where and why you are engaging.  And two, to uncover insights that will help you grow your business in ways that you could not see before.