Month: September 2011

The 2011 Mckinsey Big Data report (see the Mckinsey.com website for a free copy) covers a broad range of issues with Big Data.  Given their expertise on all thing technical, their over 300 page report provides some significant gravitas on the topic.  Of course they will be fishing for consulting assignments as a result, but they touch the nerve of a real issue.  How can organizations stay on the leading edge?

The Mckinsey answer is to provide technology to look at all aspects of your business and systematize your approach.  The data that might be accumulating without being analyzed may hold the key for getting closer to your customer by understanding their behavior better, or perhaps understanding your suppliers better, or providing a mechanism to be more methodical creating more predicable results and less variability in your operations.

The report doesn’t spend any significant time on the mechanics of Big Data technology like Dyanamo from Amazon or the open software Hadoop to analyze massive amounts of data.  The paper is aimed at identifying the pain of not properly using Big Data, and building a business case why a significant amount of money should be spent to master your data.   In all fairness, they do provide some additional granularity, with some industries being more appropriate than others.  They do spend time discussing four examples: health care, government services, manufacturing and telecom/mobile.

Major ways that Big Data can add value include mechanisms for both companies and consumers to become better informed and create transparency for pricing and trends.  It can provide the platform to create new business models that are disruptive for existing approaches.  It might also take more human decision making out of the process, and replace it with algorithms.  The intent is not to dehumanize the world, but to get over the reinvention of the wheel for the 6 billionth time.

The report also looks at the problems with Big Data implementations.  There are legal and structural reasons why the Big Data approach may hit roadblocks.  Data security, talent shortages and technology infrastructure are three areas that will slow or stop Big Data.

The report ends with a smidgen of their consulting approach, starting with a methodology outline of how to create a value potential index for your company, and some of the suitability metrics that they would employ to see if Big Data is appropriate.

The potential benefits for this approach are significant, if you are in an appropriate industry, have some budget available, and the stomach for some real disruptive changes.  But this is what separates the leaders from the pack.

Are Social Media companies like Facebook, LinkedIn, and the like underutilizing the data they’ve accumulated?  I think marketers from all media companies and indeed all companies want to know more about their customers and about prospective customers.  Data Warehousing was an attempt to understand their customers better, but it’s not adequate for today’s challenges of massive amounts of data.  The solution is a different approach, because even with the rapid growth of compute and storage power, there is still a chasm between what is done today and what could be done.

Smart phones and tablets do more than provide internet downloads and interesting QR links.  They are the delivery mechanism for an explosion of tweets, blogs and other unstructured data that in aggregate is useful, if only it could be captured and analyzed.  Due to the size of these datasets the traditional relational databases will have difficulty processing and indexing this data since new data is continually arriving at an increasing rate.

The solution is still unfolding, but it appears that parallel computing structures hold the best hope of attacking this mountain of data.  Hadoop is a tool to tackle this job that was spun out of Yahoo! and is an open source program that can distribute the workload to try and turn data into information.  Hadoop has spawned several startups looking to commercialize the solution to Big Data, and are joined by several large companies like IBM and EMC who already have horses in this race.

Welcome to WordPress.com. After you read this, you should delete and write your own post, with a new title above. Or hit Add New on the left (of the admin dashboard) to start a fresh post.

Here are some suggestions for your first post.

1. You can find new ideas for what to blog about by reading the Daily Post.
2. Add PressThis to your browser. It creates a new blog post for you about any interesting  page you read on the web.
3. Make some changes to this page, and then hit preview on the right. You can always preview any post or edit it before you share it to the world.