I spoke at the Boston eMetrics Symposium yesterday on the subject of Big Data; specifically, how we will build organizations that can adapt to it, extract insights and act on them in a meaningful way.
People generally discuss Big Data in technical and even aspirational terms (see this month’s Harvard Business Review dedicated to the topic), but there’s less attention to the organizational impact. Who will have access to it? What will we do with it? What challenges will we face? Who, if anyone, will “own” it? And how big a change is this, actually?
Let’s be clear: Big Data–or at least elements of it–is here already. We’re seeing it in a focused way in some of the case studies that HBR mentions. One example, in “Big Data: The Management Revolution,” addressed the topic of plane arrival times, which pilots generally (and often incorrectly) estimate. The issue is that errors either direction produce productivity impacts that translate to significant economic impact. So one major airline improved its ETAs by engaging PASSUR Aerospace to crunch tons of data on plane locations, weather patterns, flight schedules, along with proprietary data, greatly improving arrival time estimates.
But we’re also seeing the effects of Big Data beyond the type of highly sophisticated decision support application above. Social data, after all, is a type of Big Data. It meets the “Three Vs” criteria: Velocity (speed), Variety (as IBM says, “text, sensor data, audio, video, click streams, log files and more”) and, most certainly, Volume.
Social data is the “canary in the coal mine” for Big Data: we can learn a lot about how Big Data will thrive (or not) by looking at what has happened with social data thus far. I covered many of these issues in my research reports, “A Framework for Social Analytics” and “The Social Media ROI Cookbook.” Consider this (all data courtesy Altimeter Group):
- The average enterprise-class company has 178 social media accounts
- The average enterprise-class company has 13 departments actively engaged in social media
- Fifty-six percent of companies we surveyed stated that the biggest challenge to measuring revenue impact of social media was “inability to tie social media to business outcomes.”
Some of the other impacts we see, both quantitatively and via anecdotal evidence, have to do with the strain on organizations as they seek to integrate social data into decision-making:
- Lack of analytics expertise
- Poor or disparate tools
- Inconsistent analytical approaches
- Unreliable data
From a maturity model standpoint, we’re seeing this (rough) progression for social data organizations:
One drawback of this progression, at this point, anyway, is that it encompasses only social data, which threatens to become a silo of its own if it is not integrated with enterprise data such as CRM, Business Intelligence and Market Research data, not to mention the many varieties of Big Data coming from sensors, mobile devices, the Web, and so forth. We’re seeing early signs of this as companies such as SAP, eBay, Oracle, Adobe and Salesforce partner with or acquire technologies that integrate social and enterprise (and ultimately even external economic) data.
Some people have proposed the idea of a “Chief Data Officer” to oversee this process, and that is likely a fine idea, but not in isolation. While it’s a tiny sample size, most of the heat during and following my talk focused on the cultural impacts of Big Data: Who owns it? How do we share it? And how do we make the transition from a culture where decisions are made by “HiPPO” (Highest Paid Person in the Organization) to one that is data driven?
It’s a bit of a chicken-and-egg scenario (to continue the bird metaphors) as organizations need leadership to make the leap of faith needed to invest in data-driven organizations, and leadership needs the data to convince them that this is a leap worth taking.
Plenty more to mine here; I’d love your thoughts as always.