The Big Question About Big Data
Predicting the future isn’t quite a mug’s game anymore. It was once the province of futurists who talked like evangelists and fortune-tellers who paradoxically could never seem to leverage their predictive abilities with a lottery ticket at their local convenience store. Today, lots of folks are into the prediction racket these days – “predictive analysis”, anyway, with an eerie degree of accuracy. It’s all thanks to Big Data – that ambiguous little term every vendor is using to sell their technology.
What is ‘Big Data’? (For those who are having visions of a ‘roided-out Brent Spiner from Star Trek TNG, nice try – but no). It is what it sounds like: a set of data so large you might not quite know what to do with it; gigabytes and terabytes of data that firms and organizations have been gathering, often for years at a stretch. From the NSA’s logs of telephone calls from every American citizen (and Canadians as well, for all we know), to medical records from state and provincial health authorities, crime statistics, demographics of every one who ever purchased tax filing software and what you bought at the supermarket using your in-store card, we’re sitting on piles of data that would make the fictional Bill Gates of 1981 flinch (Mythbusting tip of the day: he actually never said that “640 kb ought to be enough for anybody”. It’s still a great story, though).
The promise of big data comes from two developments: first, the dramatic increase in storage capability that allows that data to be collected in the first place; and better software tools for sorting that data into usable intelligence. Companies want to use that intelligence not just to recognize patterns, but to predict future behavior (and capitalize on it). From supermarkets and chain restaurants to insurance companies and CIA spooks, that power of prediction is a powerful lure. That’s why so many vendors are adding this – or at least the promise of it, to their technology offerings.
Before you start considering technology offerings, the big question your company has to answer before you start investing in a solution is “what do you want to know?” You’ll want to be careful what you wish for.
Actual predictive power is a Pandora’s box. Think about it — true insight into data is rarely benign. Often it will have significant impact on the organization, leading to changes in their business model. Big Data can be disruptive.
Cautionary notes aside, once you’ve decided you want to move ahead with predictive analysis, you’re going to need to proceed strategically. You’ll need a team in place that includes business managers, IT resources, and senior executives; a wide range of skill sets, including business and analytical capabilities; highly-qualified coders – and most importantly, team members with the desire to learn and contribute, who can ask questions and imagine the future. These experts will build a roadmap, define clear objectives and outcomes, validate their approach. Once that legwork is done – and only then, your firm can use technology to drive your vision forward.
Dealing with a Big Data problem? You’ll need people to solve it, starting with asking the right questions. Then we can start talking about the technology.