The arrival of Big Data and “automated decision making” has lead to much hand wringing. Nassim Nicholas Taleb for example, said that his biggest problem with modernity may lie in the growing separation of the ethical and the legal. One could imagine saying the same of company’s pregnancy detecting model, hiring via algorithm or more futuristically, destruction by weapons with “artificial conscience”. However, most of us wouldn’t want to go back to the good old days of (mostly) data free decision making for the same reason one wouldn’t want to be the lone horse and buggy in a F1 race. In a world of turbo-charged decision making, opting out is hardly an option, as W. Edwards Deming wryly put it– “It is not necessary to change. Survival is not mandatory.”
What I would say (and I’m certainly not the first) is that Big Data calls for Big Decisions, as data become varied and more plentiful, we are called to make more, not fewer decisions. The lazy bum in us asked a hard question and hoped for an easy answer – why can’t it be problem in, data magic, decision out? It can’t be when “big” in Big Data denotes not just the size of the dataset but also the complex and often latent relationships in the background tying disparate pieces of information together. It can’t be when “big” in “Big Decision” means stretching your comfort zone, admitting your intuition may be wrong, and making peace with the fact that data offers directional guidance, not precise measurements (despite being numeric). Playing blackjack perfectly (by counting cards) doesn’t mean you’ll win every time. You win only when you’re in the game long enough, the same is true for Big Data.
And staying in the game means there’re a few things to keep in mind when handling Big Data:
- Big Data isn’t really for disruptive innovation. While Big Data can help you make a great laptop by tying a person’s click stream to his social media comments to actual purchase behavior, it will never tell you to make an iPad instead. Data measures what was and what is, relying on quantity to reveal patterns or spark new questions. As disruptive ideas and preferences are often first held by the minority, looking for them using Big Data is like a game of connect-the-dots with only two dots – you could draw any shape (conclusion) you want. Big Data is much better at producing sustained innovation, which admittedly, is not what makes a company famous, but what keeps it going.
- Big Data requires synthesis. Using Big Data means you are exposed to different sources, sources that may butt heads with each other because they have different viewing angles. If you ever find yourself unable to act because of contradictory data, see how Adobe overcomes data divergence via a two pronged strategy – gathering the source experts for context and building out a comprehensive customer touch point model to focus on prediction gaps. In MREB’s research study this year, we saw that business partners who were given synthesized info in a business case made the right decision 79% of the time while those who were given the same info as-they-were made the right choice only 54% of the time. A significant difference, not just statistically.
- Big Data is a people business. Besides stretching your comfort zone, Big Data also means getting various data owners out of their silos to collaborate with each other in order to achieve that multi-faceted view. This isn’t always easy – market research sometimes sees analytics as hostile, and vice versa. While politics isn’t easily overcome, as a consumer of research you can make clear your preference – your fellow marketers and other internal consumers of research indicated in our survey this year that they consider a mixture of traditional (survey, focus groups) and non-traditional sources (social media, CRM data) more relevant, valuable and accurate. On the other hand, if you happen to be in a position of influence on the research side, consider how you can make your business partners’ lives easier by putting together a Collaborative Planning Council like TIAA-CREF Financial Services.
The trending of Big Data has sometimes been compared to the CRM hype of 90s, which I could see given the amount of attention paid to both. I could also see history repeating itself if we abdicate our responsibility to decide when, where and how to use Big Data. It’s easy to blame the tulips for a Tulip Mania, but we forget it’s humans who’re at fault.
Related member resources:
- Driving Decision Quality Through Multi-Source Synthesis
- Adobe’s Mode-Based Synthesis
- TIAA-CREF’s Collaborative Planning Council