This year will be the year of Big Data. The Data Warehousing
Institute (TDWI) reported that 90 percent of the IT professionals it surveyed said they were familiar with big data analytics. And 34 percent said they
already applied analytics to Big Data. The vast hordes of data collection
during e-commerce transactions, from loyalty programs, employment records,
supply chain and ERP systems are, or are about to get, cozy. Uncomfortably
cozy.
Let me start by saying there is nothing inherently wrong
with Big Data. Big Data is a thing, and like anything, it can be used for good
or for evil. It can be used appropriately given known limitations, or stretched
wantonly until its principles fray. For now, the identification, consolidation, and governance of data is an appropriate step, as Forbes‘s Tom Groenfeldt recently
documented
with Michigan’s $19 million in data center consolidation savings. Dirk Helbing of the Swiss Federal Institute
of Technology in Zurich is more ambitious. His €1-billion
project, the topic of the December 2011 Scientific
American cover story, seeks to do nothing less than foretell the future.
The meaningful use of Big Data lies somewhere between these
two extremes. For Big Data to move from anything more than an instantiation of
databases running in logical or physical proximity, to data that can be
meaningfully mined for insight, requires new skills, new perspectives, and new
cautions.
The Big Data Dream
Dirk Helbing seeks a system that is akin to Asimov’s
Psychohistory as imagined in the Foundation
series. In broad swaths, it would anticipate the future by linking social, scientific, and economic data. This system could be used to help advise world governments on the most
salient choices to make.
Reading the article in Scientific
American reminded me of a science fiction story by
Tribble-inventor David Gerrold—When
Harlie Was One. In this book, Harlie, which stands for Human Analog Robot
Life Input Equivalents, decides that he needs answers, and that he isn’t
sophisticated enough to solve his own problem and therefore keep the corporate
interests that built him interested enough to keep him plugged in. So he
designs a new computer, the Graphic Omniscient Device, or GOD, as a proof of his
value. GOD will answer all questions submitted to it. Unfortunately, as the
human engineers building GOD eventually realize, the processing capacity is so
vast, that GOD will not be able to provide an answer to any question during the
lifetime of a human. Harlie, of course, knew this all along. He needed the
humans for three reasons: to keep him running, for engineering labor to build GOD, and to ask the questions that GOD will answer.
