Is It Possible To Analyze Digital Data If It's Growing Exponentially?

Our digital universe is morphing quickly into a cacophony of unfiltered digital noise. Is there any way to make sense of it all before we simply tune it out?

I've always been someone who thrives in a fast world. The more I've been able to access information, connect with friends and colleagues, and move information and connections in real time, the happier I've been. Heck, I still have my copy of Fast Company Issue #1. I remember thinking that the magazine was being published just for me.

But lately I've begun to wonder if the speed of information is overwhelming the institutions and devices I count on to make sense of the world.

Is it possible things are moving too fast? And, what happens if we don't slow them down? That's been what's mostly been on my mind these days. And now there's some data that back up my worst nightmares: information is spinning itself out of control.

Looking to the near future--the year 2020--it seems our "digital universe" is morphing quickly into a cacophony of unfiltered digital noise. Since 2007, research firm IDC has been tracking and reporting on our digital universe, which they describe this way: "The digital universe is made up of images and videos on mobile phones uploaded to YouTube, digital movies populating the pixels of our high-definition TVs, banking data swiped in an ATM, security footage at airports and major events such as the Olympic Games, subatomic collisions recorded by the Large Hadron Collider at CERN, transponders recording highway tolls, voice calls zipping through digital phone lines, and texting as a widespread means of communications."

In December 2012, they published the sixth of their studies, with some eye-popping findings:

  • From 2005 to 2020, the digital universe will grow by a factor of 300, from 130 exabytes to 40,000 exabytes, or 40 trillion gigabytes (more than 5,200 gigabytes for every man, woman, and child in 2020).
  • From now until 2020, the digital universe will double every two years.
  • 68% of the data created in 2012 was created and consumed by consumers-- watching digital TV, interacting with social media, sending camera phone images and videos between devices and around the Internet, and so on.
For the longest time, the promise was that software would provide an automated way to find, filter, and catagorize data and turn it into coherent information. But that promise was based on the assumption that information was being created by either professionals or devices that would create 'structured' data. But the IDC report ominously warns that "the vast majority of new data being generated is unstructured. This means that more often than not, we know little about the data." And while this means new information is being added to the cloud at a prodigious rate, information architects held a deep seated hope that social media would provide a raft of new 'signals' that give data robots new ways to organize and contextualize information.

Alas, if only that had happened.

Instead, the 'raw feeds' of information that I try to surf each day are getting increasingly harder to navigate. A Google search for a saxophone teacher for my son turns up SEO enhanced 'tutor' sites. Meanwhile, Craigslist provides links to actual tutors--in our backyard. Facebook's new timeline is like a raging river of information without a filter. Friends post family news, business links, political rants, and local business recommendations. Maybe Facebook's new 'Graph Search' product will help me search my social network, but I'm not optimistic. And my enjoyment of both Twitter and Tumblr, which had been for me edgy and exciting, are now moving faster than my curiosity can process. Which is why I'm shifting away from feeds, and back to people. People I trust.

For media news, I'm looking at individual writers and trusted curators. For travel, I've traded in Expedia for a human travel agent (though I do like TripAdvisor for advice from my friends). For news, I'm shifting to trusted journalists (AP, Reuters, The New York Times).

What does this information overload mean? It means to survive the avalanche of information, I'm narrowing my field of vision, deleting emails more ruthlessly, and exploring the outer reaches of the web with less curiosity and sense of adventure than in the past.

There's plenty of opportunity in the problem. New human news curation services that balance algorithmic filters with human editorial, new innovations in email that sort my inbox based on time, source, and urgency, and most importantly, new metrics to help me score and store what matters.

Our digital universe is spinning ever faster, and there's no chance it will slow down. Not as our FitBits spin out daily fitness data, and our refrigerator tweets it needs more milk, and our dog's collar tracks the last time we've taken Fido out for walk. We are living in a new world of unfiltered data creation--and it's our job to figure out how to filter it.

[Image: Flickr user Christian Scholz]

Add New Comment

0 Comments