Data Dump

The number of people paid to produce data keeps going up (we think). So why is it so hard to find numbers that actually mean anything?

As crack Fast Company interns, we gamely take on any number of mission-critical assignments that, besides literally making possible an issue each month, also reflect in some profound way the vagaries of business and society. So it was that, for the magazine's 10th anniversary issue in March, the three of us were drafted to fill five pages with "illuminating" (our editor's word) statistics on the state of demographics, technology, health, and other stuff. Cake, we thought: A few calls to analysts, a huddle with the Census Bureau folks, and we'd be set.

So, okay, we're here to admit we were wrong. A month and several Starbucks cards later, we've learned that even three "reasonably intelligent people" (our editor again) can't possibly get their arms around the statistical state of our world--because the state of statistics is dreadful.

You'd think it would be pretty straightforward to pinpoint the number of U.S. landfills in 2005. It's not; that statistic doesn't exist. As for a projection of future trends: "As you can see, the number of landfills had gone down, so you can probably figure out next year's numbers," a representative from the Office of Solid Waste told us. Not the sort of scientific precision we were looking for. How about the number of telephone landlines? "Timely raw data is scarce," said David Chamberlain of research firm In-Stat, explaining why 2004 information wasn't available until late 2005. When we asked about data on retirement communities, a trade association executive literally laughed at us.

Repeat those sorts of conversations every day for several weeks, and reasonably intelligent people start to wonder: Are we losing our statistical grip? Are we no longer able to keep track of ourselves--and to predict with some certainty where we're all headed? And what does that mean for businesses whose strategies depend on having a savvy understanding of where markets are headed?

We took these questions first to Katherine Wallman, chief statistician of the federal Office of Management and Budget and the person responsible for improving statistical programs and policies across 10 principal U.S. bureaus and hundreds of agencies. We asked: Why can't those offices come up with useful statistical projections? Her reply: "You need to ask the individual agencies. Why don't you tell me?"

Uh-oh.

Part of the problem, of course, is the sheer and growing complexity of our world and the speed at which it changes. We don't have good data on this (ahem), but statisticians say there's just more to keep track of, changing more quickly than ever. Getting a handle on it all takes staff and money--and by the time a measurement has been made, the storm has moved on. All of that means "we must constantly rethink and redesign the way questionnaires are structured," says Lynda Carlson from the National Science Foundation.

Imagine trying to make statistical sense of terrorism. It isn't just the types of attacks that change over time, "but it's also taking place on an ever-changing world map," says James O. Ellis III, research and program coordinator at the Memorial Institute for the Prevention of Terrorism. "What part of Germany? Which successor state of Yugoslavia?" Until 1998, the institute tracked only incidents in which terrorists struck targets abroad or associated with a foreign state. Now attacks by locals on targets in their own countries account for at least 85% of all terrorism. "It's very hard," Ellis says, "to create an accurate picture over time."

The same is at least as true in the technology realm, where a trend can explode and then wane before anyone even knows what to call it. "A lot of analysts are reluctant to make technology predictions nowadays," says Christopher Kelley, director of consumer technographics at Forrester Research. Why? Because it has become so easy to be wrong. "Take DVD players," says Kelley. "Sales for those skyrocketed in the late '90s and early 2000s. Now DVD sales are declining because consumers have new entertainment options we didn't even imagine could exist a couple of years ago. How many of us would have been thinking about that in 2000? Not many."

There's something else going on, too. The availability of data online has radically democratized the practice of research, turning everyone (interns included) into amateur statisticians. Outsell, a consultant to information providers, says Americans spent an average of 11 hours per week looking for and analyzing information in 2005, up from 8 hours in 2001. The problem is, few of us bring any real expertise to that task. We used to rely on a few researchers, statisticians, and analysts with particular field expertise to know what questions to ask and where to look. Now we're on our own. And we're overwhelmed.

"Do you have data on surveillance cameras?" we asked. "Do you have $5,000?" he snapped.

Which is why there's a robust trade in supplying current, relevant statistics and forecasts for, well, money. "Do you have projected data for the number of surveillance cameras in the United States?" we asked Joe Freeman of J.P. Freeman, a security-industry researcher. "Do you have $5,000?" he snapped. At joints like Forrester, ACNielsen, and Mintel Group, among many, many others, complexity, change, and general confusion are all positive business indicators.

But we can't buy our way out of this. If we're condemned to ever-increasing statistical fuzziness, individuals and businesses alike must become a lot more flexible about the way we consider the future. Rather than mapping strategy around a relatively certain future, we have to be able to accommodate a range of possible outcomes. That sort of resiliency is healthy, in any case.

Failing that, of course, there's a simpler solution to the statistics jam. We were clued in by an official at the United Nations Statistics Division when we sought a certain piece of agricultural data. After a pause, she asked: "Have you tried Google?"

Yeah, once or twice.

Have something to say about Fast Company magazine? Email the editor.


Statistically Significant

Employment at the 10 principal federal statistical agencies:

2004* 16,258
2005   18,675
2006   1,285

*Budget years. Source: Statistical Programs of the United States Government

Add New Comment

0 Comments