When sociologist Peter Moskos left Harvard to police East Baltimore, the scholar-cop noticed a tension of incentives. Public safety is served by commonsense policing, he says, things like walking a beat and building relationships in the neighborhood. But if things get "data-driven," the common practices get less sensical.
As Reuters columnist Felix Salmon illustrates this point:
Under statistically driven regimes, individual officers have almost no incentive to actually do that stuff [that doesn't match the metric]. Instead, they’re all too often judged on results—specifically, arrests. (Not even convictions, just arrests).
That Baltimore scene rings familiar to the working world, eh?
Educators teach "to the test," since the standardized test scores are what are taken to be indicators of the success of a classroom and a school and a school system.
Journalists are constantly attempting to perfect the "clickbait" headline, when the page view and the unique visitor count is the rubric of quality (and ad dollars spent).
And as Salmon reported in Wired, this emphasis on quantification concentrated decisions and dollars "within a relatively small group of people at a bank's headquarters"—an arrangement that helped catalyze the global financial crisis.
In each of these cases, management "guru" Peter Drucker's maxim that "what gets measured gets managed" finds its fullest fruition. When we make organizational systems that reward behavior that shows up on a spreadsheet—like arrests or page views—then it's only rational for the individuals within that system to privilege that metric over anything else.
The psychologist Donald Campbell explained how this works in a 1976 paper:
The more any quantitative social indicator ... is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor
Campbell isn't talking about corruption in the sense of taking bribes; rather, it's a case of a tail wagging the dog. As some thoughtful educators have pointed out, we use test scores as indicators because they're way easier to measure than something confusingly qualitative like the quality of an education. This is okay so long as the indicator itself doesn't become the mission. But if the indicator has a monopoly on high or low performance, then it wrenches the whole organizational system: you start running after arrests, teaching to tests, and gunning for page views.
Hat tip: Wired