How America’s Biggest Cities Make Sense Of Their Data

If you want to know if your local taco place got a D on its health inspection, or how vacant homes affect crime, you have Socrata to thank.

How America’s Biggest Cities Make Sense Of Their Data
[Photo: Flickr user Tim RT]

Want to know what the state of Connecticut has spent money on in almost real-time? Or maybe you’d find visualizations for the White House’s recently released 2017 federal budget useful. Maybe you’d be into seeing health reports for every restaurant entry on Yelp, instead.


Socrata, creator of custom data systems for civic departments and federal agencies, is the company behind those possibilities. As old data infrastructure ages and public requests for government data increase, civic departments are looking for solutions to modernize the organization of their data and put it online for the public to use. Instead of government departments dropping big bucks to build their own proprietary data networks from scratch, Socrata sells custom data infrastructure systems as a software as a service (SaaS) product. Civic departments in half of the largest 300 U.S. cities already use their systems.

Where agencies store their data and how easy it is for the public (or even the agency’s own employees) to access it varies from agency to agency. Many agencies are working with 25-year-old data storage systems, says Socrata president and CEO Kevin Merritt, and some of those systems are only navigable—and fixable—by their legacy IT directors, who will all retire someday (or maybe already did). Outsourcing data organization to SaaS and cloud-based companies like Socrata offloads the burden of modernization and ease of use. But as Socrata takes on more and more civic government data systems, they’ve narrowed down what data their agency clients will likely need and how to make that data useful.

“The question is, how do we use data to allow cities to tackle big and small problems?” says Saf Rabah, VP of product at Socrata.

Untouched and unanalyzed government data—what Rabah calls “dark data”—usually sits on enterprise file systems and databases. The city of Seattle, for example, has 1,200 different enterprise systems, says Rabah. Socrata’s job is to make that data usable—but not just for the city. Aside from other government departments, there are three groups that could benefit from data made public: citizens, developers, and advocacy groups.

“Citizens have information needs too, like, ‘I need to know how safe my neighborhood is,’ or ‘I’m about to move to a new city.’ Everyone has information needs that are very unique to them at that point in time,” says Rabah.

In the interest of transparency and accountability, civic agencies usually want to make their data accessible to the public. Some civic agencies create bespoke websites, apps, and portals for the public to access that data–with mixed results. Instead of creating portals, Socrata creates API spigots for public-facing apps, services, and advocacy groups to ingest that government info. This makes it easy for developers to pour data into their products.


“There’s a booming civic tech community taking this data and putting it into apps and services that people use in their everyday lives. We worry about the developer experience: How do they get access to that data, and what resources do they need to build mobile apps in a matter of hours?” says Rabah.

In turn, Socrata hosts the tech community’s apps on its civic app marketplace. Socrata hosts both consumer-facing and government-facing apps: Neighborhood Insight, for example, uses Socrata’s Open Data platform to make a heat map of crimes, code violations, and public service requests to educate citizens about their neighborhood.

But some apps are ingesting area data and providing results. Civic Insight partnered with the cities of Palo Alto and New Orleans, allowing users to enter those city portals to search and track the ownership and permitting progress for any location in the city.

How Government Benefits

Socrata currently works with agencies in half of the largest U.S. cities, says Socrata CMO Eric Eden, giving Socrata a decent chunk of civic agencies along with a number of federal agencies. Government modernization of agency data systems dovetailed with the passage of the Digital Accountability and Transparency Act of 2014, the congressional response to public desire for government financial transparency. The Obama Administration has led the charge, partnering with Socrata to publish visualizations explaining the federal budget for the second year in a row.

Obviously, making data more accessible between government agencies helps them coordinate, like when doing roadwork or closing bridges or issuing statements. And like any business, more data flowing between departments can lead to surprising discoveries. One of Socrata’s clients is the Los Angeles Police Department (LAPD), says Merritt, which discovered that when properties become vacant, there’s a chance that squatters or drug dealers move in within two weeks. So when the LAPD receives alerts about housing vacancies from a city housing authority, the LAPD immediately drives over and puts up signs stating that the house will be patrolled.

By outsourcing their data architecture to Socrata’s SaaS product instead of building ad hoc solutions in-house, says Merritt, governments get a more efficient system at less cost to taxpayers. Such a well-tested system is also typically easier to use for employees, increasing work efficiency, as access to data informs employees and affects their decisions, both as individuals and as groups.

Photo: Flickr user Freedom II Andres

What’s The Rush?

Socrata’s wide client base of agencies has taught it a lot, says Merritt. With all that data, Socrata has built models for new clients to speed up adoption of Socrata’s SaaS products. The same-sized cities tend to have the same-sized problems: aside from needing fewer snow plows, Seattle and Boston have very similar problems. (Side note: Check out how Seattle and Boston measure up using Socrata’s Open Data Network tool.)

There’s another reason to upgrade now: the Internet of Things. As cities consider deploying sensors to track myriad urban conditions, from weather to traffic to space usage, they must also account for how to store and make sense of all that new data.


New custom data systems built for each department, like the older ones Socrata is replacing, are likely Socrata’s biggest competition, says Rabah. But Rabah considers these “custom stovepipe systems” to be inferior at the outset: In his estimation, they’re incredibly hard to maintain and don’t trade data well (if at all) with other data systems.

“If you look at it from the perspective of a typical midsize or large government situation, they have thousands of enterprise systems out there, and every one has its own data systems and data stores. They’re incredibly disjointed, reductive, expensive to maintain, and unsustainable in the long term,” says Rabah. “Our competition is the status quo, legacy IT, and disconnected proprietary systems that are incredibly complex.”

Of course, there are other groups and vendors that are helping governments open up data. Rabah doesn’t see them as competition: They’re part of the big movement to get data out of dark systems and put it in the public domain. And they’d be wise to do it: liberating data could unlock $3.2 trillion to $5.4 trillion in economic value per year, according to a 2013 McKinsey Global Institute study. Unlocking data, the study claims, will enable greater economic efficiency, development of new products and services, and a “consumer surplus” generated by better cost comparisons and product quality.