City government data released to public mined for useful information

SAN FRANCISCO — A big pile of city crime reports is not all that useful. But what if you could combine that data with information on bars, sidewalks and subway stations to find the safest route home after a night out?

In Washington, a Web site called Stumble Safely makes that possible. It is one example of the kind of creativity that cities are hoping to mobilize by turning over big chunks of data to programmers and the public.

Many local governments are figuring out how to use the Internet to make government data more accessible. The goal is to spawn useful Web sites and mobile applications — and perhaps even have people think differently about their city and its government.

“It will change the way citizens and government interact, but perhaps most important, it’s going to change the way elected officials and civil servants deliver programs, services and promises,” said Gavin Newsom, the mayor of San Francisco, which is one of the cities leading the way in releasing government data to Web developers. “I can’t wait until it challenges and infuriates the bureaucracy.”

Advocates of these open-data efforts say they can help citizens figure out what is going on in their backyards and judge how their government is performing.

But programmers have had trouble getting their hands on some data. And some activists and software developers wonder whether historically reticent governments will release data that exposes problems or only information that makes them look good.

It is too early to say whether releasing city data will actually make civil servants more accountable, but it can clearly be useful. Even data about mundane things like public transit and traffic can improve people’s lives when it is packaged and customized in an accessible way — a situation that governments themselves may not be equipped to realize.

A Web site called CleanScores, for instance, tracks restaurant inspection scores in various cities and explains each violation. After School Special combines data from San Francisco schools, libraries and restaurants so parents can plan after-school activities and see how children’s nutritional options compare by neighborhood. AndTrees Near You, available for the iPhone, lets people identify trees on New York streets.

By releasing data in easy-to-use formats, cities and states hope that people will create sites or applications that use it in ways City Hall never would have considered.

San Francisco recently unveiled DataSF, a Web clearinghouse of raw government data that the public can download. The data sets include seismic hazard zones, street sweeping schedules and campaign finance filings. New York City’s Data Mine includes directories of sidewalk cafes, property values, horseback riding trails and historic houses.

Washington was a leader in releasing its data, and the federal government is doing it too at Data.gov.

Much of this data has always been publicly available, but until recently it has been almost impossible to find. Getting hold of it might have required tenacity, drive and endless phone calls.

The push to publicize government data goes as far back as the 1960s, but technology has made it possible for people to use the data in ways that would not have been possible even a year ago, said Eric Gundersen, president of Development Seed, the Washington company that created Stumble Safely. The company builds data and map applications for international development programs.

“The timing now with the open data movement is really critical because there are a lot of open-source tools that really make that data usable,” Mr. Gundersen said. These include the mapping tool he used to build Stumble Safely and also a site for the United States Agency for International Development that maps public health clinics.

Some activists are skeptical that governments will release politically risky data that could show that people are not doing their jobs. Mayor Newsom said he wants to release all kinds of data, and said he would not be surprised if “people who love to hate their mayor” create an application that maps his public schedule, to bolster their cases about which parts of town he neglects.

There is also the concern that people might misinterpret what the data is telling them.

“In the most basic of forms, with regard to crime stats and unemployment numbers, these kinds of bulletin boards are very useful,” said Saul Bloom, executive director of Arc Ecology, an environmental nonprofit group, who has been an activist in San Francisco for three decades. “But on detailed data dealing with very complicated material, you really have to know what you’re looking for in order to distinguish between good data and junk data.”

Mr. Bloom also worries that cities could manipulate data to gloss over things like unemployment rates by neighborhood.

Governments are trying to make data openness a more open process itself by asking people to vote for data sets they want to be released. In New York, for example, people have requested data on school violence, public restroom locations and bicycle accidents.

Still, asking for the data is often not enough. Software developers in New York have been unsuccessful in getting data feeds of pedestrian and bicyclist injuries and fatalities from the Police Department, said Noel Hidalgo, who is director of technology innovation for the New York State Senate and has been working with developers on building city-data applications. He envisions applications that overlay accident information on city bike maps.

Paul J. Browne, a deputy commissioner of the New York City Police Department, said it releases information about individual accidents to journalists and others who request it, but would not provide software developers with a regularly updated feed. “We provide public information, not data flow for entrepreneurs,” he said.

There have been other scuffles over who has the right to data. Routesy, an iPhone application, uses data from San Francisco’s Municipal Transportation Agency to show train and bus schedules and locate stations on a map. It stopped working for a while because a private contractor working with the agency wanted to charge a licensing fee for the information. The agency now requires its contractor, NextBus, to make the data freely available.

There is evidence that governments’ attitude toward publicizing data is changing. Two years ago, when a Web design and research firm called Stamen Design started a Web site, Crimespotting, that mapped crime data for Oakland, Calif., the city cut off access to the data a week after the site went up.

Bob Glaze, the city’s chief technology officer, said the frequent data requests from the site were disrupting the city’s own crime site. The city eventually changed its mind. And in August, Stamen’s designers unveiled a San Francisco version of Crimespotting with Mayor Newsom at their side.

Some government leaders are making data disclosure an official policy. Mayor Newsom signed an executive order saying city data should be released, and the White House is about to publish a directive expected to give similar instructions to federal agencies.

San Francisco, New York and Washington have all organized contests to encourage software developers to create applications with their data. And the developers are using the data to build businesses. Stamen, for example, uses Crimespotting to show potential clients what it could create for them. Other firms are selling the iPhone apps they have built.

The cities, meanwhile, are to some degree using developers to provide citizens with a service so they do not have to.

“We are increasingly governing in a time when the demand for services exceeds our resources,” said Aneesh Chopra, chief technology officer of the United States. If the contests “spur dozens of innovative applications,” he said, “then we’ve essentially achieved a policy objective at virtually no cost.”

Copyright 2009 The New York Times Company