infochimps.org - help
PriceOfFreeData (changes)

Showing changes from revision #3 to #4: Added | Removed

All of the datasets below are freely available in principle (and in some cases gathered at taxpayer expense). However, in practice they lie behind high bureaucratic and monetary barriers, and in some cases arrive with added use restrictions:

• NCDC weather
  • http://www.earthinfo.com/prices/prices.htm— Annual Subscription: $ 300 per CD Purchase: $ 900 per CD

• Materials Safety Data Sheets – $399 a year to view data (no raw data access) This is especially galling—this information can protect and improve people’s lives.

• USGS (US Geographical Survey) Map Data—$499 / 8GB data /month.

• FAA Flight history—$4550 for 24 months of data on one single airport.

• Match Zip Code to Government Representative—free from the Sunlight Foundation or echoditto, but you can pay $9300+/year or this simpler version for a mere $450/year. if you’re on a budget.

No disrespect is meant to any of the sites above; they are charging what the market will bear and in some cases adding value. But for comparison, the base bandwidth cost for industrial-strength data transfer is less than $200/Terabyte , so the markups above are in the 100x – 10,000x range (that’s 10,000% – 1,000,000%).

However, we feel that basic information about our universe and our society belongs to us all. With the right tools and the right community, we want to bring the cost of this information down to the simple bandwith charge.

• Materials Safety Data Sheets—these describe the chemical properties and exposure risks for chemicals used in the products we buy and the processes that make them. Companies are required by OSHA to prepare these so that employees know proper workplace procedures and emergency response, and they’re broadly useful for consumers, consumer/environmental advocates and chemists alike. http://www.msdsonline.com/ will give you a few for free but ask $399 a year for frequent access; and even so, it’s one at a time (no raw data access). Their price is partially justified, because these are scattered piecemeal all across the web (and even off it), and I suspect they’ve made a large investment to unify this collection. So lay most of this blame on the government—at a minimal marginal effort over what is already required by statute, we could specify 1) a uniform format, readable by person and computer, and 2) that MSDS’s be submitted to a central repository. That is, “Do what you’re doing, but use this MS word template; and when you’re done, upload it here”—not a heavy burden. Restriction of access to MSDS is especially galling, as this information can protect and improve people’s lives.

• Weather data; free from the NCDC for educational or governmental institutions, and freely redistributable for non-commercial purposes, but $2000 from the NCDC for online access http://ols.nndc.noaa.gov/plolstore/plsql/olstore.prodspecific?prodnum=C00532-TAP-A0001 http://www.earthinfo.com/prices/prices.htm asks $300 per CD for an Annual Subscription; or $900 per CD for Purchase—these are for small (very small) slices through the dataset. The whole thing is about 40GB, so in proportion the full set would be around $50,000.

• USGS (US Geographical Survey) Map Data. These are prepared by government mandate, but the USGS asks you to pay for the data http://edc.usgs.gov/products/aerial/doq.html#search—Government site; works out to about $30/GB http://data.geocomm.com/premiumdownloads.html $499 / 8GB data /month
  For comparison, the commodity cost of bandwidth is less than $200/Terabyte (http://www.amazon.com/S3-FAQs-AWS-home-page/b?ie=UTF8&node=16427271), or 20 cents per gigabyte. So the digital maps bear a markup of more than 300x, and the weather data about 7000x—for free government data.

• FAA Flight history—$4550 for 24 months of data on one single airport. http://flightaware.com/live/airport/KAUS/history/buy This is gathered by the FAA and only given out under the strictest terms (you have to qualify to even look at it). I’m not talking about sensitive national security data, though—you can go to flightaware and buy it. It’s redistributable in principle, but the FAA just won’t let us have it so that we can /pay all the costs of distributing it for free/.

Please note that the price for these increases with the bureaucratic obscurity of the data and not with its complexity. The geographic and weather data are hugely dimensional and demand specialized tools, yet even at their prices cost far less than this modestly-sized excel spreadsheet.

• Match Zip Code to Government Representative—free from the Sunlight Foundation or echoditto, but you can pay $9300+/year from http://www.spatialinsights.com/catalog/product.aspx?product=1240&content=3962 or get this simpler version from http://www.congressmerge.com/products/index.htm for a mere $450/year. This is a chart of nothing more than the elected official for each zip code. As in, people are asking almost $10,000 a year for a tool to connect citizens to their elected representatives.

Thanks to the partnership with archive.org, we’re willing to bear the full cost of indexing and redistributing every single one of the above datasets. In too many cases, though, the government won’t give us one copy for free so we can give multiple copies to thousands.