OpenStreetMap: the data behind the maps

March 6, 2009

This article was contributed by Tom Chance.

In my last article on OpenStreetMap I looked at the recent mass imports of public data — everything from British oil wells to the entire road network for the United States. But for those interested in more than an alternative to Google Maps, the ability to extract or add data to the project is what really makes OpenStreetMap shine. Whether you want to get an SVG of a campus map or import a local government's database of every building in the city, Linux users will find plenty of tools that cater to their needs.

The export tab on the web site provides the most simple way to access data. Users can draw an area on the main map view and then grab an image (in PNG, JPEG, PDF or PS formats); some HTML to embed the map into your web site; or the raw XML data. To further modify the data, either in the OpenStreetMap database or a local copy (stored as an XML .osm file on your disk) download the data using an editor like JOSM (the 'Java OpenStreetMap editor'). To make life easier when selecting the area to download, open up the preferences dialog and install the namefinder and slippy_map_chooser plugins.

Grabbing larger amounts of data would be difficult, slow and clumsy with these methods. More advanced users can get data directly through the API. Check the latitude and longitude coordinates for the area you want — an easy method for this is to use the export tab to draw an area, then note down the coordinates it records — then fire up wget or curl and download the data:

    wget http://api.openstreetmap.org/api/0.5/trackpoints?bbox=left,bottom,right,top

The main api only lets you grab 5,000 points per request; you have to page the request to get the additional data. To pull out a really large chunk of data, or to filter it (for example to just download all the pubs in the city) use the extended OSM API (XAPI, or 'zappy'). Access to really enormous amounts of data, such as the entire planet or a country, can be found in the frequently updated dumps listed on the Planet.osm wiki page.

Once you have the data there are all manner of uses - your GPS navigation device, rendering your own maps for the web or print, or converting the data into another standard GIS format with tools like the Ruby osmlib. The documentation for each tool various enormously, but the toolchains tend to be relatively straight forward.

Of course, extracting data is only half the story. Not only should all good open source citizens be contributing back, but you will get the most value from the data if you collaborate with others in developing a rich data set that will lead to tools and use cases you can later replicate.

OpenStreetMap abounds with methods and tools for entering data. You might like the "old school" method of tracing a breadcrumb GPS trail — much more fun in the early days when I mapped much of Reading with some friends from a completely blank slate. Many mappers have traced basic road layouts and buildings from aerial imagery donated from Yahoo! so that others can go in and identify street names and points of interest. The main editing tools are Potlatch, a flash interface on the main web site (just click on the 'Edit' tab once you're zoomed into your local area), and the previously-mentioned JOSM. The wiki has plenty of guidance.

When importing large sets of existing data, things get a little more complicated. The first step is to step back and have a good think. Imports can cause two kinds of headaches for other contributors if done wrong: you might put a load of new data over the top of somebody else's efforts and make a complete mess in the process; or worse, you might import data without proper permission, causing legal difficulties for the project and technical difficulties in taking the data back out again.

It's always best to begin by asking a few questions on the relevant mailing list; there are localized lists for many areas, a general (high traffic) "talk" list, and a "legal-talk" list for legal issues such as licensing for imports. It's especially important to avoid convenient interpretations of web site notices regarding copyright and database rights when deciding if you can import the data. You need to get written confirmation so that the OpenStreetMap project is immune from legal attacks. There are some nice general guidelines on the wiki, which are worth a read.

If you have data with written permission to use it, you can begin the import process. The first, and most laborious, step is to map out the data against standard OSM tags, as in this UK public transport example or this really comprehensive exercise for CanVec data. You'll notice that oftentimes source-specific data (like unique IDs for features and really niche data) is retained in a namespace like "CanVec:FID" and "naptan:StopAreaCode". This can also be useful where you don't want the data to appear until volunteers have gone through checking it against existing data in the database, for example to merge two bus stops (one crowdsourced, the other from the import).

For large chunks of data, importers have tended to write custom scripts to then bring the data in. If the data is in the OpenStreetMap format, and it is in a state suitable to go straight into the database, this bulk import script makes the process quick and painless. The Canvec2osm code shows how to pull in more complicated data; this converts 11 different shape files into themed osm files with correct tagging, which can then be worked into a suitable state for importing.

A more cautious approach can be appropriate in areas with a lot of existing data. One quite technically challenging route is to set-up your own Web Map Service (WMS) using a tool like mapserver, and then set-up the JOSM WMS plugin to pull those maps in as a layer underneath your map data so it can be traced. This Map Warper tool is in beta and tries to make this process easier. If the data is quite simple you could just put the source and editor side-by-side on your screen and use your judgement to copy over points of interest.

However you want to proceed, you're probably best off getting in touch with some local or more experienced community members. Interested people could even just lobby local government officers and public institutions to get the data, then pass it along to somebody with more of an appetite for the technical stage. Given 6 months to study, process, and import the data, you should find richly detailed maps and underlying data available under a Creative Commons BY-SA license; the license, incidentally, may soon change to one more suitable for databases. Whatever you do, just remember to have fun.

Index entries for this article
GuestArticles	Chance, Tom

to post comments

Printing a good-looking map

Posted Mar 6, 2009 22:49 UTC (Fri) by epa (subscriber, #39769) [Link]

You can easily go to the openstreetmap.org front page and print it from your browser. You get a reasonably good-looking map that is often better rendered than Google Maps or the other competition. However, because it's printed out from the bitmap tiles, it's not quite such good quality as your printer is capable of.

In theory you should be able to export a PDF and print that... but I have found it difficult to get one of the right size, or send it to the printer without odd errors. Is there a foolproof way to 'download PDF of the current view' but using your normal paper size (A4 or US-Letter)?

altitude data?

Posted Mar 7, 2009 0:00 UTC (Sat) by roskegg (subscriber, #105) [Link] (27 responses)

Does the database include GIS data for the whole earth? I am interested in getting altitude data for every square meter (or every square mile) of the earth's surface, including underwater altitudes. I want to test the old theory that either Jerusalem or Babylon are positioned directly over the earth's center of mass. So, is this altitude info available and easy to parse and access?

altitude data?

Posted Mar 7, 2009 0:11 UTC (Sat) by flewellyn (subscriber, #5047) [Link] (17 responses)

The Earth's center of mass is more or less at the center of the planet. EVERYTHING on its surface is therefor positioned "directly" over it, or close enough that variations are statistically insignificant. There wouldn't be a whole lot of point in such an exercise.

I don't know if OpenStreetMap has elevation data, but I would imagine not; it is, after all, OpenSTREETMap.

altitude data?

Posted Mar 7, 2009 0:37 UTC (Sat) by jwb (guest, #15467) [Link] (16 responses)

Having just had this argument, I feel that you should consider that Earth is not perfectly spherical, it's more of an ellipsoid. Only in certain places will the line normal to the surface pass through the center of gravity. It's still a stupid exercise, of course.

altitude data?

Posted Mar 7, 2009 0:40 UTC (Sat) by flewellyn (subscriber, #5047) [Link] (15 responses)

Of course I know that. I work in GIS. But, while the precise shape of the Earth is important in the field, and the reason for varying measures such as NAD83 and WGS84, I don't think that there's enough variation to make it meaningful to say that a certain point on the Earth is "directly over the center of mass".

Especially since the Earth's core is, y'know, liquid. It moves around.

altitude data?

Posted Mar 7, 2009 1:31 UTC (Sat) by roskegg (subscriber, #105) [Link] (14 responses)

I understand, from a physics and scientific point of view, the number I am looking for viz the point on earths surface closest to the center of mass, is useless.

But I am wanting this data for religious reasons. That particular bit of data figures in various schools of theology and prophetic interpretation. A number showing that the spot is neither Jerusalem or Babylon should put an end to a lot of bad prophetic interpretation.

Science, meet religion. Religion, science. Thank you.

altitude data?

Posted Mar 7, 2009 1:47 UTC (Sat) by jwb (guest, #15467) [Link] (1 responses)

You left out Mecca.

altitude data?

Posted Mar 7, 2009 2:01 UTC (Sat) by roskegg (subscriber, #105) [Link]

Mecca has never been in the running, but Antioch and Damascus are also contenders.

altitude data?

Posted Mar 7, 2009 1:49 UTC (Sat) by iabervon (subscriber, #722) [Link] (4 responses)

The center of volume isn't going to be useful for finding the center of mass, because the earth's core is liquid and not of constant density. I wouldn't even be too surprised if the center of mass moved randomly in a certain area around the center of volume, so there wouldn't be any way to rule out the point you're looking for being in any given location on any given day. Maybe if you could solve all the fluid dynamics, and you knew when the prophesy was going to come true, you could pick out the relevant location at that time, but that's sort of begging the question.

altitude data?

Posted Mar 7, 2009 1:59 UTC (Sat) by roskegg (subscriber, #105) [Link] (1 responses)

Thank you for helping me clarify the question. You are right that finding the center of mass is a pretty hairy job, maybe even impossible right now.

But the center of volume will be quite sufficient for theological discussions.

altitude data?

Posted Mar 7, 2009 2:37 UTC (Sat) by iabervon (subscriber, #722) [Link]

IIRC, the geographic north pole is the point where sea level is closest to the center of volume of the earth, by a healthy enough margin over equatorial locations to rule out anywhere there were early civilizations. The earth bulges out at the middle, so it's all higher (and therefore further from the center) than the poles.

I think you're trying to ask where the point on the surface is such that the center of the earth is most on that side of the earth. This makes sense for any other point inside the earth, such as the bottom of the Mariana Trench, because you can say that, to get to that point from, say, London, you'd have to go past the center of the earth to get there. But the center of the earth is defined as being the point that's just as much directly under Jerusalem as Milwaukee or Tahiti. In order to get any result at all, you need to find the geometric center, and then find the center of relevant stuff, and then you draw a line from the geometric center through the center of relevant stuff and see where it hits the surface. But you obviously can't use the same center for both things, because then the line doesn't go anywhere.

I was thinking you intended to use the center of mass as the second point, but that would give you different answers over time. Imagine a balloon with a rubber ball in it, flying through the air; there's a center of mass, which is the center of the rubber ball, and there's a center of volume, which is the center of the balloon, and you could pick the point on the surface of the balloon that the ball is most directly under. And that's great, but a moment later the relationship between the two has changed and it's a different point.

altitude data?

Posted Mar 8, 2009 0:24 UTC (Sun) by jordanb (guest, #45668) [Link] (1 responses)

Minor nitpick. The earth's core is divided into two regions. The outer core is -- indeed -- liquid. But the inner is believed to be a solid sphere some 1,500 miles in diameter.

altitude data?

Posted Mar 8, 2009 3:27 UTC (Sun) by iabervon (subscriber, #722) [Link]

Clearly my references are 2-4 billion years out of date. But, in any case, this means that there's a more dense, possibly irregular, object in the center of the earth which is not rigidly attached to the surface. People think it's rotating relative to the surface at about a degree/year, too, which implies that any theologically significant point on it moves under the surface by about 66 miles every year (in some direction).

altitude data?

Posted Mar 7, 2009 1:54 UTC (Sat) by flewellyn (subscriber, #5047) [Link] (4 responses)

I figured that it was for such reasons. Honestly, you're better off leaving the truth of mythos as mythos, and not trying to "prove" it with the mechanisms of logos. They are two separate categories of truth, and it's a category error to use one to "prove" the other. Especially since mythos is not intended to be literal and provable.

altitude data?

Posted Mar 7, 2009 1:55 UTC (Sat) by roskegg (subscriber, #105) [Link] (2 responses)

That is a very logical approach to the matter, but doesn't really help those who don't wish to sever the connection between science and religion.

altitude data?

Posted Mar 9, 2009 3:22 UTC (Mon) by BrucePerens (guest, #2510) [Link]

That is a very logical approach to the matter, but doesn't really help those who don't wish to sever the connection between science and religion.

You are well able to sever the connection between science and religion for yourself. The problem is that there will still be religionists for whom this has no meaning. One I discussed the issue with confessed a lack of belief in cause and effect.

altitude data?

Posted Mar 9, 2009 17:10 UTC (Mon) by nye (subscriber, #51576) [Link]

However hard you try, you're never going to get OpenStreetMap to show you how to find the Easter bunny.

This entire discussion is so ridiculous that I'm having trouble believing it isn't all a joke, but surely the answer is that, assuming the Earth to be an oblate spheroid of constant density (not actually true but apparently close enough), any point lying on the equator, plus the poles, would be 'directly above' the centre of mass, if by that you mean that the surface normal at that point intersects the centre (NB. this is based on my intuitive geometric understanding of the spheroid. I haven't attempted to verify it mathematically, though doing so should be straightforward).

If you actually care about the Euclidean distance from the centre, then clearly the poles are going to be the closest points, and given that the density of the Earth isn't really constant, one is likely to be closer than the other (an earlier poster claimed that the north pole is closer to the Earth's centre of mass than the south). Note that the equatorial region, though 'directly above' the centre of mass, is furthest from it.

Re: Mythos vs Logos

Posted Mar 26, 2009 9:25 UTC (Thu) by ldo (guest, #40946) [Link]

flewellyn wrote:

[mythos and logos] are two separate categories of truth, and it's a category error to use one to "prove" the other.

So the fact that there are two different categories of truthwhat kind of truth is that? Is that a mythos or a logos truth?

altitude data?

Posted Mar 7, 2009 7:44 UTC (Sat) by epa (subscriber, #39769) [Link]

...should put an end to a lot of bad prophetic interpretation.

Yeah, good luck with that.

altitude data?

Posted Mar 25, 2009 21:35 UTC (Wed) by Baylink (guest, #755) [Link]

I believe the answers you've received already amount to "you can't get there from here"; it doesn't really matter whether you were trying to prove or disprove the old assertion.

altitude data?

Posted Mar 7, 2009 0:34 UTC (Sat) by peterh (subscriber, #4225) [Link] (7 responses)

The USGS has free Digital Elevation Model (DEM) data for the US, down to 1/3 arc second (~10m
IIRC); they may have some global data up there as well (shuttle radar SRTM data is global with a 3
arc second (~90m) resolution IIRC). Google will find you want you want.

Anyone want to start OpenTopographicMaps? :-)

altitude data?

Posted Mar 7, 2009 0:41 UTC (Sat) by flewellyn (subscriber, #5047) [Link] (1 responses)

Wonder if they have LIDAR data too? That'd be really useful.

altitude data?

Posted Mar 7, 2009 13:24 UTC (Sat) by dnewcomb (guest, #3312) [Link]

Lidar data for the US, where it is available, can be downloaded from the USGS Click site, http://lidar.cr.usgs.gov/. Be advised, that if you are interested in downloading the point source data, it is a lot of data.

altitude data?

Posted Mar 7, 2009 1:29 UTC (Sat) by roskegg (subscriber, #105) [Link]

Thank you, Peter.

altitude data?

Posted Mar 7, 2009 5:32 UTC (Sat) by pabs (subscriber, #43278) [Link]

Opentopomaps already exists:

http://opentopomap.org/

The site seems a bit broken though.

altitude data?

Posted Mar 7, 2009 12:11 UTC (Sat) by Burgundavia (guest, #25172) [Link]

OpenCycleMap and OpenPisteMap both use STRM (Shuttle radar imagery), but it has some major issues, such as blank spots and obviously wrong stuff.

altitude data?

Posted Mar 7, 2009 12:11 UTC (Sat) by jhellan (guest, #17103) [Link]

The SRTM data doesn't cover latitudes above 60 degrees.
http://www.viewfinderpanoramas.org/ covers a lot of what SRTM left out.

altitude data?

Posted Mar 12, 2009 3:02 UTC (Thu) by puetzk (guest, #3318) [Link]

SRTM data is easily combined with OSM layers to make topographic contour maps:

http://wiki.openstreetmap.org/wiki/Srtm2Osm

altitude data?

Posted Mar 19, 2009 20:01 UTC (Thu) by pal (guest, #57253) [Link]

every place on earth is directly over the center of mass, because "down" is by definition direction to the center of mass

you left out OpenStreetBugs

Posted Mar 7, 2009 12:11 UTC (Sat) by vblum (guest, #1151) [Link]

... where I live, OpenStreetBugs is actually a well-used tool - even if you do not have time or the data to fix a bug precisely, you can still add a comment there, and remarkably, things do get fixed. Missing streets on a city's map are an excellent example ... and I found a bunch of those already.

http://openstreetbugs.appspot.com/ should definitely be added to the article.

SVG Output?

Posted Mar 9, 2009 14:20 UTC (Mon) by kfiles (subscriber, #11628) [Link] (3 responses)

While the lead-in for the article discussed generating SVGs, I didn't see any tools mentioned which provided this export option. Does anyone know the simplest transformation on OSM data to get an attractive SVG map?

Thanks,
--kirby

SVG Output?

Posted Mar 9, 2009 16:42 UTC (Mon) by michel (guest, #10186) [Link] (2 responses)

Some quick googling found me this:

http://wiki.openstreetmap.org/index.php/Osmarender

No clue if that's any help...

SVG Output?

Posted Mar 10, 2009 4:55 UTC (Tue) by kfiles (subscriber, #11628) [Link] (1 responses)

Actually, it looks like the best answer is the Export tab, which apparently added SVG export about a
year ago. Strange that the article lists all of the possible export formats *except* SVG.

SVG Output?

Posted Mar 10, 2009 14:38 UTC (Tue) by telex4 (guest, #21370) [Link]

My mistake!

How to download the data

Posted Mar 19, 2009 14:47 UTC (Thu) by deleteme (guest, #49633) [Link]

The link for downloading the OSM data was pointing at GPS track log database, which is interesting but not what you want, to download use:

Live map data, always current
wget http://api.openstreetmap.org/api/0.5/map?bbox=left,bottom,right,top

Filtering server that lets you choose what to download (10 min old)(read moe about xapi
curl -g http://www.informationfreeway.org/api/0.5/bbox=left,bottom,right,top -o data.osm

Delivering bulk data (10 min old) (read more about ROMA)
wget http://api1.osm.absolight.net:8080/api/0.5/map?bbox=left,bottom,right,top