Jul 09, 2010

MapQuest Opens Up - in the UK

As I write, we are in the final few days of work left to us on the new Beta site open.mapquest.co.uk.  It's an experimental site that uses the free open source OpenStreetMap data, as well as various open source tools, software and utilities to bring it all together.  We then added some of our own special sauce - our best-in-class routing algorithms, and our new user interface (a full experience of which can be found at new.mapquest.com). If you are unfamiliar with OpenStreetMap, it is mapping data gathered by volunteers, for free - over a quarter million of us by now! - across the entire world with the express purpose of this data being available to anyone for (almost) any use, and not locked in high-priced commercial vaults.  There is a great benefit to this - If you find something wrong or missing, you can go fix it or add it! if you go to openstreetmap.org, find the place on the map, and click the edit tab at the top, you can fix the roads, add the businesses, and edit the map.  There is also an OSM (OpenStreetMap) Wiki that tells you more about the OpenStreetMap foundation, the data, and how to get involved. It is truly amazing what they have accomplished. Randy Meech, our Head of Engineering, summed up why we are doing this project more succinctly than I ever could - so here's what he said: "We believe that open source is ultimately the future for AOL's local and mapping applications. And we're very excited about supporting OpenStreetMap, which powers the maps behind Patch, our local news and information platform. We believe community generated maps that are of high quality and accuracy will end up ultimately being the better mapping product for users. Allowing users to improve the areas they know and care about like streets in their neighborhood, in addition to hiking trails, parks and bike paths, we believe will lead to the best mapping experience for all users.” To prove we are serious, we announced today at the SotM conference with a $1 million fund to support the growth of open-source mapping in the United States.  This first website is also proof that this is not lip-service.  We are engaged, and actively working on integrating, using, and improving these tools and we wanted to be able to demonstrate that at the conference. The goal was to create a MapQuest experience for the United Kingdom using only OpenStreetMap data.  As much as possible we tried to use the open source software used by the OSM community, so anything we did to these tools could be contributed back.  We picked the UK first because we felt we had the best shot of getting use-able routes from the data without having to worry about a language barrier at the same time. We started out, as anyone else would, with the latest dump of the OSM Planet data. We grabbed OSM2PGSQL, a python script used for data conversion, and set ourselves up with PostgreSQL with postGIS extensions (a library that adds a lot of geographical functions and datatypes to PostgreSQL). After successful import, we needed to render and cache the tiles.  For rendering we use Mapnik and for caching, TileCache.  We turned to our good friends at Cartifact to help us re-create our MapQuest style as Mapnik style files.  At this point, we discovered there were some features that Mapnik didnt have that we could really use.  Fortunately, we were able to hire Artem Pavlenko, the very awesome chappie who created Mapnik, to improve it with the features we needed, and then release it back to the community. They included such things as SVG Symbolizer, label offsets, improved connection management, and multistyle rendering based on polygonal regions. The last one we needed because we have 3 different styles (US, Europe, Rest of World) we needed to apply. [caption id="attachment_281" align="aligncenter" width="443" caption="OSM Style of Oxford."]OSM Style of Oxford[/caption] [caption id="attachment_279" align="aligncenter" width="443" caption="MQ style of Oxford."]MQ style of Oxford[/caption] For directions and routing, we started with the raw OSM data and converted it into our own format that works with our routing engine.  Along the way, we found much duplicate data and other little issues that sometimes caused problems.  For example, we were initially unable to make directions to York in England, because the city center is surrounded by pedestrian walkways.  For a while, we could only get the Channel Tunnel going one-way, and no-one could get into, or leave France.  However, once we overcame the issue, we were remarkably impressed by how well it all worked. [caption id="attachment_277" align="aligncenter" width="620" caption="From Scotland to Istanbul, stopping in Girona along the way."]Directions from Scotland to Istanbul[/caption] Geocoding & Search turned out to be our biggest challenge when using OSM data, and is probably the area we will focus on the most in the next phase.  Normally we make a distinction between Ambiguities (Did you mean Brentwood, England or Brentwood, Alabama?) and Searches (Pubs in Brentwood) and we provide a different UI flow for search results than we do for ambiguities. The Open Source finders have only a single set of results to any query, with no distinction.  Most of the Ways (what OSM calls lines, or streets) and Nodes (OSM-speak for Points) do not have house numbers or addresses on them, which makes geocoding to a numbered street address very unlikely. Most of the businesses do not have the info that a commercial data set brings - phone numbers, hours of operation, etc.  So be prepared! The search results are not yet what you would expect from a full blown MapQuest site - but we'll get there...oh yes, we will.  To that end, we also tracked down Twain, the excellent creator of Nominatim, the search engine that OSM uses, and have shanghai'd him to make improvements he's wanted to do for a while, and contribute back, as well. This droning monologue has gone on far too long already! For anyone who has stuck with me through this post, and who happens to be at the SotM conference, feel free to accost me with advice, suggestions and frosty beverages. This project has been a very valuable lesson in all the little things that go into MapQuest that as both an employee and a user, I take for granted.  The most amazing thing to me is the incredible effort that has been put into OpenStreetMap and its tools by such a large number of dedicated volunteers - a movement that will continue to grow. I look forward to providing more details as we continue to work on this exciting new path for MapQuest after the conference and am very excited to be participating in the OSM community. Note: For a more concise post, I suggest you check the blog post of my co-conspirator, Deb Tankersley, over on our consumer blog.