OSM uses CDN for instant updates | Fastly
Wikipedia is the encyclopedia for everyone; OpenStreetMap (OSM) serves the same purpose for mapping. OSM is an open-source global map built by 6.4 million volunteers who survey towns and countrysides around the world. They built the software that powers the project, from the interface to the underlying data access API, and they use a wiki-style collaborative editing tool and GPS units to update the data. The map images and the underlying data are released under an open-content license, free to use and openly accessible.
OSM is part of our open-source program, and as such, we power their content delivery with our content delivery network (CDN). Here’s their story.
The challenge
OSM data is used all over the world for routers, couriers, disaster response (e.g. closest hospital), and of course, maps. The vast majority of OpenStreetMap’s usage comes from their Map Tile Service: an organization takes OpenStreetMap data and displays it within their own application or website. Tens of thousands of websites use OSM’s data. If you’ve looked at a business directory listing on Facebook or calculated the cost of a ride on Uber, you’ve interacted with OpenStreetMap.
The Map Tile Service pushes changes to the underlying data on a regular basis, in some cases as frequently as every minute. In addition, OSM publishes a new complete export of their data on a weekly basis, called the “planet file.” Knowing that a CDN would be crucial for the effective delivery of this content, OSM set out to build their own with donated resources.
However, they struggled with that solution from the beginning. It was difficult to obtain servers in certain regions of the world. Map editors often complained that tiles loaded very slowly. Their ability to detect and correct server issues was minimal. Then in 2019, the London Marathon released a new app that allowed users to track a runner along the route. With no warning, the huge traffic spike caused OSM’s custom-built CDN to melt down. They were forced to re-route traffic to other servers and began to look for a different solution.
Why a modern CDN?
Modern CDNs make it possible to serve truly dynamic content that changes with every request from the edge instead of the origin. By giving you the ability to run your code closer to the end user — with more strategically placed high-capacity POPs that also act as a storage pool to cache more content — it’s possible to eliminate many origin requests entirely.
The modern CDN’s ability to instantly purge (measured in milliseconds by some CDNs) and selectively purge content (as opposed to “purge all”) helps companies cache a lot of content that would otherwise be uncacheable with traditional CDNs.
Additionally, a modern CDN gives developers real-time visibility into stats and logs to program and monitor API performance. These mechanisms are powerful enablers for content delivery and help multiple facets of organizations' applications, including performance, origin offload, and overall insight.
Why Fastly?
OpenStreetMap was initially drawn to the high level of control we can provide. Due to the Map Tile’s format, many requests don’t follow specific patterns, making it more difficult to cache them. However, OpenStreetMap was able to write custom logic in our Varnish Configuration Language (VCL) to limit invalid requests sent to the backend and either serve a 404 or reroute the requests to the correct backend. Another immediate benefit was that we were able to reduce the load on OSM’s servers, allowing them to be repurposed to more advanced uses: one of their former tile network boxes was recently turned into a geocoding server powering search against OSM data. OSM was also able to entirely decommission selected older servers, reducing their overall admin load.
Before migrating their entire setup to our network, Ian Dees, OpenStreetMap’s treasurer and infrastructure wonk, decided to run a test. He set up a prototype service to sit in front of the demo tile service. Once the configurations were set, he began to roll Fastly out to a bigger user audience in the United States. The feedback was incredible — mappers noted a massive improvement in performance.
“With our previous CDN, our users would complain that sections of map images would load and then would get stuck,” Ian recently told us. “That kind of experience meant that the page took five, sometimes 10 seconds to load. That's way too long. With Fastly, that doesn’t happen. As soon as you get there, everything loads. It feels so much faster.”
Ian said that working with us has also improved observability capabilities, which have enabled their team to move from a defensive to a proactive posture. With their previous CDN, they had minimal visibility into statistics, so they were only able to publish basic stats like, “where people are looking at maps all over the world.” They’re now working with a custom logging solution that will be used to inform disaster response mapping.
So far the stats have only been used internally, but they’re working on a parser so they’re able to make them public soon. In addition, through their work with us, Ian said they feel better prepared than ever to handle unexpected usage. When they were running their own CDN, they constantly felt like they were defending against potential disruptions. Now, they’re able to offload to do planning and predictability rather than just play defense.
What’s next?
With their delivery woes behind them, OpenStreetMap is now able to spend more time working on their core purpose: mapping the world. Our CDN enabled their teams to start building more customization into their platform, including localized versions of maps. For example, previously they’ve only been able to display data in English, or a specific country's national language. Now they’re working making it so a user can peruse the entire world map in their native language.
Explore what you can do with a modern CDN in our Guide to the Modern CDN: security and performance for today’s developer.
