You Don’t Need To Integrate Google Maps or Mapbox in Your Apps
Create and host your own stylable vector tiles to make maps that stand out
In this article, I’ll introduce you to the toolset and process to create vector tiles and show you how you can host the map data statically and cheaply without the need for a server. In the end, you’ll be able to use the map tiles in web-based products and mobile apps.
Maps are cheap and most of the time — free. Not quite. There are situations in which you can use Google Maps or Mapbox complete for free or at least at a reasonable price.
However, there are some contexts in which maps become super expensive and therefore are not an option for small businesses.
Furthermore, especially if your product or site is targeted at EU customers, GDPR is always an obstacle you have to overcome. Hosting the tiles yourself makes all of the above less of an issue.
Maps have become an essential part of our daily lives. Chances are high that you’ve either encountered maps, which are being powered by Google Maps, Apple Maps, or Mapbox.
Nowadays, most maps (if not all) are some kind of derivative of data collected by the OpenStreetMap community.
When I started working on Aerwork, I already knew that I needed maps to visualise flight paths and display airport locations.
TL;DR If you don’t care about how I got to my final result, you can skip ahead and go directly to “The Way To Go”: a step-by-step guide on how to convert, serve, and display your own map.
Approach #1. Use Any Commercial Mapping Platform Like Mapbox or Google Maps
As a developer with a strong visual focus, it was a no-brainer to head over to Mapbox, install their JavaScript SDK, design some beautiful maps in Mapbox Studio, and I'm done (more or less, no flight paths yet but the basic setup is done).
When I looked at their licensing page, my initial euphoria quickly vanished. For each web seat (if I provide maps to authenticated users), I would have to pay $4. For a SaaS without a single paying customer and a Plus tier starting at only $8, it seemed to be a dealbreaker. All other mapping providers offer a product in a similar price range and therefore did not make it into the product either.
Approach #2. Host the Map Tiles Ourselves
Next, I thought about providing the map tiles by myself. I mean it can’t be that hard. Raw OSM data is available for free under an ODbL license and therefore is allowed to be used in a commercial product.
Well, things got ugly quite fast. Raw OSM data is insanely detailed and therefore huge. Trashcans, trees, buildings, airports, you name it. It’s there. Kudos to all OSM contributors.
At the same time, this is an issue if you want to work with the data. It turned out that it is a rather ambitious plan to do everything yourself as there’s no “Just add water” solution. After doing some research I figured out that I needed an MBTiles file that contains the map tiles for the specific area of the world (or the whole planet) I am interested in.
An MBTiles file is a format for storing map tiles in a single file. Technically it’s just an SQLite database. By using an MBTiles file you’re able to style the map using the Mapbox GL style specification, which is a huge benefit as there are some editors out there to visually define the map styles e.g., maputnik.
So I needed to get my hands on some MBTiles. Maptiler (which is another mapping platform) offers already prepared MBTiles for certain areas or even the whole planet.
You can subscribe and get regular updates or pay a one-time fee to download the MBTiles file for the required area. As stated above, Aerwork is a SaaS just starting out. I didn’t want to spend hundreds or even thousands of dollars along the way just to provide nice-looking maps. I’d love to do that but that’s far away in the future.
Even though Europe would be sufficient for now, I couldn’t find any MBTiles file of Europe for free (or at least a bit cheaper than what maptiler is offering).
So the plan stands. I create the tiles myself. After stumbling across the open source toolset that maptiler provides for this exact task of converting raw osm data to a usable MBTiles file, I’ve pulled the docker image from the openmaptiles repo, followed along with their documentation and started by testing it with Hamburg.
Hamburg itself is rather small and therefore a good starting point to see if the result is something I can use in our product. The conversion went well; it’s been quite fast (Macbook Pro, 2.9Ghz 6-Core i7, 32GB RAM). If Hamburg is quick, why not try the same with Germany? Well, it quickly turned out to take quite long. I was still optimistic about doing the same with Europe on my machine. Now the frustration began.
After running it for 14 hours, it just crashed with an error, telling me there’s not enough space left. 700GB of free space is not enough? The raw source data is about 30GB. 700 GBs should be enough. Well, chances are high that I did something wrong. I reread the docs thoroughly to make sure I got everything right the second time. This time it crashed after about 12 hours with some non-descriptive error. I did the same steps as in the previous iteration. Great. Thanks for nothing.
I thought about taking a different approach. As I only wanted to show flight paths and airports, I thought about getting rid of buildings and other details. Buildings are the biggest chunk in OSM data and therefore take up a lot of time and computing power.
Luckily there are tools out there to help you edit raw OSM data. First, I started out using osmosis. The latest release is from September 2020. Even though it’s not super up to date I still gave it a shot.
While trying to figure out how to properly filter the raw data, I was able to shrink the size of the raw OSM data. Not extensively but at least a bit. However, even for Hamburg, it took quite some time.
I reran the openmaptiles tools but still ran into numerous errors. OK, maybe there’s another tool to filter unwanted data? Say hi to Osmium. It’s written in C++ and should be fast. I gave it a shot.
I managed to also shrink the size of the OSM data. It is quite fast. Osmosis is nowhere near osmium, but for Europe, it still took forever, and at some point, I decided to kill the task. So, basically, I was back at square one.
After doing some further research, I found tilemaker as an alternative to openmaptiles. It’s also based on C++ and has no dependencies. Oh boy, fingers crossed. I started again with Hamburg. It went well. There it was — our MBTiles file.
I dropped it into our tileserver-gl (which reads the MBTiles file and provides the requested tiles for the specific area the client needs) and voila I had a working map with self-converted and self-hosted tiles. Yeehaw! I couldn’t believe it. Mentally preparing for converting Europe, I reread the tilemaker doc just to get it right the first time. Unfortunately, the docs say tilemaker should only be used for small extracts and not whole continents.
Good morning. It’s Groundhog day. Greetings to Bill Murray. WTF. I did some research and found this blogpost (german) by Marcel Normann.
According to his findings, tilemaker uses quite a lot of RAM. He found out after doing some benchmarks, the factor is 12 RAM to filesize of the raw data.
So, Europe would need around 256GB of RAM. This kind of RAM is not even affordable when outsourcing everything to AWS, so I nearly gave up. It can’t be that hard to just display a nicely styled map without relying on some third-party company.
After taking some deep breaths and rethinking all the options, I went back to my initial approach, using the open source openmaptiles tooling with just a bit more resources.
Luckily, AWS offers some powerful EC2 instances to make the process quite easy. This way it’s not blocking my machine the whole day because AWS offers compute time for reasonable prices.
So below, you’ll find the solution which turned out to work quite nicely in the end — providing you’ve got enough computing resources.
In the end, I learned a lot about OSM data itself, how to manipulate raw osm data, and how to trust your gut. Most of the time your intuition is right, and the first choice is sometimes the best. I had to learn it the hard way this time.
The Way To Go
- Launch an AWS EC2 Instance (I use an c5.4xlarge, which is optimised for computing tasks. It has 16 vCPUs, 32GB RAM, and a 500GB EBS gp2 SSD storage volume)
- Install docker and docker-compose
3. Clone the required tooling for working with raw OSM data and convert it to a *.mbtiles file
Now this will take quite some time. Our last run took 48 hours for Europe.
In order to provide the tiles to your frontend or mobile application, there are two options:
Option 1: Run a server (e.g., a PHP-based tileserver or tileserver-gl), which takes care of extracting the necessary tiles from the MBTiles file.
Option 2: Extract the tiles and host them statically.
I went with the latter approach as I always want to keep maintenance to a minimum, have an easy-to-grasp infrastructure, and leverage the caching mechanism of AWS Cloudfront.
Using mbutil is the way to go. It’s quite simple. Be aware that the t*.pbf files that are being extracted by mbutil are gzipped. So, if you use S3, make sure you include the flag —-content-encoding gzip when copying the files.
This adds additional metadata to the object that is being stored in S3 and helps Cloudfront to handle it correctly. You can also un-gzip the files in case you want to gzip them on the fly or don’t want to use compression at all.
Note: It took 48 hours to get an MBTiles file of Europe with an EC2 5c.4xlarge instance, with 500GB EBS gp2 Volume, running Ubuntu Server 20.04. I was constantly on edge, hoping for the process not to crash randomly in between.
Optional step —Un-gzipping the files
Optional step — Using a server to host and serve the tiles
For some reasons you might want to keep the tiles in one file (regular updates, for example) and leverage a tileserver to do all the heavy lifting for you. Serving the tiles, caching them, or even providing prerendered raster tiles are some benefits of running a tileserver yourself. It’s pretty straightforward to accomplish:
- Download/Upload TileServer PHP onto your server
- Copy the *.mbtiles file in the root dir next to the other tileserver files
- Open up your browser, navigate to your directory, and there it is.
TileServer PHP also offers you a code snippet on how to render the map in a browser using mapbox-gl-js (as of mapbox-gl-js 2.x they’ve adjusted their licensing, and now you’re not allowed to use the js library without an active subscription anymore. Therefore you either have to use 1.x or maplibre-gl-js, for example).
Tips
1. In order to speed up the process, you can manipulate the raw OSM data with the osmium-tool by filtering out buildings (buildings take up a lot of space).
2. Prevent using zoom levels beyond 14. Believe me, you don’t want to go there.
3. If you plan on using smaller extracts e.g., single countries, I’d strongly suggest using tilemaker. In comparison to the openmaptiles approach it's insanely fast.
Resources, Tools, Raw Data, and Other Useful Links
- AWS EC2 5c.4xlarge with 500GB EBS gp2 Volume, running Ubuntu Server 20.04
- Openmaptiles Tools (Converting raw OSM data into MBTiles)
- Osmium Tool (Manipulating raw OSM data)
- TileServer PHP (Serve MBTiles)
- Geofabrik OSM Data (Download raw OSM data)
- Tilemaker am Limit (Article about RAM usage of tilemaker. In German)
- Pre-Converted *.mbtiles (quite pricy though for commercial use, depends on your budget)
