Background map is incredibly old

[mfbehrens]

I am using Street Complete in an area where I added all the ways and roads some 5 Months ago however they still don't show up an the map in the background. The quests the app is showing me are fine. How often does the map get updated? Or is there maybe a bug. If not I would suggest to update the map data more frequent.

[Akasch]

At the moment I don't have the server capacities to host the whole World. Therefore some parts are served by https://www.nextzen.org/, but as far as I know the maps from them are older.

If you tell me which region you are from, I can see if it is possible to include it into my server (or you can sponsor a server or host the region yourselve)

[westnordost]

That most of the world (outside of the area highlighted in green here) is very old is a known problem, on the upside, nextzen is provided free.

Though since the up-to-dateness is a problem for many people, I could start thinking again about using a commercial vector tile hosting service and/or ask them if they'd like to sponsor this app by granting their service for free. Though right now I am busy with other urgent things regarding this app.

There are a number of other vector tile hosters around for who it must be ascertained first if their tiles are (a) regularly updated worldwide, (b) no/very few downtimes during tile re-generation and (c) are compatible with tangram-es. switch2osm lists a few, of which it looks like these offer vector tile hosting:

• https://www.thunderforest.com/
• https://stadiamaps.com/
• https://www.maptiler.com
• https://www.mapbox.com/
• https://www.jawg.io

At least the first two are from people which are active in the OSM community which is a plus. So anyone wanting to do some nonbinding enquiries about the above mentioned requirements, please feel invited to do so! For Point (c), it must be MVT format of course, but the details are more difficult to acquire. For starters, it would help to ask if they serve the identical data as mapbox (or another big player) or something custom with hopefully a documentation.

Another angle would be to find someone, like FOSSGIS, who would sponsor the costs for a dedicated vector tile hosting server which @Akasch could maintain, if he is still up to it.

[lilioid]

Hi,
I'm a friend of @Akasch and currently working myself into what he has built as server infrastructure. I'm planning to do some major code cleanup and general hosting improvements in the course of this year. This will hopefully make everything more scalable, maintainable and up-to-date. It would also make it easier for others to chime in and help with hosting resources.

Of course we wouldn't say no to additional hosting resources and gladly administer them ;)

[westnordost]

That's nice to hear!
What would make it easier for others to help with hosting resources would be a dead-simple one-click setup. So, probably a ready-made docker container these days. Though if it is just python scripts(?), shoving a package of preconfigured scripts into a certain directory of a good ol' webspace might do the trick too.
Finally, how to host resources could be included in the CONTRIBUTING.md

I don't expect that much help in hosting though. Not many users who read this tracker will have rented some (v)server with free capacities and commitment to maintain it. Right now, Oceania is hosted by someone else than @Akasch right? Who was it again? This information could perhaps be added to the map showing the servers?

[goldfndr]

Esri's OpenStreetMap vector basemap is now in general release and there is a variety of basemaps.

Best of all, the OpenStreetMap vector basemap hosted by Esri is freely available to any user or developer to use in your map or app! The OpenStreetMap vector basemap hosted by Esri is provided under a Creative Commons by Attribution (CC BY 4.0) license so that it may be used freely, and without transaction limits, in your internal and public facing maps. You just need to give appropriate credit for use of the map (i.e. “Map data © OpenStreetMap contributors, Map layer by Esri”) in your work.

There's a sample web deployment via Tangram. Android deployment might require the ArcGIS Runtime SDK which presumably involves specific licensing.

(Apologies if I've mentioned it previously; I haven't been keeping up these days.)

[lseelenbinder]

Hi @westnordost, I'm co-founder at Stadia Maps! I'm happy to answer any direct questions you have.

We use OpenMapTiles for our schema, with a few (currently only additive) adjustments. Our data update cadence (every month is our target, we're at roughly 2 months right now because of holiday break) may be problematic for your needs, but we can discuss that.

We'd be willing to find some middle-ground on costs since StreetComplete is so important to the ecosystem. Happy to talk further on here or via email: luke.seelenbinder@stadiamaps.com.

[westnordost]

Hi @lseelenbinder, thank you very much for the offer! I also met Ian on the SotM but I fear he might be a bit disappointed at me for not following up on the default speed limits research we've been doing a together a bit. As said I am currently very busy with other things so I won't be able follow up on this for some time (I mean your offer). The information about OpenMapTiles is valuable, I assume this format is used by several vector tile hosting services. It is left to find out if what of https://github.com/ENT8R/streetcomplete-mapstyle must be changed for the style to work with that format.

[lseelenbinder]

I think @ianthetechie understands—we're all insanely busy. 😄

He also is a bit better at styles than I am, so I'll let him comment on how much work would be necessary to adjust the theme to our schema.

[ianthetechie]

No worries @westnordost haha. I've been equally busy.

Regarding the style, tangram and Mapbox GL JSON styles are a bit different, so you would have to do some work to restyle this, but since your style is pretty minimalist, I don't think this would be too much work.

[RubenKelevra]

Hey guys!

What about using a Collaborative Clusters from IPFS to help the guy(s) which offers the map?

This way everyone can just set up a IPFS client and add a follower to the cluster and provide storage as well as bandwidth.

The Gateway from IPFS to HTTPS can be provided from Cloudflare.

This way the map is always fast and tiles are redundantly stored in many different places and there's much more bandwidth available.

Info about Collaborative Clusters:

https://cluster.ipfs.io/documentation/collaborative/

I'm running a Collaborative Cluster, feel free to ask me for details how to set it up and feed it with the data. :)

[lilioid]

@RubenKelevra thanks for the idea, it certainly sounds like a viable alternative and I quite like the IPFS project too :)

It might just not be the solution to our actual problem but I see that this is not really clear from outside.
Our problem is not really storage or bandwidth per se but rather the fact that we do processing on the whole OSM world dataset before committing them to storage. In addition to that, the storage is not just a dumb blob storage but a postgresql database with a lot of indices (for zoom levels and such) which makes querying the data a bit complex too.

Now we might be able to work something out by using IPFS but I don't yet see how

[RubenKelevra]

If I understand your pipeline correctly, you're downloading the OSM-XML file, process it to a PostgreSQL.

If there's a request you're the querying the PostgreSQL for the data which is inside the boundaries of the tile, like roads, houses etc. to generate the SVG, right?

So your sever is basically on it's limit if someone is querying a lot of tiles on an area you haven't processed before, because those SVGs cannot be delivered by your nginx.

Do you know the option proxy_cache_use_stale updating; combined with proxy_cache_background_update on; from Nginx?

This basically tells nginx to push old files to the clients where the time of life has already expired, while nginx asks the backend now for a new version of the file in the background, to update it's cache.

So you can basically delay the update process, while the clients gets outdated tiles but these are served very fast.

When the client asks the next time if the tile is up to date (like on the next day) nginx will deliver the new file to the client, since the backend has refreshed the tile.

[RubenKelevra]

@ftsell

Another option would be to switch to a different database. RethinkDB for example can handle geo data very well and can natively store points, lines and polygons.

The clustering of more than one server and sharding/replication of data can be done on the fly and very easy on a web interface.

Queries on a cluster is calculated in parallel by all shard members and send back to the client when completed in a transparent way, so the client doesn't have to care if he speaks to one server or a cluster. You can also use this to run stuff on multiple servers, since the dataset is writeable on all cluster members. :)

It supports native Python, Ruby, Java and JavaScript functions, so you don't have to create SQL queries, but write your queries in native functions which I found always pretty intuitive.

You can also import and export GeoJSON directly from and to the database.

A nice feature of RethinkDB is that you can stream updates to clients (called changefeeds), so you can query the database and if there are updates which match the query, they are send to the client in real time.

RethinkDB also supports map & reduce which runs inside the database cluster.

Not sure if this would fit your needs, just a heads up that there are more options than just PostgreSQL :)

https://rethinkdb.com/docs/geo-support/python/

[rugk]

Generally said, would not it also be a first step to "open-source" the server config/"pipeline process". (configs files and scripts or so used) So people can see and directly create issues/comment on the setup/files like the nginx config etc.? (just as done above, but with guessing)

[Akasch]

@rugk most of the used code is in the repositories at https://github.com/Map-Data only the nginx configs for the distribution and caching are missing. But most of the code is not really documented. I will see if I can clean up the nginx configs and publish them to.

@RubenKelevra while I think IPFS and RethinkDB are both interresting projects I don't think we can use them to make the tileserver faster/better. One of the mayor points is that we would have to rewrite the used libraries, and even if we would update the Database based on the minutely osm changeset feeds, the problem is more with the preprocessing of the data, as to apply an update the data has to be preprocessed and correlated with existing data.

The problem at the moment is mostly that we do not have enough fast storage to make it possible to host nearly the full world again. Maybe some more CPU power is required, but I think with more SSD space most of the problems would mostly solvable.

I will try to write a document about the general structure and how the different parts are interacting with each other in the next days. @ftsell and I are also planning to make it possible to host a region via a simple Docker command, provided the server has ~ 50 GB free space (SSD is way faster in the initial setup) and ~4 CPU cores (some cores can be shared if more than one region is hosted on the same server)

[RubenKelevra]

The problem at the moment is mostly that we do not have enough fast storage to make it possible to host nearly the full world again.

@Akasch host what? Tiles or the database? 🤔

[RubenKelevra]

@rugk most of the used code is in the repositories at https://github.com/Map-Data only the nginx configs for the distribution and caching are missing. But most of the code is not really documented. I will see if I can clean up the nginx configs and publish them to.

@RubenKelevra while I think IPFS and RethinkDB are both interresting projects I don't think we can use them to make the tileserver faster/better. One of the mayor points is that we would have to rewrite the used libraries, and even if we would update the Database based on the minutely osm changeset feeds, the problem is more with the preprocessing of the data, as to apply an update the data has to be preprocessed and correlated with existing data.

Yeah maybe you're right. It's interesting to get a basic overview to see how stuff can be improved. :)

[Akasch]

@Akasch host what? Tiles or the database? thinking
Host the Databases to generate the tiles on request (or if one want pre generate tiles).

To add to the requirements: while importing it is a good idea to have more than 5 GB of RAM, while running as regional tileserver, I think 2 GB should ok, but I havn't tested the RAM requirements.

[smichel17]

I have a pretty beefy coding/gaming desktop that's currently sitting around unused for most of the day. I don't have a good way to host from it, but I'm happy to donate compute power if possible.

[davidak]

@smichel17 this is unrelated, but have you considered to donate compute power to science using BOINC, for example in the World Community Grid? It only runs when the CPU/GPU is idling.

[Suburbanno]

[RubenKelevra]

@Suburbanno outch.

[michaelblyons]

Yeah, I've been lurking this issue since January. I have a similar problem wherein I can't (for instance) do any building-related quests in my local area because the building traces in OSM are newer than the data Nextzen has.

[RubenKelevra]

@westnordost how about getting in touch with Andy Allan from Thunderforest?

Using their Tile-API for an OpenStreetMap-Editor may be free of charge - JOSM at least uses their service for Tiles for years.

Their tile-services at least are quite nice to look at and really fast. :)

[lseelenbinder]

The offer from @stadiamaps is still on the table, as well. We're happy to sponsor vector tiles (MVT) for this.

I think the primary issue with switching is that it requires reworking the themes, etc. It's not as simple as a drop-in replacement.

[Akasch]

Currently I have the data preprocessing running, and will reimport the region of Europe in the next 1-2 days. After that I will look how many additional regions I can host, while doing it I try to make the process better so it can run automated and also is easier to run the parts of the process.

Switching to an other existing provider would, of course make it easier for me, but I think it would require a requite of the styles, I don't know how compatible they are.

[smichel17]

Hm. Maybe instead of DDNS I could set up a load balancer of some sort, that would forward people to my or @Helium314's home PCs if they're online and nextzen otherwise. That way we wouldn't exceed capacity and it wouldn't be as big a deal if I had less uptime. However, this has some complications -- I would want to send redirects or something rather than proxying all traffic through my hosting provider; maybe at that point we should just use bittorrent :D -- and is a bigger project that I'm not sure I have time for right now.

[Atrate]

You mention Bittorrent jokingly, but a distrivuted P2P network updated from a could of central servers could work of there was a protocol that would support quick peer discovery and reliable incremental updates

[rugk]

Don't claiming or evaluating (whether) this is the solution, but let's throw https://ipfs.io/ into the room… it's made for p2p and a lot of data.

[lilioid]

@Helium314

Like @smichel17 I'm thinking about hosting from home
I have several questions about this (@Akasch?):

Great 👍 we're planning to make it possible for self-hosters like you to contribute their hosts to our server-mesh (if you wanna call it that) but we're not quite there yet.

1. will this actually provide recent map data, or just another copy of the outdated tiles?

Yes it will provide recent map data.

2. what do I need to do? Setup from https://github.com/Map-Data/regiontileserver and get some nginx configuration?

Essentially yes since regiontileserver is the server which actually hosts the data. It does so from a postgresql database so you'll need one as well.
Currently the script for importing into postgres is in this repo too but we might change that.

Additionally osm-tile-data-extract is used to download upstream data from OSM and split it up into smaller chunks so that one tileserver can serve one such smaller chunk later.

Now because we don't use on huge machine for serving the whole world, we also have tileserver-mapping. This server knows which tileserver handles what chunk and tells nginx via X-Accell Header the correct tileserver.

3. is my computer actually good enough? 4 GB memory, i5 4300U CPU, ~70 GB to spare on SSD

To host one or two tileservers for small chunks of the map, probably but for the whole world definitely not. The whole OSM planet dump is currently around 50GB and you will need some working disk space to do the splitting into smaller chunks.

4. is 50 Mbit/s upload enough?

I don't really know but if it's not used too much probably.

---

I'll write this down properly in some documentation later this weekend ;)

[smichel17]

Thanks @ftsell, that's a great overview. One piece I'm missing: how does the data get transformed from raw OSM to vector tile format? I was looking at openmaptiles.org the other day and it seems like this process is fairly compute-intensive. Is that part of the script for importing to postgres?

[lilioid]

It's currently in regiontileserver in some bash script but it would better fit into osm-tile-data-extract.

Yes it is fairly intensive because a lot of indexes have to be generated. It would be nice to do the transformation directly after we split the chunks and then simply provide postgresql dumps for others to download.

[Helium314]

So we need to

• split up the planet into regions
• import region(s) to postgresql
• host a tileserver for one or more regions

As far as I understand it shouldn't be hard to distribute this work, e.g.

• someone splits up the planet and hosts regions (probably good bandwidth is useful)
• import is done on several computers (because it is slow), and postgresql dumps are uploaded to the data server above. The computers doing the transformation do not need to be used as tile servers. This would allow to distribute the work better, e.g. I could use my main computer.
• tile servers download/import postgresql dumps for some regions and serve tiles

Now I don't know much about how all this really works, especially the database stuff.
So is this actually possible?

[lilioid]

Yes that sums it up pretty well. It just needs to be implemented where most work needs to be done too coordinate / trigger these schedules.

[goldfndr]

The whole OSM planet dump is currently around 50GB

If this is only for the background map then presumably most of that data can be filtered out via osmfilter or something similar. Doing a cursory look at the map styles, it looks like the tags used are limited to:

name
addr:housenumber
oneway
natural
boundary
waterway
building
landuse
amenity
leisure
barrier
location
highway
railway
aerialway
route=ferry
route=piste
aeroway

(and probably further limited by more specific values, e.g. specific barriers)

But that seems incomplete, as somehow relative building heights are extruded.

[westnordost]

In fact, I evaluated a few commercial vector tile providers in the meantime, see the different branches on https://github.com/ENT8R/streetcomplete-mapstyle

Vector Tile Providers

openmaptiles (maptiler)

• barriers such as walls, fences, hedges, city walls etc. missing are missing. It sounds like they do not plan to add it into the main source.
• a property to recognize private roads is missing. But they find it relevant and I added a feature request.
• mini roundabouts, turning loops and turning circles are missing. But they find it relevant and I added a feature request.
• 3D buildings don't render as good as for nextzen tiles, there are some bugs
• mediocre appearance on medium zoom level (z8): small landcovers are not merged together but just left out
• maptiler tiles are updated weekly

openmaptiles (stadiamaps)

same as for maptiler, only:

• additionally, housenumbers are missing. They do not plan to add it to the main source but maybe provide it as a second source in the future.
• stadiamaps tiles are updated monthly. There are plans to get down to bi-weekly or even weekly updates.

thunderforest

• very good appearance on medium zoom level (z8,...): small landcovers are merged together and geometry is simplified very carefully (or not at all?)
• the tiles are very huge, 2-4 times the size of openmaptiles on high zooms, several MB on medium zooms (z8,...). They are working on further reducing the size.
• there is no height information for buildings and thus no 3D buildings at all. They are aiming at adding at least basic support for 3D buildings (so, no building parts).
• mini roundabouts, turning loops and turning circles are missing. It doesn't have a high priority for them to add it.
• the tiles are updated every few days

jawg.io

• good appearance on medium zooms (z8)
• the tiles are very lightweight, having roughly a third of the size of openmaptiles tiles. On the other hand, jawg.io is the only service of those tested that serves tiles until z16 (others till z14), meaning that there are 20x as many tiles served
• they have full 3D buildings support
• mini roundabouts, turning loops and turning circles are missing. It doesn't have a high priority for them to add it.
• no filtering between service rails and other possible. They do not plan to add this info.
• the tiles are updated daily

MapBox

I haven't looked closer at mapbox tiles yet as I didn't find the TileJSON url. What I know is that they also update daily and that they serve tiles till z16.

Next Steps

I've looked at the tile sizes, but not at what actual traffic is generated. Next step would be to look at that. Some properties currently in the stylesheet are not that important, such as mini roundabouts etc.
The things I deem important are:

• size
• up-to-dateness
• 3D buildings
• housenumbers

[ZLima12]

I could possibly host a server for the New York area.

[ENT8R]

Mapbox

The documentation of the Mapbox vector tiles seems to be at https://docs.mapbox.com/vector-tiles/reference/mapbox-streets-v8/ but as far as I understand, it is not allowed to use them in your app without using their own Mapbox Mobile SDK.

In their service terms they state the following:

7. Mobile SDKs.
   7.1. General Requirements.
   You must use the Mobile SDKs as your exclusive means of accessing the Services in mobile applications. The Mobile SDKs will periodically send location and usage data to us, which we may use for the purpose of fixing bugs and errors, accounting and generating aggregated anonymized statistics. [...]

[westnordost]

The documentation of the Mapbox vector tiles seems to be at https://docs.mapbox.com/vector-tiles/reference/mapbox-streets-v8/ but as far as I understand, it is not allowed to use them in your app without using their own Mapbox Mobile SDK.

I think this is a pricing policy. MapBox has special pricing for usage of their tiles for when using their mobile SDK: You pay by number of users, not by number of requests, as far as I know. There is also pricing by number of requests, so maybe that wouldn't be a problem.

[westnordost]

Okay, I looked at the total download size by example of zooming in from z0 to z16 to the city center of Hamburg. Relative to the current nextzen tiles, the sizes are as follows:

	zoom 1-7	zoom 8-12	zoom 13-16
openmaptiles	1.5x	0.7x	0.7x
thunderforest	11x	6x	2.5x
jawg.io	0.4x	0.3x	0.4x

Remarks:

• In real usage, sizes for lower zooms are less relevant. Also, nextzen and jawg tile size will go up, as more z15 and z16 tiles will have to be loaded when the user moves around, while with openmaptiles and thunderforest, the z14 tile is already the end of the line.

Two conclusions:

• thunderforest is quite far off from the others, especially on lower zooms
• even though jawg.io serves tiles till zoom 16, it doesn't make it bigger (than openmaptiles), each of those z16 tiles is tiny

[westnordost]

After this evaluation, I decided that the best choice for StreetComplete are jawg tiles:

• They are the smallest, which is important for use on mobile devices. That of course also means that they have the fewest features, but the map style of StreetComplete is simple anyway
• They are the most current

JawgMaps also provides the tiles to StreetComplete for free, as you can now read in the readme and in the changelog of the new beta I am compiling right now. Though, the other providers I talked with were also very forthcoming to reach an arrangement and open for feedback. I can recommend all the providers I looked at, each having some unique selling points:

• JawgMaps has the most current and lightweight tiles
• Thunderforest has the most detailed tiles with a lot of different information on it, are also very current and economically priced
• MapTiler and StadiaMaps are quite economically priced and can be considered as more compatible as they build on an open standard (OpenMapTiles)

[ianthetechie]

Thanks again for posting your detailed analysis here. This is really interesting stuff, and the feedback is really valuable for helping us improve things like tile sizes.

[westnordost]

Well this size-analysis was quite rough, I'd expect the values to very much fluctuate in different areas of the world.

But what I can definitely say from that is that there is room for improvement when handling mid and low zoom levels for all four of the services I looked at in one way or another for:

• better simplifying of geometry on lower zooms
• merging together neighbouring same(ish) landuses and -covers

JawgMaps has an advantage here because they throw many landuses and -covers into one pot, for example industrial and residential landuses are both just rehashed to "other" landuses. So if they are the same, they can be merged together for lower zoom levels.
For tile providers who want more detail, this means that it is expedient to group landcovers in a hierarchical manner. For example if amenity=school was mapped to something like layer: landcover, class: populated, populated: residential, residential: education or similar, on lower zoom levels, these polygons could be merged together with other populated polygons.

[RubenKelevra]

Thank you for investing so much time in this issue! The new service works great! It makes fun to start the app again, since you don't wait like 10 seconds on a blank screen. :)

[michaelblyons]

I'm having trouble installing the Beta overtop my F-Droid provided version on Android 10.

• I downloaded the APK from the GitHub Releases page.

• I then scanned it with ClassyShark.

• When I select it in my Android file browser, I get what appears to be an OS message saying

  Your phone and personal data are more vulnerable to attack by unknown apps. By installing this app, you agree that you are responsible for any damage to your phone or loss of data that may result from its use.

• If I tap "continue" (instead of "cancel"), I get another OS dialog saying

  Do you want to install an update to this existing application? Your existing data will not be lost.

• If I assent to that, I get an "Installing..." progress bar.

• When the progress bar completes, the text changes to "App not installed."

And now you know what I know.

[matkoniecz]

See https://www.chiark.greenend.org.uk/~sgtatham/bugs.html - it is completely unclear what is going wrong. Error message or explanation what exactly is going wrong is necessary (for example "StreetComplete crashes on start with message XYZ" is different from "install fails because I run out of memory" or "F-Droid is down"). Also, it is basically certain that it is a new unrelated issue.

[Helium314]

This is because the F-Droid version is signed with F-Droid keys, so you can only update with versions from F-Droid.
If you have root access there are some ways around this, otherwise you need to remove the F-Droid version and then install the Beta.

[smichel17]

The debug version can also be installed alongside the release version, if you're up for installing Android Studio on your computer and building the project yourself.

[michaelblyons]

21.0 is on F-Droid now. It's so great! Thank you!

[sinekonata]

After this evaluation, I decided that the best choice for StreetComplete are jawg tiles:
...

Does this mean that it is now implemented? Is StreetComplete using the tiles that would re-generate daily? In Belgium, they re-generate every 3 days, which is quite good when you see other OSM clients usually have months old tiles.

Thank you so much for your work btw, this app has gotten me into this whole rabbit hole :D

[westnordost]

Yes, since v21.0 (June)