Alternative ways to determine the list of tiles to process

[anandthakker]

Couple possibilities I could see being useful:

• Accept options.tiles as a direct list of tiles
• Something like "do all the tiles in this mbtiles file" (since getting such a list is possible with mbtiles)

[tcql]

Also, some internal tools allow supplying a geojson polygon either as a string or in a file, then generate tile-cover from that. It would make sense for this to be native to tile-reduce.

[mourner]

Yes, would be very much up to improvements on this front! I'm especially interested in the possibility to stream tile numbers one by one through the API. E.g. not specifying the area in the constructor, but calling something like job.tile(tile) in a loop afterwards or accepting a Node stream.

Direct list of tiles is supported in the current version but I need to port it over to the nextgen branch. It won't help in case of millions of tiles though — still a memory problem.

[anandthakker]

@mourner Oh yeah: sending tiles in dynamically via job.tile() or a stream would be excellent

[mourner]

I'm thinking of an even more generic cool API: if you do not specify area in constructor options, you can then job.run(options) as many times as needed, where options can have tiles, bbox, polygon etc.; it would start processing as soon you start serving tiles. Then you run job.end() to indicate that you served all the tiles you wanted to process, and now waiting for the end event. cc @tcql @morganherlocker

[tcql]

addressing the first part of this (allowing different types of job areas) in #49. I'll do a PR soon for @mourner's suggestion of streaming in tiles

[mourner]

Instead of complicating the tile-reduce job API, how about accepting a next function as the area that can be used to fetch tiles one by one in streaming fashion?

next(); // {value: [10, 10, 2], done: false}
next(); // {value: [11, 10, 2], done: true}
///

Then we could create a version of tile-cover that streams tiles like this (but without deduping because it would need hashing all tiles in memory).

[anandthakker]

👍 to the iterator approach -- would this mean options.tiles could be either an array or a next function?

[mourner]

@anandthakker yes, something like this. I'll look into this and also doing a streaming https://github.com/mapbox/tile-cover version for polygons — it should be relatively easy to do.

[mourner]

@anandthakker how about using Node readable streams that stream tile objects instead of iterable? I found out that tippecanoe can generate a list of tiles for a given mbtiles file like this (without the need to run any cover algorithms):

../tippecanoe/enumerate us-west.mbtiles | awk '{print $2,$3,$4}' > us-west.txt

Then we can turn it into object stream as easily as this:

var split = require('split');

var tileStream = fs.createReadStream('us-west.txt')).pipe(split(function (line) {
  return line.split(' ').map(Number);
}));

[anandthakker]

@mourner 💯 I think a Node object stream is ideal.

[anandthakker]

^ also note: node-mbtiles can do something similar too

[mourner]

@anandthakker I ended up implementing the API that accepts the path to a tile list file rather than a generic stream, because Node streams are really hard and confusing. This may be good enough for our use cases though. Pushed to the nextgen branch, try it out!

[mourner]

Happening in #46