Memory leak

[mourner]

When you're not accumulating results on reduce events, the memory consumption still creeps up so you can easily go out of memory on a large number of tiles. This doesn't look right — there's probably a big memory leak somewhere.

[mourner]

Found the cause. Asynchronous zlib.unzip leaks memory when you call it many thousands of times at once. Switching to unzipSync here fixes the leak (only supported by node 0.12+).

In theory, maxrate could help with this, but it is not set by default for mbtiles sources. But even if you set it, maxrate is broken anyway #44.

To fix it the proper way, we need to ditch maxrate completely and instead switch to a global queue with a max limit, so that only a certain number of tiles is processed at the same time, and a new tile doesn't start to be even loaded or parsed from MBTiles before we completed a previous tile.

[mourner]

cc @MateoV @morganherlocker @tcql

[morganherlocker]

nice catch!

[mourner]

I was wrong, unzipSync does not fix the memory issue entirely — memory still creeps up on long-running tasks. However the next-gen tile-reduce should solve all memory problems.

[mourner]

Will be fixed by #46.