protect(): stack overflow error at relatively low resource usage

[sheffe]

I'm working with a moderately sized sf polygon object (all US census tracts downloaded by the tigris package), about 73k rows and 150Mb in memory according to object.size(). I'm using ms_simplify() with default params, and getting a consistent error of the form Error: protect(): protection stack overflow. Other sf polygon objects work just fine, but they're meaningfully smaller.

I'm running MacOS with 64GB RAM and system monitors show relatively low memory usage during runtime, so I can't believe I'm hitting a hard cap somewhere. I've tried using ms_simplify() on the sf object and a conversion with geojson_json() and geojson_list(), all to the same effect.

Unfortunately this isn't a small example, but I've put full code and a devtools::session_info() in the linked gist.

Getting lots of use from the package -- thanks!

[Update: confirmed that I can get this working by splitting the complete object into groups by state and then running ms_simplify() on groups, per below, but if I'm understanding the underlying process, that could result in unwanted boundary distortions for polygons neighboring each other across a state line.]

[ateucher]

Hi @sheffe - thanks for the report. I can replicate this as well. It looks like the error occurs in geojsonio::geojson_json (which is used internally in rmapshaper. From your example, if you run just geojsonio::geojson_json(alltracts) you get the same error. In particular, if you look at the output of traceback() after the error it looks like it is occurring with jqr... I'll post more detail tomorrow once I dig into it a bit more.

Can you also open an issue over at geojsonio? (I'm not really passing the buck as I am an author on that package as well 😉 ).

And yes, you're right about the risk of splitting it up in that the relationship of shared boundaries between states will be lost and you could wind up with gaps and overlaps.

[sheffe]

@ateucher done, and thanks for all the work.