Process queue to offload ImageMagick

[icarito]

This is a note that in production we're seeing large images (I observed upwards of 15000x15000 if I believe htop) - causing convert to take a huge time and RAM and bringing the app down.

Benjamin Sugar suggested offloading to a process queue of some sort. I a not familiar with it but I think Redis is popular.

It'd be nice to not worry about this kind of downtime.
Thanks!

[icarito]

Note that alternative or interim measures to better contain ImageMagick are appreciated. Currently we use /etc/ImageMagick-6/policy.xml to limit resource usage. But I think the problem occurs because we run several processes at once.

I've further throttled to 25% cpu load and added a 2GB RAM limit per process:

  <policy domain="resource" name="thread" value="1"/>
  <policy domain="resource" name="throttle" value="30"/>
  <policy domain="resource" name="memory" value="2GiB"/>

I hope it will hold for now.

[jywarren]

I believe we can encourage people to download images one by one using this issue: publiclab/Leaflet.DistortableImage#83 But if we can also further modularize this code, we could come up with a way to "send" this to another application, potentially? https://github.com/publiclab/mapknitter/blob/master/app/models/map.rb#L204 Perhaps we could modularize this operation which runs on all images in the map: https://github.com/publiclab/mapknitter/blob/master/app/models/map.rb#L282-L302 The code run on each image is 117 lines long, even before it attempts to join the images together: https://github.com/publiclab/mapknitter/blob/master/app/models/warpable.rb#L153 That code could potentially be reworked. However, all of this is pretty astoundingly heavy, and dense code.

…

On Thu, May 3, 2018 at 1:56 AM, Sebastian Silva ***@***.***> wrote: Note that alternative or interim measures to better contain ImageMagick are appreciated. Currently we use /etc/ImageMagick-6/policy.xml to limit resource usage. But I think the problem occurs because we run several processes at once. I've further throttled to 25% cpu load and added a 2GB RAM limit per process: <policy domain="resource" name="thread" value="1"/> <policy domain="resource" name="throttle" value="30"/> <policy domain="resource" name="memory" value="2GiB"/> I hope it will hold for now. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#258 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ9SXNxTH05IdUcbA3TxAzgsb0FhGks5tupwIgaJpZM4TwhnV> .

[icarito]

I think if we could put outside processes in a queue and only run a few at a time it could work. Perhaps there's a way to transparently offload the ImageMagick commands themselves... e.g. replace those scripts with `ssh somootherserver convert` and have both servers mount the same shared directory, for instance.

…

On 03/05/18 10:35, Jeffrey Warren wrote: I believe we can encourage people to download images one by one using this issue: publiclab/Leaflet.DistortableImage#83 But if we can also further modularize this code, we could come up with a way to "send" this to another application, potentially? https://github.com/publiclab/mapknitter/blob/master/app/models/map.rb#L204 Perhaps we could modularize this operation which runs on all images in the map: https://github.com/publiclab/mapknitter/blob/master/app/models/map.rb#L282-L302 The code run on each image is 117 lines long, even before it attempts to join the images together: https://github.com/publiclab/mapknitter/blob/master/app/models/warpable.rb#L153 That code could potentially be reworked. However, all of this is pretty astoundingly heavy, and dense code. On Thu, May 3, 2018 at 1:56 AM, Sebastian Silva ***@***.***> wrote: > Note that alternative or interim measures to better contain ImageMagick > are appreciated. Currently we use /etc/ImageMagick-6/policy.xml to limit > resource usage. But I think the problem occurs because we run several > processes at once. > > I've further throttled to 25% cpu load and added a 2GB RAM limit per > process: > > <policy domain="resource" name="thread" value="1"/> > <policy domain="resource" name="throttle" value="30"/> > <policy domain="resource" name="memory" value="2GiB"/> > > I hope it will hold for now. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#258 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AABfJ9SXNxTH05IdUcbA3TxAzgsb0FhGks5tupwIgaJpZM4TwhnV> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#258 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMMSzHVlSB5B66_XId19njWdCFpjr0lks5tuyPGgaJpZM4TwhnV>.

[jywarren]

is it important that it be a separate server? I think that initially we could try to make parts of this process asynchronous, or even to just not run too many at once? A close read of the code may show that everything is actually synchronous right now and that making it asynchronous without queueing could actually be a problem. Also note that there are both ImageMagick commands and GDAL commands. Coudl be worth checking if we need to cage the GDAL commands too? On Thu, May 3, 2018 at 11:38 AM, Sebastian Silva <notifications@github.com> wrote:

…

I think if we could put outside processes in a queue and only run a few at a time it could work. Perhaps there's a way to transparently offload the ImageMagick commands themselves... e.g. replace those scripts with `ssh somootherserver convert` and have both servers mount the same shared directory, for instance. On 03/05/18 10:35, Jeffrey Warren wrote: > I believe we can encourage people to download images one by one using this > issue: publiclab/Leaflet.DistortableImage#83 > > But if we can also further modularize this code, we could come up with a > way to "send" this to another application, potentially? > > https://github.com/publiclab/mapknitter/blob/master/app/ models/map.rb#L204 > > Perhaps we could modularize this operation which runs on all images in the > map: > https://github.com/publiclab/mapknitter/blob/master/app/ models/map.rb#L282-L302 > > The code run on each image is 117 lines long, even before it attempts to > join the images together: > https://github.com/publiclab/mapknitter/blob/master/app/ models/warpable.rb#L153 > > That code could potentially be reworked. > > However, all of this is pretty astoundingly heavy, and dense code. > > On Thu, May 3, 2018 at 1:56 AM, Sebastian Silva < ***@***.***> > wrote: > > > Note that alternative or interim measures to better contain ImageMagick > > are appreciated. Currently we use /etc/ImageMagick-6/policy.xml to limit > > resource usage. But I think the problem occurs because we run several > > processes at once. > > > > I've further throttled to 25% cpu load and added a 2GB RAM limit per > > process: > > > > <policy domain="resource" name="thread" value="1"/> > > <policy domain="resource" name="throttle" value="30"/> > > <policy domain="resource" name="memory" value="2GiB"/> > > > > I hope it will hold for now. > > > > — > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub > > > <#258# issuecomment-386197873>, > > or mute the thread > > > <https://github.com/notifications/unsubscribe-auth/ AABfJ9SXNxTH05IdUcbA3TxAzgsb0FhGks5tupwIgaJpZM4TwhnV> > > . > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#258# issuecomment-386337830>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAMMSzHVlSB5B66_ XId19njWdCFpjr0lks5tuyPGgaJpZM4TwhnV>. > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#258 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ7tlaNm5FMSC14u-3NZ2etf3u-33ks5tuySNgaJpZM4TwhnV> .

[icarito]

No it's not necessary to run it separately, it was just and idea to deal with it from the system level rather than the code level. Actually I've also read some options are possible to make a queue in the command line with a wrapper script. That is also an option if code itself is complex!

…

On 03/05/18 10:45, Jeffrey Warren wrote: is it important that it be a separate server? I think that initially we could try to make parts of this process asynchronous, or even to just not run too many at once? A close read of the code may show that everything is actually synchronous right now and that making it asynchronous without queueing could actually be a problem. Also note that there are both ImageMagick commands and GDAL commands. Coudl be worth checking if we need to cage the GDAL commands too? On Thu, May 3, 2018 at 11:38 AM, Sebastian Silva ***@***.***> wrote: > I think if we could put outside processes in a queue and only run a few > at a time it could work. Perhaps there's a way to transparently offload > the ImageMagick commands themselves... e.g. replace those scripts with > `ssh somootherserver convert` and have both servers mount the same > shared directory, for instance. > > > > On 03/05/18 10:35, Jeffrey Warren wrote: > > I believe we can encourage people to download images one by one using > this > > issue: publiclab/Leaflet.DistortableImage#83 > > > > But if we can also further modularize this code, we could come up with a > > way to "send" this to another application, potentially? > > > > https://github.com/publiclab/mapknitter/blob/master/app/ > models/map.rb#L204 > > > > Perhaps we could modularize this operation which runs on all images in > the > > map: > > https://github.com/publiclab/mapknitter/blob/master/app/ > models/map.rb#L282-L302 > > > > The code run on each image is 117 lines long, even before it attempts to > > join the images together: > > https://github.com/publiclab/mapknitter/blob/master/app/ > models/warpable.rb#L153 > > > > That code could potentially be reworked. > > > > However, all of this is pretty astoundingly heavy, and dense code. > > > > On Thu, May 3, 2018 at 1:56 AM, Sebastian Silva < > ***@***.***> > > wrote: > > > > > Note that alternative or interim measures to better contain ImageMagick > > > are appreciated. Currently we use /etc/ImageMagick-6/policy.xml to > limit > > > resource usage. But I think the problem occurs because we run several > > > processes at once. > > > > > > I've further throttled to 25% cpu load and added a 2GB RAM limit per > > > process: > > > > > > <policy domain="resource" name="thread" value="1"/> > > > <policy domain="resource" name="throttle" value="30"/> > > > <policy domain="resource" name="memory" value="2GiB"/> > > > > > > I hope it will hold for now. > > > > > > — > > > You are receiving this because you are subscribed to this thread. > > > Reply to this email directly, view it on GitHub > > > > > <#258# > issuecomment-386197873>, > > > or mute the thread > > > > > <https://github.com/notifications/unsubscribe-auth/ > AABfJ9SXNxTH05IdUcbA3TxAzgsb0FhGks5tupwIgaJpZM4TwhnV> > > > . > > > > > > > — > > You are receiving this because you authored the thread. > > Reply to this email directly, view it on GitHub > > <#258# > issuecomment-386337830>, > > or mute the thread > > <https://github.com/notifications/unsubscribe-auth/AAMMSzHVlSB5B66_ > XId19njWdCFpjr0lks5tuyPGgaJpZM4TwhnV>. > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#258 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AABfJ7tlaNm5FMSC14u-3NZ2etf3u-33ks5tuySNgaJpZM4TwhnV> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#258 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMMS_U9G_BsTFZoTtyPG8QLTAX61Kc6ks5tuyYkgaJpZM4TwhnV>.

[jywarren]

I'm wondering if we could create a Google Cloud mini app that includes imagemagick, gdal, and some Ruby, to offload key portions of this into a cloud-based queue.

We're also looking at how to implement this in JavaScript, but I want to be thorough and explore both means. And JavaScript won't be able to generate GeoTiffs and map tiles the way GDAL will.

Maybe starting with this? https://github.com/geo-data/gdal-docker

Then installing imagemagick on a variant of that, and Ruby?

Maybe we could integrate it with this? https://github.com/googleapis/google-cloud-ruby

https://cloud.google.com/ruby/getting-started/hello-world

[jywarren]

Merging into #296 and #298!