[general] Unexpected error when running pipeline using crew controller

[psychelzh]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

This might be an issue of {crew} package and I am using crew version of 0.4.0. I am running my pipeline with the following targets options:

tar_option_set(
  packages = c("tidyverse", "lavaan", "collapse"),
  memory = "transient",
  garbage_collection = TRUE,
  storage = "worker",
  retrieval = "worker",
  error = "null",
  format = "qs",
  controller = crew::crew_controller_local(
    name = "local",
    workers = 8,
    seconds_idle = 60
  )
)

The pipeline takes about 15 hours to finish. But after about 8 hours or so, the pipeline errors with the following message. There is no error with the pipeline per se because it finished without error with targets::tar_make_future(workers = 8). I am sorry I cannot make a reprex for this.

Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: 'length = 31' in coercion to 'logical(1)'

BTW, currently on Windows, running with crew controller also shows popup windows for each worker just as following:

[wlandau]

Thanks for reporting this. I believe I suppressed the pop-up window in wlandau/crew@2c6e402.

I cannot reproduce the error message you see, so if you can reproduce it on your machine, I recommend peeling back layers. What happens when you run a similar workload with just crew? If you can reproduce the error with just crew, what happens if you run it with just mirai?

[robschick]

@wlandau - I'm also getting weird (but different) errors about daemons with crew, and would like to try what you suggest vis-a-vis peeling back layers. How do I run with just mirai? I had been doing this

tar_option_set( controller = crew::crew_controller_local(workers = 2) )

thanks

[wlandau]

Is it "'errorValue' int 5 | Timed out"? If so, it would be great to discuss details in a different thread (e.g. what kind of pipeline you have, what platform, and how frequently this happens). As for the steps of executing it in crew only and then mirai only, I think it will depend on how simple you can make your pipeline beforehand and still reproduce the error.