Error = 'continue' behaviour after cb85323

[stuvet]

Hi @wlandau,

I'm wondering if cb85323 had the intended effect when error = 'continue'?

Expected Behaviour

• As per documentation, when error = 'continue', the whole pipeline keeps going.

Observed Behaviour

Before cb85323

• When error = 'continue' ( & retrieval = 'main'), the pipeline keeps going, treating the errored targets as having no output
  • This allows e.g. collation/summary steps to view the overall progress of the sucessfully-completed targets, despite some targets having errored (e.g. through preempted workers).
  • This is clearly distinct behaviour from either error = 'stop' or error = 'abridge'

After cb85323

• When error = 'continue', the pipeline keeps going, but (as far as I can see) all targets downstream of an errored target throw an error too.
  • Technically correct, reflecting that one or more upstream targets errored, but it prevents collation/summary of the successfully-completed targets. This is undesireable if the collated targets are independent (e.g. summary of model diagnostics across various datasets - the errored targets could be discovered by other means).
  • The new behaviour of error = 'continue' almost mirrors setting error = 'abridge' on the erroring targets, as far as I understand it.

Is this the desired behaviour for error = 'continue'? Perhaps so, & it's the safest option in some cases, but it risks making large pipelines fragile if certain errors are considered acceptable by the user (e.g. infrastructure preemption). If this behaviour were desired, perhaps users could set error = 'abridge' instead?

[wlandau]

Yes, the current behavior is the intended behavior. If a target errors out, it should not return a value. Returning any value, no matter what the value is, implies success. Before that commit, errored targets with error = "continue" would accidentally keep a return value of NULL in memory, which was incorrect.

In your case, you might consider pre-filling the values of your targets using tarchetypes hooks (https://docs.ropensci.org/tarchetypes/reference/index.html#section-hooks) and setting deployment = "main" to avoid worker preemption when those targets are being pre-filled.

# _targets.R
library(targets)
library(tarchetypes)
tar_option_set(
  error = "continue",
  deployment = "main" # Do not run targets on the cluster the first time.
)

prefill_values <- function(code) {
  NULL # Do not run the code and just return NULL for the target.
}

list(
  tar_target(x, stop("message")),
  tar_target(y, x),
  tar_target(z, y)
) |>
  tar_hook_outer(prefill_values(.x), all_of("x")) # Drop all_of("x") to apply the hook to all targets.

> tar_make()
• start target x
• built target x
• start target y
• built target y
• start target z
• built target z
• end pipeline

Then, just remove the hook remove deployment = "main" when you deploy the targets to the cluster.

# _targets.R
library(targets)
library(tarchetypes)
tar_option_set(error = "continue")
list(
  tar_target(x, stop("message")),
  tar_target(y, x),
  tar_target(z, y)
)

> tar_make()
• start target x
x error target x
Error: message

✓ skip target y
✓ skip target z
• end pipeline

[wlandau]

By the way, would you explain the situation with your cluster? I am not sure I fully understand worker preemption, why it is instated in your case, and how much control you have over the resources.

[stuvet]

I agree about the intended behaviour. I've not looked into hooks properly & sounds like it'd be the correct approach. Very many thanks.

Worker Preemption

• (Slurm) cluster running on Google Compute Engines. Workers scale up from 0 as required, & can scale to thousands.
• I have full control of it all, but have chosen to use 'preemptible VMs' for most of the workers to keep costs down.
• When Google decides to preempt a worker VM the machine is no longer under my control & the job fails. With my current setup (& with the chunks.as.arrayjobs=TRUE workaround from Tips: Getting tar_make_future stable on Slurm #570), Slurm recognises that it has lost control of the worker & requeues the job which completes as expected, so no big deal really, but the same problem may remain unresolved:
  • on other backends where 'requeue' isn't an option
  • where Slurm clusters are configured to reject arrayjobs
  • on busy clusters where jobs are 'preempted' (by HPC scheduler, not by the infrastructure provider) by other jobs with higher priority

[stuvet]

oh, & the jobs which are affected by infrastructure preemption result in an errored target after 141859f (via a FutureError) , whereas they took down the pipeline beforehand.

[stuvet]

Hi @wlandau ,

Related to your tar_hook_outer solution above, I wondered - is there a way to extract a reference to the target from within a hook given only its code? Hashing?

Aim

From within the prefill_values function, I'd like to be able to check whether or not the target already has a value of NULL. If not, return NULL, & if so, run code. That way, the behaviour you've induced with this prefill_values would remain even as more datasets are added to the start of the pipeline, and the hook could be left in place. With the existing solution, you'd have to rerun the prefill_values hook manually for all branches downstream from added datasets.

No big deal if this is impossible, but I'd be interested to hear any suggestions.

Thanks

[wlandau]

You might have luck with tar_name(), tar_path(), and the functions at https://docs.ropensci.org/targets/reference/index.html#section-existence.