[help] format = "qs" seems to suppress storage = "worker"

[multimeric]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I'm working with some very large datasets (100GB) in targets, and so I need to be very careful that no data is duplicated as there isn't enough memory to do that. Consequently, I'm using storage = "worker" to prevent any data being sent over the network. I'm also using format = "qs" to hopefully speed up the very slow process of loading these targets to/from disk. Finally, I'm running all of this on a Slurm cluster, so I'm also specifically setting the memory limit of my workers using ulimit because it gives much clearer error messages when R attempts to use more memory than we have. Without ulimit the process just gets killed by Slurm which is much worse. However, I suspect that the underlying issues here are not specific to crew.cluster, since the format/storage stuff is implemented bytargets.

Here is a targets workflow that exhibits these same settings, that can be used to reproduce my findings:

library(targets)

tar_option_set(
  controller = crew.cluster::crew_controller_slurm(
    workers = 1,
    script_lines = c(
      "module load R/4.4.0",
      # Stop R from overallocating memory
      "ulimit -v $(( SLURM_MEM_PER_CPU * 1024))"
    ),
    slurm_memory_gigabytes_per_cpu = 10,
    slurm_log_error = NULL,
    slurm_log_output = NULL
  ),
  format = "qs",
  # Don't send data over the network
  storage = "worker",
  retrieval = "worker",
)

list(
    # This is 7.5 Gb as determined by object.size
    tar_target(vector, numeric(1E9)),
    tar_target(size, object.size(vector))
)

When you run this, targets prints this to the console:

Error:
! Error running targets::tar_make()
Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
Debugging guide: https://books.ropensci.org/targets/debugging.html
How to ask for help: https://books.ropensci.org/targets/help.html
Last error message:
    {crew} worker 1 launched 5 times in a row without completing any tasks. Either troubleshoot or raise launch_max above 5. Details: https://wlandau.github.io/crew/articles/risks.html#cras
hes

When you check the Slurm log file, you get:

Error in send(ctx, data = data, mode = 3L, block = TRUE) :
  'R_Realloc' could not re-allocate memory (1744830464 bytes)
Calls: <Anonymous> -> do.call -> <Anonymous> -> send
slurmstepd: error: *** JOB 17387391 ON il-n04 CANCELLED AT 2024-07-19T15:03:09 ***

This worries me for two reasons:

1. We seem to be using send() despite storage = "worker". I certainly don't want to send these datasets over the network. The failing snippet seems to be from here in mirai so I'm inclined to think this is caused by targets behaviour
2. When using send, large amounts of data seem to be duplicated in memory. Here we have an attempted allocation of 1.75 GB, but in my own real life code, the allocation is actually 105092481024 bytes = 100 GB
3. targets/mirai will retry when it hits these failures, because didn't occur during the command itself. I wish they could realise that this is not a transient failure. Although I guess it's rare that we have detailed enough information

[wlandau]

I am not sure why the R_Realloc error happened, but targets does correctly implement storage = "worker" and retrieval = "worker". I ran a local version of your pipeline:

library(targets)

tar_option_set(
  controller = crew::crew_controller_local(),
  format = "qs",
  storage = "worker",
  retrieval = "worker",
)

list(
  tar_target(vector, numeric(1e9)),
  tar_target(size, object.size(vector))
)

while running tar_make(callr_function = NULL) with browser() statements exactly where targets sends and receives data to and from the crew controller. When storage = "worker" and retrieval = "worker", the data objects are around 50 KB at most, and htop shows non-worker processes stay at or below 135 MB used. When I comment out storage = "worker" and retrieval = "worker", the memory of the mirai dispatcher explodes, and the pipeline crashes. All this is expected. My company's grid engine cluster is totally occupied as usual, so I can't try there unfortunately.

[shikokuchuo]

@wlandau I'm not understanding the question. 'R_Realloc' could not re-allocate memory in an R error that is thrown from base R, and I assume correctly, as you are trying to allocate a large object in serialization.

The send(ctx, data = data, mode = 3L, block = TRUE) is a call from the daemon process, i.e. you're trying to send this object back from the worker.

[wlandau]

@multimeric, thanks, I can reproduce the error now.

@shikokuchuo, the problem is that this large object is not actually supposed to be sent over the network. With storage = "worker" and retrieval = "worker", targets is trying to store it locally to a file and have the next process retrieve it from that file instead of sending it over an NNG socket.

[wlandau]

So apparently targets may be trying to send the whole value after all, but only when there is an error. Somehow the target is not relinquishing the return value because of that error. If I'm right, it's definitely a bug in targets.

[wlandau]

b680b8f fixes the issue so that now the data sent over the network is very light in the event of a storage error. Now when you run the example in #1304 (reply in thread), the crew worker does not crash, and tar_make() prints the error message "store std::bad_alloc". The underlying error comes from qs::qsave(), which is running into the memory constraint set by unix::rlimit_as().

[multimeric]

Woo! Thanks for investigating this further, it's much appreciated.

[multimeric]

Although I still don't understand why this error happens with qs and not rds. Why is it trying to allocate all this memory at all?

[wlandau]

I'm not sure, all I know is that the underlying error is coming from qs::qsave() when it tries to save a large object. I can reproduce it with:

unix::rlimit_as(cur = 10 * 1024 ^ 3)
x <- numeric(1e9)
qs::qsave(x, tempfile())
#> Error in qs::qsave(x, tempfile()) : std::bad_alloc

So it is a qs issue at this point on.

By the way, the maintainer of qs is working on a successor called qs2: https://github.com/traversc/qs2. There will be native integration in targets when that happens, likely a new format = "qs2" to guarantee back-compatibility.