[help] flatten list of lists, in dynamic branching, without loading everything into memory

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

(in pseudocode) I have such a target

some upstream dynamic target process produces this target -> a huge list of lists

 ll = list(
 list("every_element_is_20Gb", "every_element_is_20Gb"),
 list("every_element_is_20Gb", "every_element_is_20Gb")
)

And I want to flatten this list without loading this whole thing into memory, ideally.

tar_target(a, ll |> do_something(), pattern = flatten_map(ll))

or more elegantly

tar_target(a, ll |> do_something(), pattern = map(ll, unlist = TRUE))

Without doing a redundant

tar_target(ll_flatten, ll |> unlist())

Which I assume will load everything into memory and become unmanageable, plus it would duplicate I guess the whole target on the hard-drive for not good reason.

Thanks

[multimeric]

I had a look into how this works in targets. Firstly, I'm not convinced that using pattern = map() over a list target does actually lazy load. From what I can tell, it loads the whole list into memory and then slices it up. You can see this by debugging the target_produce_junction.tar_stem method.

I don't think you even can partially load objects from an RDS file. I'm trying to think of a workaround. For now I would still suggest working with these large objects as files, so that the list of large objects is just a character vector that you can easily manipulate in memory, and then load the paths only as needed.

[stemangiola]

I had a look into how this works in targets. Firstly, I'm not convinced that using pattern = map() over a list target does actually lazy load. From what I can tell, it loads the whole list into memory and then slices it up. You can see this by debugging the target_produce_junction.tar_stem method.

I would be very surprised if this was the case. Do you mean it get called once as a whole and sliced, or gets called every time as whole, and subset?

I don't think you even can partially load objects from an RDS file. I'm trying to think of a workaround.

I am not sure that use format = "parquet" will help with lists.

For now I would still suggest working with these large objects as files, so that the list of large objects is just a character vector that you can easily manipulate in memory, and then load the paths only as needed.

Yes I am using HDF5 now, but even loading a long list of HDF5, can be memory heavy.

[wlandau]

For a pipeline like this, targets needs to load the whole upstream object into memory:

list(
  tar_target(upstream, big_object()),
  tar_target(downstream, upstream, pattern = map(upstream))
)

But for this pipeline where the branches already exist, only one branch of upstream is loaded into memory at a time.

tar_option_set(memory = "transient", garbage_collection = TRUE)
list(
  tar_target(index, seq_len(100)),
  tar_target(upstream, create_branch(index), pattern = map(index)),
  tar_target(downstream, upstream, pattern = map(upstream))
)

Now if you have a flattened target which mentions downstream and does not branch over it, then targets loads all the branches into memory. Regardless of targets, R needs all the full list in memory in order to unlist it.

tar_option_set(memory = "transient", garbage_collection = TRUE)
list(
  tar_target(index, seq_len(100)),
  tar_target(upstream, create_branch(index), pattern = map(index)),
  tar_target(downstream, upstream, pattern = map(upstream)),
  tar_target(flattened, unlist(downstream)) # loads all branches of `downstream`
)

If you want to flatten each branch at a time, you can do:

tar_option_set(memory = "transient", garbage_collection = TRUE)
list(
  tar_target(index, seq_len(100)),
  tar_target(upstream, create_branch(index), pattern = map(index)),
  tar_target(downstream, upstream, pattern = map(upstream)),
  tar_target(flattened_branches, unlist(downstream), pattern = map(downstream))
)

[stemangiola]

If I apply the last suggestion (it would be great to ru unlist in separate processes)

list(
  tar_target(index, seq_len(100)),
  tar_target(upstream, create_branch(index), pattern = map(index)),
  tar_target(downstream, upstream, pattern = map(upstream)),
  tar_target(flattened_branches, unlist(downstream), pattern = map(downstream))
)

I get still that the "flatten" list of the same length of the "downstream", so for me the flattening did not happen

...,
    tar_target(
      list_of_lists,
      pseudobulk_group_list  |> 
       
      split_summarized_experiment(chunk_size = 10) , 
      pattern = map(pseudobulk_group_list), 
      iteration = "list"
    ), 
    
   tar_target(
    flattened, 
     list_of_lists |> unlist() , 
     pattern = map(list_of_lists) 
    )

Is it because I am using iteration = "list"?

[wlandau]

Is it because I am using iteration = "list"?

Could be, but a runnable reprex would make it clearer to see the exact output that is generated vs what you are expecting.