How to combine branches by mapped groups in targets ?

[sdechaumet]

Hello !

I have used drake since the start and switched to targets early this year to construct analytical pipelines used on HPC. I'm a very big fan of all your work, it is incredibly useful to us !

I have a simple design for which I can't find an answer in all the documentation and previously posted questions I read.

Description of the design (all as dynamic patterns since the number of input datasets are dynamic)

I have 3 datasets with different number of rows (y) in input. For each dataset I want to apply similar steps so pattern = map() on those.
For each dataset I want to branch on each row to parallelize some computational steps before regrouping them. Since I can't find a way to use cross pattern on mapped branches with different number of y elements, I first combine the 3 datasets and use map on each row.
Then I want to combine the mapped branches according to the 3 input datasets. In drake, we could use group_by to do this, but I can't find a way with targets.
My workaround is to combine all the mapped branches and then subset them by their provenance, but in the full workflow, each dataset is incredibly heavy and it is not efficient to combine them just to redispatch them afterward.

I'm sure there is a better way to do this.

Code

library(targets)

# Set target-specific options such as packages.
tar_option_set(packages = c("dplyr", "data.table", "magrittr"))

# End this file with a list of target objects.
list(
  tar_target(Manip, c("A", "B", "C")),
  tar_target(Manip_row, c(90, 73, 121)),
  tar_target(data_in, data.frame(data = Manip, x = sample.int(Manip_row), y = sample.int(Manip_row)), pattern = map(Manip, Manip_row)),
  tar_target(data_combined, data_in),
  tar_target(data_map_each, {data_combined$STEP01 <- sample.int(100, 1) ; data_combined}, pattern = map(data_combined), iteration = "list"),
  tar_target(data_map_each_step02, {data_map_each$STEP02 <- sample.int(100, 1) ; data_map_each}, pattern = map(data_map_each), iteration = "list"),
  tar_target(data_Groups, {
    temp <- Reduce(function(x, y) {rbind(x, y)}, data_map_each_step02)
    temp[temp$data == Manip,]
  }, pattern = map(Manip), iteration = "list"),
  tar_target(next_steps_on_group, {
    data_Groups$STEP03 <- sample.int(1000, nrow(data_Groups))
    data_Groups
  }, pattern = map(data_Groups), iteration = "list")
)

Current output

Desired behavior(s)

• Could it be possible to use cross on a map pattern with variable y groups for each mapped branches like in the schemes below without loosing the original mapping :

• Or maybe I can use the slice function to select which branches to combine for each group ? But as I understand it, we can't populate tar_targets inside pattern = slice()

Well, any orientation towards a possible solution or discussion addressing this subject is very welcomed :)

Thanks you again for all of this !

[wlandau]

Since you have a small number of datasets and each dataset is large, I recommend using static branching over datasets and dynamic branching over rows or row groups within each dataset: https://books.ropensci.org/targets/static.html#dynamic-within-static-branching

[wlandau]

Something like this:

# _targets.R file:
library(rlang)
library(targets)
library(tarchetypes)
list(
  tar_map(
    values = list(name = syms(c("airquality", "mtcars"))),
    tar_target(data, name),
    tar_target(row, data, pattern = map(data))
  )
)

[sdechaumet]

Thank you for your quick suggestion and the clear example provided ! It solves indeed the cross-over-map patterns brilliantly 👍 I would like your advises on the two following cases (the second one was not originally asked) :

• What if the number of datasets is not known in advance and I'd like to apply the tar_map on each ?
• How could I combine the results of all the datasets after the tar_map ? In your example above it would be to combine row_airquality and row_mtcars ?

[wlandau]

If you do not know the datasets in advance, then this workaround is not possible and branching would need to be completely dynamic. This is more difficult for exactly the reasons you identify, but it is not a common use case, and I hesitate to develop a grouping feature because it was clunky and brittle in drake.

To combine all the results across all the datasets, you can use tarchetypes::tar_combine(): https://books.ropensci.org/targets/static.html#combine

[sdechaumet]

I didn't use tarchetypes since I always found some workaround directly with targets but it seems the right time to start experimenting with it !

In the present case I may be able to guess the number of datasets prior to writing the _targets.R scripts and implementing something like you did in targets-shiny, namely: using tar_helper() to rewrite the full script on-demand based on user inputs (and datasets number in the present case). I hoped for a simpler solution with a 'generic' pipelines working for any kind of inputs, but I also love the tar_helper idea, so...

A big thanks for your time and answers ! Looking forward the next updates.

[sdechaumet]

For future reference, if anyone searching how to implement this kind of pipeline, I think that the targetopia with target factories is of great interest. A nice exemple of what I tried to do can be found here (with minor modifications): see the multiple models part.