In what contexts are exotic patterns like cross useful versus map with tibbles?

[wdkrnls]

I'm trying to wrap my head around dynamic branching and from reading the documentation a lot of emphasis is placed on using cross and slice. However, it's not clear to me what their use cases are versus mapping across tibbles or groups within tibbles. After banging my head with these fancy pattern functions, I have found hashed branches hard to map back to their corresponding inputs in the simulation. Are these functions likely to be deprecated in the future because of this, or are there use cases for them where their complexity is worth it? Are there subtle methods of recovering their origins that I have overlooked? Integer indexing seems like an R anti-pattern to me: good for a spot check that things are working, but not for providing tangible assurance that the code is doing what you hope it is.

[wlandau]

cross() comes up when you want all combinations of upstream branches and cannot pre-compute the expanded grid in advance (for example, if each branch has a lot of data and it would be infeasible to put all of it in a data frame to map over, especially a grid with multiple copies of each branch). slice() is for odd situations and debugging where a specific set of branches is desired, and it is not very common. There is no feasible way to get around integer indexing for slice() because of how dynamic branching already works in targets.

[wdkrnls]

Thanks for being open to explaining. I still don't understand, but I imagine I just don't have enough experience the use cases you are describing. My experience with running simulations on HPC is that its trivially easy to exhaust system resources, both memory and disk space. Presumably moving to cloud based workflows makes that problem even more frequent. Why you would want to aggregate all the response data in a target instead of just a minimal target with only provenance information in a data frame?

Maybe I am just missing how to use tar_meta to get that provenance. Right now it just seems like guesswork to figure out what all these hashed results are when using cross. However, I think the real answer you have been hinting at is to always return a data frame with this info.

[wlandau]

I recommend pursuing a use case and seeing what works for you. In most cases, pattern = map() is enough, and you don't have to worry about tar_meta() or hashes to set up the pipeline. In most cases, a dynamic branch target should return a data frame with any relevant provenance information you will need in order to do the downstream postprocessing to meet your needs. You can write provenance information, such as the simulation scenario in a simulation study, directly into the target output using your custom functions.

[davidr-PA]

@wdkrnls Let me give you an example use case. Imagine you wanted to run 3 types of analyses on 5 datasets, each in its own dataframe/tibble. One target defines a list of tibbles (or alternatively, dynamic branches of tibbles made using the map pattern), and another defines the analyzer functions.
tar_target(
name = analyses,
command = analyzers(tibbles),
patterns = cross(tibbles, analyzers)
)

This will give you 3 x 5 = 15 branches, each containing 1 of 3 analyses of 1 of the 5 datasets you analyzed. You could then save each analysis in the next step.

That being said, you could also just have 3 targets using the map pattern, and keep each set of analyses separate.