How do you make the targets graph re-converge after applying branching pattern?

[davidr-PA]

Hello,

I am struggling to figure out how to make a targets data pipeline which dynamically branches then re-converges. I assumed it was able to re-converge, since it's a DAG and DAG's can branch then re-converge (since they're directed, and therefore cannot form cycles).

For instance, I might want a starting branch A which is a collection, then branch out to dynamic branches on B, then re-converge to a collection on C. For B, I can pass map(A) to the pattern parameter of that tar_target. But there's no reduce pattern which lets me re-converge the dynamic branches. Is there another way of achieving this with dynamic branches? I know I can call B without a pattern. For tibbles/dataframes, this automatically concatenates them into one large tibble/dataframe. But what I want is a list of tibbles/dataframes, not a massive concatenated tibble/dataframe.

Are dynamic branches effectively trees, not full DAGs?

[wlandau]

In dynamic branching, the "reduce" step happens automatically whenever you need the combined object. The points target from https://books.ropensci.org/targets/dynamic.html#example is an example. Behind the scenes, the points target has dynamic branches, but whenever you call tar_read(points) or just reference all of points in a downstream target, you will automatically get all the dynamic branches together as one big data frame (a data frame because the branches are data frames). No explicit "reduce" step is necessary.

In static branching, you can use tar_combine() for reduction/aggregation.

[davidr-PA]

@wlandau Is there a way to customize the reduce step, so it does not concatenate all dataframes together, but instead takes lists of dataframes and makes one large list of dataframes?

My use-case is that I have some different ways of processing data based upon the characteristics of that dataset. Then I bring them back together in one large list to save them.

I am currently saving in multiple targets steps, because I can't bring those dynamic branches (which contain 1-3 aggregated versions of the original dataset, based on its characteristics) into one combined list, and save them all in one step.

[wlandau]

You can with iteration = "list": https://books.ropensci.org/targets/dynamic.html#iteration. For the features you are trying to use, the dynamic and static branching chapters of the manual are worth a read.

[davidr-PA]

I'll look back at that again, thanks for the quick reply!