[general] Improved reporting of branches

[multimeric]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I have a lot of dynamic branches, and my workflow often involves debugging errors in particular branches but not others. When I get a failure in one branch, it says something like ▶ dispatched branch instance_path_44d946c111c4f3e1, and then it fails. However I'm not sure how to work backwards to determine which values of each upstream dependency were used in that branch. This is particularly bad in cases where the task fails outside of the command (e.g. here), because then I don't even get a traceback.

I wonder if I could request a reporter that prints out the values of each dependency for dynamic branches. So it says something like:

▶ dispatched branch instance_path_44d946c111c4f3e1 using x = 1 and y = 3

[mattwarkentin]

Hi @multimeric,

I think workspaces (tar_workspaces() and tar_workspace()) and tar_branches() provide the functionality you need. Setting the option as:

tar_option_set(workspace_on_error = TRUE)

Then if a dynamic branch fails, you can identify the saved "workspaces" with tar_workspaces() and then select the dynamic branch name and load it into the global environment for debugging with tar_workspace("name_of_branch_1234"). This will load all of the globals (functions and objects) required to debug your code/pipeline.

Lastly, if you want just the names of the upstream dependencies for branches, you can use tar_branches():

tar_branches(target_name, pattern = <<enter pattern here>>)

Lame example but showcases things:

targets::tar_dir({
  targets::tar_script({
    library(targets)

    foo <- function(x) {
      if (x == 5) rlang::abort('Something went wrong')
      x
    }

    tar_option_set(
      workspace_on_error = TRUE,
      error = 'continue'
    )

    list(
      tar_target(
        a,
        1:10
      ),
      tar_target(
        b,
        a,
        pattern = map(a)
      ),
      tar_target(
        c,
        foo(b),
        pattern = map(b)
      )
    )
  }, ask = FALSE)
  
  targets::tar_make()

  print(targets::tar_workspaces())
  # Could load workspace with `tar_workspace("c_24be1142")`

  targets::tar_branches(c, map(b))
})
#> ▶ dispatched target a
#> ● completed target a [0.001 seconds]
#> ▶ dispatched branch b_0a91b2ed
#> ● completed branch b_0a91b2ed [0 seconds]
#> ▶ dispatched branch b_847c5ae2
#> ● completed branch b_847c5ae2 [0 seconds]
#> ▶ dispatched branch b_1619f1a0
#> ● completed branch b_1619f1a0 [0 seconds]
#> ▶ dispatched branch b_9401bbb6
#> ● completed branch b_9401bbb6 [0 seconds]
#> ▶ dispatched branch b_d24115c8
#> ● completed branch b_d24115c8 [0 seconds]
#> ▶ dispatched branch b_fb63d300
#> ● completed branch b_fb63d300 [0 seconds]
#> ▶ dispatched branch b_3138aba9
#> ● completed branch b_3138aba9 [0 seconds]
#> ▶ dispatched branch b_ad3eda3e
#> ● completed branch b_ad3eda3e [0 seconds]
#> ▶ dispatched branch b_bf196919
#> ● completed branch b_bf196919 [0 seconds]
#> ▶ dispatched branch b_b0795e65
#> ● completed branch b_b0795e65 [0 seconds]
#> ● completed pattern b
#> ▶ dispatched branch c_66acaf33
#> ● completed branch c_66acaf33 [0 seconds]
#> ▶ dispatched branch c_02f9cbce
#> ● completed branch c_02f9cbce [0.001 seconds]
#> ▶ dispatched branch c_c46c3a46
#> ● completed branch c_c46c3a46 [0 seconds]
#> ▶ dispatched branch c_0b5d87dc
#> ● completed branch c_0b5d87dc [0 seconds]
#> ▶ dispatched branch c_24be1142
#> ▶ recorded workspace c_24be1142
#> ✖ errored branch c_24be1142
#> Error: Something went wrong
#> 
#> ▶ dispatched branch c_0ff7342d
#> ● completed branch c_0ff7342d [0 seconds]
#> ▶ dispatched branch c_7de5013b
#> ● completed branch c_7de5013b [0 seconds]
#> ▶ dispatched branch c_54814e50
#> ● completed branch c_54814e50 [0 seconds]
#> ▶ dispatched branch c_241b47cd
#> ● completed branch c_241b47cd [0.001 seconds]
#> ▶ dispatched branch c_419518f2
#> ● completed branch c_419518f2 [0 seconds]
#> ✖ errored pipeline [0.167 seconds]
#> [1] "c_24be1142"
#> # A tibble: 10 × 2
#>    c          b         
#>    <chr>      <chr>     
#>  1 c_66acaf33 b_0a91b2ed
#>  2 c_02f9cbce b_847c5ae2
#>  3 c_c46c3a46 b_1619f1a0
#>  4 c_0b5d87dc b_9401bbb6
#>  5 c_24be1142 b_d24115c8
#>  6 c_0ff7342d b_fb63d300
#>  7 c_7de5013b b_3138aba9
#>  8 c_54814e50 b_ad3eda3e
#>  9 c_241b47cd b_bf196919
#> 10 c_419518f2 b_b0795e65

[wlandau]

Should be fixed now.

[multimeric]

Unfortunately I now get the following error with the above example script:

Error:
!  is not a target in _targets.R

[wlandau]

Should fix itself if you supply a name argument.

[multimeric]

Thanks. However, because the target that I want to query hasn't completed all branches, it uses the incomplete metadata, and I get this:

> tar_branches(name = instance_path)
Error in data.frame(..., check.names = FALSE) : 
  arguments imply differing number of rows: 18, 26

This is because children (which is derived from metadata, and so is missing some rows), is shorter than out (which is derived form the pipeline spec only)

[wlandau]

Unfortunately we can't get around this limitation (and others) of tar_branches(). tar_branches() is only an approximation.