targets.qmd

---
title: "A workflow manager for {{< fa brands r-project >}}"
subtitle: "`targets`"
date: today
date-format: "dddd, [the] D[<sup style='font-size:65%;font-style:italic;'>th</sup>] [of] MMMM, YYYY"
institute: "University of Luxembourg"
author: "Aurélien Ginolhac"
image: "https://docs.ropensci.org/targets/reference/figures/logo.png"
format:
  unilu-theme-revealjs:
    logo_url: https://basv53.uni.lu
    code-block-height: 850px
from: markdown+emoji
title-slide-attributes: 
  data-background-image: "https://docs.ropensci.org/targets/reference/figures/logo.png"
  data-background-position: "50% 10%"
  data-background-size: "150px"
execute: 
  echo: true
dpi: 300
---

```{r}
#| label: setup
#| include: false
```


## Workflow managers

:::: {.columns}
::: {.column width="60%" .incremental}

- Workflow Managers are designed to compose and execute a series of computational steps.
- Workflows are typically represented as a visual graph where nodes are connected together.
- Workflow managers support abstractions and provide automation.
- Scientific workflow systems enable large scale scientific experiments.
- They make computational methods reproducible, portable, maintainable, and shareable.

:::
::: {.column width="30%"}
![](img/static_dag.png)
:::
::::


## Makefiles

It started with **Makefile**, when computers power was limiting.
Compile objects (`*.o`) only when needed: source (`*.c`) modified.
`make` first release is April 1988.

:::: {.columns}
::: {.column width="45%"}
Dependency rules.

``` c
target: dependencies
      commands
```

![](img/dag_makefile.svg)

:::
::: {.column width="55%" .fragment}
``` c
# This is a comment line
CC=gcc
# CFLAGS will be the options passed to the compiler.
CFLAGS= -c -Wall
all: prog

prog: main.o factorial.o hello.o
    $(CC) main.o factorial.o hello.o -o prog

main.o: main.c
    $(CC) $(CFLAGS) main.c

factorial.o: factorial.c
    $(CC) $(CFLAGS) factorial.c

hello.o: hello.c
    $(CC) $(CFLAGS) hello.c

clean:
    rm -rf *.o
```

Compile with `make` (rule `all`)

:::
::::

## {targets} and companion package tarchetypes

![](https://docs.ropensci.org/targets/reference/figures/logo.png){.absolute height="150px" top="1em" right="1em"}
![](https://docs.ropensci.org/tarchetypes/reference/figures/logo.png){.absolute height="150px" top="6em" right="1em"}

- Saving you time and stress

::: {.incremental}

- Understand how it is implemented in `targets`
    + Define your `targets`
    + Connect `targets` to create the **dependencies** 
    + Check **dependencies** with `visnetwork`
    + Embrace either or combined
        + **Dynamic** branching 
        + **Static** branching 
    + Run **only** what needs to be executed and in fresh session with `{callr}`
    + Embrace [literate programming](https://books.ropensci.org/targets/literate-programming.html) with `qmd` or `Rmd` docs
    + Bundle **dependencies** in a documents with [`tar_render()`](https://docs.ropensci.org/tarchetypes/reference/tar_render.html)/[`tar_quarto()`](https://docs.ropensci.org/tarchetypes/reference/tar_quarto.html)
- Be better at scheduling your work

:::


## Folder structure


![](https://raw.githubusercontent.com/rstudio/renv/master/man/figures/logo.svg){.absolute height="150px" top="0.2em" right="1em"}
![](https://upload.wikimedia.org/wikipedia/commons/thumb/d/d0/RStudio_logo_flat.svg/1200px-RStudio_logo_flat.svg.png){.absolute height="150px" top="17em" right="1em"}
![](https://git-scm.com/images/logos/logomark-black@2x.png){.absolute height="150px" top="11em" right="1em"}


:::: {.columns}
::: {.column width="30%"}

``` bash
├── .git/
├── _targets.R
├── _targets/
├── Repro.Rproj
├── R
│   ├── functions.R
│   └── utils.R
├── run.R*
├── renv/
├── renv.lock
└── report.qmd
```
:::
::: {.column width="70%"}
### Targets

- With [`renv`](https://rstudio.github.io/renv/). Snapshot your package environment
- `_targets.R` is the only mandatory file
- Use a `R` sub-folder for functions, gets closer to a `r fontawesome::fa("r-project")` package
- `Rmd`/`qmd` file allows to gather results in a report
- In a RStudio project
- Version tracked with `git`
- An executable `run.R` allows to use Build Tools in RStudio
:::
::::

## DatasauRus example, smart animation caching

This example is available at the [`target_demos` repo](https://github.com/ginolhac/targets-demos)


:::: {.columns}
::: {.column width="50%"}
#### targets script [`_targets_ds_fun1.R`](https://github.com/ginolhac/targets-demos/blob/main/_targets_ds_fun1.R)

``` r
library(targets)
library(tarchetypes)
source("R/plotting.R")
# load the tidyverse quietly for each target
# which each runs in a fresh R session
tar_option_set(packages = "tidyverse")

list(
  # track if distant file has changed
  tar_url(ds_file, "https://raw.githubusercontent.com/jumpingrivers/datasauRus/main/inst/extdata/DatasaurusDozen-Long.tsv"),
  tar_target(ds, read_tsv(ds_file, show_col_types = FALSE)),
  tar_target(all_facets, facet_ds(ds)),
  # animation is worth caching  ~ 1 min
  tar_target(anim, anim_ds(ds), 
             packages = c("ggplot2", "gganimate", "gifski")),
  tar_file(gif, {
    anim_save("ds.gif", animation = anim, title_frame = TRUE)
    # anim_save returns NULL, we need to get the file output path
    "ds.gif"},
             packages = c("gganimate")),
  tar_quarto(report, "ds1.qmd")
)
```
:::
::: {.column width="50%" .fragment}
#### Corresponding Directed Acyclic Graph

- **Directed**: each node has a one-way direction.
- **Acyclic**: no loop, no ambiguity.

![](img/targets_dag_linear.png)





:::
::::



::: {.footer}
Animation code is presented as function, full code in [`_targets_ds_1.R`](https://github.com/ginolhac/targets-demos/blob/main/_targets_ds_1.R)
:::


## Manifest, a good companion to the DAG


![](img/targets_dag_linear.png){height=320}

Precise description of steps in a table

``` r
> tar_manifest()
# A tibble: 6 × 2
  name       command     
  <chr>      <chr>  
1 ds_file    "\"https://raw.gi[...]n/inst/extdata/DatasaurusDozen-Long.tsv\""
2 ds         "read_tsv(ds_file, show_col_types = FALSE)"    
3 anim       "anim_ds(ds)"
4 all_facets "facet_ds(ds)"
5 gif        "{\n     anim_save(\"ds.gif\", animation = anim, title_frame = TRUE)\n     \"ds.gif\"\n }"  
6 report     "tarchetypes::tar_quarto_run(args = list(input = \"ds1.qmd\", \n     execute = TRUE,
```




## Literate programming

We recommend using it [within a target](https://books.ropensci.org/targets/literate-programming.html#literate-programming-within-a-target) and not the [Target Markdown](https://books.ropensci.org/targets/literate-programming.html#target-markdown) that overloads the document.

:::: {.columns}
::: {.column width="50%"}
![](img/targets_ds1_source.png)
:::
::: {.column width="50%" .fragment}
![](img/targets_ds1_rendered.png)
:::
::::

## Multi-projects in one folder

Like the `targets_demos` repo which has 4 projects

:::: {.columns}
::: {.column width="55%"}
### Config file: `_targets.yaml`

`targets` needs a R script and a store location

``` yaml
ds_linear:
  store: _ds_1
  script: _targets_ds_1.R
ds_fun_linear:
  store: _ds_fun1
  script: _targets_ds_fun1.R
ds_dynamic:
  store: _ds_2
  script: _targets_ds_2.R
ds_static:
  store: _ds_3
  script: _targets_ds_3.R
  reporter_make: verbose_positives # do not display skipped targets
```
:::
::: {.column width="45%" .fragment}
### Usage

In your Rmd/qmd/console, one env variable to set:

``` r
Sys.setenv(TAR_PROJECT = "ds_fun_linear")
```
:::
::::


## Custom Building Tool


:::: {.columns}
::: {.column width="50%"}
#### Tools > Projects Options > Custom

![](img/targets_custom_run.png)

`run.R`: 
``` r
#!/usr/bin/env Rscript
# Optional var env for > 1 _targets.R
Sys.setenv(TAR_PROJECT = "ds_fun_linear")
targets::tar_make()
```
:::
::: {.column width="50%" .fragment}
#### Running targets

- Useful shortcut {{< kbd Shift-Ctrl-B >}}

![](img/targets_fun_ds1_run.png)

- Animation takes the most time

::: {.callout-warning}
## Issue on Windows

Seems that a custom script is not working on {{< fa brands windows >}}
:::

:::
::::


## Re-running, same shortcut only what is needed

:::: {.columns}
::: {.column width="50%"}
#### Without changes

![](img/targets_ds1_rerun.png)
:::
::: {.column width="50%" .fragment}
### Change in `facet_ds()`
``` r
facet_wrap(vars(dataset), ncol = 4) # <- 3
```
![](img/targets_rerun_allfacets.png)
:::
::::

::: {.footer}
Change that are **comments** are not invalidating a `target`
:::


## Dynamic branching

:::: {.columns}
::: {.column width="50%"}
### Often we start from multiple files

``` bash
data/
├── dset_10.tsv
├── dset_11.tsv
├── dset_12.tsv
├── dset_13.tsv
├── dset_1.tsv
├── dset_2.tsv
├── dset_3.tsv
├── dset_4.tsv
├── dset_5.tsv
├── dset_6.tsv
├── dset_7.tsv
├── dset_8.tsv
└── dset_9.tsv
```
And we want to apply the same treatment to each
:::
::: {.column width="50%" .fragment}
#### Functional programming again, iteration for what's needed

Done by the `pattern = map()` keyword. Use `cross()` for combinations.

``` r
tar_target(ds, read_tsv(dset, show_col_types = FALSE),
           pattern = map(dset)),
```


Directly with `tar_files_input()` (pair of targets)

![](img/targets_dag_dynamic_input.png)
:::
::::


## Changing one input file

:::: {.columns}
::: {.column width="50%"}
![](img/targets_dynamic_change_input.png)
:::
::: {.column width="50%" .fragment}
#### Re-run only one file and downstream dependencies


``` r
✔ skipped target dset_files
[...]
✔ skipped branch dset_1357daeb5edc5b3b
▶ dispatched branch dset_376af7da24ddcfc7
● completed branch dset_376af7da24ddcfc7 [0.001 seconds]
✔ skipped branch dset_fc156975d3544187
[...]
✔ skipped branch ds_4bc1a3d4ea6fdf12
▶ dispatched branch ds_501bf242796ba6b2
● completed branch ds_501bf242796ba6b2 [0.892 seconds]
✔ skipped branch ds_c601ea8afad80c5f
● completed pattern ds
✔ skip branch summary_stat_ad2f392a
[...]
✔ skipped branch summary_stat_aad2733c0eca3cae
▶ dispatched branch summary_stat_0f7ac98a50809586
● completed branch summary_stat_0f7ac98a50809586 [0.02 seconds]
✔ skipped branch summary_stat_9cefee38f54d6115
[...]
✔ skipped branch plots_aad2733c0eca3cae
▶ dispatched branch plots_0f7ac98a50809586
● completed branch plots_0f7ac98a50809586 [0.031 seconds]
● completed pattern plots
▶ dispatched target report
● completed target report [13.378 seconds]
▶ ended pipeline [16.281 seconds]
```
:::
::::


## Automatic aggregation

:::: {.columns}
::: {.column width="50%"}
#### For vectors/tibbles happens directly

``` r
> tar_read(ds)
# A tibble: 1,846 × 3
   dataset     x     y
   <chr>   <dbl> <dbl>
 1 away     32.3  61.4
 2 away     53.4  26.2
 3 away     63.9  30.8
```

#### Use branches for subsetting
``` r
> tar_read(ds, branches = 2L)
# A tibble: 142 × 3
   dataset     x     y
   <chr>   <dbl> <dbl>
 1 star     58.2  91.9
 2 star     58.2  92.2
 3 star     58.7  90.3
```
:::
::: {.column width="50%" .fragment}
#### For plots, use `iteration = "list"` 

```{r}
#| code-line-numbers: "5"
#| eval: false
tar_target(plots, ggplot(ds, aes(x, y)) +
             geom_point() +
             labs(title = unique(ds$dataset)),
           pattern = map(ds),
           iteration = "list")
```


``` r
> tar_read(plots, branches = 2L)
## $plots_a55f1afc
```

Then this list can be used by `patchwork`

``` r
library(patchwork)
wrap_plots(tar_read(plots)) +
   plot_annotation(title = "13 datasets bundled 
     with patchwork") & theme_void()
```
:::
::::


## Static branching, with dynamic inside

Dynamic branch names are not meaningful, just hashes

:::: {.columns}
::: {.column width="70%"}
#### Multi-folders input data

We still have multiple files per folder

``` bash
circles/
├── dset_2.tsv
└── dset_3.tsv
lines/
├── dset_11.tsv
├── dset_12.tsv
├── dset_13.tsv
├── dset_6.tsv
├── dset_7.tsv
├── dset_8.tsv
└── dset_9.tsv
others/
├── dset_10.tsv
├── dset_1.tsv
├── dset_4.tsv
└── dset_5.tsv
```
:::
::::


::: {.footer}
`tar_map()` is from `{tarchetypes}`
:::

## Dynamic vs Static

| Dynamic                                  |	Static                                             |
|------------------------------------------|-----------------------------------------------------|
| Pipeline creates new targets at runtime. | All targets defined in advance.                     |
| Cryptic target names.                    | Friendly target names.                              |
| Scales to hundreds of branches.          | Does not scale as easily for `tar_visnetwork()` etc.|
| No metaprogramming required. 	           | Familiarity with metaprogramming is helpful.        |

: {.striped}


> static branching is most useful for smaller number of heterogeneous targets.

::: {.footer}
Source: [`targets` manual](https://books.ropensci.org/targets/static.html#branching) by **William Landau**
:::


## Dynamic within static, best of both worlds

More difficult to write with `tar_map()` (see [example](https://github.com/ginolhac/targets-demos/blob/main/_targets_ds_3.R))

But meaningful names and combine when needed:

![](img/targets_dag_static.png){fig-align="center"}

Use `tar_manifest()` to display exactly the command to be run


::: {.footer}
Command used `tar_visnetwork(label = c("description", "branches"))`
:::

## Parallel static branches and combine

:::: {.columns}
::: {.column width="70%"}
#### From [`_targets_ds_3.R`](https://github.com/ginolhac/targets-demos/blob/main/_targets_ds_3.R), static branches:

``` r
# Static branching with dynamic branching inside
values <- tibble(
  folders = c("lines", "circles", "others")
)

# tar_map() generates R expressions, and substitute the desired 'values'
mapped <- tar_map(
  values = values,
  names = "folders", # to avoid targets reporting "files_lines_lines"
  tar_target(filenames, fs::dir_ls(folders, glob = "*tsv")),
  # filenames is not of format file, no checksum is done
  # we need a dynamic pattern at this step to read them dynamically too
  tar_target(files, format = "file", filenames, 
             pattern = map(filenames)),
  # Dynamic within static
  tar_target(ds, read_tsv(files, show_col_types = FALSE),
             pattern = map(files)),
  tar_target(summary_stat, summarise(ds, m_x = mean(x), m_y = mean(y)),
             pattern = map(ds)),
  tar_target(plots, ggplot(ds, aes(x, y)) +
               geom_point(),
             pattern = map(ds),
             iteration = "list"),
  # Patchwork each group into one plot
  tar_target(patch_plots, 
             wrap_plots(plots) + 
               # Title the last bit of path_plots_{circles,lines,others}
               plot_annotation(title = stringr::str_split_i(tar_name(), '_', -1)),
             packages = "patchwork")
)
```
:::
::::

## Combining step

::::{.columns}
::: {.column width="60%"}
``` r
# We want to combined in one tibble the 3 tibble of summary stats
# Each of one them is actually composed of 2, 4 and 7 tibbles
stat_combined <- tar_combine(
  stat_summaries,
  mapped[["summary_stat"]],
  # Force evaluation using triple bang (!!!)
  command = dplyr::bind_rows(!!!.x, .id = "ds_type")
)
# And the plots now, a patchwork of patchwork
plot_combined <- tar_combine(
  plots_agg,
  mapped[["patch_plots"]],
  # Force evaluation of all patchwork plots again with triple bang!
  command = {wrap_plots(list(!!!.x), ncol = 2) + 
               plot_annotation(title = "Master Saurus")},
  packages = "patchwork"
)
# Wrap all targets in one list
list(mapped, 
     stat_combined, 
     plot_combined, 
     tar_quarto(report, "ds3.qmd"))
```
:::

::: {.column width="40%"}

![](img/targets_tar_combine.png){fig-align="center"}
:::
::::

::: {.footer}
- `!!!` is the _unquote-splice_ operator from `{rlang}`
- `tar_combine()` is from `{tarchetypes}`
:::

## Manifest

#### `tar_manifest()` (paged version in [`ds3.qmd`](https://github.com/ginolhac/targets-demos/blob/main/ds3.qmd))

``` r
# A tibble: 21 × 4
   name                 command                                                                pattern description
   <chr>                <chr>                                                                  <chr>   <chr>      
 1 filenames_circles    "fs::dir_ls(\"circles\", glob = \"*tsv\")"                             NA      circles    
 2 filenames_others     "fs::dir_ls(\"others\", glob = \"*tsv\")"                              NA      others     
 3 filenames_lines      "fs::dir_ls(\"lines\", glob = \"*tsv\")"                               NA      lines      
 4 files_circles        "filenames_circles"                                                    map(fi… circles    
 5 files_others         "filenames_others"                                                     map(fi… others     
 6 files_lines          "filenames_lines"                                                      map(fi… lines      
 7 ds_circles           "read_tsv(files_circles, show_col_types = FALSE)"                      map(fi… circles    
 8 ds_others            "read_tsv(files_others, show_col_types = FALSE)"                       map(fi… others     
 9 ds_lines             "read_tsv(files_lines, show_col_types = FALSE)"                        map(fi… lines      
10 summary_stat_circles "summarise(ds_circles, m_x = mean(x), m_y = mean(y))"                  map(ds… circles    
11 plots_circles        "ggplot(ds_circles, aes(x, y)) + geom_point()"                         map(ds… circles    
12 summary_stat_others  "summarise(ds_others, m_x = mean(x), m_y = mean(y))"                   map(ds… others     
13 plots_others         "ggplot(ds_others, aes(x, y)) + geom_point()"                          map(ds… others     
14 plots_lines          "ggplot(ds_lines, aes(x, y)) + geom_point()"                           map(ds… lines      
15 summary_stat_lines   "summarise(ds_lines, m_x = mean(x), m_y = mean(y))"                    map(ds… lines      
16 patch_plots_circles  "wrap_plots(plots_circles) + plot_annotation(title = stringr::str_spl… NA      circles    
17 patch_plots_others   "wrap_plots(plots_others) + plot_annotation(title = stringr::str_spli… NA      others     
18 patch_plots_lines    "wrap_plots(plots_lines) + plot_annotation(title = stringr::str_split… NA      lines      
19 stat_summaries       "dplyr::bind_rows(summary_stat_lines = summary_stat_lines, \n     sum… NA      NA         
20 plots_agg            "wrap_plots(list(patch_plots_lines = patch_plots_lines, \n  …          NA      Key step t…
21 report               "tarchetypes::tar_quarto_run(args = list(input = \"ds3.qmd\", \n…      NA      Rendering …
```

## Final plot



![](img/targets_mastersaurus.png){fig-align="align"}


## Descriptions, free text field

Recent addition, showing up in `tar_manifest()` and network

``` r
plot_combined <- tar_combine(
  plots_agg,
  mapped[["patch_plots"]],
  command = wrap_plots(list(!!!.x), ncol = 2) + plot_annotation(title = "Master Saurus"),
  packages = "patchwork",
  description = "Key step to wrap plots"
)

list(mapped, stat_combined, plot_combined, tar_quarto(report, "ds3.qmd", description = "Rendering quarto doc"))
```

. . .

Also useful for selection of `targets` using `tar_described_as()`:

``` r
tar_manifest(names = tar_described_as(starts_with("survival model")))
```

::: {.footer}
From `{target}` version 1.6.0
:::

## Static-in-static

Dynamic branches still have cryptic names. What is we want to go **full static** where all steps are known upfront.

. . .

:::: {.columns}
::: {.column width="65%"}
Nested `tar_map()`: toy example:

``` r
library(targets)
library(tarchetypes)
mapped <- tar_map(
  #unlist = FALSE, # Return a nested list from tar_map()
  values = list(model = c("mod_1", "mod_2")),
  tar_target(
    distrib,
    tar_name(),
  ),
  # static in static
  tar_map(
    values = list(sim = c("A", "B")),
    tar_target(
      estim,
      paste(distrib, tar_name()),
    )
  )
)
combined <- tar_combine(combi, 
                        # select all estimations
                        tar_select_targets(mapped, starts_with("estim")), 
                        command = paste(!!!.x))
list(mapped, combined)

```
:::
::: {.column width="35%" .fragment}
![](img/targets_full_static.png){fig-align="center"}

No more square `targets`, no `pattern = map(...)`
:::
::::


## Full static for datasauRus, `_targets_ds_4.R`


```{.r}
mapped <- tar_map(
  values = values,
  names = "names", # to avoid targets reporting "files_data.lines"
  # special pair of targets
  # readr is in charge of the aggregation (bind_rows())
  tar_file_read(files, fs::dir_ls(folders, glob = "*tsv"), read_tsv(file = !!.x, show_col_types = FALSE)),
  # nested tar_map
  tar_map(
    values = list(funs = c("mean", "sd")),
    tar_target(summary, summarise(files, x_sum = funs(x), y_sum = funs(y)))
  )
)
mcombined <- tar_combine(mean_combine, 
                         # tarchetypes helper to select all averages 
                         tar_select_targets(mapped, contains("_mean_")),
                         # .x placeholder all matching targets
                         # !!! unquote-splice operator
                         command = bind_rows(!!!.x, .id = "set"))

scombined <- tar_combine(sd_combine, 
                         # tarchetypes helper to select all averages 
                         tar_select_targets(mapped, contains("_sd_")),
                         # .x placeholder all matching targets
                         # !!! unquote-splice operator
                         command = bind_rows(!!!.x, .id = "set"))

combi <- tar_combine(stats, mcombined, scombined)

list(mapped, mcombined, scombined, combi)
```

## Corresponding DAG

:::: {.columns}
::: {.column width="65%"}
![](img/targets_ds_full_static.png){fig-align="center"}
:::
::: {.column width="35%" .fragment}
```{.r}
> tar_read(mean_combine)
# A tibble: 3 × 3
  set                  x_sum y_sum
  <chr>                <dbl> <dbl>
1 summary_mean_circles  54.3  47.8
2 summary_mean_lines    54.3  47.8
3 summary_mean_others   54.3  47.8
```

And final `stat` object:

```{.r}
> tar_read(stats)
# A tibble: 6 × 3
  set                  x_sum y_sum
  <chr>                <dbl> <dbl>
1 summary_mean_circles  54.3  47.8
2 summary_mean_lines    54.3  47.8
3 summary_mean_others   54.3  47.8
4 summary_sd_circles    16.7  26.9
5 summary_sd_lines      16.7  26.9
6 summary_sd_others     16.7  26.9
```
:::
::::

## Better project design

Thinking at what is a [good `targets`](https://books.ropensci.org/targets/targets.html#what-a-target-should-do) helps tremendously the coding

::: {.center}

>1. Are large enough to subtract a decent amount of runtime when skipped.
2. Are small enough that some targets can be skipped even if others need to run.
3. Invoke no side effects (tar_target(format = "file") can save files.)
4. Return a single value that is:
    + Easy to understand and introspect.
    + Meaningful to the project [...]


— _William Landau_

:::


## Data storage, `rds` is the default, but quite slow 

:::: {.columns}
::: {.column width="50%"}
::: {.callout-warning}
## Watch out

For malicious **promises**!

:::

![](https://files.mastodon.social/media_attachments/files/112/360/182/948/124/986/original/c5529d407f519690.png){height=400}


Relevant blog post: [CVE-2024-27322 Should Never Have Been Assigned And R Data Files Are Still Super Risky Even In R 4.4.0](https://rud.is/b/2024/05/03/cve-2024-27322-should-never-have-been-assigned-and-r-data-files-are-still-super-risky-even-in-r-4-4-0/) by **Bob Rudis**

:::
::: {.column width="50%" .fragment}
From `{tarchetypes}`:

- `tar_fst_tbl()` for **tibbles** ([`{fst}`](https://github.com/fstpackage/fst))

![](https://raw.githubusercontent.com/fstpackage/fst/develop/man/figures/fst.png){height=120}

- `tar_qs()` for **lists** (Quick serialization of {{< fa brands r-project >}} objects [`{qs}`](https://github.com/traversc/qs))

:::
::::


::: {.footer}
Source: [Konrad Rudolph](https://mastodon.social/@klmr/112360501388055184) about CVE-2024-27322
:::


## Excellent possibilities for debugging {.smaller}


::: {.incremental}
- Finish the pipeline anyway
    + `tar_option_set(error = "null")`
    + Useful for dynamic branching
- Error messages
    + `tar_meta(fields = error, complete_only = TRUE)`
- Save a targets workspace
    + `tar_option_set(workspace_on_error = TRUE)`
    + list workspaces: `tar_workspaces()`
    + load one: `tar_workspace(analysis_02de2921)` all object, variables are visible interactively
    + also: `tar_traceback(analysis_02de2921)`
- Pause the pipeline with the `targets debug` option.
    + `tar_option_set(debug = "analysis_58_b59aa384")`
    + see [example](https://books.ropensci.org/targets/debugging.html#pause-the-pipeline-with-the-debug-option)

:::


::: {.footer}
Further reading: [debugging chapter](https://books.ropensci.org/targets/debugging.html)
:::

## Simplify the layers

Remember that all code run in a fresh session, so needs to load its package dependencies.

To avoid it:

- Remove `{callr}`: `tar_make(callr_function = NULL)`

- Or the opposite, remove `{targets}`:

``` r
# What about just {callr} without {targets}?
callr::r( # same error
  func = function() {
    set.seed(-1012558151) # from tar_meta(name = dataset1, field = seed)
    library(targets)
    suppressMessages(tar_load_globals())
    data <- simulate_data(units = 100)
    analyze_data(data)
  },
  show = TRUE
)
```

::: {.footer}
Source: [debug repo](https://github.com/wlandau/targets-debug) by **William Landau**
:::

## Missing parts

:::: {.columns}
::: {.column width="50%"}
### [HPC](https://books.ropensci.org/targets/crew.html)

- [`{crew}`](https://wlandau.github.io/crew/) for autoscaling on workers

![](https://wlandau.github.io/crew/logo.svg){height=150}

``` r
library(targets)
library(crew)
tar_option_set(
  controller = crew_controller_local(workers = 2)
)
```

``` r
tar_crew()
#> # A tibble: 10 × 5
#>    controller    worker launches seconds targets
#>    <chr>          <int>    <int>   <dbl>   <int>
#>  1 my_controller      1        1   103.      104
#>  2 my_controller      2        1   100.      100
```


:::
::: {.column width="50%"}
- `{crew.cluster}` for job scheduler submission
    + `sge`
    + `slurm`

![](https://wlandau.github.io/crew.cluster/logo.svg){height=150}

### [Cloud computing](https://books.ropensci.org/targets/cloud-storage.html)

- AWS
- GCP
:::
::::

## Before we stop

::::: {.columns}
:::: {.column width="50%"}
:::{.callout-tip icon=false} 
## Highlights

- `targets`, dependencies manager, re-run what's needed

**William Landau** intro:

- Get started in four minutes: [vimeo video](https://vimeo.com/700982360)
- Example project: [targets-four-minutes](https://github.com/wlandau/targets-four-minutes)

:::


:::{.callout-tip icon=false} 
## Further reading 📚

- [Main website](https://docs.ropensci.org/targets/)
- [Targetopia](https://wlandau.github.io/targetopia/packages.html) **Landau** universe of targets-derived (stan/jags)
- [Video](https://www.youtube.com/watch?v=odcBA4ETLn8) from Bayes Lund by **William Landau**. October 2021
- [Documentation](https://books.ropensci.org/targets/) as bookdown by **Landau**
- [Debugging guide](https://books.ropensci.org/targets/debugging.html) by **Landau**

:::
::::

:::: {.column width="50%"}
::: {.callout-note icon=false} 
## Acknowledgments  🙏 👏

* **Eric Koncina** early adopter of `targets`
* [**William Landau**](https://github.com/wlandau) main developer of `targets`
:::
[Thank you for your attention!]{.thanks}
::::
:::::