Create quarto reports and branching - different useful names of reports?

[rkrug]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

This is a follow up to #1296 - in a nutshell, I do dynamic branding over files and create for each file graphs and some summary statistics.

Now I would like to have at the end have a report which is based on measures calculated as a target as well as the graphs created. I have e.g. tar_read(figure_oa_status_ias) in the quarto document to include a graph, tar_read(bibliography) to read the data. How can I do this when I am using dynamic branching?

my pipeline looks as follows:

tar_files(
        files,
        c(
            file.path(".", "input", "IPBES ILK.csv"),
            file.path(".", "input", "IPBES IAS.csv")
        )
    ),
    tar_target(bibliography, load_bibliography(files), pattern = map(files)),
    tar_target(figure_pub_year, plot_publication_year(bibliography), pattern = map(bibliography)),
    tar_target(figure_oa_status, plot_oa_status(bibliography), pattern = map(bibliography)),
    tar_target(figure_top_journ_data, plot_top_journals_data(bibliography), pattern = map(bibliography)),
    tar_target(figure_top_journals, plot_top_journals(figure_top_journ_data), pattern = map(figure_top_journ_data)),
    tar_target(figure_top_country_data, plot_top_country_data(bibliography), pattern = map(bibliography)),
    tar_target(figure_top_country, plot_top_country(figure_top_country_data), pattern = map(figure_top_country_data))

and I would like to add something along the lines of

tar_quarto(
        report,
        "bibliography_report.qmd",
        quiet = FALSE,
        map = (????)
    )

So that at the end I have one report for each branch.

[rkrug]

I switched to Static Branching because of the file names - makes more sense in this context.

I now have:

values <- data.frame(
    files = list.files(
        path = "input",
        pattern = "\\.csv$",
        full.names = FALSE
    ) |>
        gsub(pattern = "\\.csv$", replacement = "")
    # c(
    #     ILK = file.path(".", "input", "IPBES ILK.csv"),
    #     IAS = file.path(".", "input", "IPBES IAS.csv")
    # )
)
tar_map(
    values = values,
    tar_target(bibliography, load_bibliography(files)),
    # tar_target(metrics, dois_measures(bibliography)),
    tar_target(figure_pub_year, plot_publication_year(bibliography)),
    tar_target(figure_oa_status, plot_oa_status(bibliography)),
    tar_target(figure_top_journ_data, plot_top_journals_data(bibliography)),
    tar_target(figure_top_journals, plot_top_journals(figure_top_journ_data)),
    tar_target(figure_top_country_data, plot_top_country_data(bibliography)),
    tar_target(figure_top_country, plot_top_country(figure_top_country_data)),
    tar_quarto(
        report,
        "bibliography_report.qmd",
        quiet = FALSE
    )
)

And everything except of the tar_quarto() works. It aborts when trying the first tar_read() command, i.e. bibliography <- targets::tar_read(bibliography) which makes quite sense, as it is trying to read bibliography which does not exist.

[ginolhac]

if you use static branching, you should know upfront the different things you iterate over. So you should have different reports.
Or you can use parametrized reports if the skeleton is almost identical

[rkrug]

I think parametrised report is the way to go. The easiest would be to give the branch name (the part after the _ and then in the report use something along the lines of tar_read(bibliography_THE_BRANCH_NAME) but I tried this already - and did not succeed. How can I make tar_read(paste0("bibliography_", "IPBES.IAS")) read the target named bibliography_IPBES.IAS? I think then I have it (mainly).

[ginolhac]

you can use tar_name() that returns the name of the current branch as character. You should maybe provide a reprex so we can test it for real

[rkrug]

Yes - but I am getting somewhere. I now have the branch name in the quarto file as a parameter

values <- data.frame(
    files = list.files(
        path = "input",
        pattern = "\\.csv$",
        full.names = FALSE
    ) |>
        gsub(pattern = "\\.csv$", replacement = "")
)
names <- data.frame(
    name = gsub(values$files, pattern = " ", replacement = ".")
)
list(
    tar_map(
        values = values,
        tar_target(bibliography, load_bibliography(files)),
        tar_target(metrics, get_bibliography_measures(bibliography)),
        tar_target(figure_pub_year, plot_publication_year(bibliography)),
        tar_target(figure_oa_status, plot_oa_status(bibliography)),
        tar_target(figure_top_journ_data, plot_top_journals_data(bibliography)),
        tar_target(figure_top_journals, plot_top_journals(figure_top_journ_data)),
        tar_target(figure_top_country_data, plot_top_country_data(bibliography)),
        tar_target(figure_top_country, plot_top_country(figure_top_country_data))
    ),
    tar_eval(
        tar_quarto(
            name = name,
            path = "bibliography_report.qmd",
            execute_params = list(branch = tar_name())
        ),
        values = names
    )
)

and I read the targets using tar_read_raw() which works b ut gives a waring (not nice, but I can live with this).

My last 2 problems:

1. The tar_quarto() target is executed directly after the bibliography - which does not work as it depends on the other targets. Can I force it to run last?
2. The second time it runs, it overwrites the first one, and I do not manage to specify the output file name of the quarto rendering. I can not specify output in tar_quarto() and when I use quarto_args = 'output_file = your_output_filename.html' I get a panic error. I lean towards making my own target which calls quarto_render() and at the end remains the resulting report to what I want it to be.

[ginolhac]

Hi,
I had to work hard on this one, simplify it to something easier.
You really want static branching while the dynamic one would be clearer in the dependency graph. Yes branch names are more complex but now we need a tar_combine() step. Nevertheless, see below the sketch for static + parametrized quarto docs.
And I think the best for parametrized quarto docs is to use tar_quarto_rep().

report.qmd:

---
title: "biblio"
format: html
params:
  name: "default_name"
  value: "default_val"
---


## discussion 1297
```

```{r}
print(params$name)
```

_targets.R file

library(targets)
library(tarchetypes)

values <- data.frame(
  val = 1:3,
  names = c("A", "B", "C")
)

mapped <- tar_map(
  values = values,
  tar_target(bibliography, paste(val, "bib")),
  tar_target(summary, paste(bibliography, "sum"))
)

list(
  mapped,
  tar_combine(combi, mapped[["summary"]], 
              command = dplyr::bind_rows(!!!.x) |> 
                tidyr::pivot_longer(cols = dplyr::everything()) |> 
                dplyr::mutate(output_file = paste0(name, ".html"))),
  tar_quarto_rep(
    name = report,
    path = "report.qmd",
    execute_params = combi
  )
)

DAG:

After tar_make() the folder contains the 3 rendered HTML files using the provided output_file in tar_combine()

report_files
report.qmd
summary_1_A.html
summary_2_B.html
summary_3_C.html
_targets
_targets.R

Example of summary_1_A.html

summary_2_B.html

[rkrug]

Thanks - and sorry for causing that much work for you. I think I am out of my depth now (dplyr was never my strong side).

Final question: You said that dynamic branching would be better in this case - how would your example look in the case of dynamic branching? Maybe I get used to the hashed branch names...

[ginolhac]

No problem, I like playing with this package.
To ease yourself and also others, doing reprex is great. By simplifying, things usually become clearer and others can run code on their machine.
See this reprex for your last demand, reading biblio csv (fake but created inside the reprex) in a dynamic context.

# Parameterized Quarto:
lines <- c(
  "---",
  "title: 'biblio'",
  "format: html",
  "params:",
  "  author: \"default value\"",
  "  value: \"default value\"",
  "---",
  "## discussion 1297",
  "```{r}",
  "print(params$author)",
  "print(params$value)",
  "```"
)
writeLines(lines, con = "report.qmd")
# fake files
writeLines(c("author,year", "a,2012"), con = "bib1.csv")
writeLines(c("author,year", "bb,2014", "bb,2015"), con = "bib2.csv")
writeLines(c("author,year", "ac,2016", "ac,2010", "ac,1998"), con = "bib3.csv")
# the dynamic pipeline
targets::tar_script({
  library(tarchetypes)
  list(
    tar_files_input(bibfiles, fs::dir_ls(glob = "*.csv")),
    tar_target(bib, readr::read_csv(bibfiles, show_col_types = FALSE) |> 
                 # create the variable for the output filename based on the 
                 # input files,
                 dplyr::mutate(output_file = stringr::str_replace(bibfiles, ".csv$", ".html")),
               pattern = map(bibfiles)),
    tar_target(bibcount, dplyr::count(bib, output_file, author, name = "value"),
               pattern = map(bib)),
    tar_quarto_rep(
      report,
      "report.qmd",
      execute_params = bibcount
    )
  )
}, ask = FALSE)

> targets::tar_make()
▶ dispatched target bibfiles_files
● completed target bibfiles_files [0.001 seconds]
▶ dispatched branch bibfiles_a854fcb2edcbcc69
● completed branch bibfiles_a854fcb2edcbcc69 [0 seconds]
▶ dispatched branch bibfiles_37dc96391e254039
● completed branch bibfiles_37dc96391e254039 [0 seconds]
▶ dispatched branch bibfiles_dd5ecd4757c5539f
● completed branch bibfiles_dd5ecd4757c5539f [0 seconds]
● completed pattern bibfiles
▶ dispatched branch bib_009cfea8ffb7cde0
● completed branch bib_009cfea8ffb7cde0 [0.253 seconds]
▶ dispatched branch bib_463491e38f68f4fd
● completed branch bib_463491e38f68f4fd [0.006 seconds]
▶ dispatched branch bib_7ab6f4a25026850d
● completed branch bib_7ab6f4a25026850d [0.006 seconds]
● completed pattern bib
▶ dispatched branch bibcount_bac1924fb7edecfc
● completed branch bibcount_bac1924fb7edecfc [0.026 seconds]
▶ dispatched branch bibcount_60d955e73ee0b234
● completed branch bibcount_60d955e73ee0b234 [0.006 seconds]
▶ dispatched branch bibcount_62ba7581982a4a01
● completed branch bibcount_62ba7581982a4a01 [0.008 seconds]
● completed pattern bibcount
▶ dispatched target report_params
● completed target report_params [0.002 seconds]
▶ dispatched branch report_b8aa2b8c7e9d8782
● completed branch report_b8aa2b8c7e9d8782 [2.564 seconds]
▶ dispatched branch report_b769eb575a9413a0
● completed branch report_b769eb575a9413a0 [2.743 seconds]
▶ dispatched branch report_f1fab0c1bf797437
● completed branch report_f1fab0c1bf797437 [2.677 seconds]
● completed pattern report
▶ ended pipeline [8.69 seconds]

targets::tar_visnetwork(label = "branches")

the files once done:

❯ ls -1
bib1.csv
bib1.html
bib2.csv
bib2.html
bib3.csv
bib3.html
dynamic.R
report_files
report.qmd
_targets
_targets.R

• first report:
  

• second report
  

Branch names are cryptic but the filenames are clear and the DAG looks just great as one line