[help] Cross-projects Reference for ETL

[Jiefei-Wang]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Is it possible to monitor the change of a target object from another project without loading it, and only load the object if the current target is outdated and needs an update? The use case is that I have an ETL project which has three tar projects:

1. tar project 1: Extract
2. tar project 2: Transform
3. tar project 3: Load

Since they are defined in separate projects. The downstream projects must know if the upstream data has been changed or not. The current solution is to call tar_load_everything and load all data from the previous project. This method has a significant overhead and makes parallel computing not possible. Therefore, I'm hoping to have a simple and fast solution for it.

I'm not a targets expert, but I can imagine two solutions for that:

1. Add an additional argument in tar_make to declare dependencies(e.g. project_deps). Therefore, the data can be loaded when needed.
2. Create a new function to explicitly do it(e.g. tar_load_project). This can load the project's meta data only. When tar_make is called, it can make the required data available.

Any suggestions will be appreciated.

[wlandau]

If you need to split the workflow into 3 projects as you say, then each successive project could reference the data files from _targets/objects/ from the previous one.

[Jiefei-Wang]

Thanks for the suggestion. I never thought about this. I read the tutorial https://books.ropensci.org/targets/data.html#external-files. The example provided seems promising, but is a little bit verbose

# _targets.R
library(targets)
library(tarchetypes)

create_output <- function(file) {
  data <- read.csv(file)
  output <- head(data)
  write.csv(output, "output.csv")
  "output.csv"
}

list(
  tar_target(name = input, command = "data.csv", format = "file"),
  tar_target(name = output, command = create_output(input), format = "file")
)

The target input only holds a character file path, so the downstream targets are still responsible for reading the file. Is it possible to hand the file reading over to targets package? For example, what I hope to have is something like

## This is still an external file, it will be read when the output target needs to run
tar_target(name = input, command = "data.csv", format = "file", read_func = read.csv)

[Jiefei-Wang]

I followed @wlandau's suggestion and made this verbose code. The idea is to use the external file + customized read/write function. For each external object, I need two targets

• Target 1: Claim and trace the external file
• Target 2: Read the external object, and only write metadata for the current project(customized write function). When the object is needed for building any target, read the metadata, find its project, and read the data from its project (customized read function)

By doing this, the external object wouldn't be stored in my current project. I can still trace its change. Is this something we can add to the targets package? So other users can also use this for their ETL process.

library(glue)
## For target command
## It takes a file path from another project as input, 
## read the file and return the data
read_upstream_file <- function(project, file_path){
    target <- basename(file_path)
    store <- tar_config_get("store", project=project)
    tar_read_raw(target, store=store)
}

## Customized read function
read_project_file <- function(path){
        file_meta <- readRDS(path)
        store <- file_meta$store
        target <- file_meta$target
        targets::tar_read_raw(target, store=store)
    }

## Customized write function, only save the file meta data
## I have to use the eval(parse(text=...)) to create the function
## as the enclosed environment does not work
write_project_file <- function(project){
    store <- targets::tar_config_get("store", project=project)
    eval(parse(text=
    glue(
      "
      function(object, path){{
        target_name <- basename(path)
        meta <- targets::tar_meta(store='{store}')
        target_idx <- which(meta$name == target_name)
        if(length(target_idx) == 0){{
            stop(paste0(path, ' Target not found'))
        }}
        target_hash <- meta$data[target_idx]
        file_meta <- list(
          store = '{store}',
          target = target_name,
          hash = target_hash
        )
      saveRDS(file_meta, path)
    }}
      "
    )
    ))
}

## Load all objects from another project as targets
tar_load_project <- function(project){
    store <- tar_config_get("store", project=project)
    format_project <- 
      tar_format(
        read = read_project_file,
        write = write_project_file(project),
        marshal = NULL,
        unmarshal = NULL,
        convert = NULL,
        copy = NULL,
        substitute = list(),
        repository = NULL
      )

    ## list all files in the project
    files <- list.files(glue("{store}/objects"))

    file_targets <- c()
    for(i in seq_along(files)){
        target_name <- files[i]
        target_file_referece <- tar_target_raw(
          name = glue("{target_name}_upsteam_file"),
          glue("{store}/objects/{target_name}"), 
          format = "file")
        target_data <- 
        eval(parse(text = 
        glue("
        tar_target({target_name}, 
          read_upstream_file('{project}', {target_name}_upsteam_file), 
          format = format_project)")
        ))
        file_targets <- c(file_targets, target_file_referece, target_data)
    }
    file_targets
}


project_extract <- tar_load_project("step1")
project_transform <- tar_load_project("step2")
c(
  project_extract,
  project_transform
)

[wlandau]

If you have a solution you like, you can build it in a separate package that depends on targets. https://github.com/databio/unitar did something similar.

[Jiefei-Wang]

This sounds great! I can work on it. I have another question that I hope you can kindly help me with: Is there any way to change the hash function for a particular target? I see you have tar_format to help objects read and load, but I cannot find a function to replace the hash function. Thanks!

[wlandau]

I do not have plans to make the hash function configurable in targets.