Local config file is deleted when using tar_resources_aws()?

[philiporlando]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hello, and thank you for creating this package—it's been a tremendous help and has greatly simplified much of my work!

I’m currently integrating tar_resources_aws() into my project and encountering some unexpected behavior. My {targets} pipeline is built around the {config} package, which reads from a config.yml file. To detect changes in the config.yml, I’ve been using a tar_target(format = "file") target. This has worked seamlessly when using the local file system to store {targets} objects/metadata: any changes to the config.yml file would invalidate the config_file target, triggering a rebuild of this target and its downstream dependencies, such as config_value.

Here’s what my config.yml looks like:

default:
  config_value: "my-config-value"

And here is what my previous _targets.R file looked like:

library(targets)

list(
  tar_target(
    config_file,
    command = "config.yml",
    format = "file"
  ),
  tar_target(
    config_value,
    config::get(value = "config_value", file = config_file)
  )
)

Running this interactively:

tar_make()

tar_read(config_value)
# [1] "my-config-value"

Everything works as expected.

However, when I switch to using tar_resources_aws(), I experience an issue where the local config.yml is unexpectedly removed.

Here's my new _targets.R file:

library(targets)

resources <- tar_resources(
  tar_resources_aws(
    bucket = "my-bucket",
    prefix = "my-project"
    )
  )
)

tar_option_set(
  repository = "aws",
  resources = resources
)

list(
  tar_target(
    config_file,
    command = "config.yml",
    format = "file"
  ),
  tar_target(
    config_value,
    config::get(value = "config_value", file = config_file)
  )
)

If I run tar_make(config_file), the config.yml file is actually deleted from my local file system. I initially thought I had accidentally removed the file, but I later discovered that a config_file object had been created in my S3 bucket. This behavior prevents me from easily editing the local config.yml file to adjust the pipeline configuration at runtime.

I'm hoping to get a better understanding of why this is happening, and learn what I can do to ensure that local config files are preserved when using tar_resources_aws().

Edit: I'm seeing the below in the storage formats doc:

If repository is not "local" and format is "file", then the character vector returned by the target must be of length 1 and point to a single file. (Directories and vectors of multiple file paths are not supported for dynamic files on the cloud.) That output file is uploaded to the cloud and tracked for changes where it exists in the cloud. The local file is deleted after the target runs.

It looks like this is the intended behavior of file-based targets when using a remote repository. Are there any recommended workarounds for this behavior when using the {config} package? It would be helpful if I could continue using a local config.yml to provide necessary config at runtime to my pipeline, while also making use of AWS S3 to store its objects/metadata. Would switching over to _targets.yaml help avoid this issue? Any input is appreciated!

[wlandau]

The assumption with repository = "aws" is that these are large and either uncomfortable or unfeasible to keep on disk. Your config_file target could make a temporary local copy of config.yml and return that copy as the file path.

[wlandau]

Happy to review PRs for the docs.

[philiporlando]

Thanks for clarifying this! I wasn’t entirely sure how to implement the temporary local copy of the config.yml approach you suggested, but I believe I've found a workable solution by overwriting the default repository for this target.

library(targets)

resources <- tar_resources(
  tar_resources_aws(
    bucket = "my-bucket",
    prefix = "my-project"
    )
  )
)

tar_option_set(
  repository = "aws",
  resources = resources
)

list(
  tar_target(
    config_file,
    command = "config.yml",
    format = "file",
    repository = "local" # overwrite default repository to prevent the local file from being deleted
  ),
  tar_target(
    config_value,
    config::get(value = "config_value", file = config_file)
  )
)

This setup does exactly what I was aiming for: it allows me to work with a local config.yml file while enabling targets to track any changes made to it via format = "file", ensuring out-of-date targets are properly re-materialized. At the same time, it uploads all other target objects and metadata to my S3 bucket.

[wlandau]

Yeah, if you don't actually need to upload a copy of config.yml, then repository = "local" fits nicely.