Dynamic files as specialized formats

[wlandau]

Summary

file_out() and friend handle static files: ones that you hard-code in your plan and you have to know in advance. This PR equips drake to watch dynamic files that you do not need to know in advance. Dynamic files can be local files or local directories, and they are fully compatible with dynamic branching.

Usage

1. Set format = "file" in target() within drake_plan().
2. Return the paths to local files from the target.
3. To link targets together in dependency relationships, reference target names and not literal character strings.

Example

library(drake)

bad_plan <- drake_plan(
  upstream = target({
      writeLines("one line", "my file") # Make sure the file exists.
      "my file" # Must return the file path.
    },
    format = "file" # Necessary for dynamic files
  ),
  downstream = readLines("my file") # Oops!
)

plot(bad_plan)

good_plan <- drake_plan(
  upstream = target({
      writeLines("one line", "my file") # Make sure the file exists.
      "my file" # Must return the file path.
    },
    format = "file" # Necessary for dynamic files
  ),
  downstream = readLines(upstream) # Use the target name.
)

plot(good_plan)

make(good_plan)
#> ▶ target upstream
#> ▶ target downstream

# Make changes...
good_plan <- drake_plan(
  upstream = target({
      writeLines("different line", "my file") # Change the file.
      "my file"
    },
    format = "file"
  ),
  downstream = readLines(upstream)
)

# The downstream target automatically reruns!
make(good_plan)
#> ▶ target upstream
#> ▶ target downstream

^{Created on 2020-02-21 by the reprex package (v0.3.0)}

cc

A bunch of folks have shown interest. Here are some.

• @billdenney.
• @vkehayas
• @MilesMcBain (This is a precursor to cloud data storage.)

Related GitHub issues and pull requests

• Ref: Dynamic files #1168, Special format for files? #1127

Checklist

• I understand and agree to drake's code of conduct.
• I have listed any substantial changes in the development news.
• I have added testthat unit tests to tests/testthat for any new functionality.
• This pull request is not a draft.

[codecov-io]

Codecov Report

Merging #1178 into master will increase coverage by 0.11%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #1178      +/-   ##
=========================================
+ Coverage   99.88%    100%   +0.11%     
=========================================
  Files          48      48              
  Lines        7871    7946      +75     
=========================================
+ Hits         7862    7946      +84     
+ Misses          9       0       -9

Impacted Files	Coverage Δ
R/drake_config.R	100% <ø> (ø)	⬆️
R/drake_plan.R	100% <ø> (ø)	⬆️
R/handle_triggers.R	100% <100%> (+3.46%)	⬆️
R/hpc.R	100% <100%> (ø)	⬆️
R/local_build.R	100% <100%> (ø)	⬆️
R/dynamic.R	100% <100%> (ø)	⬆️
R/drake_meta_.R	100% <100%> (ø)	⬆️
R/outdated.R	100% <100%> (ø)	⬆️
R/backend_clustermq.R	100% <100%> (ø)	⬆️
R/logger.R	100% <100%> (ø)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a4dfbd...649e9d2. Read the comment docs.

[billdenney]

This looks great!

Can it work with a vector of files, a vector of directories, or a mixed vector of files and directories? (I'll admit that the mixed vector isn't something that I've needed, but it seemed worth adding for completeness.)

[wlandau]

Can it work with a vector of files, a vector of directories, or a mixed vector of files and directories?

It sure can!