[help] target type parquet: hive partitioned and tar_read() return dataset not tibble?

[rkrug]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi
I have two questions concerning the parquet target format.

1. Is it possible to write the results from a target to a hive partition parquet dataset instead of a single parquet file?
2. How can I return the dataset instead of the nibble when reading the target? tar_read() returns the nibble from the parquet file, which works for smaller datasets, but may cause problems when the parquet dataset is larger and does not fit into memory.
   Thanks

Rainer

[Aariq]

targets expects targets to be saved as a single file in _targets/. I don't think this will work unless you use format = "file" and write your target function to save to a partitioned parquet dataset and then open that dataset in any downstream targets

[rkrug]

Oh - thanks for the clarification - makes sense.

At the moment, I have defined the target function that it saves the hive partitioned dataset and returns the file patch - but as a normal target (without format specification).

save_hive_parquet <- function(data){

path <- "./somewhere"

unlink(path, recursive = TRUE)
save_to_hive(data, path)

return(path)

and

tar_target(parquet_hive, save_hive_parquet(data)),

Do I just have to specify format = "file", or do I have to change something else? What are the differences between my approach and the one using format = "file"?

[Aariq]

format = "file" tracks the files in path for changes. The pipeline may not work with format = "rds" (the default) because when you update the data, the file paths may not change and downstream targets may not be invalidated correctly.

[rkrug]

OK - setting format = "file" and not changing anything (i.e. still returning the path to the dataset) should work.

Thanks.

[rkrug]

And it does work.