running without archiving

[gofford]

As I've mentioned in other threads, targets is great.

However, sometimes I want to just 'run' a pipeline. This is mainly for interactive use. Ideally I'd want to run the pipeline, keeping all artefacts in-memory and without archiving them with Disk/cloud-based IO.

Is this something possible currently? I have scoured the docs, and as far as I can tell a 'format = NULL' or equivalent isn't really supported. Can you advise?

[wlandau]

In development targets, Target Markdown in interactive mode does most of what you describe. A {targets} code chunk can have an entire target list, and when it runs, the objects end up in memory. It does still store and retrieve the data from disk though because it is important to test if the output can be saved in the selected format (helps avoid surprise bugs when you run the pipeline for real).

[gofford]

I can see the need for that from a production scenario. In general, is it possible to have a memory-only target? i.e., one that does not get saved to disk?

Use case here is that there are some quirks with serialisation of LightGBM models where R-native serialisation files (rds, qs, etc) fail-on-load unless the serialisation is carried out with lightgbm::lgb.save() which passes through C (I believe). See:

microsoft/LightGBM#4296
curso-r/treesnip#33

When using LightGBM with tidymodels, the workaround for this is thus to dump the fit itself to a text file, save the overall workflow as a separate normal serialised object, then load the two objects separately and re-create the workflow manually. There are two ways to then use this model in a target

1. Load and recombine the fit and workflow objects within each of the dependent functions and target step. This incurs a lot of IO overhead as the object(s) are continually pushed and pulled from S3. However, it does work.

2. An alternative solution that I've not been able to get to work is to have a discrete step in the targets pipeline that pulls the two files from S3, recombines them, then holds the workflow in memory for use in subsequent steps (from memory). However, while the recombination does succeed it also incurs a corresponding artefact that is not useful, and subsequent steps fail as they seem to try using the broken serialised object (even with memory = 'persistent').

It would be useful here to have a method of deeming a given step in the pipeline as completely transient, and without IO to disk. I can see this being useful in other model-based scenarios where I just want to pull a pre-trained model from S3 to the local machine without re-achiving -- if that makes sense?

[wlandau]

This is similar to keras and torch, which create non-exportable objects: https://cran.r-project.org/web/packages/future/vignettes/future-4-non-exportable-objects.html. Because keras and torch are popular machine learning packages and have no plans for compatibility with saveRDS()/readRDS(), I implemented format = "keras", format = "torch", format = "aws_keras", and format = "aws_torch". One approach here would be format = "lightgbm" and format = "aws_lightgbm". However, since the lightgbm developers expressed interest in supporting saveRDS(), I would prefer not to support special lightgbm formats in targets because of the long-term maintenance burden. When the lightgbm developers introduce compatibility with R's base serialization, you will be able to select the default format = "rds" (or format = "aws_rds") which should just work at that point.

Targets without storage are out of scope because they do not align with the intent of the package. Without storage, everything in the pipeline will vanish when you restart your session, which does not serve the majority of projects. Even in Target Markdown interactive mode, it is best to test storage and retrieval so you can catch surprises like this one before running the fully scaled-up pipeline for real.

[gofford]

That's fair enough, and I appreciate the reply. Completely understand the reasoning as outside of this specific example (which can be characterised as an issue with LightGBM more than anything!) I've not yet encountered a similar problem where storage of artefacts is undesired 👍