Integration with reticulate package

[jmbuhr]

I am really excited about the targets package and love the functional style. Because R's reticulate package makes it so easy to include python functions in R and it handles all the conversion automatically, it got me thinking how we could use python scripts in targets. I am sure the interest will be there, but I want to make sure the design principles are correct before I jump in and potentially build something for the targetopia. Or maybe you have also already though about this and have other plans.

Here is the state of what is currently possible with python in targets:

Say we have a python file py/test.py with the following content:

def make_zeros():
  return [0,0,0]

We can load all python scripts in our _targets.R file, inspired by the template of @MilesMcBain in tflow (https://github.com/MilesMcBain/tflow), via:

lapply(list.files("py", full.names = TRUE), source_python, envir = globalenv())

However, because the reticulated python function looks the same to targets even if the underlying implementation of make_zeros changes (i.e. <function make_zeros at 0x7f11df131430>), we need to tell targets to explicitly watch the source file for changes and mention it in the target:

tar_plan(
  tar_file(py_script, "py/test.py"),
  tar_target(zeros, {py_script; make_zeros()})
)

I would love input from reticulate experts on whether it is possible for targets to be able to track the reticulated functions as it does for R functions. And if this is not the case, if it would be a sufficient usecase for a targetopia package to bundle the source-file-watching and function call into a target factory.

[wlandau]

Yeah, I think lapply(list.files("py", full.names = TRUE), source_python, envir = globalenv()) and a Python-specific target would belong in an R Targetopia pacage. I do not have time to implement one personally, but I would be happy to chime where useful.

The trickiest part is the serialization of non-exportable objects. Those Python pointers are not valid outside the current R session, and this could interfere with distributed computing scenarios. targets makes special exceptions for Keras and Torch (e.g. tar_target(..., format = "keras")) because these are common use cases. I hesitate to implement something for reticulate immediately because of the upcoming work to marshal R objects in serialization.

One workaround might be to define a project-level .Rprofile file that loads all the python code in advance. That way, each HPC worker node should have pointers valid for its own R session.

[jmbuhr]

Oh indeed. I think this marshaling / serialization thing is above my current python and HPC knowledge 😕

[Aariq]

I was able to use a python function via reticulate using tar_hook_before() to source the function in the same session as the target as a workaround.

E.g. something like

library(targets)
library(tarchetypes)
library(reticulate)

tar_plan(
  tar_target(zeroes, make_zeroes())
) |> tar_hook_before(hook = source_python("py/test.py"), names = "zeroes")

Not super ideal, but if you only have a few targets that use python functions this seems to work, but doesn't track changes to the python script.