Preferred method for handling interactive authentication?

[lindsayplatt]

I am currently working on a targets pipeline that pushes and pulls data from a service requiring the pipeline user to authenticate. I have been able to use my secrets manager to automate the login when I run the pipeline, but I am adding additional pipeline users who will not be able to use this approach. Instead, they will need to use the login function's interactive prompts and enter their username and password.

I have failed with two methods and succeeded with one. While I did find success, I am unsure if this is the preferred way to include interactive authentication within a pipeline and would love to hear other perspectives on this subject. Maybe this can generally be thought of as "what is the preferred method for accepting user input in a pipeline build?".

Failures:

1. Use the login function and then call tar_make(). Since targets::tar_make() builds the pipeline in a new environment, I cannot login first and then run the pipeline.
2. Run tar_make() and have it pause when the code prompts for user input. This also did not work - the code carried on running without waiting for user input which results in the login function throwing an error about invalid username and password input.

Success:

1. Run tar_make(callr_function = NULL) as you would during debugging with browser() to have the pipeline pause and wait for the user. This allowed the user to enter their username and password when prompted. Seems a bit risky considering the documentation states that the callr_function should not be NULL for serious reproducible work.

[wlandau]

targets is a very opinionated tool, and with the exception of debugging, interactivity in pipelines is out of scope. I recommend including instructions in your project's documentation to help your collaborators establish their own credentials before running the pipeline. Maybe it would also help to include a target or pre-pipeline step that asserts those credentials before launching into the full workflow.

[lindsayplatt]

Yes, it makes sense that targets has to limit it's scope somewhere and can't do everything!

I'm curious what risks you see with using the debugging callr_function=NULL approach? You don't guarantee a clean start environment, but the instructions for running this could be "first, clear the environment and then run tar_make(callr_function=NULL). These instructions would be much simpler to pass along compared to getting a user set up with credentials.

Could you elaborate on what you mean by asserts those credentials before launching into the workflow? To me, that sounds like what I tried and failed at first.

[wlandau]

I'm curious what risks you see with using the debugging callr_function=NULL approach? You don't guarantee a clean start environment, but the instructions for running this could be "first, clear the environment and then run tar_make(callr_function=NULL). These instructions would be much simpler to pass along compared to getting a user set up with credentials.

It is easy to forget to start with a clean environment, and accidentally invalidating expensive targets is egregious. These factors were constantly at play with drake:make(), which does not run in a new session: https://twitter.com/MilesMcBain/status/1101302743895961600. So even though callr_function = NULL works in your particular case, it always carries risk, and I have decided to discourage it as a matter of general policy.

Could you elaborate on what you mean by asserts those credentials before launching into the workflow? To me, that sounds like what I tried and failed at first.

It sounded like the user needs to set some local configuration with secret before attempting to run the pipeline. Maybe the script that runs tar_make() could first check that the required credentials exist in the local vault. And if the remote service provides a cheap way to test those credentials, that might be worth implementing too.