Improve subprocess CRDS calls for easier debugging

[mcara]

Recently there were a couple of helpdesk issues (INC0155647 and INC0156501) when users reported errors with running CRDS commands using subprocess. One difficulty in understanding the cause of the errors was that suprocess in some notebooks is not set up to output errors. This PR modifies calls to subprocess.check_output()so that messages from CRDS can be read in the notebook.

[pllim]

Is there a way to do this without shell=True?

[pllim]

Isn't CRDS written in Python? Can't we call its Python API directly?

[mcara]

CC: CRDS experts @eslavich @jaytmiller

[mcara]

@pllim May I ask: what is the issue with shell=True?

[pllim]

https://docs.python.org/3/library/subprocess.html#security-considerations

[mcara]

Just out of academic curiosity, is there benefit in sorting the glob results first in this usage?

I do not see why would it matter here.

[pllim]

Well, let's say if the bestref is usually named in a way that would appear last in listing, and the function do a simple linear search, then sorted(..., reverse=True) would make the search faster? Anyway, I have no idea how it works, so I am just thinking aloud. And of course, it is probably out of topic as far as this PR is concerned.

[eslavich]

The order shouldn't matter -- the glob result is a list of datasets, and we have to make the best references determination for each of them.

Incidentally, I don't think shell=True is a security concern in this case. We wouldn't want to use it in web server code (for example) that interpolates user input into a command, but here the user is already in an environment where they can execute arbitrary shell commands so it's no more unsafe than a notebook in general.

[pllim]

It is not as much as a safety in this case, as to promote best practices. What if someone copy-paste what is here into server code? 😉

And given that it is possible to call Python API directly, there is really no reason to have shell=True at all in the first place.

[pllim]

p.s. Yeah, just ignore my comment about sorting. Sorry for the noise!

[bhilbert4]

I don't know if it is relevant or useful here or not, but here's an example of using the CRDS API: https://github.com/spacetelescope/mirage/blob/master/mirage/reference_files/crds_tools.py#L141

[mcara]

I would say that I just want to improve one aspect of existing notebooks to address a specific issue. If there is desire to re-design the notebooks, that could be done too as a separate PR.