generate a diff file with differences between supplements

[javierggt]

This PR adds functionality to generate an ECSV file with the (effective) differences between supplements.

I say "effective" because, if a star is not in the supplement, then the catalog magnitude is used, so strictly speaking it is not just the difference between supplements. It is the difference in magnitude reported by agasc when using the supplement.

Description

Interface impacts

Provides a new data file intended to be used by other groups (sot mp).

Testing

Unit tests

Independent check of unit tests by Jean

• Mac

(ska3) flame:agasc jean$ pytest
==================================================== test session starts =====================================================
platform darwin -- Python 3.10.8, pytest-8.0.2, pluggy-1.4.0
rootdir: /Users/jean/git
configfile: pytest.ini
plugins: timeout-2.2.0, anyio-3.6.2
collected 72 items                                                                                                           

agasc/tests/test_agasc_1.py .....                                                                                      [  6%]
agasc/tests/test_agasc_2.py ..........sssss..........ss...............                                                 [ 65%]
agasc/tests/test_agasc_healpix.py ...........                                                                          [ 80%]
agasc/tests/test_obs_status.py ..............                                                                          [100%]

=============================================== 65 passed, 7 skipped in 13.57s ===============================================
(ska3) flame:agasc jean$ git rev-parse HEAD
9575fc9f068361d3b236385152d0218b2c23a49c

Functional tests

Testing the diffs

Comparing between flight and RC:

python -m agasc.scripts.supplement_diff --from $SKA/data/agasc/agasc_supplement.h5 --to $SKA/data/agasc/rc/agasc_supplement.h5 -o agasc_supplement_diff.ecsv

Comparing when there are no diffs:

python -m agasc.scripts.supplement_diff --from $SKA/data/agasc/agasc_supplement.h5 --to $SKA/data/agasc/agasc_supplement.h5 -o empty.ecsv

Testing sending email

From the local repo:

export PYTHONPATH=`pwd`
mkdir -p ska/www/ASPECT/agasc/supplement
mkdir -p ska/data/agasc/rc/promote
ln -s $SKA/data/* ska/data/
cp $SKA/data/agasc/* ska/data/agasc/
export SKA=`pwd`/ska
cp ska/data/agasc/agasc_supplement.h5 ska/data/agasc/rc/promote
python -m agasc.scripts.supplement_tasks update-rc

This script fails in the last step (updating the supplement in rc) but does everything in the PR.

[jeanconn]

My only go-back is that it seems like this should write the diffs out to date directories .

[taldcroft]

I completely missed this. Overall it looks good.

👍 to @jeanconn's comment to write the diffs out to date-based directories. The existing directory for the corresponding supplement report seems like a good location to me, but whatever.

[taldcroft]

It's not super important but I don't understand why mag_aca_err is treated differently from mag_aca. Only the latter gets filled from the catalog if it is missing.

agasc_id mag_aca_from mag_aca_err_from last_obs_time_from mag_aca_to mag_aca_err_to last_obs_time_to mag_aca bad_star_source_from bad_star_source_to d_mag d_mag_err
...
27921360 8.840762 0.031336494 759648038.866 8.836912 0.03262599 831848273.596 8.75863 -9999 -9999 -0.0038499832 0.0012894981
27921920 8.933575 "" "" 8.970234 0.03473679 831848273.596 8.933575 -9999 -9999 0.03665924 ""

Meaning the last row could be like

27921920 8.933575 <mag_aca_err> "" 8.970234 0.03473679 831848273.596 8.933575 -9999 -9999 0.03665924 <d_mag_aca_err>