Add capability to use regularly updated bad star list and history file

[jeanconn]

Could we agree on a spec for https://github.com/sot/ska-projects/issues/92 and set proseco and aca_preview to use it all in proseco 4.4? I'm also talking about having Mark add it to the list of files that would be synced.

We could finish process for updating the file and such after the release.

This would be a stretch for 4.4.

[taldcroft]

Yes great idea. We can agree on the spec for 4.4 and at least make a stub filled.

[taldcroft]

Without thinking too hard, a starting point is an hdf5 file like proseco_agasc.h5. then as few things TBD.

[jeanconn]

Agreed on an hdf5. By "like" do you mean basically keeping those columns and maybe adding a couple including something like a "just don't use this star" column?

We probably do have to figure out how we'd use a mag and mag_err in such a table in proseco before figuring out just what is helpful.

[jeanconn]

I think we want separate "just don't use this star" for acq and guide. Part of me wants a "why" in that column(s) but probably cleaner to just have a bool or int.

Then, do we want entries in the table to override proseco agasc in an arbitrary fashion? For the first example of the stars from GAIA that have bad proper motion in the AGASC, I wondered if we'd want to override the PM values and correct those star positions for selection (not very code complex but perhaps putting too much power into the update process for this supplementary table?). Or if we are only considering overriding for maybe mag/mag_err ... and for the rest of the kinds of things that would puts stars in the table it would just be a "don't use this star".

[jeanconn]

Or maybe we only get as far as "don't use this star" for this release.

[taldcroft]

I'm pondering this. I had about 5 disagreements with myself already... later.

[jeanconn]

I'm leaning toward starting with the "just don't use this star" implementation as a first complete item, and then, if we have time, going with arbitrary override of any proseco agasc column, but with extra checks of reasonable-ness on any stars with override information (which aca_review should be able to handle) and good extra info logging and debug statements for the user.

[taldcroft]

What about starting with an HDF5 file with two columns (initially):

• agasc_id
• status

The status field would be uint64. Each of the 64 bits would indicate some reason for being in the table. Here, because of performance concerns, we would have a supplemental "lookup table" that can be included as a table in the meta of the HDF5 file.

• bit 0 (0x1): do not use for guide
• bit 1 (0x2): do not use for acquisition
• bit 2: incorrect proper motion in AGASC
• bit 3: incorrect magnitude in AGASC (too bright)
• bit 4: incorrect magnitude in AGASC (too faint)
• bit 5: not found as GS, DSS shows a double star
  etc.

[jeanconn]

Sure. So you are leaning only to the "don't use this star" first draft. I was think the magnitude histories would really be helpful for star review, but if you want to make this the fast bad-star-list supplement and we'll add a separate file table with mag histories and such at some point in the future, that's fine too.

[jeanconn]

Or I suppose for a fast history, I'll be 99% of the way there for selection and review without mag histories as we nail down a definition for 'incorrect magnitude in AGASC'.

For example for a star with MAG_ACA of 8.0 and MAG_ACA_ERR of 0.01 and observed mag of 7.0 we could set bit 3, but not set bit 0 or 1 (or not if it isn't really interesting?).

[taldcroft]

Good thoughts. Adding to the "starting" point, columns:

• n_guide_obs (uint16)
• n_acq_obs (uint16)
• n_acq_fail (uint16) (though needs serious analysis to interpret)
• mag_obs (float32) (this would be a derived quantity based on a not-entirely-simple code to generate our best estimate of the magnitude based on all available data)
• mag_obs_max (float32) (same caveat, where this is not literally the max observed, but something like 99th percentile)
• mag_obs_min (float32)

So then this table would include one row for every star ever observed. Most would have status=0.

Getting the functionality to support generating and using this table has a lower priority than other PRs currently in work.

[jeanconn]

Indeed. Just brainstorming a bit while it was fresh. Thanks!

[jeanconn]

What are your thoughts for appropriate values for entries with no observed mag data?