Data Validation Tool Enhancements

[bsunnquist]

Some future enhancements for the GSEG Data Validation Notebook:

• Verify header APERNAME based on APT Subarray (currently APERNAME is only checked to be consistent with DETECTOR, but not with the APT file, so this has to be checked manually)
• Modify adjust_exptype() to work with more than EXP_TYPE=NRC_IMAGE and NRC_GRISM (e.g. NRC_TSIMAGE, NRC_TSGRISM)
• Include checks for dark.fits file for files with EXP_TYPE=NRC_DARK, rather than assuming all uncal files have corresponding rate files
• Fix EFFEXPTM estimate calculation, which is giving wrong answers for some files
• Remove assumption that all detectors will be used for a given observation (currently missing files have to be checked for manually because of this)
• Remove dependence on MAST Portal (astroquery, read access to MAST directories)?
• Clean up unnecessary messages in output (i.e. only show warnings and mismatches), and save output to a text file
• Check TFRAME values to make sure they are computed for the correct number of outputs (see JIRA ticket JWSTDMS-19)
• enhancement (nice to have): check for more than just *rate.fits files (e.g., exposures with NINTS>1 will also have a *rateints.fits file, data products are here: https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/product_types.html)
• Check that any associations (ASNTABLE) exist and that its members are assigned the correct designation

[bsunnquist]

per @aliciacanipe, the following seems to be the correct equation:
EFFEXPTM = (TFRAME * (NGROUPS * NFRAMES + (NGROUPS - 1) * GROUPGAP +DRPFRMS1) * NINTS)

[aliciacanipe]

Just to capture some other things we talked about:

• check TFRAME values to make sure they are computed for the correct number of outputs
• enhancement (nice to have): check for more than just *rate.fits files (e.g., exposures with NINTS>1 will also have a *rateints.fits file, data products are here: https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/product_types.html)

[bsunnquist]

@bhilbert4 just fyi, we've started storing future enhancements for the data validation tool here

[bhilbert4]

Sounds good. Along the lines of checking for rateints file, what about expanding to check versions with any suffix that happen to be present? I guess really that just means cal and calints.

TFRAME should be correct (assuming it's using the function in Mirage). The only complication is for a few of the grism subarrays, where they can use either 1 or 4 amplifiers. The information on the number of outputs actually used is not captured in the APT xml file. For Mirage I fall back on the user having to specify that information for those few subarrays. Otherwise all subarrays use 1 output, and full frame uses 4.

The other issue that will be tough is not assuming all detectors are used. This is also information that is not in the APT xml file, which is why Mirage just blindly assumes that all detectors are used. I suppose the easiest fix would just be a dictionary listing the detectors used for a given aperture.

[aliciacanipe]

Yeah, my TFRAME suggestion was because I was worried about DMS and not Mirage (based on this ticket we filed last time, and it seems the numbers were still wrong this time): https://jira.stsci.edu/browse/JWSTDMS-19

[bsunnquist]

The APT sql file lists the number of apertures used and the aperture name for each observation (though there is only one entry per observation, so the latter isn't always accurate when using multiple detectors in the observation); also it only gives the number of SCAs used and not their actual names.

So is it true that given a module and subarray, the detectors used will always be constant (e.g. For an observation with Module=B and Subarray=SUB400P, the SCAs will always be B1, B5 like in observation #114 of the most recent gseg test)? Any chance this info is part of siaf already? Otherwise, is it available online in a table somewhere?

[bsunnquist]

and yeah, adding cal image checks should be very easy the way the code is currently structured - it might take a little more work incorporating the int.fits and dark.fits checks however.

[bhilbert4]

For the SCAs used, I think that's true in most cases. The special (engineering only?, WFSC) cases are where we need to be careful though. There are apertures there where for example, only A3 is used. Maybe for GSEG testing we don't need to worry about these special cases. But for commissioning we probably should.

[bsunnquist]

I've started incorporating these enhancements (#109) and checking them off as I go.

Minimally tested, but so far things are looking good. It's tailored to the GSEG3 data for now - I'm checking with John about the planned future GSEG observations to make sure this will work for all of them (some changes/assumptions will have to change in this tool for Commissioning though, I'm sure).

I'm also talking with John about a way to predict the EXP_TYPE based on the APT file; the Engineering Imaging template we use for the GSEG tests is making it tough to predict, and there isn't much useful info in the APT xml to help much with this.