Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Al-522: Adding a DQ flags parameter to ramp_fit to make them pipeline indepedent #25

Merged
merged 8 commits into from
May 25, 2021

Conversation

kmacdonald-stsci
Copy link
Collaborator

The DQ flags were originally defined in the JWST pipeline and defined globally. The move to STCAL for ramp fitting to be used by multiple pipelines, there is a possibility the flags could be different. To accommodate for this, DQ flags are now passed as a parameter to ramp_fit as a dictionary with five required keywords.

@kmacdonald-stsci kmacdonald-stsci requested review from nden and dmggh May 25, 2021 16:14
nden
nden previously approved these changes May 25, 2021
Copy link
Collaborator

@nden nden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. A couple minor comments.

constants.dqflags["SATURATED"] = dqflags["SATURATED"]
constants.dqflags["JUMP_DET"] = dqflags["JUMP_DET"]
constants.dqflags["NO_GAIN_VALUE"] = dqflags["NO_GAIN_VALUE"]
constants.dqflags["UNRELIABLE_SLOPE"] = dqflags["UNRELIABLE_SLOPE"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could do constants.dqflags.update(dqflags) to update the entire dict at once.
We probably need to make sure all flags have a valid value

if None in constants.dqflags.values():
   raise ValueError

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created an update function and a NoneType check.

CHANGES.rst Outdated
@@ -14,6 +14,7 @@ ramp_fitting
------------

- Added ramp fitting code [#6]
- Added DQ flag parameter to `ramp_fit` [#25]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Start a new section in the Changes.rst and move the log there.

0.2.2 (Unreleased)
==================

ramp_fitting
------------

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made these changes to the change log.

nden
nden previously approved these changes May 25, 2021
@nden nden merged commit ea9f63e into spacetelescope:main May 25, 2021
Copy link
Contributor

@dmggh dmggh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Eventually both mission's ramp_fit_step.py will also need to be modified for this new parameter. I don't recall what other changes are also needed for parameter files.

@kmacdonald-stsci
Copy link
Collaborator Author

I opened a PR on JWST changing the unit tests and step code to handle this additional parameter: spacetelescope/jwst#6072

@kmacdonald-stsci kmacdonald-stsci deleted the al_522_dqflags branch May 25, 2021 20:24
Copy link
Collaborator

@jdavies-st jdavies-st left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize this review is a bit late, as now this is merged. But here it is anyway.

Comment on lines +1 to +7
dqflags = {
"DO_NOT_USE": None,
"SATURATED": None,
"JUMP_DET": None,
"NO_GAIN_VALUE": None,
"UNRELIABLE_SLOPE": None,
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If dqflags is a required parameter to pass to ramp_fit now, what's the purpose of having flags defined as constants in a separate module? Shouldn't all the flag mnemonics be happening in ramp_fit and then passed along directly to any function it calls?

Having these defined in a module constants also implies they are constants, which clearly they no longer are if they can be changed.


n_int, ngroups, nrows, ncols = data.shape

num_bad_slices = 0 # number of initial groups that are all DO_NOT_USE

while np.all(np.bitwise_and(groupdq[:, 0, :, :], DO_NOT_USE)):
while np.all(np.bitwise_and(groupdq[:, 0, :, :], constants.dqflags["DO_NOT_USE"])):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jump_flag is a pointer to the mnemonic `dqflags["JUMP_DET"], but here you use the mnemonic itself. Consistency would be good. I.e. either always use long-form mnemonics directly from the dict, or always use a variable, preferably defined at top of function.

@@ -82,6 +87,11 @@ def ramp_fit(model, buffsize, save_opt, readnoise_2d, gain_2d,
Object containing optional GLS-specific ramp fitting data for the
exposure
"""

constants.update_dqflags(dqflags)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like constants are no longer constants. ;-)

This is sort-of-a round-about way of getting the dict passed to the calling code might be better achieved by just passing the dict directly to ols_ramp_fit. The indirection here is a bit confusing.

Clearly this is a dict that needs to be shared by a number a functions, so a class structure would be better encapsulate what is going on here. But that may be too much of a leap for this PR.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the long term goal. In JP-553 I am creating an internal class. The needed flags are there. I will eventually remove this global variable. Not only are they not constants anymore, using global variables that can change can have bad side effects. This PR will be the last that requires changes to the JWST code. The changes regarding the internal class and the proper passing around of all the needed information, such as the DQ flags will not effect JWST anymore, i.e., a mirror PR in JWST will not be needed when a PR is opened in STCAL ramp fitting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants