GroupDQ index bug in ramp_fit.py

[chriswillott]

This is a simple issue where the same effect has been introduced twice in subsequent lines of code, so can be fixed by changing one of the lines.
The group indices of the GROUPDQ section need to be shifted by 1 in order to line up with the differences array.
Line 348 does this shift by 1 by extracting the GROUPDQ section with gdq_sect[1:,:,:].

jwst/jwst/ramp_fitting/ramp_fit.py

Lines 348 to 349 in ad87fa4

	i_group,i_yy,i_xx, = np.where( np.bitwise_and(gdq_sect[1:,:,:],
	dqflags.group['JUMP_DET']) != 0)

Line 355 does the same shift again by setting the indices as i_group-1 instead of i_group.

jwst/jwst/ramp_fitting/ramp_fit.py

Line 355 in ad87fa4

first_diffs_sect[ i_group-1, i_yy, i_xx ] = np.NaN

Only one of the above shifts is required.

Running version 0.13.7.

[nden]

@stscijgbot-jp

[stscijgbot-jp]

This issue is tracked on JIRA as JP-1950.

[kmacdonald-stsci]

Chris, do you have test data you used to find this bug? I have been assigned this issue.

[chriswillott]

Hi Ken,
I have an example that shows there is an error in the code. It doesn't actually change the final rate image for this test data because this part of the code is only setting the first differences and in many cases the error here does not change the final result. I don't have an example handy of a case where it does affect the final rate image, since it was a few years ago I found the bug.
Test:
I added a series of large artificial jumps in nearby pixels to a NIRISS 13 group dark frame like this:

    #add a jump at each group in ramp at a pixel separated by 5 pixels from previous jump 
    for j in range(numgroups):        
        jumps[0,j:,800,(800+5*j)]=25000
 

I ran this through the detector1 pipeline twice. The second time I edited ramp_fit.py changing line 355 above (now at line 725 in v1.1.0) from
first_diffs_sect[ i_group-1, i_yy, i_xx ] = np.NaN
to
first_diffs_sect[ i_group, i_yy, i_xx ] = np.NaN

I also added a print statement before and after this line in the code to print the pixel values of the first 4 pixels that I added the jumps to:

print ('Before setting Nans',first_diffs_sect[:4,800,[800,805,810,815,820]])
print ('After setting Nans',first_diffs_sect[:4,800,[800,805,810,815,820]])

Here are the results, first with the original code:
Before setting Nans
[[2.3902344e+01 2.6105770e+04 2.1851049e+01 1.5539833e+01 2.6077328e+01]
[2.1332031e+01 1.1505859e+01 2.6460771e+04 2.7400354e+01 8.4289856e+00]
[3.0751953e+01 2.7750000e+01 3.5666016e+01 2.6588086e+04 2.9540726e+01]
[3.8849609e+01 2.5761719e+01 2.2011719e+01 2.1576172e+01 2.6548316e+04]]
After setting Nans
[[2.3902344e+01 2.6105770e+04 nan 1.5539833e+01 2.6077328e+01]
[2.1332031e+01 1.1505859e+01 2.6460771e+04 nan 8.4289856e+00]
[3.0751953e+01 2.7750000e+01 3.5666016e+01 2.6588086e+04 nan]
[3.8849609e+01 2.5761719e+01 2.2011719e+01 2.1576172e+01 2.6548316e+04]]

Then with the edited code:
After setting Nans
[[23.902344 nan 21.85105 15.539833 26.077328]
[21.332031 11.505859 nan 27.400354 8.428986]
[30.751953 27.75 35.666016 nan 29.540726]
[38.84961 25.761719 22.011719 21.576172 nan]]

You can see the NaNs now line up correctly with the high 2.6e+04 jump values.

I can provide the raw fits file used for this test if it's useful.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed in spacetelescope/stcal#12