issues in processing 20230309

[abhi0395]

Hi all,
There seem to be some issues with the processing of 20230309. The night started with some issues in r1A that affected the first few exposures (upto expid: 170817) evening calibrations and the first three tiles of the night (26084: bright, 26081: bright, 11775:dark). The night had to be re-processed (after flagging those exposures) with the second set of flat/arcs taken in the morning. @julienguy helped generate morning-darks..pdf

https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20230309/morningdark-20230309.pdf

and they look bad in r4, r5, z5, z8, z1, r9 (the ccd shows excess e- issue, @araichoor also pointed out this issue on slack space) We also re-ran with the morning set of zeros and we still see that excess electron issue in ccd! On the other hand, the morning darks for 20230310, and 20230311 look fine

https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20230310/morningdark-20230310.pdf
https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20230311/morningdark-20230311.pdf

So we are not sure if there was anything particularly bad on 20230309...

@julienguy also checked and confirmed from the VCCDSEC header keyword on 20230309, only r1 had its voltages reset. So, r1 reset should not have any impact on other ccd...

desi_header -i ~/redux/daily/exposures/20230309/00170{816,819}/frame-{r1,r4,r5,r9,z1,z5,z8,z9}-00170*.fits.gz  -k VCCDSEC
# VCCDSEC filename
 11541.7 /global/homes/j/jguy/redux/daily/exposures/20230309/00170816/frame-r1-00170816.fits.gz
 11541.7 /global/homes/j/jguy/redux/daily/exposures/20230309/00170816/frame-r4-00170816.fits.gz
 11541.7 /global/homes/j/jguy/redux/daily/exposures/20230309/00170816/frame-r5-00170816.fits.gz
 11541.7 /global/homes/j/jguy/redux/daily/exposures/20230309/00170816/frame-r9-00170816.fits.gz
 11534.9 /global/homes/j/jguy/redux/daily/exposures/20230309/00170816/frame-z1-00170816.fits.gz
 11530.7 /global/homes/j/jguy/redux/daily/exposures/20230309/00170816/frame-z5-00170816.fits.gz
 13695.1 /global/homes/j/jguy/redux/daily/exposures/20230309/00170816/frame-z8-00170816.fits.gz
 7771024.9 /global/homes/j/jguy/redux/daily/exposures/20230309/00170816/frame-z9-00170816.fits.gz
 2813.5 /global/homes/j/jguy/redux/daily/exposures/20230309/00170819/frame-r1-00170819.fits.gz
 14512.0 /global/homes/j/jguy/redux/daily/exposures/20230309/00170819/frame-r4-00170819.fits.gz
 14512.0 /global/homes/j/jguy/redux/daily/exposures/20230309/00170819/frame-r5-00170819.fits.gz
 14512.0 /global/homes/j/jguy/redux/daily/exposures/20230309/00170819/frame-r9-00170819.fits.gz
 14505.2 /global/homes/j/jguy/redux/daily/exposures/20230309/00170819/frame-z1-00170819.fits.gz
 14501.0 /global/homes/j/jguy/redux/daily/exposures/20230309/00170819/frame-z5-00170819.fits.gz
 16665.3 /global/homes/j/jguy/redux/daily/exposures/20230309/00170819/frame-z8-00170819.fits.gz
 7773995.2 /global/homes/j/jguy/redux/daily/exposures/20230309/00170819/frame-z9-00170819.fits.gz

@julienguy also checked and found that, for z1, the offset in the bias is visible in both exposures 170816 (before r1 issue) and 170819 (after r1 issue), so we do not understand if r1 issue has impacted any other ccd??

[julienguy]

Copying here a comment made on slack:
Looking back at the numbers, VCCD had been ON for only 2.6 hours for the first science exposures for most CCDs. That is as simple as that: we have to have VCCD ON for many hours to get sub-electron residual bias.

NIGHT EXPID CAMERA VCCDSEC
 20230309 170813 b0 9469.3
 20230309 170813 r0 7623390.9
 20230309 170813 z0 9466.5
 20230309 170813 b1 609609.6
 20230309 170813 r1 9473.3
 20230309 170813 z1 9466.5
 20230309 170813 b2 9469.3
 20230309 170813 r2 9473.4
 20230309 170813 z2 4280558.4
 20230309 170813 b3 437640.1
 20230309 170813 r3 437643.7
 20230309 170813 z3 9466.6
 20230309 170813 b4 9469.3
 20230309 170813 r4 9473.3
 20230309 170813 z4 9466.5
 20230309 170813 b5 609614.9
 20230309 170813 r5 9473.4
 20230309 170813 z5 9462.3
 20230309 170813 b6 9464.0
 20230309 170813 r6 609618.9
 20230309 170813 z6 9466.5
 20230309 170813 b7 9469.3
 20230309 170813 r7 609618.9
 20230309 170813 z7 9466.6
 20230309 170813 b8 11118.8
 20230309 170813 r8 11199.6
 20230309 170813 z8 11626.7
 20230309 170813 b9 7768959.3
 20230309 170813 r9 9473.3
 20230309 170813 z9 7768956.6

I would declare bad the dark tiles taken that night of 20230309, and have a second look at the others. We can try to mitigate this issue by setting the option --bkgsub-for-science to desi_preproc if we want to improve the reduction of the special program tiles.

[akremin]

@abhi0395 has this been addressed? It looks like there is a linked desisurveyops ticket which is closed.

[abhi0395]

Hi @akremin Yes, this has been addressed! The desisurveyops comments are the latest ones! You can close this one! Initially, I didn't know so I opened a ticket here but Eddie asked me to open on desisurveyops and that has been resolved.