Develop new calibration procedure for Hyperspectral Imagers

[dlebauer]

@czender commented on Thu Mar 02 2017

Description

We discussed pitfalls of current hyperspectral (HS) calibration procedure and agreed to attempt to improve calibration by expanding the temporal/angular resolution of the Spectralon reflectance imagery, utilizing the spectral downwelling irradiance from the Environmental Logger (EL) sensor, and possibly incorporating radiative transfer modeled (RTM) fluxes to extend the effective bandwidth of the EL sensor beyond 1 micron, so the technique can be used for the SWIR imagery.

Details

1. Lemnatec developed the current (someday to be "old") HS calibration procedure, employed since 201609. This procedure has been based on a single set of exposures made 20161021, and does not use any "real time" data from the EL. The script that "reduces" the spectralon exposures to the form used in the HS workflow is here
2. An older HS calibration procedure utilizes "real time" irradiance from the EL. This was never fully implemented due to lack of input data (spectralon imagery in particular, but also spectral irradiance data).
3. The "new" calibration procedure will combine elements from both of the above procedures, and yield reflectance data that better accounts for zenith-angle and cloud effects, which are both large.

Completion Criteria

• New hyperspectral calibration algorithm documentation #282 Update and document new calibration procedure
• New hyperspectral calibration algorithm documentation #282 New data reduction procedure for Spectralon calibration imagery #289 Update new calibration procedure to account for dark counts
• Hyperspectral Calibration #208 Collect clear sky diurnal hyperspectral spectralon images for calibration
• Hyperspectral Calibration #208 Collect overcast sky diurnal hyperspectral spectralon images for calibration
• New data reduction procedure for Spectralon calibration imagery #289 Create new reduction procedure based on current reduction here
• Interpolate EL irradiance to time of HS image @FlyingWithJerome
• Interpolate Spectralon exposure to cosine zenith angle of HS image
• Modify HS calibration to use interpolated irradiance and spectralon exposure (VNIR+SWIR)
• Validate/quality control VNIR reflectance for clear sky (Milestone 1)
• Compute clear sky RTM fluxes for zenith angle of HS image
• Extend EL-observed spectral irradiance with RTM irradiance (SWIR only)
• Validate/quality control SWIR reflectance for clear sky (Milestone 2)
• Obtain or diagnose cloud fraction
• Compute all sky (with clouds if present) RTM fluxes for zenith angle of HS image
• Use interpolated cloudy sky spectralon exposure, RTM fluxes in HS calibration
• Validate/quality control VNIR+SWIR reflectance for all sky (Milestone 3)
• Shadow detection algorithm/correction?

---

@czender commented on Thu Mar 02 2017

As y'all can see, I have created this issue as an "epic" for the new calibration procedure. Please comment add modify. In the next few days I'll start to add some sub-issue that correspond to the completion criteria. If you know what needs to be done, please feel free to add those sub-issues yourself and link back to this epic.

[smarshall-bmr]

At the moment I'm still working with LemnaTec trying to get the SWIR and VNIR to take full data stripes. I'll get to this as soon as possible but I'm working through issues one at a time.

[dlebauer]

Do the sensors need to take full data stripes to generate the calibration images? Or can this be prioritized (please)?

[smarshall-bmr]

@dlebauer We tend to get help from Headwall in bursts so I'm try to take advantage of their help while we have it.

@rjstrand Who do I need to talk to about the 50% grey target again?

Our current plan is to calibrate the VNIR on a 50% target, does this have to be a spectralon target or do we have other options on material? The calibration run will now include the SWIR as well, I haven't heard if there is a consensus on a target for that calibration. Maybe we should have a call to talk about that like we have with the VNIR? Of course I can just gather every target I can get my hands on and scan them all on the assumption that one will work for our purposes, or at least inform what type of target we should be using.

[dlebauer]

@smarshall-bmr makes sense. I think that the 'every target you can get your hands on' approach makes sense. From the summary of a previous call #208 (comment) it sounded like both a 4% and a 50% target were available and would be sufficient, but that having a target in between these values would also be useful.

[smarshall-bmr]

@dlebauer It occurred to me that the 50% target may not be spectralon, hopefully @rjstrand is able to chime in here.

[czender]

The brand of, or material used for, the target is immaterial as far as I can tell. If a non-Spectralon target is used then we should (and will) reword some of the documentation and scripts to use a more generic term (e.g., "target") than Spectralon, so as to prevent confusion.

[rjstrand]

Hi, Sorry for my absence from this conversation...

@dlebauer @smarshall-bmr As I remember, Nadia has money budgeted to buy panels specifically for this purpose. I think she is only waiting for someone to tell her how many, what size, and what reflectance values... within the budget constraints and then she can order them.

ALARC has panels and tarps. As I remember, the panels are 50 (Spectralon) and 20% (Not sure). Tarps are 68, 48, 8, and 4%.

@dlebauer - Stuart is out of town for a while. I'll contact you once I get to Maricopa next week.

[NewcombMaria]

The ALARC set of big tarps have been deployed near the scanner field on some of the UAV flight days around solar noon. The tarps are too large to place within the scanner field. While Stuart is out of town I've offered to help with the calibration scan, with Bob's help to get the correct targets in the correct positions etc.

[dlebauer]

@remotesensinglab will 20% and 40% suffice? Could you advise @nshakoor on which additional standards (if any) we will need?

@rjstrand thanks for your help! I understand we also these 95% reflectance and and color targets:
SphereOptics-Diffuse-Reflectance-Standards-Targets-Materials-Zenith-Polymer.pdf
spectralon_color_standards_data_sheet.pdf

[NewcombMaria]

@remotesensinglab and @dlebauer we are tentatively planning for April 15 for the 'clear sky diurnal hyperspectral spectralon images for calibration' scan. Weather permitting. @smarshall-bmr wrote a scan for sun-up to sun-down passes over targets before he left. We've borrowed spectralon targets from Kelly Thorp (ALARC). @remotesensinglab please advise on protocol. If Solmaz is able to bring the spectrometer with her, should the targets be validated using the spectometer at a couple time points (perhaps 10am and 2pm)? @rjstrand has offered to help and has been working with the SWIR and VNIR. We want to ensure that we follow the proper protocol since we may don't have many available sun-up to sun-down windows. Thanks.

[remotesensinglab]

@NewcombMaria this is great, the planning on April 15.
I have given a list of targets to purchase to Nadia months ago. Having multiple targets are good (99%, 75%, 50%, 40%, and 2%), but 75% and/or 50% targets would be great if we have just one or two are available.

ALARC Tarps 68, 48, 8, and 4% may work but good to collect spectroscopy reflectance of these tarps as well just to confirm their reflectance. Yes, a 10am, 12PM, and 2Pm measurement of these tarps using the spectroscopy Solmaz has would be great.

Thanks

[dlebauer]

Everyone: very excited that there is a date for this, thanks for making it happen!

@remotesensinglab and @czender I'd like to confirm that we need to / plan to collect images of the spectralon targets at 15 minutes. I believe I saw a protocol written up somewhere but now am unable to locate said protocol.

[rjstrand]

@remotesensinglab - This is essentially the same set of targets that we used last fall, with the exception of replacing the 99% with yours. So, panels at 99, 75, 20%, Tarps at 68, 48, 8, 4%. I believe that some of these same tarps are documented in the attached paper published by my former USDA colleagues Moran, Clarke, and Qi.
2001_mar_273-286.pdf

[dlebauer]

@remotesensinglab The importance of a target near the max for vegetation (50-60%) has been repeatedly discussed. The paper that @rjstrand linked to shows that tarps are not substitutes for spectralon targets. Can you please confirm that the available targets are sufficient?

[czender]

Calibration targets should have reflectances in the range of plant reflectances. Targets that are too reflective (like the 99% Spectralon) provide information that is never used (unless you want to detect and produce reflectances for shiny metallic or specular surfaces that are also in the FOV). Pick calibration reflectances that bound the desired reflectances (so we get accurate brights and darks) and also have at least one calibration target near the mean- or median-expected reflectance. If we had only one target it would be optimal to have it there so that most of the pixels are well calibrated. With the luxury of multiple targets, choose them insofar as possible to be evenly spaced in reflectance between the maximum and minimum expected/desired reflectances (e.g., 10, 20, 30, 40, 50, 60%). The HS retrieval algorithm will do weighted interpolations between the nearest neighboring calibration values.

[czender]

Regarding temporal frequency of sampling, 15 minutes sounds good to me. It gives no worse than 360/(24*4) = 3.75 degree zenith angle resolution. It might be overkill if the zenith angle dependence is weak. But we've never measured it before so we don't know what to expect. There is no written protocol to my knowledge other than this, #208, and #289. And, to be clear, what I try to stress is that max(plant_reflectance)*exposure_time*max(counts_per_time) is ~2^16. Otherwise we waste bits and reduce measurement precision.

[NewcombMaria]

Thanks @remotesensinglab @czender and @dlebauer for helpful input. We discussed our plan this morning and still have a couple questions:

1. Is it better to adjust the height of the sensor box so that the panels are in sun for much of the day?
   During the winter season and still at this time, the sun angle allows for sunshine in the field of view. However during the summer sorghum season with the change in sun angle, the field of view is under the shade of the sensor box. We can alter sun/shade exposure by altering the height of the sensor box, but the optimal height for best focus at this date puts the field of view mostly in the sunshine.

2. What camera exposure setting is best for the diurnal scan? We can use the same exposure settings used most recently (April 7-8) when Stuart ran a hyperspectral scan, which was an exposure for sunshine. What's the best exposure setting?

We will have to go with the targets that we have at this time point, which for the middle ranges include spectralon panels at 20% and 75%, and also tarps at 68, 48, 8 and 4%. We'll put out all the targets that we have, and if new ones are obtained in the future the diurnal scan can be repeated. We'll be able to get 15 min temporal frequency.

[czender]

Sun/shade is undoubtedly an important issue, perhaps as or more important as cloud/clear. I listed Sun/shade after cloud/clear in the #281 completion criteria for mainly subjective reasons. Ideally we will obtain one set of calibration images for direct sun and another with targets shaded by the sensor box, in both clear and overcast skies (i.e., 4 calibration images for each zenith angle, for each target, although 3 might suffice since the sun/shade distinction is much smaller on overcast days). We intend to use a sun/shade pixel mask (developed by @remotesensinglab?) to determine on which pixels to use the sun and which the shaded calibration. Is it possible on Monday to get a clean set of direct sunlight exposures (all target in sunlight) and another "clean set" of sensor-shaded (all target in shade) exposures?

Re: exposure setting. The calibration exposures are most useful when they are the same duration as the regular imagery. I though Monday's calibration exposures would be made at a range of durations that covers all the actual durations of the Hyperspectral exposures, e.g., 10ms, 15ms, 20ms, 25ms, 30ms. When we process a regular exposure we want to use the calibration exposure for the same amount of time and not have to interpolate/extrapolate to different times.

[NewcombMaria]

Thanks @czender. Our plan is do run the full-day scan tomorrow (Saturday) because it may be the only day before planting on the 19th and sprinkler irrigation after planting when we have clear skies and no equipment or sprinklers through the field. If we run the scan from sun-up to sun-down, we need to select either sun or shade exposure. Or do you mean run a sun-scan in the morning and a shade-scan in the afternoon? I know we have an opportunity Saturday the 15th, and there will be another opportunity for a second full-day scan at some point, but I don't know when.

Regarding exposure, typically the exposure is set for the entire script at one setting. I'm not sure what is meant by the range of durations of the hyperspectral exposures. Perhaps @rjstrand understands.

[NewcombMaria]

Suzette and Kelly Thorp (USDA ALARC) provided the following information about the 20% and 75% panels. Both are from LabSphere. There's a calibration certificate for the 75% panel and an ID tag for the 20% panel. Both are attached here, along with the spectralon target specs from the website.
spectralon_targets (from website).pdf

75_Panel Certificate.pdf

[dlebauer]

I've added data for the 75% target to the 'Target Calibration Datasets' folder on google drive. I noticed that the color target spectral reflectance response curves are also available in there, but those are only calibrated to 830nm

I also opened terraref/reference-data#132 to address compiling the calibration data and making it available.

[czender]

This is a good start. The 75% target data looks usable.

[dlebauer]

FYI I parsed the calibration table to CSV https://drive.google.com/open?id=0By_PDCcY5g2JWlU4R1VzMHFhejg It is in the target calibration folder on Google drive alongside the measurements Solmaz made Label here doesn't have calibration files for the 20% target. They suggested they might not be valid and could be recalibrated.

…

On Thu, May 25, 2017 at 10:41 AM Charlie Zender ***@***.***> wrote: This is a good start. The 75% target data looks usable. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#281 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcX56z4Ty3K0ZLKXxYmvfsbluTGKKfoks5r9aEegaJpZM4MXbGM> .

[solmazhajmohammadi]

@czender I have not heard from you regarding the repetitive pattern in the target positions in the calibration files. I am tagging @FlyingWithJerome also, if he is still working on this.
Please see the plotted output from the multiple scans below. The file size is changing, however, if you set the origin to the 20% target, the targets seems to be staying at the same position:

My first guess is that the file size and pattern should be same in the datasets with the same exposure time(or at least close enough that if you choose a ROI in the center of targets, we should be safe). Please try this, let me know how it goes.

[czender]

@hmb1 and @FlyingWithJerome can you please find the largest possible hyperslab for each target that remains on the target for the entire day (20170415)? It looks like it may not be possible for the 75% target which is relatively small, but might be possible for the 20%, 48%, and 95% targets. And keep @solmazhajmohammadi in the loop.

[hmb1]

@FlyingWithJerome, @czender is there any way I can view nc/image files on roger without downloading to my local machine as some of the files are huge ?

[czender]

I don't know of methods than ncview and panoply. Both should be able to display remotely from roger if your X DISPLAY environment is suitable. Maybe @ZongyangLi or @yanliu-chn know a better way.

[ghost]

@czender - Do you think this can be completed by then end of July? I want to make sure that it goes into the V1 release.

[czender]

I doubt it will be complete by the end of July.

[yanliu-chn]

@czender I tried to install ncview on ROGER before, but it requires system upgrade to install dependent X11 libraries. So I gave up.

If this is a need from TerraRef, I will ask ROGER admin to take care of the system upgrade. Let me know.

[dlebauer]

@yanliu-chn Some method of viewing large netcdf files is required. Ncview and Panoply are two options - but if we can deploy a THREDDS server ( @robkooper is working on #155, what is ETA) then Panoply and a few other options will be available.

[yanliu-chn]

ok. I will follow up with ROGER admin to get ncview installed.

[yanliu-chn]

ncview 2.1.7 has been installed. ROGER admin found a time to install a dependent library libXaw-devel on ROGER.

To use, please do the following:
On your computer:
ssh -Y youraccount@roger-login.ncsa.illinois.edu

On ROGER:
module purge; module load gdal2-stack udunits ncview
ncview /path/to/your/netcdffile.nc

On mac, you need XQuartz to set up X b/w mac and remote X on ROGER; on Windows, you need an ssh client with X support.

[czender]

This is great. Works for me. Thanks much @yanliu-chn

[hmb1]

@yanliu-chn
I tried to run ncview on the login node on ROGER and got the following:
should i try "startx" with my ROGER login ?

[butowskh@cg-cmp14 rgr]$ ncview i1.nc & [1] 42728 [butowskh@cg-cmp14 rgr]$ Ncview 2.1.7 David W. Pierce 29 March 2016 http://meteora.ucsd.edu:80/~pierce/ncview_home_page.html Copyright (C) 1993 through 2015, David W. Pierce Ncview comes with ABSOLUTELY NO WARRANTY; for details type ncview -w'.
This is free software licensed under the Gnu General Public License version 3; type `ncview -c' for redistribution details.

Note: could not open file /home/butowskh/.ncviewrc for reading
Error: Can't open display: '

[hmb1]

OK Yan,
got it working -great ...Thanks

[hmb1]

@smarshall-bmr
@solmazhajmohammadi

Have the following observations about the VNIR calibration data.
Comments below are by Solmaz.

1. Exposure time cycles as 25ms,30ms,35ms,40ms,44ms
   ( except for The First two series where exposure starts at 20ms)

2. all exposures have samples==1600 (thats the X dimension)
   exposures at 40ms have lines ( 17 to 48 ) (thats Y dimension)
   If there is a data with 17 to 48 line scan, please create a github issue and assign it to @smarshall-bmr He should be able to fix the script. This seems odd to me, since the script was iterating over all the iterations. No matter how much is the exposure time, you should at least collect more than 1000 lines which is about 1 meter.

   all other exposures have lines ( 1463 to 1818 )

3. The 'start-time' is "later" than the "time-stamp" in the directory name
   Which time should I use ?
   Start-time and time-stamp should be close enough anyway. It is a different between two clock signal, and it is different because they are not exactly synchronized. time-stamp is based on the sensor's clock signal and start-time is based on gantry server's clock signal. I would say use the gantry clock signal, which would be easier in future to map it with other clocks in the system.

4. Missing exposure times 40ms, 50ms, 55ms, 65ms in the calibration
   Technically in the sensor exposure time should be 5ms less than frame period otherwise, the camera might start dropping the frames. Frameperiod is set to 50ms. I have not personally checked to make sure that the exposure times over 45ms is valid or not. Again you can create a github issue and ask @smarshall-bmr to make sure that there is no frame drop when the exposure is more than 45ms. However, there is not much data collected with exposure time over 45ms.

[hmb1]

@smarshall-bmr
@solmazhajmohammadi

I have averaged the data over the targets and created the variable:
ushort xps_img_wht(time, exposure, target, wavelength)

In the afternoon 14.07 to 17.14 the reading seem too high - regardless of the target (20%,48%, 95%,99%) or exposure time ( 20ms,30ms,35ms,45ms ) if somebody else could please take a look
that would be grand.

I have checked and rechecked my code and it seems OK .
The file is on ROGER at : /tmp/xps_img_wht_tm_exp_trg.nc

[hmb1]

@smarshall-bmr
@solmazhajmohammadi
this is a plot of max wavelength count through the calibration days.
target 48% Exposure 25ms
X-Axis is Hour:min
Y-Axis counts

[solmazhajmohammadi]

@hmb1
Hi Henry,
I am trying to understand your question... you are asking why the average of digital raw counts, in a single spectralon target is changing?
So different time in a day, different sun angle and we want to see this change... digital counts should be max around solar noon. However, this might not be true in our case, since we have the effect of gantry shading the targets.
If the change is so dramatic, I would assume that you are not looking to make sure that all the target is in shade or sun. Targets might be partially in the shade and partially in the sun.

[hmb1]

@dlebauer
@solmazhajmohammadi
@czender
@smarshall-bmr

From what I can tell with the 2017-04-15 VNIR calibration data we have

1. 6:11 - 13.39 targets in the shade of the Gantry
2. 14:07 - 16.21 targets in direct sunlight
   ( can this be confirmed ? )

This means we have sufficient zenith angles to calibrate for 1) but NOT for 2)

From the above comments I understand, theres a 1 hour window when the VNIR gets out of the sensor box shadow and the SWIR does the same ( can this be confirmed ? )

...Henry