Fails with a buffer underflow when attempting to "use" a particular URL.

[noaaroland]

This URL:

https://mynasadata.larc.nasa.gov/thredds/dodsC/EarthSystemLAS/atmos_rad_net/data_usr_local_fer_dsets_data_CERES_EBAF-TOA_Ed4.1_Subset_200003-202105_Net.nc_Count_Globe_init_toa_net_all_mon.jnl_expr_%7B%7D%7Bletdeq1+toa_net_all_mon_1_transformed%3Dtoa_net_all_mon%5Bd%3D1%2Cx%3D-245.390625%3A114.609375%40Average%2Cy%3D-24.609375%3A155.390625%40Average%5D_cr_ATTRCMD+toa_net_all_mon+toa_net_all_mon_1_transformed%7D

can be accessed using ncdump -c, but when accessed using:

use "https://mynasadata.larc.nasa.gov/thredds/dodsC/EarthSystemLAS/atmos_rad_net/data_usr_local_fer_dsets_data_CERES_EBAF-TOA_Ed4.1_Subset_200003-202105_Net.nc_Count_Globe_init_toa_net_all_mon.jnl_expr_%7B%7D%7Bletdeq1+toa_net_all_mon_1_transformed%3Dtoa_net_all_mon%5Bd%3D1%2Cx%3D-245.390625%3A114.609375%40Average%2Cy%3D-24.609375%3A155.390625%40Average%5D_cr_ATTRCMD+toa_net_all_mon+toa_net_all_mon_1_transformed%7D"

in the latest DEV release of PyFerret it fails with a buffer underflow.

[ACManke]

I don't have much to contribute here, but one thing to look at here is netCDF versions. In my case, PyFerret is linked with v4.7.4,

yes? show symbol netcdf_version
NETCDF_VERSION = "4.7.4 of Oct 21 2020 19:55:48 $"

and my installation of ncdump indicates the same version. In my case, both PyFerret and ncdump fail with "NetCDF: DAP server error"

[seasage]

I got this same error from PyFerret with netCDF 4.74. I upgraded to 4.8.1 and I got:

yes? use "https://mynasadata.larc.nasa.gov/thredds/dodsC/EarthSystemLAS/atmos_rad_net/data_usr_local_fer_dsets_data_CERES_EBAF-TOA_Ed4.1_Subset_200003-202105_Net.nc_Count_Globe_init_toa_net_all_mon.jnl_expr_%7B%7D%7Bletdeq1+toa_net_all_mon_1_transformed%3Dtoa_net_all_mon%5Bd%3D1%2Cx%3D-245.390625%3A114.609375%40Average%2Cy%3D-24.609375%3A155.390625%40Average%5D_cr_ATTRCMD+toa_net_all_mon+toa_net_all_mon_1_transformed%7D"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/osborne_local/PyFerret/lib/python3.7/site-packages/pyferret/__init__.py", line 489, in init
    result = run()
  File "/Users/osborne_local/PyFerret/lib/python3.7/site-packages/pyferret/__init__.py", line 905, in run
    retval = libpyferret._run(str_command)
RuntimeError: 

**ERROR Ferret crash; signal = 4 (SIGILL)
Enter Ctrl-D to exit Python

@ACManke any ideas?

[ACManke]

No, I don't really have any ideas, but here's a little bit of exploration.

With the ncdump from netCDF 4.7.4 that I have installed I am getting a "NetCDF: DAP Server error" when I try the URL with "ncdump -c"

De-encripting the URL,, and using the fact that in F-TDS

LETDEQ1 means "LET/D=1" and
ATTRCMD is an alias for SET ATT/LIKE=

this is equivalent to these commands:

use "https://mynasadata.larc.nasa.gov/thredds/dodsC/EarthSystemLAS/atmos_rad_net/data_usr_local_fer_dsets_data_CERES_EBAF-TOA_Ed4.1_Subset_200003-202105_Net.nc_Count_Globe_init_toa_net_clr_c_mon.jnl"
let/d=1 toa_net_all_mon_1_transformed=toa_net_all_mon[d=1,x=-245.390625:114.609375@Average,y=-24.609375:155.390625@Average]
set attributes/like=toa_net_all_mon toa_net_all_mon_1_transformed

These commands do works for me with the PyFerret I have on my system, V7.65 built from the current GitHub PyFerret code base, and linked with netCDF 4.7.4. I can go on and make a plot

plot/L=1:100 toa_net_all_mon_1_transformed

Now, the F-TDS url is going to execute and load the transformed variable on the server at mynasadata, right, so doesn't this all mean that it's something at that end which may be returning something that doesn't work?

[noaaroland]

For what it's worth:
NETCDF_VERSION = "4.6.3 of Mar 21 2019 12:26:12 $"
works with this data set when loading it directly from mynasadata.gov. At least at version 4.6.3 of the client library it's sending the right stuff.

[ACManke]

It sounds like it might be the right thing to do, to go back to that previous version of netCDF for the PyFerret builds. I don't believe that in the last 3 years at least, that there was ever a reason we linked with a version of NetCDF because of either features we needed that were newly available or bug fixes in a particular netCDF version (or bugs in a release that we were avoiding by staying away from a particular release.)

Release notes for NetCDF releases should be checked to see what they included. Also look for any mention of netCDF in Ferret and PyFerret GitHub issues. Also I don't know if there may be comments in the recipe for updating the PyFerret build or Anaconda process. Karl handled these things most recently and likely was following good general practices of keeping up with library releases. For myself I don't have a recommendation either way.

I don't know how this behavior would be reported to Unidata -- creating a self-contained example that can be seen using ncdump/ ncgen is how I made bug reports in earlier years

[noaaroland]

This build of PyFerret works with this data source. It is essentially the latest PyFerret code linked with a slightly older version fo the netCDF library (4.6.3) which does not have the bug reported above.

To use this version of PyFerret, you can build an python environment which is compatible using the attached conda environment file. First build the environment:

conda env create -f envi_pyferret.yml.txt

You can change the name of the environment by editing the yml.txt file. The name in there now is pyferret. Then activate the resulting environment:

conda activate pyferret

Now you can expand the tar file and descend into the resulting directory to set up PyFerret to use the local version you just untarred.

cd pyferret-latest-local/
./bin/Finstall

The answer to the FER_DIR will be something like: /home/schweitzer/pyferret-latest-local depending on where you expanded the tar.

The answer to the datasets directory will be something like: /home/schweitzer/miniconda3/envs/pyferret/share/fer_dsets

And where to put the ferret_paths is up to you. I put mine in /home/schweitzer/etc/pyferret for this exmaple.

Once that is all done you should be able to plot the URL in initial report.

To configure this into an LAS 8 instance, use the values of the section of you FerretBackendConfig.xml file. E.g.

        <variable>
            <name>FER_DIR</name>
             <value>/home/schweitzer/pyferret-latest-local</value>
        </variable>

PyFerret-7.6.5-RHEL7-Python3.6-NetCDF4.6.3.tar.gz
envi_pyferret.yml.txt