Potential issue with METv11.0.1 build on Hera #2090

michelleharrold · 2023-03-16T16:08:35Z

michelleharrold
Mar 16, 2023
Collaborator

Good Morning!

Several DTC and NCAR tasks are doing echo top verification via the UFS SRW App on Hera (NOAA). Recently, we have been having issues with our grid-stat output -- stat files are empty or the values are totally wonky. After some exploring and using plot_data_plane, it looks like MET (on Hera) is having issues with correctly reading the MRMS observation file.

I am loading MET on Hera like this:

module use -a /contrib/anaconda/modulefiles
module load intel/2022.1.2
module load anaconda/latest
module use -a /contrib/met/modulefiles/
module load met/11.0.1

Here is the command I am executing on Hera to plot a sample MRMS observation files:

plot_data_plane EchoTop_18_00.50_20210525-000000.grib2 et.ps 'name="EchoTop18"; level="Z500";'

Here is the resulting print-to-screen info:

DEBUG 1: Start plot_data_plane by Michelle.Harrold(10315) at 2023-03-16 15:50:36Z  cmd: plot_data_plane EchoTop_18_00.50_20210525-000000.grib2 et.ps name="EchoTop18"; level="Z500"; 
DEBUG 1: Opening data file: EchoTop_18_00.50_20210525-000000.grib2
libpng warning: Application built with libpng-1.5.13 but running with 1.6.34
DEBUG 1: Creating postscript file: et.ps
DEBUG 1: Finish plot_data_plane by Michelle.Harrold(10315) at 2023-03-16 15:50:38Z

Note, the libpng warning!

Here is the resulting plot:

When I do a wgrib2 -V on the obs file, I see:

wgrib2 -V EchoTop_18_00.50_20210525-000000.grib2
1:0:vt=2021052500:500 m above mean sea level:anl:EchoTop18 Echo Top - 18 dBZ [km MSL]:
    ndata=24500000:undef=0:mean=-1.65701:min=-3:max=19

Where min/max is -3 and 19, respectively...so it appears something is happening when MET is reading the file, maybe due to the libpng warrning?

To confirm, I ran the same plot_data_plane command on Cheyenne (METv11.0.1), and the result plot looks like this:

It seems like there might be an issue with the Hera METv11.0.1 install?

Thanks for any help you can provide!
Michelle Harrold

Answered by JohnHalleyGotway

Mar 16, 2023

Here's what I've found:

Confirm the problem

Running plot_data_plane at -v 4 illustrates the problem via log messages:

module use -a /contrib/anaconda/modulefiles
module load intel/2022.1.2
module load anaconda/latest
module use -a /contrib/met/modulefiles/
module load met/11.0.1

/contrib/met/11.0.1/bin/plot_data_plane \
/scratch2/BMC/fv3lam/ens_design_RRFS/obs_data/mrms/proc/20210525/EchoTop_18_00.50_20210525-000000.grib2 \
plot.ps  'name="EchoTop18"; level="Z500";' -v 4

On my laptop I get the correct range of values (-3, 19):

DEBUG 4: Data plane information:
DEBUG 4:    plane min: -3
DEBUG 4:    plane max: 19

On hera with 11.0.0, it's correct:

DEBUG 4: Data plane information:
DEBUG 4:…

View full answer

jprestop · 2023-03-16T16:39:10Z

jprestop
Mar 16, 2023
Maintainer

Hi @michelleharrold. Thank you for all of this information and for comparing with Cheyenne. Can you please tell us the location of your file EchoTop_18_00.50_20210525-000000.grib2 on Hera?

0 replies

jprestop · 2023-03-16T17:01:29Z

jprestop
Mar 16, 2023
Maintainer

Hi @michelleharrold.

When I look at the what version of libpng plot_data_plane links with, it seems to be using the correct and desired version:

-bash-4.2$ ldd plot_data_plane  | grep libpng
	libpng16.so.16 => /contrib/met/11.0.1/external_libs//lib/libpng16.so.16 (0x00007fd69d5bb000)

which is created from version libpng-1.6.34.

However, I went ahead and rebuilt MET with a slight change to the compilation script, although I'm not sure if that will solve this problem or not. I couldn't find your file, but I ran plot_data_plane with a different file:

-bash-4.2$ plot_data_plane /scratch1/BMC/dtc/METplus/METplus-5.0_sample_data/model_applications/medium_range/reduced_model_data/20141215/gfs_4_20141215_0000_036.grb2 TMP.ps 'name="TMP"; level="Z2";'
DEBUG 1: Start plot_data_plane by Julie.Prestopnik(10845) at 2023-03-16 16:55:38Z  cmd: plot_data_plane /scratch1/BMC/dtc/METplus/METplus-5.0_sample_data/model_applications/medium_range/reduced_model_data/20141215/gfs_4_20141215_0000_036.grb2 TMP.ps name="TMP"; level="Z2"; 
DEBUG 1: Opening data file: /scratch1/BMC/dtc/METplus/METplus-5.0_sample_data/model_applications/medium_range/reduced_model_data/20141215/gfs_4_20141215_0000_036.grb2
DEBUG 1: Creating postscript file: TMP.ps
DEBUG 1: Finish plot_data_plane by Julie.Prestopnik(10845) at 2023-03-16 16:55:38Z

and did not receive the same warning that you did. Could you please get a new shell for a clean environment, load the following, and rerun:

module use -a /contrib/anaconda/modulefiles
module load intel/2022.1.2
module load anaconda/latest
module use -a /contrib/met/modulefiles/
module load met/11.0.1
module use /contrib/METplus/modulefiles
module load metplus/5.0.1

and see if the file creation is now as expected? Please let us know how it goes.

0 replies

jprestop · 2023-03-16T17:22:15Z

jprestop
Mar 16, 2023
Maintainer

Hi @michelleharrold. Thanks for passing along the location of your file. It's strange. When I run the command with your file, I DO get the warning:

-bash-4.2$ plot_data_plane /scratch2/BMC/fv3lam/ens_design_RRFS/obs_data/mrms/proc/20210525/EchoTop_18_00.50_20210525-000000.grib2 et.ps 'name="EchoTop18"; level="Z500";'
DEBUG 1: Start plot_data_plane by Julie.Prestopnik(10845) at 2023-03-16 17:20:09Z  cmd: plot_data_plane /scratch2/BMC/fv3lam/ens_design_RRFS/obs_data/mrms/proc/20210525/EchoTop_18_00.50_20210525-000000.grib2 et.ps name="EchoTop18"; level="Z500"; 
DEBUG 1: Opening data file: /scratch2/BMC/fv3lam/ens_design_RRFS/obs_data/mrms/proc/20210525/EchoTop_18_00.50_20210525-000000.grib2
libpng warning: Application built with libpng-1.5.13 but running with 1.6.34
DEBUG 1: Creating postscript file: et.ps
DEBUG 1: Finish plot_data_plane by Julie.Prestopnik(10845) at 2023-03-16 17:20:12Z

This is quite confusing. Unfortunately, I am about to go on vacation through 3/27, so I need to ask @JohnHalleyGotway or @georgemccabe to take a look and see if they can figure out what's going on.

1 reply

michelleharrold Mar 16, 2023
Collaborator Author

@jprestop -- thank you so much for taking an initial look at this! I really appreciate it.

JohnHalleyGotway · 2023-03-16T18:09:19Z

JohnHalleyGotway
Mar 16, 2023
Maintainer

Here's what I've found:

Confirm the problem

Running plot_data_plane at -v 4 illustrates the problem via log messages:

module use -a /contrib/anaconda/modulefiles
module load intel/2022.1.2
module load anaconda/latest
module use -a /contrib/met/modulefiles/
module load met/11.0.1

/contrib/met/11.0.1/bin/plot_data_plane \
/scratch2/BMC/fv3lam/ens_design_RRFS/obs_data/mrms/proc/20210525/EchoTop_18_00.50_20210525-000000.grib2 \
plot.ps  'name="EchoTop18"; level="Z500";' -v 4

On my laptop I get the correct range of values (-3, 19):

DEBUG 4: Data plane information:
DEBUG 4:    plane min: -3
DEBUG 4:    plane max: 19

On hera with 11.0.0, it's correct:

DEBUG 4: Data plane information:
DEBUG 4:    plane min: -3
DEBUG 4:    plane max: 19

On hera with both 11.0.1 and 11.1.0-beta1 I get an INCORRECT range of values (-3, -3):

DEBUG 4: Data plane information:
DEBUG 4:    plane min: -3
DEBUG 4:    plane max: -3

So I agree that this is a problem with the installation of 11.0.1 and 11.1.0-beta1, and is almost certainly related to the libpng warning message you see.

Comparing the installation of 11.0.0 to 11.0.1, I do see some small environment differences. I'll keep digging and let you know what I find.

The most likely explanation are differences in the compile_MET_all.sh script used between versions 11.0.0 and 11.0.1. The structure of the external libraries look unchanged. So I'll keep looking at that script.

Unrelated but worth noting

Running wgrib2 on my laptop prints a warning... but this is likely unrelated:

wgrib2 EchoTop_18_00.50_20210525-000000.grib2 
** Warning: reference time includes non-zero minutes/seconds **
1:0:d=2021052500:EchoTop18:500 m above mean sea level:anl:

There's an extra 4 seconds in the reference time for some reason:

wgrib2 -T EchoTop_18_00.50_20210525-000000.grib2 
1:0:D=20210525000004

3 replies

JohnHalleyGotway Mar 17, 2023
Maintainer

Check MET's linker map

I tested further by copying 11.0.1 into /contrib/met/11.0.1-libpng and compiling with

export CXXFLAGS="-Xlinker --print-map"

And all references to the libpng library point to the right one:

grep 'libpng\.' /contrib/met/11.0.1-libpng/MET-11.0.1/make.log | sort -u
/contrib/met/11.0.1-libpng/external_libs/lib/libpng.so

And that points to -> libpng16.so which points to -> libpng16.so.16.34.0.
So MET is compiled the way we want, and that's a dead end.

Compile 11.0.1 using the 11.0.0 libraries

Recompile using:

export EXTERNAL_LIBS=/contrib/met/11.0.0/external_libs

And that FIXES the problem:

DEBUG 4: Data plane information:
DEBUG 4:       plane min: -3
DEBUG 4:       plane max: 19

So the problem is in how the external libs are configured/compiled and NOT in the configuration/compilation of MET itself.

Problem in compiling the GRIB2C library

I found the problem by comparing these log files:

/contrib/met/11.0.0/external_libs/g2clib/g2clib-1.6.0/make.log includes

-I/contrib/met/11.0.0/external_libs/include -I/contrib/met/11.0.0/external_libs/include/jasper

/contrib/met/11.0.0/external_libs/g2clib/g2clib-1.6.0/make.log DOES NOT INCLUDE THAT! And that's why 11.0.1 links to the wrong library.

JohnHalleyGotway Mar 17, 2023
Maintainer

@michelleharrold MET version 11.0.1 should be working fine again on Hera. I re-ran the following commands to confirm:

module use -a /contrib/anaconda/modulefiles
module load intel/2022.1.2
module load anaconda/latest
module use -a /contrib/met/modulefiles/
module load met/11.0.1

/contrib/met/11.0.1/bin/plot_data_plane \
/scratch2/BMC/fv3lam/ens_design_RRFS/obs_data/mrms/proc/20210525/EchoTop_18_00.50_20210525-000000.grib2 \
plot.ps  'name="EchoTop18"; level="Z500";' -v 4

Now reports:

DEBUG 4: Data plane information:
DEBUG 4:       plane min: -3
DEBUG 4:       plane max: 19

The problem was a wrinkle in the compilation script that's been ironed out.

Once you test and confirm that it also works fine for you, please make this discussion as being answered.

And THANK YOU very much for letting us know about this issue!

michelleharrold Mar 17, 2023
Collaborator Author

@JohnHalleyGotway (and @jprestop) -- Thank you SO much for tracking this down and finding a solution! I was able to make a beautiful, correct-looking plot of observed radar echo top.

Also, thank you for pointing out that there is an extra 4 seconds in the reference time...I am looking into that now.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Potential issue with METv11.0.1 build on Hera #2090

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Potential issue with METv11.0.1 build on Hera #2090

michelleharrold Mar 16, 2023 Collaborator

Confirm the problem

Replies: 4 comments · 4 replies

jprestop Mar 16, 2023 Maintainer

jprestop Mar 16, 2023 Maintainer

jprestop Mar 16, 2023 Maintainer

michelleharrold Mar 16, 2023 Collaborator Author

JohnHalleyGotway Mar 16, 2023 Maintainer

Confirm the problem

Unrelated but worth noting

JohnHalleyGotway Mar 17, 2023 Maintainer

Check MET's linker map

Compile 11.0.1 using the 11.0.0 libraries

Problem in compiling the GRIB2C library

JohnHalleyGotway Mar 17, 2023 Maintainer

michelleharrold Mar 17, 2023 Collaborator Author

michelleharrold
Mar 16, 2023
Collaborator

Replies: 4 comments 4 replies

jprestop
Mar 16, 2023
Maintainer

jprestop
Mar 16, 2023
Maintainer

jprestop
Mar 16, 2023
Maintainer

michelleharrold Mar 16, 2023
Collaborator Author

JohnHalleyGotway
Mar 16, 2023
Maintainer

JohnHalleyGotway Mar 17, 2023
Maintainer

JohnHalleyGotway Mar 17, 2023
Maintainer

michelleharrold Mar 17, 2023
Collaborator Author