Problems with some of the S2S use cases on Hera

[LydiaStefanova-NOAA]

Hi,

I am trying to test (with no modifications) the thirteen s2s use cases on Hera with METplus-4.1.1, loading the modules as in https://dtcenter.org/community-code/metplus/metplus-4-1-existing-builds. I have success for four of the s2s cases, and failure for the other nine (eight of these failures are of one type, the ninth is of a different type). The details of what I am doing are below:

for each use case in pathtocase=/contrib/METplus/METplus-4.1.1/parm/use_cases/model_applications/s2s/ , I run
run_metplus.py -c $pathtocase/$case -c hera.lstefanova.conf

where hera.lstefanova.conf is the same as /scratch1/BMC/dtc/Julie.Prestopnik/METplus/hera.jpresto.conf , with only OUTPUT_BASE changed to my path

When trying this, I get:

a) 'METplus has successfully finished running' for TCGen_fcstGFSO_obsBDECKS_GDF_TDF.conf, UserScript_obsERA_obsOnly_Stratosphere.conf, UserScript_obsPrecip_obsOnly_CrossSpectraPlot.conf, GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf

b) for UserScript_fcstGFS_obsERA_Blocking.conf, UserScript_fcstGFS_obsERA_OMI.conf, UserScript_fcstGFS_obsERA_WeatherRegime.conf, UserScript_obsERA_obsOnly_Blocking.conf, UserScript_obsERA_obsOnly_OMI.conf, UserScript_obsERA_obsOnly_RMM.conf, UserScript_obsERA_obsOnly_WeatherRegime.conf, UserScript_obsPrecip_obsOnly_Hovmoeller.conf, an error that seems related to a missing version of a library component of the type :

"ImportError: /scratch1/BMC/dtc/miniconda/miniconda3/lib/python3.9/site-packages/pandas/_libs/window/../../../../../libstdc++.so.6: version GLIBCXX_3.4.30' not found (required by /scratch1/BMC/dtc/miniconda/miniconda3/lib/python3.9/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-39-x86_64-linux-gnu.so)" or "/scratch1/BMC/dtc/miniconda/miniconda3/lib/python3.9/site-packages/scipy/spatial/../../../../libstdc++.so.6: version GLIBCXX_3.4.30' not found (required by /scratch1/BMC/dtc/miniconda/miniconda3/lib/python3.9/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-39-x86_64-linux-gnu.so)" or
"ImportError: /scratch1/BMC/dtc/miniconda/miniconda3/lib/python3.9/site-packages/PIL/../../../././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /scratch1/BMC/dtc/miniconda/miniconda3/lib/python3.9/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-39-x86_64-linux-gnu.so)"

c) For UserScript_obsERA_obsOnly_PhaseDiagram.conf, errors that seems related to a missing file scratch1/BMC/dtc/Julie.Prestopnik/METplus/METplus-4.1_sample_data/model_applications/s2s/UserScript_obsERA_obsOnly_PhaseDiagram/time_list_lead000.txt:

"ERROR: (command_builder.py:1362) MET command returned a non-zero return code:/contrib/METplus/METplus-4.1.1/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_PhaseDiagram/save_input_files_txt.py 20120331-000000 /scratch1/BMC/dtc/Julie.Prestopnik/METplus/METplus-4.1_sample_data/model_applications/s2s/UserScript_obsERA_obsOnly_PhaseDiagram/time_list_lead000.txt"

Can you tell if I am doing something wrong in setting up or in running, and is there any additional information I can provide (e.g. logs or environment) that can help answer this?

Thanks in advance!
Lydia

[georgemccabe]

Hi @LydiaStefanova-NOAA,

There was an issue with one of the Python packages in the Conda environment used for METplotpy/METcalcpy/METdatadb that was responsible for the error you are seeing for (b). I just applied a fix to that environment, so those use cases should work properly now. I tested using the UserScript_obsERA_obsOnly_RMM.conf use case, but I have not run the rest that failed. Please let me know if you run into any other errors with those cases and I can take a look again.

For (c), there is a bug/mistake in that use case that is causing the error you are seeing. I am also seeing that error when I run locally. The use case writes output files into {INPUT_BASE}, which is not writable by users by design to avoid corrupting the input data. All use cases should write output under {OUTPUT_BASE}. I created GitHub issue #1706 to fix this use case.

[LydiaStefanova-NOAA]

Thanks @georgemccabe,

I re-tested after the environment fix you made, and now three of the (b) cases run successfully:
UserScript_fcstGFS_obsERA_OMI.conf
UserScript_obsERA_obsOnly_OMI.conf
UserScript_obsERA_obsOnly_RMM.conf

I now get a "ModuleNotFoundError: No module named 'statsmodels'" for these:
UserScript_fcstGFS_obsERA_Blocking.conf
UserScript_obsERA_obsOnly_Blocking.conf
UserScript_obsPrecip_obsOnly_Hovmoeller.conf

And a "ModuleNotFoundError: No module named 'sklearn'" for these:
UserScript_fcstGFS_obsERA_WeatherRegime.conf
UserScript_obsERA_obsOnly_WeatherRegime.conf

[georgemccabe]

Hi @LydiaStefanova-NOAA, I just installed the missing packages on Hera. I confirmed that the Blocking use cases run successfully.

The WeatherRegime use cases require a bugfix to METcalcpy. The change has been implemented, but the bugfix release has not yet been created and installed on Hera. This is in progress. I can update you when the bugfix release is in place.

The Hovmoeller use case was failing but I was able to fix that as well.

Let me know if everything else is working as expected and I will reply here when the fix required for the WeatherRegime use cases is in place. Thank you for bringing this to our attention!

[georgemccabe]

Hi @LydiaStefanova-NOAA, I was able to install the METcalcpy 1.1.1 bugfix release on Hera and successfully run the WeatherRegime use cases. To load this version, you will have to update the module load command to load this version instead of 1.1.0:

module use /contrib/METcalcpy/modulefiles
module load metcalcpy/1.1.1

All of the s2s use cases should run successfully now, but please let me know if you are still unable to run any of them.

[LydiaStefanova-NOAA]

@georgemccabe - thank you very much!

I checked and confirmed that I can now run all s2s use cases out the box, except the PhaseDiagram one (the one that attempts to write into a subdirectory of {INPUT_BASE}). It doesn't look like the version installed on Hera has been updated with #1709 (don't know if it's meant to be) - but if I copy the .conf file and apply these changes (and manually create the output directory), the case runs successfully.

Should I mark your latest response as "the answer"? (I'm new here, not sure of the proper procedure)

[georgemccabe]

Hi @LydiaStefanova-NOAA, I'm glad to hear the use cases are running successfully. We cut the METplus 4.1.3 bugfix release yesterday which includes the fix to the PhaseDiagram use case. The fix also includes a patch so the output directory will be created automatically if it does not already exist. I can install the release on Hera later today.

You can mark this response as the answer if you don't need any more assistance with this discussion. Once a discussion topic has been marked with an answer, we lock the conversation to prevent future posts so follow-up questions don't get lost.

[LydiaStefanova-NOAA]

Thanks, @georgemccabe, for this explanation.
I have a (2-part) followup question, but if you think this is turning into topic drift, we can close this discussion, and I can start a separate discussion for it:
a) When you install the METplus bugfix release later today, will it contain the changes referenced in https://github.com/dtcenter/METplotpy/issues?q=213 ? Without it, it doesn't seem that I can't run the Hovmoeller use case with personalized hovmoeller.yaml specifications for the plot details (like the problem that was described in #1645).
b) Once you install the bugfix release on Hera, will the setup instructions at https://dtcenter.org/community-code/metplus/metplus-4-1-existing-builds be updated to point to it (or will it the setup stay the same)?

[georgemccabe]

The METplus 4.1.3 bugfix release only includes the METplus wrappers, not METplotpy or the other METplus components.
The changes from the METplotpy issue you linked are available in the METplotpy 2.0.0-beta1 release, which is likely not compatible with the METplus 4.1.3 release. We are, however, planning on creating the METplus 5.0.0-beta2 release next week on August 3. This release will be installed on Hera and will be compatible with the changes to the Hovmoeller plotting in METplotpy.

Yes, we will update the existing builds page to include instructions to install the latest bugfix release.

[LydiaStefanova-NOAA]

Thank you @georgemccabe, I really appreciate your help and explanations! I now have the answers to all the questions I had for this discussion.

[georgemccabe]

Not a problem! METplus 4.1.3 is now available on Hera and the instructions on the existing builds page has been updated.

module load intel/2022.1.2
module use /contrib/METplus/modulefiles
module load metplus/4.1.3