Running ATM_INIT: "Variable not found: get_variable_num_dimension: file:INPUT/sfc_data.nc variable: tref" error

[muradme]

Hi all,

I am running HAFS_rt_regional_atm test based on parm/rt_conf/hafs_regional_atm.conf and encountering an issue while running HAFS atm_init, where the process fails with the following error message:

And these are GFS input files that I am using:

I have run the previous steps several times to check upto atm_ic, which successfully completed. This is the screenshot of hafs-rt-status:

I have attached the hafs_atm_init.log file for more detailed logs related to this error. I would greatly appreciate any insights or suggestions on how to resolve this issue. Thank you in advance for your help!

Best regards,
Murad

atm_init log file:
hafs_atm_init.log

[muradme]

Hi everyone,
I wanted to follow up on the discussion I started. I haven’t received a response yet and was wondering if there is any suggestions or further insights available on this topic.

Any updates or additional information would be greatly appreciated.

Best regards,
Murad

[ZhanZhang-NOAA]

Hi Murad, It's difficult to debug without accessing the NCAR Derecho Cluster. I do see the following error message in your atm_init log file: Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "mkl" Try: "module spider mkl" to see how to load the module(s). Regards

…

-Zhan

On Fri, Aug 16, 2024 at 10:08 AM Murad Khondaker ***@***.***> wrote: Hi everyone, I wanted to follow up on the discussion I started. I haven’t received a response yet and was wondering if there is any suggestions or further insights available on this topic. Any updates or additional information would be greatly appreciated. Best regards, Murad — Reply to this email directly, view it on GitHub <#283 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE73MQQLJIABCZXHZEU3XR3ZRYBVVAVCNFSM6AAAAABMMSF6PKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZVHE3DCMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[gspetro-NOAA]

It looks like there is also an MPI/MPICH error in lines 86 and 10194 and several other lines. If there's something wrong with loading the MPI, that could cause a host of problems.

[muradme]

Thanks a lot for both of your inputs. I have checked and found that MPICH is not loaded properly. I am working on it fix these library issues and checking if the issue still persists. These suggestions really helped me a lot.

[muradme]

Hi, I have fixed the library issue for both MPICH and mkl. But the issue is still persisting. For some reason, atm_ic is not creating some variables like "TREF" and some others in "sfc_data.tile7.nc" file, but the workflow shows atm_ic to be successful. Can you suggest me the reasons why cant it create these variables? I am attaching both atm_ic and atm_init log files here.

hafs_atm_ic.log
hafs_atm_init.log

[BijuThomas-NOAA]

It seems like the modules were not properly loaded and perhaps the outputs from the jobs before atm_init were corrupted or incomplete. Take a look at outputs from hafs_utils_chgres_cube.x (in atm_ic) to see if they were completed successfully. Do ncdump -h sfc_data.tile7.nc to see if the variable tref exists.

[muradme]

Hi,

Thank you for your suggestion. I checked the sfc_data.tile7.nc file, and I couldn't find the tref variable in it. The variables present in this file are:

xaxis_1, yaxis_1, zaxis_1, Time, geolon, geolat, slmsk, tsea, sheleg, tg3, zorl, alvsf, alvwf, alnsf, alnwf, facsf, facwf, vfrac, canopy, f10m, t2m, q2m, vtype, stype, uustar, ffmm, ffhh, hice, fice, tisfc, tprcp, srflag, snwdph, shdmin, shdmax, slope, snoalb, stc, smc, slc.

The tref variable is not listed in this file. However, there are no errors in the atm_ic.log file except for some library-related issues. The libraries, including MKL and Ruby, are directly exported inside ush/hafs_pre_job.sh.inc, which allowed me to run atm_prep and other steps successfully. You can see that the MKL library path is exported at line 219 in the atm_ic.log file.

Could the issue be related to any specific libraries, or might there be another source causing this problem? And is this "tref" coming from fix files or gfs file and what does this refer to?

Thanks a lot,
Murad

hafs_atm_ic.log

[BijuThomas-NOAA]

I checked one of sfc_data.tile7.nc file in our run. It has several other variables including tref in addition to those you listed. My understanding is that tref refers to foundation temperature.

I see the forecast executable complains that "Variable not found: get_variable_num_dimension: file:INPUT/sfc_data.nc variable: tref"

[muradme]

I have fixed this issue. Thanks for all your suggestions. The problem was originated from two issues: 1. Initially it was due to two improper library usage [MKL and MPICH] like you have suggested above 2. and finally, the issue was solved when I have set "ictype=gfsnetcdf" in parm/rt_conf/hafs_regional_atm.conf.

Thanks again everyone for your suggestions.

Best regards,
Murad