Error building CCPP file radiation_aerosols.f with cime. #169
Comments
|
Interesting ... When I ran the regression tests yesterday for the ufs-weather-model release/public-v1 branch, everything worked on Cheyenne. See ufs-community/ufs-weather-model#186. Are you using the correct versions of the compiler? This is the modulefile for cheyenne.intel: https://github.com/ufs-community/ufs-weather-model/blob/develop/modulefiles/cheyenne.intel/fv3 |
|
@climbfuji CIME is using intel/19.0.5 and mpt/2.19. We did not change anything particular in the build system but it might be nice to check again build options. If you don't mind and if you have, please let me know the run and build directories on Cheyenne and I could use them to compare the configurations. |
|
@ufuk Turuncoglu - NOAA Affiliate <ufuk.turuncoglu@noaa.gov> Dom is on
leave for 10 days.
My understanding is that you need to use this module file for
cheyenne/intel:
https://github.com/ufs-community/ufs-weather-model/blob/release/public-v1/modulefiles/cheyenne.intel/fv3.
Note that this is from the *release/public-v1 *branch. In Dom's last
comment he pointed you to the file from the develop branch (this may or not
work - I don't know).
<https://github.com/ufs-community/ufs-weather-model/blob/release/public-v1/modulefiles/cheyenne.intel/fv3>
In the public/release-v1 branch version of the cheyenne/intel file, you
will see
module use -a /glade/p/ral/jntp/GMTB/tools/modulefiles/intel-19.0.5/mpt-2.19
module load NCEPlibs/1.1.0
It is important to use the NCEPlibs/1.1.0 for the release (has the upgraded
cghres_cube that ingests netCDF), along with the rest of the environment
described in that file.
See if that helps. If not, I'll find some help in DTC to compile/run the
release branch of the ufs-weather-model on Cheyenne for you. Let me know.
…On Fri, Aug 14, 2020 at 11:15 AM Ufuk Turunçoğlu ***@***.***> wrote:
@climbfuji <https://github.com/climbfuji> CIME is using intel/19.0.5 and
mpt/2.19. We did not change anything particular in the build system but it
might be nice to check again build options. If you don't mind and if you
have, please let me know the run and build directories on Cheyenne and I
could use them to compare the configurations.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#169 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAXVLLNFNPWYYA5DNITSAVWJNANCNFSM4P65GJAA>
.
|
|
@ligiabernardet it is fine. I'll try to run the model outside the CIME and find the differences. I'll keep you posted about the progress and if I need any help. |
|
@ligiabernardet I just wonder that how can i clean existing build? |
|
@laurie Carson <carson@ucar.edu> Can you help answer this question?
…On Mon, Aug 17, 2020 at 10:24 AM Ufuk Turunçoğlu ***@***.***> wrote:
@ligiabernardet <https://github.com/ligiabernardet> I just wonder that
how can i clean existing build?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#169 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQARYY2LYN2RLDJ4G4F3SBFKRXANCNFSM4P65GJAA>
.
|
|
@climbfuji @ligiabernardet it seems that the error is gone whan I update the model to the latest. I am still testing it and I'll double check. |
|
Hi, ufuk, the option "Yes" is to set the clean up before and after the build. |
|
Okay. Thanks @panll. |
|
Hi all, it seems that I was wrong and I am getting same error with updated code. If I try to create a new case and try to do a fresh build, it fails with
error but if I run ./case.build again. It compiles without any problem. I also try to find any possible difference but I could not fina one that could case the issue. Any suggestion? @climbfuji @ligiabernardet |
|
Hi Ufuk, Unfortunately I do not know what could be causing this problem. My
understanding is that the model is building fineas a standalone (outside of
CIME). Have you built the model as standalone successfully? - if so, then
you could look at any differences. If you need DTC to build it on Cheyenne
for you, let me know.
…On Wed, Aug 19, 2020 at 2:55 PM Ufuk Turunçoğlu ***@***.***> wrote:
Hi all, it seems that I was wrong and I am getting same error with updated
code. If I try to create a new case and try to do a fresh build, it fails
with
Building atm with output to
/glade/scratch/turuncu/ufs-mrweather-app-workflow.c96v2/bld/atm.bldlog.200819-142748
/glade/scratch/turuncu/ufs-mrweather-app-workflow.c96v2/bld/atm/obj/FV3/ccpp/physics/physics/radiation_aerosols.f(4840):
error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: )
error but if I run ./case.build again. It compiles without any problem. I
also try to find any possible difference but I could not fina one that
could case the issue. Any suggestion? @climbfuji
<https://github.com/climbfuji> @ligiabernardet
<https://github.com/ligiabernardet>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#169 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAV2NRM5EOZIOEARWKDSBQ33HANCNFSM4P65GJAA>
.
|
|
Yes, I could build standalone model outside of CIME without any problem. @jedwards4b is also looking for the possible differences now. |
|
@uturuncoglu and @jedwards4b I was able to build the App and did not encounter any problems wrt compiling /ccpp/physics/physics/radiation_aerosols.f. I could not reproduce this problem. |
|
@ligiabernardet i had a problem on Cheyenne but it happens randomly. If it fails, i am running case.build again and it builds fine. I am not sure about the cause at this point. Let's keep test the app and see what happens. |
|
@ligiabernardet I run the full test suite on Cheyenne using following command,
but all builds failing with same error. Could you test it in your side? I am not sure about the error but it seems model related because we did not change the build options as I know. @jedwards4b what do you think? Also, running ./case.build to fix the issue seems strange too. |
|
@uturuncoglu sometimes when a build suddenly fixes itself after a second run, its because it was built in parallel first and the dependencies are not correct. Is the CIME build in parallel while the standalone build serial? |
|
@rsdunlapiv i am not sure because in the first case the autogenerated radiation_aerosols.f is not well structured Fortran file and it causes error during compile step. There could be some bug in CCPP autogeneration step but I am not sure at this point. I'll test serial build, I think CIME uses 4 core to build the model. @ligiabernardet I just wonder that how UFS model build works? It builds parallel or serial? Also, I am not sure how CCPP preprocessing steps works. |
|
@uturuncoglu The ccpp_prebuild.py step just generates the physics caps for the requested suite(s). It is described here. Can @llpcarson shed any light on whether the UFS Weather Models builds in serial or parallel mode? |
|
@ligiabernardet I don't think that this has anything to do with build order - you can clearly see the misformatted lines in the file. |
|
ufs-weather-model uses parallel make, so I don't think this is the issue
(although it could still be an unusual timing thing)
radiation_aerosols.f is not auto-generated code, though, as Ligia notes,
CCPP builds the caps, not this one.
Laurie
…On Fri, Aug 21, 2020 at 3:47 PM ligiabernardet ***@***.***> wrote:
@uturuncoglu <https://github.com/uturuncoglu> The ccpp_prebuild.py step
just generates the physics caps for the requested suite(s). It is described
here <https://ccpp-techdoc.readthedocs.io/en/v4.0/CCPPPreBuild.html>.
Can @llpcarson <https://github.com/llpcarson> shed any light on whether
the UFS Weather Models builds in serial or parallel mode?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#169 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB2OWIVCNZTXM4WD5U5HK7LSB3TPDANCNFSM4P65GJAA>
.
|
|
@jim Edwards <jedwards@ucar.edu> There is Doxygen markup in the file for
generating documentation. This has never been a problem in the
platforms/compilers tested. This did not change from the v1.0.0 release.
And even on Cheyenne, you said it builds when you attempt twice. Do you
think the Fortran is illegal - but works on a second attempt? Hmmm -
baffled.
On Fri, Aug 21, 2020 at 3:57 PM Laurie Carson <notifications@github.com>
wrote:
… ufs-weather-model uses parallel make, so I don't think this is the issue
(although it could still be an unusual timing thing)
radiation_aerosols.f is not auto-generated code, though, as Ligia notes,
CCPP builds the caps, not this one.
Laurie
On Fri, Aug 21, 2020 at 3:47 PM ligiabernardet ***@***.***>
wrote:
> @uturuncoglu <https://github.com/uturuncoglu> The ccpp_prebuild.py step
> just generates the physics caps for the requested suite(s). It is
described
> here <https://ccpp-techdoc.readthedocs.io/en/v4.0/CCPPPreBuild.html>.
>
> Can @llpcarson <https://github.com/llpcarson> shed any light on whether
> the UFS Weather Models builds in serial or parallel mode?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <
#169 (comment)
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AB2OWIVCNZTXM4WD5U5HK7LSB3TPDANCNFSM4P65GJAA
>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#169 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAUROJMHTGE6A6L3B53SB3UVFANCNFSM4P65GJAA>
.
|
|
@climbfuji Do you have any idea what can cause this problem that Jim and Ufuk reported? They cannot compile the code, but at a second attempt it works. Linlin and I were not able to reproduce the problem using the updated CIME on Hera or Cheyenne. |
|
Not sure. Maybe some default libraries loaded in the user environment? Always a bad idea. We usually do "module purge" before we build anything.
… On Aug 24, 2020, at 10:26 AM, ligiabernardet ***@***.***> wrote:
@climbfuji <https://github.com/climbfuji> Do you have any idea what can cause this problem that Jim and Ufuk reported? They cannot compile the code, but at a second attempt it works. Linlin and I were not able to reproduce the problem using the updated CIME on Hera or Cheyenne.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#169 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RM72C47W5B6BGY6G33SCKIE3ANCNFSM4P65GJAA>.
|
|
@climbfuji module purge also exist in the CIME interface. I also compared that specific source file with old version of model that was used in 1.0 and it seems that source file is same and did not change since last release. It is strange but it gives error sometimes. |
|
Can I check out the latest version using branch release/public-v1 of the ufs-mrweather-app and then use the manage externals utility? |
|
@climbfuji yes you could test it with latest version of app but you need to use ufs_fix branch for CIME. This will be merged with CIME later. Here is the instructions to test, |
|
BTW, currently I am running full CIME test suite again. I also did it at the weekend and only two test failed with the compile error even with sequential build (make -j 1). So, the error is not consistent. |
|
Don't think it matters but are you using tcsh or bash on cheyenne? |
jedwards4b
changed the title
|
It is historical - that is what was believed at the time. Thanks, Jim, for changing the subject.
…On Fri, Aug 28, 2020 at 7:19 AM Dom Heinzeller ***@***.***> wrote:
Why is this issue called "Error building CCPP" - this is not a CCPP error.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#169 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAQBTFQYYZB6WBRKNYTSC6VEZANCNFSM4P65GJAA>
.
|
|
@climbfuji Thanks. |
|
See the following PRs: NOAA-EMC/NCEPLIBS-external#67 ufs-community/ufs-weather-model#192 |
|
All these PRs were merged, should be ready to test. |
|
And I can confirm that |
|
@climbfuji okay, i am updating the app. I'll let you know when it is ready |
|
@climbfuji @ligiabernardet @fossell @hertneky Here is the instruction to test the updated app. I tested on Cheyenne and it seems working export UFS_DRIVER=nems BTW, my first build attempt failed with same way and if you want to look at the build log, it is in Notes:
|
|
@uturuncoglu Under notes 2. I don't see any files existing in icfiles/201908/20190829. Am I missing something? |
|
@hertneky it must be in $UFS_INPUTDATA/ufs_inputdata/icfiles/201908/20190829. If you are trying on Cheyyene it must work fine but for other platforms you need to copy the file from following FTP link. https://ftp.emc.ncep.noaa.gov/EIB/UFS/inputdata/201908/20190829/ This just for IC and other files will be downloaded automatically. , BTW, i defined following variables on Orion. export WORK=/work/noaa/nems/tufuk |
|
@uturuncoglu I am running on Cheyenne. What should $UFS_INPUT be on Cheyenne? For v1.0, I did setenv $UFS_INPUT $CESMDATAROOT -> giving a full path of /glade/p/cesmdata/cseg/ufs_inputdata/icfiles/201908/20190829, which does not contain any files. Does $UFS_INPUT need to be changed to point somewhere else. |
|
@hertneky sorry, i moved them for testing and I forgot it. could you try again. sorry again. |
|
@uturuncoglu that works now - thanks |
|
@uturuncoglu Testing flat file customization complete and working as visualized/intended for updated v1.1 on Cheyenne. See #174 for details. |
|
@ligiabernardet @climbfuji I am testing model on Stampede and trying to create a new baseline with Intel compiler. The build is giving similar error also in that platform. It is not reproducible and some cases build without any problem. |
|
@ligiabernardet @climbfuji |
|
@uturuncoglu how exactly are you getting the code onto orion (and cheyenne)? |
|
@climbfuji i am just using manage externals. Most of the test cases are failing with build error. |
|
@climbfuji For example following test failed with build error, |
|
@climbfuji FYI, only 6/19 tests are PASSED and others are failed with build error on Orion. |
|
@climbfuji I compared application versions 1.0 and the 1.1 to find any difference in the compile options for the radiation_aerosols.f file and both of them are compiled exactly same options. Also, I run full test suite with 1.0 also and I did not see any compile time error. The CHGRES is also failing for some cases in 1.0, which was working in the 1.0 release time (#179). This indicates some changes in the system level. At this point the only difference seems model versions. I'll also compare the buildlib between 1.0 and 1.1 to find any particular reason that could cause the build error. |
I do get the build error as well when I use the CIME regression tests. The changes in the model code itself between 1.0 and 1.1 are tiny and have nothing to do with actual source code (Python 3 support for the CCPP code generator, changes in namelist templates). This smells a lot like some changes in CIME causing the compilation to fail. |
|
@climbfuji Okay that is fine. I am looking into CIME to find anything. I'll try to test the build by setting GMAKE_J=1 with xmlchange. The default value is 8. To see, I could reproduce the issue also with GMAKE_J=1. |
|
This problem stopped happening after @uturuncoglu reverted to the MRW 1.0 buildlib. |
@ligiabernardet @climbfuji We are getting following error when we try to build UFS MR Weather App under CIME. I am not sure it is a know issue or not.
In this case, i am using release/public-v1 branch for the model and last commit is
If you could also create a case on Cheyenne using UFS MR Weather Model (or I could do it if you provide me the instructions) that could help to compare the possible differences in the build. BTW, I am using intel compiler and MPT combination.
The text was updated successfully, but these errors were encountered: