Building hpc-stack-nco branch in ubuntu 20.04 container with gcc 9 and mpich 3.3.2 #71

MinsukJi-NOAA · 2020-11-17T16:12:39Z

Describe the bug
Using build_stack.sh fails to install hpc-stack-nco in ubuntu 20.04 container

To Reproduce

(terminal 1) git clone --branch feature/hpc-stack-nco https://github.com/NOAA-EMC/hpc-stack
(terminal 2) docker run --rm -it --name hpc-container noaaemc/ubuntu-base:v1
(terminal 1) cd hpc-stack
(terminal 1) docker cp . hpc-container:/home/builder/hpc-stack
(terminal 2) cd hpc-stack
(terminal 2) export HPC_MPI=mpich/3.3.2 &&
./build_stack.sh -p /home/builder/opt -c config/config_custom.sh -y config/stack_custom.yaml

Build fails with "[[: not found" error message.

Change all lib/build_*.sh from #!/bin/sh to #!/bin/bash

Repeat above steps and build fails with:

-- Could NOT find NetCDF_Fortran (missing: NetCDF_Fortran_LIBRARY NetCDF_Fortran_INCLUDE_DIR)
CMake Error at src/flib/CMakeLists.txt:275 (message):
Must have PnetCDF and/or NetCDF Fortran libraries
-- Configuring incomplete, errors occurred!
See also "/home/builder/hpc-stack/pkg/pio-2.5.1/build/CMakeFiles/CMakeOutput.log".
BUILD FAIL! Lib: pio Error:1

System:
Ubuntu 20.04, gcc 9, mpich 3.3.2

climbfuji · 2020-11-17T22:06:33Z

Maybe because ubuntu is using dash instead of bash as default shell? Just a wild guess.

aerorahul · 2020-11-17T22:08:13Z

Just to be clear, this is an issue in the branchfeature/hpc-stack-nco and not with develop.

MinsukJi-NOAA · 2020-11-17T22:11:15Z

Yeah, in ubuntu /usr/bin/sh is linked to /usr/bin/dash. But I doubt that sh is able to handle double bracket notation "[[" either.

…

On Tue, Nov 17, 2020 at 5:06 PM Dom Heinzeller ***@***.***> wrote: Maybe because ubuntu is using dash instead of bash as default shell? Just a wild guess. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANUTPO3FVQKAXR4RC732LPLSQLXXNANCNFSM4TYY265Q> .

climbfuji · 2020-11-17T22:12:21Z

I usually relink /usr/bin/sh to /usr/bin/bash - can you try?

MinsukJi-NOAA · 2020-11-17T22:14:04Z

That should work. But the second issue still remains: missing: NetCDF_Fortran_LIBRARY NetCDF_Fortran_INCLUDE_DIR

aerorahul · 2020-11-17T22:17:08Z

Why is this not raised in the CI which uses ubuntu 20.04?

climbfuji · 2020-11-17T22:20:18Z

Why is this not raised in the CI which uses ubuntu 20.04?

It's possible that the CI version does this relinking, too - specifically to address the dash shortcomings. I remember having seen something like this in the past.

aerorahul · 2020-11-17T22:21:18Z

No such special treatment in the hpc-stack workflow.
https://github.com/NOAA-EMC/hpc-stack/blob/develop/.github/workflows/build_ubuntu.yaml

aerorahul · 2020-11-17T22:22:36Z

I am concerned that a branch that did not pass CI or any kind of review went into the UFS PR.
I am more concerned that the UFS will be susceptible to issues until this is resolved and the stack will be the one to be blamed.

MinsukJi-NOAA · 2020-11-17T22:23:25Z

Current CI is using a container that does not use hpc-stack.
I tested a container with hpc-stack develop branch, and it worked fine.
The hpc-stack-nco branch, which takes care of the UPP, does not build, however.

aerorahul · 2020-11-17T22:24:33Z

Exactly my point. hpc-stack-nco branch was used in the UFS PR that just went in.

MinsukJi-NOAA · 2020-11-17T22:27:23Z

If hpc-stack-nco was built on NOAA machines successfully, that's fine, as far as ufs-weather-model is concerned, I think. I am just encountering an issue trying to build hpc-stack-nco branch in a container.

climbfuji · 2020-11-17T22:31:20Z

I am not sure if any of this is helpful to solve the issues we have right now. I'll try something different.

Minsuk, these variables are usually set by FindNetCDF.cmake - do you have the ominous export NCO_V=false in the stack config file (e.g. config/config_hera.sh) - can you compare your config file with one of the existing ones where the stack built successfully?

MinsukJi-NOAA · 2020-11-17T22:40:42Z

diff hpc-stack/config/config_custom.sh hpc-nco/config/config_custom.sh 7a8 > export NCO_V=false

MinsukJi-NOAA · 2020-11-17T23:48:31Z

That didn't do it. Same error without export NCO_V=false

edwardhartnett · 2020-11-18T12:51:52Z

This must be resolved before #72

@MinsukJi-NOAA how do we reproduce this error on a regular ubuntu workstation?

MinsukJi-NOAA · 2020-11-18T13:05:47Z

@edwardhartnett, the 'steps to reproduce' above assumes you have docker installed on your ubuntu workstation.

edwardhartnett · 2020-11-18T20:29:30Z

Can we do a 1.1 release of hpc-stack without resolving this issue? Or must this issue be resolved before the release?

@kgerheiser would like to do a release tomorrow or the next day...

MinsukJi-NOAA · 2020-11-18T20:35:08Z

@edwardhartnett , this issue is with feature/hpc-stack-nco branch, and not with develop branch. So, if #72 is not merging the feature/hpc-stack-nco into develop, I don't see why this needs to be resolved before 1.1 release.

edwardhartnett · 2020-11-19T15:51:32Z

OK, hpc-stack-nco is indeed going to be merged. So does this need to be resolved before the imminent 1.1.0 release?

aerorahul · 2020-11-19T15:56:22Z

I don't see how it will be merged in its current state.
There are multiple "features" in this branch that need to be their own PR's

We can discuss this later today.

edwardhartnett · 2020-11-19T21:23:07Z

We believe the develop branch will work in the ubuntu container. Can you try this please.

MinsukJi-NOAA · 2020-11-19T21:36:42Z

The develop branch worked in ubuntu container when I tested it two days ago. I was trying to use hpc-stack-nco branch because it has upp in it (which the weather model is using now).

If I use the develop branch, and specify upp in config files, would that work?

kgerheiser · 2020-11-20T15:40:50Z

UPP has been added to the develop branch

MinsukJi-NOAA · 2020-11-20T15:45:52Z

That's great! I will test the latest weather model in an ubuntu 20.04 container, and report the outcome.

MinsukJi-NOAA · 2020-11-20T19:49:21Z

I verified that this works by installing hpc-stack/develop in ubuntu 20.04 container and running ufs-weather-model restart unit test.

MinsukJi-NOAA · 2020-11-23T17:30:59Z

Latest develop branch failed to build in ubuntu 20.04. Error messages are:

[ 39%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/AVIATION.f.o
cd /home/builder/hpc-stack/pkg/upp-upp_v10.0.0/build/sorc/ncep_post.fd && /usr/bin/gfortran  -I/home/builder/opt/include_4 -I/home/builder/opt/include -I/usr/include/x86_64-linux-gnu/mpich  -g -fbacktrace -ffree-form -ffree-line-length-none -fconvert=big-endian -O3 -Jinclude   -fopenmp -c /home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/AVIATION.f -o CMakeFiles/upp.dir/AVIATION.f.o
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:106:15:

  106 |       allocate(QC_BL(im,jsta_2l:jend_2u,lm))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:202:15:

  202 |       allocate(hail_maxhailcast(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:231:15:

  231 |       allocate(shdmin(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:232:15:

  232 |       allocate(shdmax(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:233:15:

  233 |       allocate(lai(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:275:15:

  275 |       allocate(rainc_bucket1(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:277:15:

  277 |       allocate(rainnc_bucket1(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:279:15:

  279 |       allocate(pcp_bucket1(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:281:15:

  281 |       allocate(snow_bucket1(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/sorc/ncep_post.fd/ALLOCATE_ALL.f:283:15:

  283 |       allocate(graup_bucket1(im,jsta_2l:jend_2u))
      |               1
Error: Allocate-object at (1) is neither a data pointer nor an allocatable variable
make[2]: *** [sorc/ncep_post.fd/CMakeFiles/upp.dir/build.make:76: sorc/ncep_post.fd/CMakeFiles/upp.dir/ALLOCATE_ALL.f.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/build'
make[1]: *** [CMakeFiles/Makefile2:133: sorc/ncep_post.fd/CMakeFiles/upp.dir/all] Error 2
make[1]: Leaving directory '/home/builder/hpc-stack/pkg/upp-upp_v10.0.0/build'
make: *** [Makefile:130: all] Error 2
BUILD FAIL!  NCEPlib: upp Error:2

I did not write down which version passed my test on 11/20/2020, but I am guessing either 10a8431 or 941250a. I can find out which commit is the culprit if it is helpful.

kgerheiser · 2020-11-23T17:34:49Z

The issue is that it's building two UPP's, the one with the commit dcea26 and UPP v10.0.0.

They have the same Fortran module names (vrbls2d, etc), and the build of UPP v10.0.0 is picking up decea26's module and not its own.

The solution is to disable building nceppost.

Maybe we should disable that in the stack file that you are using? Is it stack_ufs_weather_ci.yaml? Do you need both of them?

MinsukJi-NOAA · 2020-11-23T17:37:21Z

Thank you @kgerheiser . No, I don't think so. I am building with nceppost disabled in stack_ufs_weather-ci.yaml right now. I will report back.

kgerheiser · 2020-11-23T17:39:42Z

It works fine when you build in a hierarchical structure with lmod, but when installing in a flat structure, like in a container, the names conflict.

MinsukJi-NOAA · 2020-11-23T19:04:05Z

I was able to build the latest develop branch with nceppost disabled in stack_ufs_weather_ci.yaml.

MinsukJi-NOAA added the bug Something isn't working label Nov 17, 2020

MinsukJi-NOAA mentioned this issue Nov 19, 2020

rt update ufs-community/ufs-weather-model#261

Merged

edwardhartnett assigned aerorahul Nov 19, 2020

edwardhartnett mentioned this issue Nov 19, 2020

Update to new prod_util-1.2.2 #88

Closed

edwardhartnett mentioned this issue Nov 19, 2020

add UPP to hpc-stack #63

Closed

kgerheiser closed this as completed Nov 20, 2020

edwardhartnett reopened this Nov 23, 2020

kgerheiser mentioned this issue Nov 23, 2020

Disable building nceppost for container #104

Merged

kgerheiser closed this as completed in #104 Nov 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Building hpc-stack-nco branch in ubuntu 20.04 container with gcc 9 and mpich 3.3.2 #71

Building hpc-stack-nco branch in ubuntu 20.04 container with gcc 9 and mpich 3.3.2 #71

MinsukJi-NOAA commented Nov 17, 2020

climbfuji commented Nov 17, 2020

aerorahul commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020 via email

climbfuji commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020

aerorahul commented Nov 17, 2020

climbfuji commented Nov 17, 2020

aerorahul commented Nov 17, 2020

aerorahul commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020 •

edited

Loading

aerorahul commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020 •

edited

Loading

climbfuji commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020

edwardhartnett commented Nov 18, 2020

MinsukJi-NOAA commented Nov 18, 2020

edwardhartnett commented Nov 18, 2020

MinsukJi-NOAA commented Nov 18, 2020

edwardhartnett commented Nov 19, 2020

aerorahul commented Nov 19, 2020

edwardhartnett commented Nov 19, 2020

MinsukJi-NOAA commented Nov 19, 2020

kgerheiser commented Nov 20, 2020

MinsukJi-NOAA commented Nov 20, 2020

MinsukJi-NOAA commented Nov 20, 2020 •

edited

Loading

MinsukJi-NOAA commented Nov 23, 2020

kgerheiser commented Nov 23, 2020 •

edited

Loading

MinsukJi-NOAA commented Nov 23, 2020 •

edited

Loading

kgerheiser commented Nov 23, 2020

MinsukJi-NOAA commented Nov 23, 2020

Building hpc-stack-nco branch in ubuntu 20.04 container with gcc 9 and mpich 3.3.2 #71

Building hpc-stack-nco branch in ubuntu 20.04 container with gcc 9 and mpich 3.3.2 #71

Comments

MinsukJi-NOAA commented Nov 17, 2020

climbfuji commented Nov 17, 2020

aerorahul commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020 via email

climbfuji commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020

aerorahul commented Nov 17, 2020

climbfuji commented Nov 17, 2020

aerorahul commented Nov 17, 2020

aerorahul commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020 • edited Loading

aerorahul commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020 • edited Loading

climbfuji commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020

MinsukJi-NOAA commented Nov 17, 2020

edwardhartnett commented Nov 18, 2020

MinsukJi-NOAA commented Nov 18, 2020

edwardhartnett commented Nov 18, 2020

MinsukJi-NOAA commented Nov 18, 2020

edwardhartnett commented Nov 19, 2020

aerorahul commented Nov 19, 2020

edwardhartnett commented Nov 19, 2020

MinsukJi-NOAA commented Nov 19, 2020

kgerheiser commented Nov 20, 2020

MinsukJi-NOAA commented Nov 20, 2020

MinsukJi-NOAA commented Nov 20, 2020 • edited Loading

MinsukJi-NOAA commented Nov 23, 2020

kgerheiser commented Nov 23, 2020 • edited Loading

MinsukJi-NOAA commented Nov 23, 2020 • edited Loading

kgerheiser commented Nov 23, 2020

MinsukJi-NOAA commented Nov 23, 2020

MinsukJi-NOAA commented Nov 17, 2020 •

edited

Loading

MinsukJi-NOAA commented Nov 17, 2020 •

edited

Loading

MinsukJi-NOAA commented Nov 20, 2020 •

edited

Loading

kgerheiser commented Nov 23, 2020 •

edited

Loading

MinsukJi-NOAA commented Nov 23, 2020 •

edited

Loading