-
Notifications
You must be signed in to change notification settings - Fork 117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update miniconda3/4.12.0 module and regional_workflow environment on Cheyenne, Hera, Jet, Orion #431
Update miniconda3/4.12.0 module and regional_workflow environment on Cheyenne, Hera, Jet, Orion #431
Conversation
remove loading of system python3
remove loading system python module
Load updated miniconda3 and ask to activate regional_workflow enviroment
Load an updated miniconda3 and ask to activate regional_workflow environment
Update miniconda3 module location and ask to activate regional_workflow
Update miniconda3/4.12.0 module location and ask to activate regional_workflow environment
Run an additional cycle of "conda deactivate" and "conda activate regional_workflow". It ensures that _python3_ binary path from the *regional_workflow* environment becomes prepended to the search $PATH, and is found first, before the _python3_ from miniconda3/4.12.0 from the *base* environment.
"conda activate regional_workfow"
…orkflow.lua use new miniconda3/4.12.0 with regional_workflow environment
all the requested packages for the python3 are found in regional_workflow environment
load updated miniconda3/4.12.0 with regional_workflow environment
Load an updated miniconda3/4.12.0 with the regional_workflow environment
LmodMsgRaw([===[Please do the following to activate conda: | ||
> conda activate /glade/p/ral/jntp/UFS_SRW_app/conda/regional_workflow | ||
> conda activate regional_workflow |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good to see Cheyenne is similar to other systems in this regard now! I think you would need to do the same in ush/load_modules_wflow.sh
lines 63-69
-if [[ "${machine}" == 'cheyenne' ]]; then
- conda activate /glade/p/ral/jntp/UFS_SRW_app/conda/regional_workflow
-else
- if [ ! -z $(command -v conda) ]; then
- conda activate regional_workflow
- fi
+if [ ! -z $(command -v conda) ]; then
+ conda activate regional_workflow
fi
conda activate ${SRW_ENV} | ||
if [ $machine = "gaea" ]; then | ||
# if [ $machine = "gaea" ]; then | ||
conda deactivate | ||
conda activate ${SRW_ENV} | ||
fi | ||
# fi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like now you had to do the deactivate->activate
process for all systems now -- we were hoping for the opposite with your PR! I guess it must be the transition from TCL to Lua of miniconda3 file epic installations that is causing this. Did you have to do this with your previous PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@natalie-perlin Looking at PR #419, I didn't see this modification. Is this required due to only using the new miniconda3 modules? @mark-a-potts has opened PR #428 to remove this logic, along with the conda deactivate
and conda activate ${SRW_ENV}
. It would be best to remove this, rather than make all machines require it. Would you be able to run tests of removing this logic and making similar changes that @mark-a-potts made for the Gaea local modulefiles in PR #428?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@danielabdi-noaa - this fix was temporary that we found would work on all systems. It is likely related to additional loads of miniconda3 in *.local.lua files if/when they are used. This module loading changes the binaries search $PATH once again. The use of *local.lua files may still need revisiting, checking the options now (@mark-a-potts does testing as well)
need to have miniconda3 loaded in build module
need to have miniconda3 loaded in the build module
need to have miniconda3 loaded in build module
need to have miniconda3 loaded in build module
need to have miniconda3 loaded in the build modulefile
need to have miniconda3 loaded in build modulefile
conda activate command same across the platforms
miniconda3 is now loaded in build_<system>_<compiler>, not in *.local files
miniconda3 is now loaded in build_<system>_<compiler>, not in *.local files
miniconda3 is now loaded in build_<system>_<compiler>, not in *.local files
miniconda3 is now loaded in build_<system>_<compiler>, not in *.local files
The changes addressed in PR-444. |
PR-444 (#444) merged into develop |
DESCRIPTION OF CHANGES:
Update SRW modulefiles and use miniconda3/4.12.0 conda/python updated modules that have identical installations in EPIC-managed common space on the following HPC systems: Cheyenne, Hera, Orion, Jet. The installations are similar to that on Gaea, which has been already merged PR-352, PR-353. A regional_workflow virtual environment is installed under miniconda3 module and contains all the python packages needed to run the SRW and prepare basic plots.
Miniconda3/4.12.0 locations on the RDHPCs, with the regional_workflow environment created containing all the packages for running, plotting, and met verification.
Cheyenne
:module use /glade/work/epicufsrt/contrib/miniconda3/modulefiles
module load miniconda3/4.12.0
Hera:
module use /scratch1/NCEPDEV/nems/role.epic/miniconda3/modulefiles
module load miniconda3/4.12.0
Orion:
module use /work/noaa/epic-ps/role-epic-ps/miniconda3/modulefiles
module load miniconda3/4.12.0
Jet:
module use /mnt/lfs4/HFIP/hfv3gfs/role.epic/miniconda3/modulefiles
module load miniconda3/4.12.0
Earlier similar installation on
Gaea
; the repo changes have been already merged:module use /lustre/f2/dev/role.epic/contrib/modulefiles
module load miniconda3/4.12.0
Type of change
TESTS CONDUCTED:
It is a part of a larger PR-419. The tests have been made as reported in PR-419, and additional successful tests have been run on Cheyenne.
DEPENDENCIES:
DOCUMENTATION:
ISSUE:
This PR is referenced in SRW-Issue-427
CHECKLIST
BASH_SOURCE[1]: unbound variable
. This was observed following recent PR merges.LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):