can't generate regional workflow due to 'jinja2' module (at Derecho)

[SarahLu-NOAA]

When I try generating regional workflow (by ./generate_FV3LAM_wflow.py) at derecho, I encounter the following error msg:
File "/glade/work/clu/ufs/ufs-srweather-app/ush/python_utils/config_parser.py", line 28, in
import jinja2
ModuleNotFoundError: No module named 'jinja2'

I then re-build the SRW v2.2.0 and encounter the same error msg. How to go around this? Thanks

[MichaelLueken]

Good morning, @SarahLu-NOAA!

Before generating the regional workflow via ./generate_FV3LAM_wflow.py, did you load the wflow_derecho modulefile and activate the workflow_tools conda environment? For the SRW v2.2.0 release, after building the SRW, using the following commands on Derecho should allow you to activate the necessary conda environment to generate the regional workflow:

module use $PWD/modulefiles
module load wflow_derecho
conda activate workflow_tools

Once the workflow_tools conda environment has been loaded, you should be able to generate the regional workflow without the missing 'jinja2' module error you have been encountering.

Please let me know if you continue to encounter this issue once the workflow_tools conda environment has been loaded.

[SarahLu-NOAA]

Good morning, @MichaelLueken

I did activate conda env, as stated above.
I was able to run SRW v2.2.0 a couple of weeks, but now encounter the jinja2 module error. Thanks

[SarahLu-NOAA]

@MichaelLueken ,
Sorry that I did not make it sure. I did the following:
Source $PWD/etc/lmod-setup.sh derecho
module use $PWD/modulefiles
module load wflow_derecho
conda activate workflow_tools

[MichaelLueken]

Very interesting, @SarahLu-NOAA,

Cloning a copy of release/public-v2.2.0 on Derecho, I was able to follow your steps and the workflow_tools conda environment was properly loaded. I have copied your ush/config.yaml file into my cloned copy, updated the account, and attempted to generate the workflow via ./generate_FV3LAM_wflow.py. The error message I received was:

  File "/glade/derecho/scratch/mlueken/ufs-srweather-app/ush/setup.py", line 105, in load_config_for_setup
    raise Exception(errmsg)
Exception: Invalid key(s) specified in /glade/derecho/scratch/mlueken/ufs-srweather-app/ush/config.yaml:
oetadata = {'description': 'Sample community config'}

Replacing oetadata: with metadata: on line 1 of the config.yaml file, I was able to successfully generate the workflow from your config.yaml file.

Additionally, I had to move:

    task_plot_allvars:
      PLOT_FCST_START: 0
      PLOT_FCST_INC: 6
      PLOT_FCST_END: 12

out of the rocoto: section. This leads to the following in the current task_plot_allvars: sections in config.yaml:

task_plot_allvars:
  COMOUT_REF: ""
  PLOT_FCST_START: 0
  PLOT_FCST_INC: 6
  PLOT_FCST_END: 12

With these changes, I was able to get your config.yaml file to run on Derecho.

My config.yaml file can be found - /glade/derecho/scratch/mlueken/ufs-srweather-app/ush/config.yaml

[SarahLu-NOAA]

Many thanks, @MichaelLueken
I'll try your approach to replace oetadata. I have a couple of working yaml to run srw v2.2.0. But the config.yaml that you copied was not yet tested due to the jinja2 error. Thanks for pointing out that my config.yaml needs fixes.

[gspetro-NOAA]

Hi @SarahLu-NOAA ,
I can't tell if you've gotten the environment loaded yet or not, but if not, you could try running conda deactivate and then module restore to reset everything to a clean slate. Then you can try the env activation steps again (adjusting the path accordingly):

conda deactivate
module restore
source /path/to/ufs-srweather-app/etc/lmod-setup.sh derecho
module use /path/to/ufs-srweather-app/modulefiles
module load wflow_derecho
conda activate workflow_tools

[SarahLu-NOAA]

Hi @gspetro-NOAA After fixing config.yaml, I'm able to run srwv2.2.0. Your answer is very helpful to re-load needed modules. Many thanks.