zppy weekly testing log #634

forsyth2 · 2024-10-18T15:55:17Z

forsyth2
Oct 18, 2024
Maintainer

Starting this page to log the process/results for the weekly runs of zppy's integration testing: bundles, comprehensive_v2, comprehensive_v3

forsyth2 · 2024-10-18T16:28:57Z

forsyth2
Oct 18, 2024
Maintainer Author

Friday 2024-10-18

Note that before we get to testing a release candidate for Unified, we'll need to get v3 test data on to Perlmutter and Compy too.

On Chrysalis:

# Set up branch
cd ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241018 upstream/main
git log
# Last commit, from 10/15: Improve carryover dependency handling (#623)

# Set up diags to use
cd ../e3sm_diags
git checkout main
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/15: make --closedcontourcmd resolution specific (#824)
mamba clean --all
# Press `y` twice
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241018
conda activate e3sm_diags_20241018
pip install .

# Set up testing files
cd ../zppy
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241018"
# For get_chyrsalis_expansions:         "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241018",
# Keep this as is:     generate_cfgs(unified_testing=False, dry_run=False)

# Set up zppy environment
mamba clean --all
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241018 
conda activate zppy_dev_weekly_20241018 
pip install .

# Run unit tests
python -u -m unittest tests/test_*.py
# All 22 tests pass

# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg

# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg

# Once that finishes:
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241018/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241018/v2.LR.historical_0201/post/scripts
grep -v "OK" *status

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241018/v3.LR.historical_0051/post/scripts
grep -v "OK" *status

# Run integration tests
python -u -m unittest tests/integration/test_*.py

This gets us these results:

======================================================================
FAIL: test_bash_generation (tests.integration.test_bash_generation.TestBashGeneration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_bash_generation.py", line 14, in test_bash_generation
    self.assertEqual(
AssertionError: 256 != 0

----------------------------------------------------------------------
Ran 14 tests in 1291.690s

FAILED (failures=1)

python -u -m unittest tests/integration/test_bash_generation.py

We can see the diff is from the formatting change introduced in #627, so this is expected. Let's update the expected results.

cd /lcrc/group/e3sm/public_html
cp -r zppy_test_resources zppy_test_resources_previous/expected_results_until_20241018
# Currently, copying over entire expected results directory to keep a snapshot history 
# But we could just copy over subdirs that changed in a given week.

cd /home/ac.forsyth2/ez/zppy
chmod u+x tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh

That gives:

----------------------------------------------------------------------
Ran 1 test in 0.835s

OK

✅ This week's testing of main is successful.

0 replies

forsyth2 · 2024-10-25T19:12:52Z

forsyth2
Oct 25, 2024
Maintainer Author

Friday 2024-10-25

Commits since 2024-10-18:
zppy:

None

e3sm_diags:

Fixing African easterly wave density plots in TC analysis (Fixing African easterly wave density plots in TC analysis e3sm_diags#851)

# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241025 upstream/main
git log
# Last commit, from 10/15: Improve carryover dependency handling (#623)

# Set up diags to use
cd ../e3sm_diags
git checkout main
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/23: Fixing African easterly wave density plots in TC analysis (#851)
mamba clean --all --y
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241025
conda activate e3sm_diags_20241025
pip install .

# Set up testing files
cd ../zppy
# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241025"
# For get_chyrsalis_expansions:         "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241025",
# Keep this as is:     generate_cfgs(unified_testing=False, dry_run=False)

# Set up zppy environment
mamba clean --all --y
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241025 # This command took a while
conda activate zppy_dev_weekly_20241025
pip install .

# Run unit tests
python -u -m unittest tests/test_*.py
# All 22 tests pass

# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241025
# unified_testing=False
# diags_environment_commands=source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241025
# e3sm_to_cmip_environment_commands=
# environment_commands=
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish: (took about 1h15m)
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes: (took about 10m)
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:ERROR (1)
cat e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o614397 
# cp: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/lnd/180x360_aave/clim/2yr/v3.LR.historical_0051_*_1987??_1988??_climo.nc': No such file or directory

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241025/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.status:ERROR (1)
cat e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.o614424 
# cp: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241025/v2.LR.historical_0201/post/lnd/180x360_aave/clim/2yr/v2.LR.historical_0201_*_1982??_1983??_climo.nc': No such file or directory

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/scripts
grep -v "OK" *status

# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.py

That gives:

======================================================================
ERROR: test_comprehensive_v2_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 36, in test_comprehensive_v2_images
    test_images(self, "comprehensive_v2", V2_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 99, in check_mismatched_images
    compare_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 65, in compare_images
    shutil.copy(
  File "/home/ac.forsyth2/miniconda3/envs/zppy_dev_weekly_20241025/lib/python3.9/shutil.py", line 427, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/ac.forsyth2/miniconda3/envs/zppy_dev_weekly_20241025/lib/python3.9/shutil.py", line 266, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20241025/v2.LR.historical_0201/image_check_failures_comprehensive_v2/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/tc_analysis/aew-density-map.png_actual.png'

======================================================================
FAIL: test_comprehensive_v3_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 39, in test_comprehensive_v3_images
    test_images(self, "comprehensive_v3", V3_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 121, in check_mismatched_images
    test.assertEqual(missing_images, [])
AssertionError: Lists differ: ['e3sm_diags/lnd_monthly_mvm_lnd/model_vs_[43955 chars]png'] != []

First list contains 336 additional elements.
First extra element 0:
'e3sm_diags/lnd_monthly_mvm_lnd/model_vs_model_1987-1988/viewer/viewer/e3sm_logo.png'

Diff is 45015 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------
Ran 14 tests in 983.368s

FAILED (failures=1, errors=1)

The tc_analysis error is likely because of the new commit in e3sm_diags, so that will likely just be a matter of updating the expected results.

The lnd_monthly_mvm_lnd error is to be expected, considering the cannot stat error detected earlier.

Let's look at the image check failures:

v3 is empty.
v2 is empty.

Let's try the directory paths:

$ ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20241025/v3.LR.historical_0051/image_c
heck_failures_comprehensive_v3
# Empty

$ ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20241025/v2.LR.historical_0201/image_c
heck_failures_comprehensive_v2
# Empty

Clearly, image check failures aren't being routed to the right place.

1 reply

forsyth2 Oct 25, 2024
Maintainer Author

Bug report is at #635.

forsyth2 · 2024-11-01T21:21:37Z

forsyth2
Nov 1, 2024
Maintainer Author

Friday 2024-11-01

Commits since 2024-10-25:
zppy:

Fix dir issues in comparing images (Fix dir issues in comparing images #638)
Updates to diags handling (Update to diags handling #633)

e3sm_diags:

Convert H2OLNZ units to ppm by volume (Convert H2OLNZ units to ppm by volume e3sm_diags#874)
add_TCO_60S60N to default lat-lon (https://github.com/E3SM-Project/e3sm_diags/pull/#879)

# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241101 upstream/main
git log
# Last commit, from 10/30: Updates to diags handling (#633)

# Set up diags to use
cd ../e3sm_diags
git checkout main
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/29: add_TCO_60S60N to default lat-lon (#879)
mamba clean --all --y
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241101
conda activate e3sm_diags_20241101
pip install .

# Set up testing files
cd ../zppy
# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241101"
# For get_chyrsalis_expansions:         "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241101",
# Keep this as is:     generate_cfgs(unified_testing=False, dry_run=False)

# Set up zppy environment
mamba clean --all --y
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241101 # This command took a while
conda activate zppy_dev_weekly_20241101
pip install .

# Run unit tests
python -u -m unittest tests/test_*.py
# All 23 tests pass (increased from 22)

# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241101
# unified_testing=False
# diags_environment_commands=source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241101
# e3sm_to_cmip_environment_commands=
# environment_commands=
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish: (took about 1h15m)
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes: (took about 10m)
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241101/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241101/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241101/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors

# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.py

That gives:

======================================================================
FAIL: test_bash_generation (tests.integration.test_bash_generation.TestBashGeneration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_bash_generation.py", line 14, in test_bash_generation
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_comprehensive_v2_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 36, in test_comprehensive_v2_images
    test_images(self, "comprehensive_v2", V2_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 132, in check_mismatched_images
    test.assertEqual(mismatched_images, [])
AssertionError: Lists differ: ['e3sm_diags/atm_monthly_180x360_aave_mvm/[2858 chars]png'] != []

First list contains 25 additional elements.
First extra element 0:
'e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/tc_analysis/aew-density-map.png'

Diff is 2985 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_comprehensive_v3_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 39, in test_comprehensive_v3_images
    test_images(self, "comprehensive_v3", V3_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 132, in check_mismatched_images
    test.assertEqual(mismatched_images, [])
AssertionError: Lists differ: ['e3sm_diags/atm_monthly_180x360_aave_mvm/[2019 chars]png'] != []

First list contains 16 additional elements.
First extra element 0:
'e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/zonal_mean_xy/model_vs_model/v3.LR.historical_0051-Z3-500-SON-global.png'

Diff is 2119 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------
Ran 14 tests in 1691.117s

FAILED (failures=3)

python -u -m unittest tests/integration/test_bash_generation.py

The diff is from the template file change in https://github.com/E3SM-Project/zppy/pull/633/files#diff-fbd1002ca5f9034c321e896884792a69fc7e4a9fc14e826a3954dc81a3d6d76d, so this is expected. Let's update the expected results.

cd /lcrc/group/e3sm/public_html
cp -r zppy_test_resources zppy_test_resources_previous/expected_results_until_20241101
# Currently, copying over entire expected results directory to keep a snapshot history 
# But we could just copy over subdirs that changed in a given week.

cd /home/ac.forsyth2/ez/zppy
chmod u+x tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh

That test passes now.

As for the image check failures (attempted to format this hierarchically, but only semi-successfully):

v2 diags
1.1 model_vs_obs
1.1.1. tc_analysis. These changes are from Fixing African easterly wave density plots in TC analysis e3sm_diags#851 (that was merged last week, but expected results weren't updated because of [Bug]: Errors in 2024-10-25 test of main #635).
1.1.2. zonal_mean_2d_stratosphere. The ERA5 and MERRA2 diffs appear to be from Convert H2OLNZ units to ppm by volume e3sm_diags#874 and thus expected.
1.1.2.1. ERA5
1.1.2.2. MERRA2
1.2. model_vs_model
1.2.1. tc_analysis. Again, from Fixing African easterly wave density plots in TC analysis e3sm_diags#851.
1.2.2. zonal_mean_2d_stratosphere. Here, the issue is with the Q plots, but still from Convert H2OLNZ units to ppm by volume e3sm_diags#874.
v3 diags
2.1. model_vs_obs zonal_mean_2d_stratosphere. Again, from Convert H2OLNZ units to ppm by volume e3sm_diags#874.
2.1.1. ERA5
2.1.2. MERRA2
2.2. model_vs_model
2.2.1. zonal_mean_2d_stratosphere. Again, from Convert H2OLNZ units to ppm by volume e3sm_diags#874.
2.2.2. zonal_mean_xy. This has one diff, which actually appears blank. It doesn't look like zonal_mean_xy has been modified recently. I suspect this is just a very minor matplotlib pixel shift that has appeared randomly -- not a concern.

So, all image check failures are accounted for. Let's update the expected results.

# We already copied over the previous expected results

cd /home/ac.forsyth2/ez/zppy
chmod u+x tests/integration/generated/update_weekly_expected_files_chrysalis.sh
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh

After about 20 minutes, that gives:

.
----------------------------------------------------------------------
Ran 5 tests in 1298.721s

OK

All expected results have been updated.

✅ This week's testing of main is successful.

0 replies

forsyth2 · 2024-11-22T21:46:29Z

forsyth2
Nov 22, 2024
Maintainer Author

Friday 2024-11-22

This is the first Weekly test since the merging of major refactor splitting zppy into zppy itself (workflow coordination) and zppy-interfaces ("last-mile" code improving E3SM-specific usability of external packages).

Check commit history

Commits since 2024-11-01 (last Weekly test of main):

zppy:

zppy-interfaces refactor (zppy-interfaces refactor #642)

zppy-interfaces:

Initial commit
Initial implementation (Initial implementation zppy-interfaces#1)

e3sm_diags:

Release 2.12.1 (Release 2.12.1 e3sm_diags#893)

However, I'm going to conciously test e3sm_diags from before this commit, because then we can be sure that any issues appearing on this week's testing are specifically related to the zppy/zppy-interfaces split.

Set up branch

# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241122 upstream/main
git log
# Last commit, from 11/22: zppy-interfaces refactor (#642)

Set up zppy-interfaces env

cd ~/ez/zppy-interfaces
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zi_weekly_20241122 upstream/main
git log
# Last commit, from 11/22: Initial implementation (#1)
conda clean --all --y
conda env create -f conda/dev.yml -n zi_dev_weekly_20241122
conda activate zi_dev_weekly_20241122
pip install .

Set up e3sm_diags env

cd ~/ez/e3sm_diags
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git checkout main
git reset --hard upstream/main
git log
# Last commit, from 11/20: Release 2.12.1 (#893)
# NOTE: As mentioned earlier, let's use the Diags code that was used for the last testing
git reset --hard 3f5b03627aef703b4f7d8463cd5faed3c0806566
git log
# Last commit, from 10/29: add_TCO_60S60N to default lat-lon (#879)
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm_diags_from_20241029
conda activate e3sm_diags_from_20241029
pip install .

Set up zppy env

cd ~/ez/zppy
conda clean --all --y
conda env create -f conda/dev.yml -n zppy_dev_weekly_20241122
conda activate zppy_dev_weekly_20241122
pip install .

Unit tests for zppy-interfaces

cd ~/ez/zppy-interfaces
conda activate zi_dev_weekly_20241122
pytest tests/global_time_series/test_*.py
# 7 passed in 25.86s

Unit tests for zppy

cd ~/ez/zppy
conda activate zppy_dev_weekly_20241122
pytest tests/test_*.py
# 23 passed in 0.63s

Integration testing for zppy

# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241122"
# For get_chyrsalis_expansions:
#        "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_from_20241029",
#        "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20241122",
# Keep this as is:     generate_cfgs(unified_testing=False, dry_run=False)


# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241122
# unified_testing=False
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_from_20241029
# e3sm_to_cmip_environment_commands=
# environment_commands=
# Reminder: `e3sm_to_cmip_environment_commands=''` => the environment of the `ts` task will be used
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

# NOTE: Created https://github.com/E3SM-Project/zppy/issues/643 to also print out `global_time_series_environment_commands`

zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes:
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241122/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241122/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241122/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors

# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.py
# This needs to be converted to pytest still -- see https://github.com/E3SM-Project/zppy/issues/644

That gives:

Errors

======================================================================
FAIL: test_bash_generation (tests.integration.test_bash_generation.TestBashGeneration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_bash_generation.py", line 14, in test_bash_generation
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_campaign_cryosphere (tests.integration.test_campaign.TestCampaign)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_campaign.py", line 17, in test_campaign_cryosphere
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_campaign_cryosphere_override (tests.integration.test_campaign.TestCampaign)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_campaign.py", line 39, in test_campaign_cryosphere_override
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_campaign_none (tests.integration.test_campaign.TestCampaign)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_campaign.py", line 71, in test_campaign_none
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_campaign_water_cycle (tests.integration.test_campaign.TestCampaign)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_campaign.py", line 87, in test_campaign_water_cycle
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_campaign_water_cycle_override (tests.integration.test_campaign.TestCampaign)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_campaign.py", line 109, in test_campaign_water_cycle_override
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_defaults (tests.integration.test_defaults.TestDefaults)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_defaults.py", line 19, in test_defaults
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_bundles_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 42, in test_bundles_images
    test_images(self, "bundles", V3_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 131, in check_mismatched_images
    test.assertEqual(missing_images, [])
AssertionError: Lists differ: ['global_time_series/global_time_series_19[243 chars]png'] != []

First list contains 3 additional elements.
First extra element 0:
'global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_original.png'

+ []
- ['global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_original.png',
-  'global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_glb_original.png',
-  'global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_n_original.png']

======================================================================
FAIL: test_comprehensive_v2_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 36, in test_comprehensive_v2_images
    test_images(self, "comprehensive_v2", V2_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 132, in check_mismatched_images
    test.assertEqual(mismatched_images, [])
AssertionError: Lists differ: ['global_time_series/global_time_series_19[420 chars]png'] != []

First list contains 5 additional elements.
First extra element 0:
'global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_n_original.png'

+ []
- ['global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_n_original.png',
-  'global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_s_lnd.png',
-  'global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_n_lnd.png',
-  'global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_glb_lnd.png',
-  'global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_s_original.png']

======================================================================
FAIL: test_comprehensive_v3_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 39, in test_comprehensive_v3_images
    test_images(self, "comprehensive_v3", V3_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 132, in check_mismatched_images
    test.assertEqual(mismatched_images, [])
AssertionError: Lists differ: ['global_time_series/global_time_series_19[1269 chars]png'] != []

First list contains 14 additional elements.
First extra element 0:
'global_time_series/global_time_series_1985-1995_results/v3.LR.historical_0051_s_lnd_1.png'

Diff is 1363 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------
Ran 14 tests in 1336.704s

FAILED (failures=10)

ls tests/integration/test_*.py
# tests/integration/test_bash_generation.py  tests/integration/test_defaults.py   tests/integration/test_weekly.py
# tests/integration/test_campaign.py         tests/integration/test_last_year.py


python -u -m unittest tests/integration/test_bash_generation.py
# FAILED (failures=1)
# The filled-out global-time-series templates have of course changed
# `inclusions/` has been added to relevant paths
# We just need to update the expected results

python -u -m unittest tests/integration/test_campaign.py
# FAILED (failures=5)
# `inclusions/` has been added to relevant paths
# We just need to update the expected results

python -u -m unittest tests/integration/test_defaults.py
# FAILED (failures=1)
# The filled-out global-time-series templates have of course changed
# `inclusions/` has been added to relevant paths
# We just need to update the expected results

python -u -m unittest tests/integration/test_last_year.py
# OK

For Weekly tests, let's look at the image check failures

https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20241122/v3.LR.historical_0051/image_check_failures_comprehensive_v3/global_time_series/global_time_series_1985-1995_results/
- We appear to have lost the y axis units on the plots -- but only for the lnd plots; the original plots seem to only differ by minimal pixel variation.
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20241122/v2.LR.historical_0201/image_check_failures_comprehensive_v2/global_time_series/global_time_series_1980-1990_results/
- Essentially the same issues
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_bundles_www/test_zppy_weekly_20241122/v3.LR.historical_0051/image_check_failures_bundles/
- Empty -- however, we know test_bundles_images failed. Looks like the global time series plots didn't generate. (I imagine zppy-interfaces refactor #642 simply didn't update environment_commands for this test properly).

These issues should be simple to fix. Next week, I'll address them and then re-run the Weekly test suite. Specifically:

Stop y axis units from getting suppressed on Land plots
Get global_time_series plots to generate for the bundles test.
Add printout of global_time_series_environment_commands #643
Convert integration tests to pytest #644 (may do this as a later pull request)

1 reply

forsyth2 Nov 26, 2024
Maintainer Author

Rerun on branch `test-fixes`

See #646 and E3SM-Project/zppy-interfaces#7.

Unit tests for zppy-interfaces

cd ~/ez/zppy-interfaces
conda activate zi_dev_weekly_20241122
pip install .
pytest tests/unit/global_time_series/test_*.py
# 7 passed in 28.34s

Unit tests for zppy

cd ~/ez/zppy
conda activate zppy_dev_weekly_20241122
pip install .
pytest tests/test_*.py
# 23 passed in 0.13s

Integration testing for zppy

# Edit tests/integration/utils.py:
# UNIQUE_ID = "test-fixes-20241125"
# For get_chyrsalis_expansions:
#        "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_from_20241029",
#        "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20241122",
# Keep this as is:     generate_cfgs(unified_testing=False, dry_run=False)


# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test-fixes-20241125
# unified_testing=False
# diags_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_from_20241029
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_dev_weekly_20241122
# e3sm_to_cmip_environment_commands=
# environment_commands=
# Reminder: `e3sm_to_cmip_environment_commands=''` => the environment of the `ts` task will be used
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used

zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg

# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg

# Once that finishes:
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test-fixes-20241125/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test-fixes-20241125/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test-fixes-20241125/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors

# Run integration tests
cd ~/ez/zppy
pytest tests/integration/test_weekly.py

That gives:

============================================================================ short test summary info ============================================================================
FAILED tests/integration/test_weekly.py::test_comprehensive_v2_images - AssertionError
FAILED tests/integration/test_weekly.py::test_comprehensive_v3_images - AssertionError
FAILED tests/integration/test_weekly.py::test_bundles_images - AssertionError
=================================================================== 3 failed, 2 passed in 1248.72s (0:20:48) ====================================================================

Looking at the image check failures:

https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test-fixes-20241125/v3.LR.historical_0051/image_check_failures_comprehensive_v3/global_time_series/global_time_series_1985-1995_results/
- Looks like the land plots now show the y axis units.
- Looks like the differences are now all because of slight shifts in the plots, which can be expected different matplotlib versions.
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test-fixes-20241125/v2.LR.historical_0201/image_check_failures_comprehensive_v2/global_time_series/global_time_series_1980-1990_results/
- Same thing.
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_bundles_www/test-fixes-20241125/v3.LR.historical_0051/image_check_failures_bundles/global_time_series/global_time_series_1985-1995_results/
- Same thing.

Let's run the other tests now.

ls tests/integration/test_*.py
# tests/integration/test_bash_generation.py  tests/integration/test_defaults.py   tests/integration/test_weekly.py
# tests/integration/test_campaign.py         tests/integration/test_last_year.py
pytest tests/integration/test_bash_generation.py
# Expected errors (1 failed)
pytest tests/integration/test_campaign.py
# Expected errors (5 failed, 1 passed)
pytest tests/integration/test_defaults.py
# Expected errors (1 failed)
pytest tests/integration/test_last_year.py
# (1 passed)

Now, we can update expected results.

chmod u+x tests/integration/generated/update_*_chrysalis.sh
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# (1 passed)

# Remembered to copy over the old expected results!
cd /lcrc/group/e3sm/public_html
cp -r zppy_test_resources zppy_test_resources_previous/expected_results_until_20241101_but_with_updated_bash_generation
mv zppy_test_resources_previous/expected_results_until_20241101_but_with_updated_bash_generation zppy_test_resources_previous/expected_results_until_20241125_but_with_updated_bash_generation

cd ~/ez/zppy
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 1 failed, 5 passed
rm -rf /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
mkdir -p /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
mv test_campaign_high_res_v1_output/post/scripts/*.settings /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_high_res_v1_expected_files
pytest tests/integration/test_campaign.py
# 6 passed

./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed

./tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# 5 passed in 1222.49s (0:20:22)

All tests now pass. We can merge #646 and E3SM-Project/zppy-interfaces#7.

forsyth2 · 2024-12-14T01:05:12Z

forsyth2
Dec 14, 2024
Maintainer Author

Note that #520 is to automate the zppy weekly tests.

0 replies

forsyth2 · 2024-12-18T20:34:34Z

forsyth2
Dec 18, 2024
Maintainer Author

NOTE: The weekly tests' expected results were updated as part of the merging of #650 today (2024-12-18). All tests pass.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zppy weekly testing log #634

{{title}}

Replies: 6 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

zppy weekly testing log #634

forsyth2 Oct 18, 2024 Maintainer

Replies: 6 comments · 2 replies

forsyth2 Oct 18, 2024 Maintainer Author

Friday 2024-10-18

forsyth2 Oct 25, 2024 Maintainer Author

Friday 2024-10-25

forsyth2 Oct 25, 2024 Maintainer Author

forsyth2 Nov 1, 2024 Maintainer Author

Friday 2024-11-01

forsyth2 Nov 22, 2024 Maintainer Author

Friday 2024-11-22

Check commit history

Set up branch

Set up zppy-interfaces env

Set up e3sm_diags env

Set up zppy env

Unit tests for zppy-interfaces

Unit tests for zppy

Integration testing for zppy

forsyth2 Nov 26, 2024 Maintainer Author

Rerun on branch test-fixes

Unit tests for zppy-interfaces

Unit tests for zppy

Integration testing for zppy

forsyth2 Dec 14, 2024 Maintainer Author

forsyth2 Dec 18, 2024 Maintainer Author

forsyth2
Oct 18, 2024
Maintainer

Replies: 6 comments 2 replies

forsyth2
Oct 18, 2024
Maintainer Author

forsyth2
Oct 25, 2024
Maintainer Author

forsyth2 Oct 25, 2024
Maintainer Author

forsyth2
Nov 1, 2024
Maintainer Author

forsyth2
Nov 22, 2024
Maintainer Author

forsyth2 Nov 26, 2024
Maintainer Author

Rerun on branch `test-fixes`

forsyth2
Dec 14, 2024
Maintainer Author

forsyth2
Dec 18, 2024
Maintainer Author