Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zppy-interfaces refactor #642

Merged
merged 13 commits into from
Nov 22, 2024
Merged

zppy-interfaces refactor #642

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
33d0691
zppy-interfaces refactor
forsyth2 Nov 5, 2024
ceddedc
Better organize dirs
forsyth2 Nov 5, 2024
7e0339b
Update
forsyth2 Nov 5, 2024
70c07fb
Trying to run zi-global-time-series
forsyth2 Nov 5, 2024
106919b
zppy calling zi works
forsyth2 Nov 21, 2024
8a6a76a
Plots generating via zppy
forsyth2 Nov 21, 2024
506416d
Updates
forsyth2 Nov 21, 2024
23d8f75
Unit tests passing
forsyth2 Nov 21, 2024
01b8c21
Unit tests with pytest
forsyth2 Nov 22, 2024
70db840
Add logger
forsyth2 Nov 22, 2024
cc533fe
Update
forsyth2 Nov 22, 2024
be06191
Add lat_lon_conus cfg to inclusions
forsyth2 Nov 22, 2024
dbd74f3
Exclude explicit conda paths and names
forsyth2 Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build_workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ jobs:
# Does not run the integration tests, which require server access
- name: Run Unit Tests
run: |
python -m unittest tests/test_*.py
pytest tests/test_*.py

# If the branch updates documentation, then the docs will need to be updated.
publish-docs:
Expand Down
1 change: 0 additions & 1 deletion .vscode/settings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,4 @@
"python.linting.flake8Enabled": true,
"python.linting.flake8Args": ["--config=setup.cfg"],
"python.linting.mypyEnabled": true,
"python.pythonPath": "/opt/miniconda3/envs/zstash_dev/bin/python"
}
35 changes: 23 additions & 12 deletions conda/dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,32 @@ channels:
- conda-forge
- defaults
dependencies:
# Base
# =================
# Build
# =======================
- python=3.9.13
- pip=22.2.2
# Base
# =================
- configobj=5.0.6
- jinja2=3.1.2
- mache>=1.5.0
- mpas_tools>=0.15.0
- pillow=9.2.0
# Developer Tools
# Testing
# =======================
- pytest
- pytest-cov
# Documentation
# If versions are updated, also update in `.github/workflows/build_workflow.yml`
# =================
- sphinx=5.2.3
- sphinx_rtd_theme=1.0.0
- sphinx-multiversion=0.2.4
# Need to pin docutils because 0.17 has a bug with unordered lists
# https://github.com/readthedocs/sphinx_rtd_theme/issues/1115
- docutils=0.16
# Quality Assurance Tools
# =======================
# If versions are updated, also update 'rev' in `.pre-commit-config.yaml`
- black=24.10.0 # version from https://anaconda.org/conda-forge/black
- flake8=7.1.1 # version from https://anaconda.org/conda-forge/flake8
Expand All @@ -22,13 +37,9 @@ dependencies:
- mypy=1.11.2 # version from https://anaconda.org/conda-forge/mypy
- pre-commit=4.0.1 # version from https://anaconda.org/conda-forge/pre-commit
- tbump=6.9.0
# Documentation
# If versions are updated, also update in `.github/workflows/build_workflow.yml`
# =================
- sphinx=5.2.3
- sphinx-multiversion=0.2.4
- sphinx_rtd_theme=1.0.0
# Need to pin docutils because 0.17 has a bug with unordered lists
# https://github.com/readthedocs/sphinx_rtd_theme/issues/1115
- docutils=0.16
# Developer Tools
# =======================
- tbump=6.9.0
- ipykernel

prefix: /opt/miniconda3/envs/zppy_dev
2 changes: 2 additions & 0 deletions setup.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ ignore =
E501
# line break before operator
W503
# comparison to False should be 'if cond is False:' or 'if not cond:'
E712
# Max width of Github code review is 119 characters
max-line-length = 119
max-complexity = 18
Expand Down
1 change: 1 addition & 0 deletions tests/integration/generated/test_min_case_global_time_series_custom_chrysalis.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ years = "1985:1995:5",

[global_time_series]
active = True
environment_commands = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zppy_interfaces_dev_pr_1_20241121v4"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably, this will be removed once there are shared environments that support zppy-interfaces?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, realistically this will remain because we'll always want to test zppy with the latest zppy-interfaces code, not whatever the last release was. But yes, for cfgs written by users they could rely on Unified having zppy-interfaces in it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem I see is that this environment is only on Chrysalis (or Anvil). What if you want to test somewhere else?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or is this just a placeholder?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's the responsibility of tests/integration/utils.py. This file is auto-generated by that one. I currently only have a Chrysalis environment set up, so it's the only one defined.
Screenshot 2024-11-22 at 11 27 46 AM

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think it is a little odd to have machine and user specific path here, but don't know how to get around.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well we could use $USER to get my name out of it but ultimately whoever runs this cfg will need to create their own conda environment for the latest developments or else use Unified.

I think maybe the best solution is to always keep the environment_commands lines on the main branch filled in as <path to YOUR conda>; conda activate <environment YOU create>.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, the commit dbd74f3 implements the idea I mentioned in the previous comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like it would be better to have some examples somewhere but not to update them very often. This is the approach we use in MPAS-Analysis.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with that is I fear they will very quickly go out-of-date. This approach (the auto-generation of scripts from templates) updates everything together.

experiment_name = "v3.LR.historical_0051"
figstr = "v3.LR.historical_0051"
plots_original=""
Expand Down
1 change: 1 addition & 0 deletions tests/integration/generated/test_min_case_global_time_series_original_8_chrysalis.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ walltime = "00:30:00"
[global_time_series]
active = True
climo_years = "1985-1989", "1990-1995",
environment_commands = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zppy_interfaces_dev_pr_1_20241121v4"
experiment_name = "v3.LR.historical_0051"
figstr = "v3.LR.historical_0051"
moc_file=mocTimeSeries_1985-1995.nc
Expand Down
1 change: 1 addition & 0 deletions tests/integration/generated/test_min_case_global_time_series_original_8_no_ocn_chrysalis.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ walltime = "00:30:00"

[global_time_series]
active = True
environment_commands = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zppy_interfaces_dev_pr_1_20241121v4"
experiment_name = "v3.LR.historical_0051"
figstr = "v3.LR.historical_0051"
plots_original="net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,net_atm_water_imbalance"
Expand Down
1 change: 1 addition & 0 deletions tests/integration/generated/test_weekly_bundles_chrysalis.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ years = "1985:1989:2",
[global_time_series]
active = True
bundle = "bundle2"
environment_commands = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zppy_interfaces_dev_pr_1_20241121v4"
experiment_name = "v3.LR.historical_0051"
figstr = "v3.LR.historical_0051"
plots_original="net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,net_atm_water_imbalance"
Expand Down
1 change: 1 addition & 0 deletions tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ walltime = "00:30:00"
[global_time_series]
active = True
climo_years ="1980-1984", "1985-1990",
environment_commands = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zppy_interfaces_dev_pr_1_20241121v4"
experiment_name = "v2.LR.historical_0201"
figstr = "v2.LR.historical_0201"
moc_file=mocTimeSeries_1980-1990.nc
Expand Down
1 change: 1 addition & 0 deletions tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ walltime = "00:30:00"
[global_time_series]
active = True
climo_years = "1985-1989", "1990-1995",
environment_commands = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zppy_interfaces_dev_pr_1_20241121v4"
experiment_name = "v3.LR.historical_0051"
figstr = "v3.LR.historical_0051"
moc_file=mocTimeSeries_1985-1995.nc
Expand Down
1 change: 1 addition & 0 deletions tests/integration/template_min_case_global_time_series_custom.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ years = "1985:1995:5",

[global_time_series]
active = True
environment_commands = "#expand global_time_series_environment_commands#"
experiment_name = "#expand case_name#"
figstr = "#expand case_name#"
plots_original=""
Expand Down
1 change: 1 addition & 0 deletions tests/integration/template_min_case_global_time_series_original_8.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ walltime = "#expand mpas_analysis_walltime#"
[global_time_series]
active = True
climo_years = "1985-1989", "1990-1995",
environment_commands = "#expand global_time_series_environment_commands#"
experiment_name = "#expand case_name#"
figstr = "#expand case_name#"
moc_file=mocTimeSeries_1985-1995.nc
Expand Down
1 change: 1 addition & 0 deletions tests/integration/template_min_case_global_time_series_original_8_no_ocn.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ walltime = "00:30:00"

[global_time_series]
active = True
environment_commands = "#expand global_time_series_environment_commands#"
experiment_name = "#expand case_name#"
figstr = "#expand case_name#"
plots_original="net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,net_atm_water_imbalance"
Expand Down
1 change: 1 addition & 0 deletions tests/integration/template_weekly_bundles.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ years = "1985:1989:2",
[global_time_series]
active = True
bundle = "bundle2"
environment_commands = "#expand global_time_series_environment_commands#"
experiment_name = "#expand case_name#"
figstr = "#expand case_name#"
plots_original="net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,net_atm_water_imbalance"
Expand Down
1 change: 1 addition & 0 deletions tests/integration/template_weekly_comprehensive_v2.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ walltime = "#expand mpas_analysis_walltime#"
[global_time_series]
active = True
climo_years ="1980-1984", "1985-1990",
environment_commands = "#expand global_time_series_environment_commands#"
experiment_name = "#expand case_name_v2#"
figstr = "#expand case_name_v2#"
moc_file=mocTimeSeries_1980-1990.nc
Expand Down
1 change: 1 addition & 0 deletions tests/integration/template_weekly_comprehensive_v3.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ walltime = "#expand mpas_analysis_walltime#"
[global_time_series]
active = True
climo_years = "1985-1989", "1990-1995",
environment_commands = "#expand global_time_series_environment_commands#"
experiment_name = "#expand case_name#"
figstr = "#expand case_name#"
moc_file=mocTimeSeries_1985-1995.nc
Expand Down
3 changes: 3 additions & 0 deletions tests/integration/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ def get_chyrsalis_expansions(config):
"e3sm_to_cmip_environment_commands": "",
"environment_commands_test": "",
"expected_dir": "/lcrc/group/e3sm/public_html/zppy_test_resources/",
"global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zppy_interfaces_dev_pr_1_20241121v4",
"mpas_analysis_walltime": "00:30:00",
"partition_long": "compute",
"partition_short": "debug",
Expand Down Expand Up @@ -182,6 +183,7 @@ def get_compy_expansions(config):
"e3sm_to_cmip_environment_commands": "",
"environment_commands_test": "",
"expected_dir": "/compyfs/www/zppy_test_resources/",
"global_time_series_environment_commands": "",
"mpas_analysis_walltime": "00:30:00",
"partition_long": "slurm",
"partition_short": "short",
Expand Down Expand Up @@ -211,6 +213,7 @@ def get_perlmutter_expansions(config):
"e3sm_to_cmip_environment_commands": "",
"environment_commands_test": "",
"expected_dir": "/global/cfs/cdirs/e3sm/www/zppy_test_resources/",
"global_time_series_environment_commands": "",
"mpas_analysis_walltime": "01:00:00",
"partition_long": "",
"partition_short": "",
Expand Down
Loading
Loading