Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extend functionality of run_jedi_exe.py #223

Closed
CoryMartin-NOAA opened this issue Dec 5, 2022 · 11 comments · Fixed by #311
Closed

Extend functionality of run_jedi_exe.py #223

CoryMartin-NOAA opened this issue Dec 5, 2022 · 11 comments · Fixed by #311
Assignees
@RussTreadon-NOAA
Milestone
GDASApp Testing S...

Comments

@CoryMartin-NOAA
Copy link
Contributor

Make run_jedi_exe script more generic/flexible to exercise more components
@CoryMartin-NOAA CoryMartin-NOAA added this to the GDASApp Testing Sprint milestone Dec 5, 2022
@CoryMartin-NOAA
Copy link
Contributor Author

Sounds like we are going to 'abandon' run_jedi_exe

@RussTreadon-NOAA
Copy link
Contributor

While we may abandon run_jedi_exe.py in the long term, I have changes which restore the 3dvar functionality. Branch feature/runjedi has been created to hold these updates. We can decide later if we want to open a PR to merge feature/runjedi into develop.

RussTreadon-NOAA added a commit that referenced this issue Jan 17, 2023
@RussTreadon-NOAA
Restore 3dvar capability to run_jedi_exe.py (#223)
c8e5f5e
@guillaumevernieres
Copy link
Contributor

While we may abandon run_jedi_exe.py in the long term, I have changes which restore the 3dvar functionality. Branch feature/runjedi has been created to hold these updates. We can decide later if we want to open a PR to merge feature/runjedi into develop.

I vote for merging your changes @RussTreadon-NOAA . I added a script that will run the soca jjobs here that takes as input an ordered list of j-jobs, but it will take some work to make it generic enough to run the other jedi applications.

RussTreadon-NOAA added a commit that referenced this issue Jan 17, 2023
@RussTreadon-NOAA
Correct python norm errors (#223)
3d3298a
RussTreadon-NOAA added a commit that referenced this issue Jan 17, 2023
@RussTreadon-NOAA
Remove yaml_template from 3dvar_orion.yaml (#223)
4528030
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Jan 18, 2023
Merged
RussTreadon-NOAA added a commit that referenced this issue Jan 18, 2023
@RussTreadon-NOAA
add genYAML.py; clean up run_jedi_exe.py (#223)
92d21cc
@RussTreadon-NOAA
Copy link
Contributor

92d21cc adds ush/ufsda/genYAML.py and updates run_jedi_exe.py to access genYAML via import.

ush/genYAML and ush/ufsda/genYAML.py are virtually identical. It does not make sense to retain both. Removal of ush/genYAML needs to be done with care. GDASApp dependencies on ush/genYAML need to be updated and tested. It is also necessary to update and test UFS-DA scripts in global-workflow which reference genYAML.

@CoryMartin-NOAA
Copy link
Contributor Author

@RussTreadon-NOAA I think just the last 5 lines of ush/genYAML + a new import line are all that are needed to keep downstream uses of genYAML satisfied without redundant lines of code. But this will need to be tested.

@RussTreadon-NOAA
Copy link
Contributor

OK, let me try. Thank you @CoryMartin-NOAA for your help in sorting this out.

RussTreadon-NOAA added a commit that referenced this issue Jan 18, 2023
@RussTreadon-NOAA
Update genYAML to use ufsda genYAML.py (#223)
d306591
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Jan 19, 2023
Closed
5 tasks
RussTreadon-NOAA added a commit that referenced this issue Jan 19, 2023
@RussTreadon-NOAA
add ufsda.genYAML to __init__.py (#223)
55f4217
RussTreadon-NOAA added a commit that referenced this issue Jan 19, 2023
@RussTreadon-NOAA
remove genYAML from yamltools.py (#223)
2cda39e
guillaumevernieres pushed a commit that referenced this issue Jan 24, 2023
@RussTreadon-NOAA
Enable run_jedi_exe.py to run 3dvar (#278)
f7c23af 
* Restore 3dvar capability to run_jedi_exe.py (#223)

* Correct python norm errors (#223)

* Remove yaml_template from 3dvar_orion.yaml (#223)

* add genYAML.py; clean up run_jedi_exe.py (#223)

* Update genYAML to use ufsda genYAML.py (#223)

* add ufsda.genYAML to __init__.py (#223)

* remove genYAML from yamltools.py (#223)

* remove __main__ from genYAML.py

* revert changes to ush/genYAML

* remove stage from init, simplify genYAML

* import specific ufsda.stage functions in run_jedi_exe.py

* replace exclusive with mem=0 in sbatch jobcard

* add ufsda.stage import to marine_analysis_prep script

* use RussTreadon-NOAA/global-workflow feature/ufsda_stage in automated tests

* Revert "use RussTreadon-NOAA/global-workflow feature/ufsda_stage in automated tests"

This reverts commit 3c828e2.

* update keywords in hera 3dvar yaml

* provide default for NMEM_ENKF in run_jedi_exe.py
@RussTreadon-NOAA
Copy link
Contributor

PR #278 restored the ability of run_jedi_exe.py to run 3dvar. feature/runjedi was deleted upon closure of PR #278.

Continued work in this issue will be done in feature/runexe.

@RussTreadon-NOAA RussTreadon-NOAA changed the title Add LETKF to run_jedi_exe Extend functionality of run_jedi_exe.py Jan 26, 2023
@RussTreadon-NOAA
Copy link
Contributor

Change title of issue to more accurately describe the scope of this issue.

@guillaumevernieres
Copy link
Contributor

Change title of issue to more accurately describe the scope of this issue.

@RussTreadon-NOAA , we should remove the part that I added a while back that runs the "ex_script", it isn't needed anymore.

@RussTreadon-NOAA
Copy link
Contributor

@guillaumevernieres , are you referring to the gw_scripts GDASApp mode?

./run_jedi_exe.py:    supported_app_modes = ['hofx', 'variational', 'gw_scripts']
./examples/run_scripts/soca1deg.yaml:GDASApp mode: gw_scripts
./examples/run_scripts/soca025.yaml:GDASApp mode: gw_scripts
./examples/run_scripts/soca5deg.yaml:GDASApp mode: gw_scripts

@guillaumevernieres
Copy link
Contributor

@guillaumevernieres , are you referring to the gw_scripts GDASApp mode?

./run_jedi_exe.py:    supported_app_modes = ['hofx', 'variational', 'gw_scripts']
./examples/run_scripts/soca1deg.yaml:GDASApp mode: gw_scripts
./examples/run_scripts/soca025.yaml:GDASApp mode: gw_scripts
./examples/run_scripts/soca5deg.yaml:GDASApp mode: gw_scripts

Yes, gw_scripts isn't needed anymore. What's under the else statement run_jedi_exe.py#L149 can be removed ... I think.

RussTreadon-NOAA added a commit that referenced this issue Jan 27, 2023
@RussTreadon-NOAA
clean up run_jedi_exe, extend genYAML for 4dhox, update 3d and 4dhofx…
3198057 
… example yamls (#223)
RussTreadon-NOAA added a commit that referenced this issue Feb 1, 2023
@RussTreadon-NOAA
Add GDASApp mode letkf to run_jedi_exe.py (#223)
0b70950
RussTreadon-NOAA added a commit that referenced this issue Feb 1, 2023
@RussTreadon-NOAA
Minor changes required by python norms (#223)
04d772a
RussTreadon-NOAA added a commit that referenced this issue Feb 1, 2023
@RussTreadon-NOAA
change keyword for background path, clean up prints (#223)
fab061e
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Feb 1, 2023
Merged
@RussTreadon-NOAA RussTreadon-NOAA linked a pull request Feb 1, 2023 that will close this issue
Extend run_jedi_exe.py to run various fv3-jedi apps #311
Merged
RussTreadon-NOAA added a commit that referenced this issue Feb 1, 2023
@RussTreadon-NOAA
remove extraneous prints (#223)
74a58e9
RussTreadon-NOAA added a commit that referenced this issue Feb 3, 2023
@RussTreadon-NOAA
Add run_jedi_exe ctests, add example letkf yaml for orion (#223)
4a42609
RussTreadon-NOAA added a commit that referenced this issue Feb 3, 2023
@RussTreadon-NOAA
address failures with automated run_jedi_exe ctests (#223)
0b5676c
RussTreadon-NOAA added a commit that referenced this issue Feb 3, 2023
@RussTreadon-NOAA
correct errors is gw-rt implementation of run_jedi_exe ctests (#223)
221a99d
RussTreadon-NOAA added a commit that referenced this issue Feb 3, 2023
@RussTreadon-NOAA
address permission issue with run_jedi_exe_letkf (#223)
1893d66
RussTreadon-NOAA added a commit that referenced this issue Feb 4, 2023
@RussTreadon-NOAA
temporary fix to address satbias_cov problem (#223)
70b7d65
RussTreadon-NOAA added a commit that referenced this issue Feb 4, 2023
@RussTreadon-NOAA
enhance run_jedi_exe ctest error checking and optimize job configurat…
7a75fb5 
…on (#223)
RussTreadon-NOAA added a commit that referenced this issue Feb 4, 2023
@RussTreadon-NOAA
fix orion jobcard and path problems (#223)
9029966
RussTreadon-NOAA added a commit that referenced this issue Feb 6, 2023
@RussTreadon-NOAA
streamline run_jedi_exe ctests (#223)
9dbc973
RussTreadon-NOAA added a commit that referenced this issue Feb 7, 2023
@RussTreadon-NOAA
preliminary removal of CDATE and GDATE (#223)
b349d32
RussTreadon-NOAA added a commit that referenced this issue Feb 7, 2023
@RussTreadon-NOAA
remove oldcommit check from ci drivers (#223)
59bd649
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels
None yet
Projects
None yet
Milestone
GDASApp Testing Sprint
Development

Successfully merging a pull request may close this issue.

Extend run_jedi_exe.py to run various fv3-jedi apps NOAA-EMC/GDASApp
3 participants
@CoryMartin-NOAA @guillaumevernieres @RussTreadon-NOAA