Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add additional GEFS variables needed for AI/ML applications #3221

Merged
merged 10 commits into from
Jan 16, 2025
Merged

Add additional GEFS variables needed for AI/ML applications #3221

merged 10 commits into from
Jan 16, 2025

Conversation

EricSinsky-NOAA
Copy link
Contributor

@EricSinsky-NOAA EricSinsky-NOAA commented Jan 10, 2025 •
edited
Loading

Description

This PR covers a few items that were merged to the feature/gefs_reforecast branch and should be brought to the develop branch:

  • Add additional GEFS variables needed for AI/ML applications
  • Add umax, vmax and wndmax as requested by CPC
  • Add ensemble header information for atmosphere products
  • Remove duplicate OCNSPPT and EPBL settings

With the additional variables, the data size has increased from 7.41GB to 7.96GB for a given member in atmos products for the C48_S2SWA_gefs CI case.

Resolves #3205
Refs #3183

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

  • This is being cloned, built and run on WCOSS2.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@EricSinsky-NOAA
Remove duplicate OCNSPPT and EPBL settings
0e11790
@EricSinsky-NOAA
Add ensemble header information
2c020a9
@EricSinsky-NOAA
Add umax/vmax/wndmax to diag tables
0ce87ae 
The umax, vmax and wndmax variables have been added to the diag tables. Also maxuw has been recognized as components of a vector variable in exglobal_atmos_products.sh.
@EricSinsky-NOAA
Add umax/vmax/wndmax to gefs parm file
88e0b6c
@EricSinsky-NOAA
Add AI/ML related variables to GEFS parm file
e965969
aerorahul
aerorahul reviewed Jan 10, 2025
View reviewed changes
ush/forecast_predet.sh Outdated Show resolved Hide resolved
aerorahul
aerorahul reviewed Jan 10, 2025
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me.
just one comment in the review.

@EricSinsky-NOAA
Copy link
Contributor Author

Marking as ready for review. Also, added to the PR description how much more space is required for additional variables.

@EricSinsky-NOAA EricSinsky-NOAA marked this pull request as ready for review January 15, 2025 12:55
aerorahul
aerorahul approved these changes Jan 15, 2025
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

WenMeng-NOAA
WenMeng-NOAA approved these changes Jan 15, 2025
View reviewed changes
Copy link
Contributor

@WenMeng-NOAA WenMeng-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good for me.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Jan 16, 2025
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Jan 16, 2025
@emcbot
Copy link

emcbot commented Jan 16, 2025

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3221


Experiment C48_ATM_65bff4b5 Completed 2 Cycles: *SUCCESS* at Thu Jan 16 04:40:54 CST 2025
Experiment C48mx500_3DVarAOWCDA_65bff4b5 Completed 2 Cycles: *SUCCESS* at Thu Jan 16 04:47:09 CST 2025
Experiment C48mx500_hybAOWCDA_65bff4b5 Completed 2 Cycles: *SUCCESS* at Thu Jan 16 04:53:13 CST 2025
Experiment C96_S2SWA_gefs_replay_ics_65bff4b5 Completed 1 Cycles: *SUCCESS* at Thu Jan 16 05:17:26 CST 2025
Experiment C96_atm3DVar_65bff4b5 Completed 3 Cycles: *SUCCESS* at Thu Jan 16 05:47:46 CST 2025
Experiment C96C48_hybatmDA_65bff4b5 Completed 3 Cycles: *SUCCESS* at Thu Jan 16 05:47:51 CST 2025
Experiment C48_S2SW_65bff4b5 Completed 2 Cycles: *SUCCESS* at Thu Jan 16 06:11:50 CST 2025
Experiment C48_S2SWA_gefs_65bff4b5 Completed 1 Cycles: *SUCCESS* at Thu Jan 16 06:12:17 CST 2025

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit eb10ee8 into NOAA-EMC:develop Jan 16, 2025
10 of 11 checks passed
@EricSinsky-NOAA EricSinsky-NOAA deleted the feature/addgefsvars branch January 16, 2025 18:49
tsga added a commit to tsga/global-workflow that referenced this pull request Jan 22, 2025
@tsga
Merge branch 'develop' into feature/lnd_regrid_iau
dbebb2e 
* develop:
  Add echgres as a dependency only for RUN=enkfgdas, not enkfgfs (NOAA-EMC#3246)
  Add domain level to wave gridded COM path (NOAA-EMC#3137)
  CI JJOB Tests using CMake (NOAA-EMC#3214)
  Make assorted updates to waves (NOAA-EMC#3190)
  Move WCOSS2 LD_LIBRARY_PATH patches to load_ufsda_modules.sh (NOAA-EMC#3236)
  Adding a gefs_arch task to GEFS workflow (NOAA-EMC#3211)
  Add additional GEFS variables needed for AI/ML applications  (NOAA-EMC#3221)
  Add bmat task dependency to marine LETKF task (NOAA-EMC#3224)
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
  Separate use of initial increment/perturbation file from REPLAY/+03 ICs  (NOAA-EMC#3119)
  Update gsi_enkf hash and gsi_ver (NOAA-EMC#3207)
  Remove cpus-per-task from APRUN_OCNANALECEN on WCOSS2 (NOAA-EMC#3212)
  Remove 5WAVH from AWIPS GRIB2 parm files (NOAA-EMC#3146)
  Remove multi-grid wave support (NOAA-EMC#3188)
  Add echgres as a dependency for earc (NOAA-EMC#3202)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WenMeng-NOAA WenMeng-NOAA WenMeng-NOAA approved these changes

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@aerorahul aerorahul aerorahul approved these changes

@junwang-noaa junwang-noaa Awaiting requested review from junwang-noaa junwang-noaa is a code owner

Assignees
No one assigned
Labels
CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

New GEFS variables requested by CPC and PSL need to be added to atmos_prod
5 participants
@EricSinsky-NOAA @emcbot @aerorahul @WalterKolczynski-NOAA @WenMeng-NOAA