Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add BUFR2IODA python API converter to prepoceanobs task #914

Merged
merged 46 commits into from
Mar 28, 2024
Merged

Add BUFR2IODA python API converter to prepoceanobs task #914

merged 46 commits into from
Mar 28, 2024

Conversation

ShastriPaturi
Copy link
Collaborator

This PR adds BUFR2IODA python API converter to prepoceanobs task and does the following:

  1. Subsampled monthly bathy, tesac and trkob in situ BUFR files are tested.
  2. The subsampled files have been renamed (from .201804.dcom_subsampled) as .tz..tm00.bufr_d
  3. Runs for one cycle: 2018041512.

closes #892

Overlapping windows application will be added in a subsequent PR

AndrewEichmann-NOAA
AndrewEichmann-NOAA reviewed Feb 8, 2024
View reviewed changes
scripts/exglobal_prep_ocean_obs.py Outdated Show resolved Hide resolved
parm/soca/obsprep/obsprep_config.yaml Outdated Show resolved Hide resolved
scripts/exglobal_prep_ocean_obs.py Outdated Show resolved Hide resolved
@ShastriPaturi ShastriPaturi mentioned this pull request Feb 15, 2024
Closed
AndrewEichmann-NOAA
AndrewEichmann-NOAA reviewed Feb 20, 2024
View reviewed changes
Copy link
Collaborator

@AndrewEichmann-NOAA AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the additions to ush/ioda - I guess the bufr2ioda converters - already exist in a different repo? I wonder if we can import them from there

scripts/exglobal_prep_ocean_obs.py Outdated Show resolved Hide resolved
scripts/exglobal_prep_ocean_obs.py Show resolved Hide resolved
@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Mar 26, 2024
@AndrewEichmann-NOAA
Copy link
Collaborator

I started the CI forgetting about the outstanding g-w build issue on Rocky 8 - gdasapp at least builds:

build_gfs_utils.sh completed successfully!
build_ufs_utils.sh failed with status 2!
build_ww3prepost.sh failed with status 2!
build_gsi_monitor.sh completed successfully!
build_ufs.sh failed with status 2!
build_gsi_utils.sh completed successfully!
build_upp.sh failed with status 2!
build_gdas.sh completed successfully!
BUILD ERROR: One or more components failed to build
  Check the associated build log(s) for details.

@AndrewEichmann-NOAA
Copy link
Collaborator

AndrewEichmann-NOAA commented Mar 27, 2024 •
edited
Loading

@guillaumevernieres @CoryMartin-NOAA @ShastriPaturi The branch as is builds and completes all the global-workflow ctests (besides vrfy) on Hera while delivering populated in-situ obs files to the ROTDIR. I've changed the ctests so that this can run without the associated PR to global-workflow.

@guillaumevernieres guillaumevernieres self-requested a review March 28, 2024 13:16
@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Failed Automated testing with global-workflow failed on Orion labels Mar 28, 2024
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Mar 28, 2024
@emcbot
Copy link

emcbot commented Mar 28, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Thu Mar 28 14:00:35 UTC 2024 on hfe12
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Thu Mar 28 14:51:22 UTC 2024
Build: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/914/global-workflow/sorc/log.build

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Mar 28, 2024
@RussTreadon-NOAA
Copy link
Contributor

Hera GW-RT cloned g-w develop at f0b912be. This is an older hash. We should clone g-w develop at d6be3b5. Unfortunately even with this change GW-RT will still fail during the build step. upp.fd still does not build on Hera Rocky-8.

Here's the build log from g-w develop at d6be3b5

Running "module reset". Resetting modules to system default. The following $MODULEPATH directories have been removed: None
Building gsi_enkf, ufs, gfs_utils, gdas, ww3prepost, ufs_utils, gsi_utils, gsi_monitor, upp
Starting build_gsi_enkf.sh
Starting build_ufs.sh
Starting build_gfs_utils.sh
Starting build_gdas.sh
Starting build_ww3prepost.sh
Starting build_ufs_utils.sh
Starting build_gsi_utils.sh
Starting build_gsi_monitor.sh
Starting build_upp.sh
build_gsi_enkf.sh completed successfully!
build_gfs_utils.sh completed successfully!
build_ufs_utils.sh completed successfully!
build_gsi_utils.sh completed successfully!
build_gsi_monitor.sh completed successfully!
build_ww3prepost.sh completed successfully!
build_upp.sh failed with status 2!
build_ufs.sh completed successfully!
build_gdas.sh completed successfully!
BUILD ERROR: One or more components failed to build
  Check the associated build log(s) for details.

@emcbot
Copy link

emcbot commented Mar 28, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Thu Mar 28 09:01:07 CDT 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Mar 28 10:07:59 CDT 2024
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Thu Mar 28 10:38:24 CDT 2024
Tests: 100% tests passed, 0 tests failed out of 53

@emcbot emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Mar 28, 2024
guillaumevernieres
guillaumevernieres approved these changes Mar 28, 2024
View reviewed changes
Copy link
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@guillaumevernieres guillaumevernieres merged commit 33b4cb9 into develop Mar 28, 2024
7 checks passed
danholdaway added a commit that referenced this pull request Apr 8, 2024
@danholdaway
Merge remote-tracking branch 'origin/develop' into feature/use_jcb_atm
b256b0f 
* origin/develop:
  Use <filesystem> on a non c++17 supported machine (WCOSS ACORN) (#1026)
  Change generate_com to declare_from_tmpl (#1025)
  Commenting out more of the marine bufr 2 ioda stuff (#1018)
  make driver consistent with workflow driver (#1016)
  Update hashes now that GSI-B is working for EnVar (#1015)
  Add GitHub CLI to path for CI (#1014)
  Use _anl rather than _ges dimensions for increments in FV3 increment converter YAML (#1013)
  Fix inconsistent VIIRS preprocessing test (#1012)
  remove gdas_ prefix from executable filename in test_gdasapp_fv3jedi_fv3inc (#1010)
  Bugfix on Broken GHRSST Ioda Converter (#1004)
  Moved the marine converters to a "safe" place (#1007)
  restore ATM local ensemble ctest functionality (#1003)
  Add BUFR2IODA python API converter to prepoceanobs task (#914)
  Remove sst's from obs proc (#1001)
  JEDI increment write to cubed sphere history (#983)
  [End- to End Test code sprint] Add SEVIRI METEOSAT-8 and METEOSAT-11 to end-to-end testing (#766)
@guillaumevernieres guillaumevernieres deleted the feature/marine_addBUFR2IODAconverter branch April 16, 2024 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guillaumevernieres guillaumevernieres guillaumevernieres approved these changes

@apchoiCMD apchoiCMD Awaiting requested review from apchoiCMD

@AndrewEichmann-NOAA AndrewEichmann-NOAA Awaiting requested review from AndrewEichmann-NOAA

@CoryMartin-NOAA CoryMartin-NOAA Awaiting requested review from CoryMartin-NOAA

Assignees
No one assigned
Labels
hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Passed Automated testing with global-workflow successful on Orion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Make BUFRI2IODA conversion part of ocean prep obs task
6 participants
@ShastriPaturi @emcbot @AndrewEichmann-NOAA @RussTreadon-NOAA @CoryMartin-NOAA @guillaumevernieres