Merge remote-tracking branch 'origin/develop' into feature/version-files

* origin/develop:
  Fix RTD python requirements path
  Clean up RTD python install settings
  Use RTD python system packages
  Fix python RTD parameter name
  Update RTD python install
  Restore RTD python version and add requirements
  Revert python version for RTD (NOAA-EMC#1598)
  Move RTD config to root of repo (NOAA-EMC#1597)
  Add RTD config (NOAA-EMC#1596)
  Add `schema` library to manage schema for variety of input configurations. (NOAA-EMC#1567)
  Add S2SA to the allowed app list in setup expt (NOAA-EMC#1591)
  Avoid parsing group name when checking RadMon diagnostic files (NOAA-EMC#1559)
  Remove gldas from global-workflow (NOAA-EMC#1590)
  Add GEFS capability to setup_expt (NOAA-EMC#1582)
  Redo v16.3 GSI script updates to scripts/exglobal_atmos_analysis.sh (NOAA-EMC#1535)
  Fix bugs in the COM refactor of marine DA (NOAA-EMC#1566)
  Retire `getic.sh` and `init.sh` jobs from global-workflow (NOAA-EMC#1578)
  New UFS_UTILS hash for gdas_init COM reorg updates (NOAA-EMC#1581)
  Update documentation for PR standards (NOAA-EMC#1573)
  Update prepobs install copy for atm log filename fix (NOAA-EMC#1571)
  Update LandDA related job post COM refactor plus other minor fixes (NOAA-EMC#1564)
  Replace `CDATE` with `PDYcyc` combo (NOAA-EMC#1561)
  Update init.rst cycled coupled IC info (NOAA-EMC#1562)
  Use correct number of tasks in CICE when using threads (NOAA-EMC#1558)
  Remove gdas bump fix files (NOAA-EMC#1553)
  Remove remnant WAVE_RUN from archive (NOAA-EMC#1556)
  Update UFS to develop as of 2023 Apr 17 (NOAA-EMC#1509)
  Adapt the marine DA to the new COM structure (NOAA-EMC#1554)
  Update aerosol DA to use new COM structure (NOAA-EMC#1551)
  Update UFSDA ATM ens for new COM directory structure (NOAA-EMC#1538)
  Run an ensemble forecast of the coupled model (NOAA-EMC#1545)
  Fix ocean anl path in staging (NOAA-EMC#1544)
  Two minor bugfixes (NOAA-EMC#1542)
  Move guts of ocean analysis post out of j-job (NOAA-EMC#1539)
  Update UFSDA ATM anl for new COM directory structure (NOAA-EMC#1537)
  Fix incorrect ocean history tmpl during workflow generation (NOAA-EMC#1533)
  Fix ocean stating from flat structure (NOAA-EMC#1531)
  Fix warm-start IC staging (NOAA-EMC#1529)
  Quiet generate_com (NOAA-EMC#1526)
  Add ocnanlvrfy job and bugfixes. (NOAA-EMC#1514)
  Reorganize COM and refactor to use templates (NOAA-EMC#1421)
  Move GDASApp hash to stable version (NOAA-EMC#1508)
  update GDASApp hash to d34f616 (NOAA-EMC#1505) (NOAA-EMC#1506)
  ignore archiving sfluxgrbf00[124578] files when they are not present (NOAA-EMC#1498)
  Add initial land DA cycling scripts (NOAA-EMC#1351)
  Add CI cron jobs (NOAA-EMC#1476)
  Refactor UFS-DA ATM ens component to use python g-w (NOAA-EMC#1373)
  Added Fit2Obs to S4. NOAA-EMC#1489 (NOAA-EMC#1497)
  Add new task to post-process marine DA (NOAA-EMC#1485)
  Create fix file issue template (NOAA-EMC#1495)
  Update TC_tracker version to v1.1.15.6
  Update GSI-Monitor hash to reflect recent assimilation changes.
  Fit2Obs updates for package reorganization and invocation via module (NOAA-EMC#1484)
  Refactor UFS-DA ATM var component to use python g-w (NOAA-EMC#1372)
  Port the global workflow to Jet (NOAA-EMC#1301)
  Consolidate wave parm files (NOAA-EMC#1477)
  Split MPMD stdout into tasks on slurm (NOAA-EMC#1469)
  GFS fixed-file YAMLs. (NOAA-EMC#1471)

.github/ISSUE_TEMPLATE/fix_file.md

@@ -0,0 +1,24 @@
+---
+name: Fix File Update
+about: Use this template for adding, updating, or removing fix files from global dataset
+title:
+labels: Fix Files
+assignees:
+  - KateFriedman-NOAA
+  - WalterKolczynski-NOAA
+
+---
+
+**Description**
+<!-- Provide a consise description of the fix file update being requested. -->
+<!-- State if this is adding, updating, and/or removing fix files from dataset. -->
+<!-- Include related issues in component repositories. -->
+<!-- Are there related workflow or component changes to use the affected fix files? -->
+<!-- Provide pickup locations for new/altered fix files or list files to be removed. -->
+
+**Tasks**
+<!-- Update task list as needed for request. -->
+- [ ] Discuss needs with global-workflow developer assigned to request.
+- [ ] Add/update/remove fix file(s) in fix sets on supported platforms (global-workflow assignee task).
+- [ ] Update "Fix File Management" spreadsheet (https://docs.google.com/spreadsheets/d/1BeIvcz6TO3If4YCqkUK-oz_kGS9q2wTjwLS-BBemSEY/edit?usp=sharing).
+- [ ] Make related workflow/component updates.

.gitignore

@@ -129,6 +129,7 @@ ush/global_chgres.sh
 ush/global_chgres_driver.sh
 ush/global_cycle.sh
 ush/global_cycle_driver.sh
+ush/jediinc2fv3.py
 ush/mkwfsgbl.sh
 ush/ufsda
 ush/wafs_blending.sh

.readthedocs.yaml

@@ -0,0 +1,21 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/source/conf.py
+
+python:
+  install:
+    - requirements: docs/requirements.txt
+  system_packages: true

.shellcheckrc

@@ -11,3 +11,6 @@ disable=SC1090
 
 # Disable non-existent binary
 disable=SC1091
+
+# Disable -p -m only applies to deepest directory
+disable=SC2174

Externals.cfg

@@ -1,7 +1,7 @@
 # External sub-modules of global-workflow
 
 [UFS]
-tag = c22aaad
+tag = 2247060
 local_path = sorc/ufs_model.fd
 repo_url = https://github.com/ufs-community/ufs-weather-model.git
 protocol = git
@@ -15,7 +15,7 @@ protocol = git
 required = True
 
 [UFS-Utils]
-hash = 5b67e4d
+hash = 72a0471
 local_path = sorc/ufs_utils.fd
 repo_url = https://github.com/ufs-community/UFS_UTILS.git
 protocol = git
@@ -43,26 +43,19 @@ protocol = git
 required = False
 
 [GSI-Monitor]
-hash = 8cf16de
+hash = 45783e3
 local_path = sorc/gsi_monitor.fd
 repo_url = https://github.com/NOAA-EMC/GSI-monitor.git
 protocol = git
 required = False
 
 [GDASApp]
-hash = db2f998
+hash = aaf7caa
 local_path = sorc/gdas.cd
 repo_url = https://github.com/NOAA-EMC/GDASApp.git
 protocol = git
 required = False
 
-[GLDAS]
-tag = fd8ba62
-local_path = sorc/gldas.fd
-repo_url = https://github.com/NOAA-EMC/GLDAS.git
-protocol = git
-required = False
-
 [EMC-gfs_wafs]
 hash = 014a0b8
 local_path = sorc/gfs_wafs.fd

ci/cases/C96C48_hybatmDA.yaml

@@ -0,0 +1,15 @@
+experiment:
+  mode: cycled
+
+arguments:
+  app: ATM
+  resdet: 96
+  resens: 48
+  comrot: ${RUNTESTS}/${pslot}/COMROT
+  expdir: ${RUNTESTS}/${pslot}/EXPDIR
+  icsdir: ${ICSDIR_ROOT}/C96C48
+  idate: 2021122018
+  edate: 2021122200
+  nens: 2
+  gfs_cyc: 1
+  start: cold

ci/cases/C96_atm3DVar.yaml

@@ -0,0 +1,14 @@
+experiment:
+  mode: cycled
+
+arguments:
+  app: ATM
+  resdet: 96
+  comrot: ${RUNTESTS}/${pslot}/COMROT
+  expdir: ${RUNTESTS}/${pslot}/EXPDIR
+  icsdir: ${ICSDIR_ROOT}/C96C48
+  idate: 2021122018
+  edate: 2021122100
+  nens: 0
+  gfs_cyc: 1
+  start: cold

ci/environments/hera.sh → ci/platforms/hera.sh

@@ -1,10 +1,7 @@
 #!/usr/bin/bash
 export GFS_CI_ROOT=/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT
-export GFS_MODULE_USE="${GFS_CI_ROOT}/global-workflow/modulefiles"
 export SLURM_ACCOUNT=fv3-cpu
 export SALLOC_ACCOUNT="${SLURM_ACCOUNT}"
 export SBATCH_ACCOUNT="${SLURM_ACCOUNT}"
 export SLURM_QOS=debug
-export repo_url="https://github.com/NOAA-EMC/global-workflow.git"
-#export repo_url="https://github.com/TerrenceMcGuinness-NOAA/global-workflow.git"
 export ICSDIR_ROOT="/scratch1/NCEPDEV/global/glopara/data/ICSDIR"

ci/environments/orion.sh → ci/platforms/orion.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/bash
 
-export GFS_CI_ROOT="TDB" #TODO
-export GFS_MODULE_USE="${GFS_CI_ROOT}/global-workflow/modulefiles"
+export GFS_CI_ROOT=/work2/noaa/global/mterry/GFS_CI_ROOT
+export ICSDIR_ROOT=/work/noaa/global/glopara/data/ICSDIR
 export SLURM_ACCOUNT=fv3-cpu
 export SALLOC_ACCOUNT=${SLURM_ACCOUNT}
 export SBATCH_ACCOUNT=${SLURM_ACCOUNT}

ci/scripts/check_ci.sh

@@ -0,0 +1,115 @@
+#!/bin/bash
+set -eux
+#####################################################################################
+#
+# Script description: BASH script for checking for cases in a given PR and 
+#                     running rocotostat on each to determine if the experiment has
+#                     succeeded or faild.  This script is intended
+#                     to run from within a cron job in the CI Managers account
+# Abstract TODO
+#####################################################################################
+
+HOMEgfs="$(cd "$(dirname  "${BASH_SOURCE[0]}")/../.." >/dev/null 2>&1 && pwd )"
+scriptname=$(basename "${BASH_SOURCE[0]}")
+echo "Begin ${scriptname} at $(date -u)" || true
+export PS4='+ $(basename ${BASH_SOURCE})[${LINENO}]'
+
+GH=${HOME}/bin/gh
+REPO_URL=${REPO_URL:-"https://github.com/NOAA-EMC/global-workflow.git"}
+
+#########################################################################
+#  Set up runtime environment varibles for accounts on supproted machines
+#########################################################################
+
+source "${HOMEgfs}/ush/detect_machine.sh"
+case ${MACHINE_ID} in
+  hera | orion)
+   echo "Running Automated Testing on ${MACHINE_ID}"
+   source "${HOMEgfs}/ci/platforms/${MACHINE_ID}.sh"
+   ;;
+ *)
+   echo "Unsupported platform. Exiting with error."
+   exit 1
+   ;;
+esac
+set +x
+source "${HOMEgfs}/ush/module-setup.sh"
+module use "${HOMEgfs}/modulefiles"
+module load "module_gwsetup.${MACHINE_ID}"
+module list
+set -x
+rocotostat=$(which rocotostat)
+if [[ -z ${rocotostat+x} ]]; then
+  echo "rocotostat not found on system"
+  exit 1
+else
+  echo "rocotostat being used from ${rocotostat}"
+fi
+
+pr_list_file="open_pr_list"
+
+if [[ -s "${GFS_CI_ROOT}/${pr_list_file}" ]]; then
+  pr_list=$(cat "${GFS_CI_ROOT}/${pr_list_file}")
+else
+  echo "no PRs to process .. exit"
+  exit 0
+fi
+
+#############################################################
+# Loop throu all PRs in PR List and look for expirments in
+# the RUNTESTS dir and for each one run runcotorun on them
+#############################################################
+
+for pr in ${pr_list}; do
+  id=$("${GH}" pr view "${pr}" --repo "${REPO_URL}" --json id --jq '.id')
+  echo "Processing Pull Request #${pr} and looking for cases"
+  pr_dir="${GFS_CI_ROOT}/PR/${pr}"
+
+  # If there is no RUNTESTS dir for this PR then cases have not been made yet
+  if [[ ! -d "${pr_dir}/RUNTESTS" ]]; then
+     continue
+  fi
+  num_cases=$(find "${pr_dir}/RUNTESTS" -mindepth 1 -maxdepth 1 -type d | wc -l) || true
+
+  #Check for PR success  when ${pr_dir}/RUNTESTS is void of subfolders
+  # since all successfull ones where previously removed
+  if [[ "${num_cases}" -eq 0 ]] && [[ -d "${pr_dir}/RUNTESTS" ]]; then
+    "${GH}" pr edit --repo "${REPO_URL}" "${pr}" --remove-label "CI-${MACHINE_ID^}-Running" --add-label "CI-${MACHINE_ID^}-Passed"
+    "${GH}" pr comment "${pr}" --repo "${REPO_URL}" --body-file "${GFS_CI_ROOT}/PR/${pr}/output_${id}"
+    sed -i "/${pr}/d" "${GFS_CI_ROOT}/${pr_list_file}"
+    # Completely remove the PR and its cloned repo on sucess of all cases
+    rm -Rf "${pr_dir}" 
+    continue 
+  fi
+
+  for cases in "${pr_dir}/RUNTESTS/"*; do
+    pslot=$(basename "${cases}")
+    xml="${pr_dir}/RUNTESTS/${pslot}/EXPDIR/${pslot}/${pslot}.xml"
+    db="${pr_dir}/RUNTESTS/${pslot}/EXPDIR/${pslot}/${pslot}.db"
+    rocoto_stat_output=$("${rocotostat}" -w "${xml}" -d "${db}" -s | grep -v CYCLE) || true
+    num_cycles=$(echo "${rocoto_stat_output}" | wc -l) || true
+    num_done=$(echo "${rocoto_stat_output}" | grep -c Done) || true
+    num_succeeded=$("${rocotostat}" -w "${xml}" -d "${db}" -a | grep -c SUCCEEDED) || true
+    echo "${pslot} Total Cycles: ${num_cycles} number done: ${num_done}" || true
+    num_failed=$("${rocotostat}" -w "${xml}" -d "${db}" -a | grep -c -E 'FAIL|DEAD') || true
+    if [[ ${num_failed} -ne 0 ]]; then
+      {
+        echo "Experiment ${pslot} Terminated: *FAILED*"
+        echo "Experiment ${pslot} Terminated with ${num_failed} tasks failed at $(date)" || true
+      } >> "${GFS_CI_ROOT}/PR/${pr}/output_${id}"
+      "${GH}" pr edit --repo "${REPO_URL}" "${pr}" --remove-label "CI-${MACHINE_ID^}-Running" --add-label "CI-${MACHINE_ID^}-Failed"
+      "${GH}" pr comment "${pr}" --repo "${REPO_URL}" --body-file "${GFS_CI_ROOT}/PR/${pr}/output_${id}"
+      sed -i "/${pr}/d" "${GFS_CI_ROOT}/${pr_list_file}"
+    fi
+    if [[ "${num_done}" -eq  "${num_cycles}" ]]; then
+      {
+        echo "Experiment ${pslot} completed: *SUCCESS*"
+        echo "Experiment ${pslot} Completed at $(date)" || true
+        echo -n "with ${num_succeeded} successfully completed jobs" || true
+      } >> "${GFS_CI_ROOT}/PR/${pr}/output_${id}"
+      "${GH}" pr comment "${pr}" --repo "${REPO_URL}" --body-file "${GFS_CI_ROOT}/PR/${pr}/output_${id}"
+      #Remove Experment cases that completed successfully
+      rm -Rf "${pr_dir}/RUNTESTS/${pslot}"
+    fi
+  done
+done