From 3cbca7d7ebba77e0b13d201c9f2669a47ccd7c90 Mon Sep 17 00:00:00 2001 From: Kate Friedman Date: Tue, 31 Jan 2023 15:27:05 -0500 Subject: [PATCH] Updates to RTD documentation based on full review (#1287) Reviewed documentation to correct spelling mistakes and update contents for current state of system. Refs #1272 --- docs/source/clone.rst | 76 ++++++--- docs/source/components.rst | 31 ++-- docs/source/configure.rst | 62 +++----- docs/source/development.rst | 48 ++++-- docs/source/errors_faq.rst | 5 +- docs/source/hpc.rst | 8 +- docs/source/index.rst | 21 ++- docs/source/init.rst | 283 ++++++++++++++++++++------------- docs/source/jobs.rst | 12 +- docs/source/monitor_rocoto.rst | 51 ++++-- docs/source/run.rst | 3 +- docs/source/setup.rst | 219 ++++++++++++++----------- docs/source/start.rst | 7 +- docs/source/view.rst | 8 +- 14 files changed, 502 insertions(+), 332 deletions(-) diff --git a/docs/source/clone.rst b/docs/source/clone.rst index e261cea00d..c31968ec2e 100644 --- a/docs/source/clone.rst +++ b/docs/source/clone.rst @@ -6,9 +6,14 @@ Clone and build Global Workflow Quick Instructions ^^^^^^^^^^^^^^^^^^ -Quick clone/build/link instructions (more detailed instructions below). Note, Here we are making the assumption that you are using the workflow to run an experiment and so are working from the authoritative repository. If you are going to be a developer then follow the instructions in :doc: `development.rst`. Once you do that you can follow the instructions here with the only thing different will be the repository you are cloning from. +Quick clone/build/link instructions (more detailed instructions below). -For forecast-only (coupled or uncoupled):: +.. note:: + Here we are making the assumption that you are using the workflow to run an experiment and so are working from the authoritative repository. If you are using a development branch then follow the instructions in :doc:`development.rst`. Once you do that you can follow the instructions here with the only difference being the repository/fork you are cloning from. + +For forecast-only (coupled or uncoupled): + +:: git clone https://github.com/NOAA-EMC/global-workflow.git cd global-workflow/sorc @@ -16,7 +21,9 @@ For forecast-only (coupled or uncoupled):: ./build_all.sh ./link_workflow.sh -For cycled (GSI):: +For cycled (w/ data assimilation): + +:: git clone https://github.com/NOAA-EMC/global-workflow.git cd global-workflow/sorc @@ -32,31 +39,47 @@ Clone workflow and component repositories Workflow ******** -https method:: +There are several ways to clone repositories from GitHub. Below we describe how to clone the global-workflow using either the ssh or https methods. **The ssh method is highly preferred and recommended.** - git clone https://github.com/NOAA-EMC/global-workflow.git +ssh method (using a password protected SSH key): -ssh method (using a password protected SSH key):: +:: git clone git@github.com:NOAA-EMC/global-workflow.git -Note: when using ssh methods you need to make sure that your GitHub account is configured for the computer from which you are accessing the repository (See `this link `_) +.. note:: + When using ssh methods you need to make sure that your GitHub account is configured for the computer from which you are accessing the repository (See `this link `_) + +https method: + +:: + + git clone https://github.com/NOAA-EMC/global-workflow.git + +Check what you just cloned (by default you will have only the develop branch): -Check what you just cloned (by default you will have only the develop branch):: +:: cd global-workflow git branch * develop -You now have a cloned copy of the global-workflow git repository. To checkout a branch or tag in your clone:: +You now have a cloned copy of the global-workflow git repository. To checkout a branch or tag in your clone: + +:: git checkout BRANCH_NAME -Note: Branch must already exist. If it does not you need to make a new branch using the ``-b`` flag:: +.. note:: + Branch must already exist. If it does not you need to make a new branch using the ``-b`` flag: + +:: git checkout -b BRANCH_NAME -The ``checkout`` command will checkout BRANCH_NAME and switch your clone to that branch. Example:: +The ``checkout`` command will checkout BRANCH_NAME and switch your clone to that branch. Example: + +:: git checkout my_branch git branch @@ -67,27 +90,35 @@ The ``checkout`` command will checkout BRANCH_NAME and switch your clone to that Components ********** -Once you have cloned the workflow repository it's time to checkout/clone its components. The components will be checked out under the /sorc folder via a script called checkout.sh. Run the script with no arguments for forecast-only:: +Once you have cloned the workflow repository it's time to checkout/clone its components. The components will be checked out under the ``/sorc`` folder via a script called checkout.sh. Run the script with no arguments for forecast-only: + +:: cd sorc ./checkout.sh -Or with the `-g` switch to include GSI for cycling:: +Or with the ``-g`` switch to include data assimilation (GSI) for cycling: + +:: cd sorc ./checkout.sh -g -If wishing to run with the operational GTG UPP and WAFS (only for select users) provide the -o flag with checkout.sh:: +If wishing to run with the operational GTG UPP and WAFS (only for select users) provide the ``-o`` flag with checkout.sh: + +:: ./checkout.sh -o -Each component cloned via checkout.sh will have a log (checkout-COMPONENT.log). Check the screen output and logs for clone errors. +Each component cloned via checkout.sh will have a log (``/sorc/logs/checkout-COMPONENT.log``). Check the screen output and logs for clone errors. ^^^^^^^^^^^^^^^^ Build components ^^^^^^^^^^^^^^^^ -Under the /sorc folder is a script to build all components called ``build_all.sh``. After running checkout.sh run this script to build all components codes:: +Under the ``/sorc`` folder is a script to build all components called ``build_all.sh``. After running checkout.sh run this script to build all components codes: + +:: ./build_all.sh [-a UFS_app][-c build_config][-h][-v] -a UFS_app: @@ -103,17 +134,20 @@ A partial build option is also available via two methods: a) modify gfs_build.cfg config file to disable/enable particular builds and then rerun build_all.sh - b) run individual build scripts also available in /sorc folder for each component or group of codes + b) run individual build scripts also available in ``/sorc`` folder for each component or group of codes ^^^^^^^^^^^^^^^ Link components ^^^^^^^^^^^^^^^ -At runtime the global-workflow needs all pieces in place within the main superstructure. To establish this a link script is run to create symlinks from the top level folders down to component files checked out in /sorc folders. +At runtime the global-workflow needs all pieces in place within the main superstructure. To establish this a link script is run to create symlinks from the top level folders down to component files checked out in ``/sorc`` folders. -After running the checkout and build scripts run the link script:: +After running the checkout and build scripts run the link script: + +:: ./link_workflow.sh [-o] - where: - -o: Run in operations (NCO) mode. This creates copies instead of using symlinks and is generally only used by NCO during installation into production. + +Where: + ``-o``: Run in operations (NCO) mode. This creates copies instead of using symlinks and is generally only used by NCO during installation into production. diff --git a/docs/source/components.rst b/docs/source/components.rst index 0f392fc9b5..6b947b3432 100644 --- a/docs/source/components.rst +++ b/docs/source/components.rst @@ -13,7 +13,7 @@ The major components of the system are: * Post-processing * Verification -The Global Workflow repository contains the workflow layer and, after running the checkout script, the code and scripts for the analysis, forecast, and post-processing components. Any non-workflow component is known as a sub-module. All of the sub-modules of the system reside in their respective repositories on GitHub. The global-workflow sub-modules are obtained by running the checkout script found under the /sorc folder. +The Global Workflow repository contains the workflow and script layers. After running the checkout script, the code and additional offline scripts for the analysis, forecast, and post-processing components will be present. Any non-workflow component is known as a sub-module. All of the sub-modules of the system reside in their respective repositories on GitHub. The global-workflow sub-modules are obtained by running the checkout script found under the /sorc folder. ====================== Component repositories @@ -21,18 +21,19 @@ Component repositories Components checked out via sorc/checkout.sh: +* **GFS UTILS** (https://github.com/ufs-community/gfs_utils): Utility codes needed by Global Workflow to run the GFS configuration * **UFS-Weather-Model** (https://github.com/ufs-community/ufs-weather-model): This is the core model used by the Global-Workflow to provide forecasts. The UFS-weather-model repository is an umbrella repository consisting of cooupled component earth systeme that are all checked out when we check out the code at the top level of the repoitory * **GSI** (https://github.com/NOAA-EMC/GSI): This is the core code base for atmospheric Data Assimilation -* **GSI UTILS** (https://github.com/NOAA-EMC/GSI-UTILS): Utility codes needed by GSI to create analysis +* **GSI UTILS** (https://github.com/NOAA-EMC/GSI-Utils): Utility codes needed by GSI to create analysis * **GSI Monitor** (https://github.com/NOAA-EMC/GSI-Monitor): These tools monitor the GSI package's data assimilation, detecting and reporting missing data sources, low observation counts, and high penalty values * **GLDAS** (https://github.com/NOAA-EMC/GLDAS): Code base for Land Data Assimiation * **GDAS** (https://github.com/NOAA-EMC/GDASApp): Jedi based Data Assimilation system. This system is currently being developed for marine Data Assimilation and in time will replace GSI for atmospheric data assimilation as well * **UFS UTILS** (https://github.com/ufs-community/UFS_UTILS): Utility codes needed for UFS-weather-model -* **GFS UTILS** (https://github.com/ufs-community/gfs_utils): Utility codes needed by Global Workflow to run the GFS configuration * **Verif global** (https://github.com/NOAA-EMC/EMC_verif-global): Verification package to evaluate GFS parallels. It uses MET and METplus. At this moment the verification package is limited to providing atmospheric metrics only * **GFS WAFS** (https://github.com/NOAA-EMC/EMC_gfs_wafs): Additional post processing products for Aircrafts -Note, when running the system in forecast mode only the Data Assimilation conmponents are not needed and are hence not checked out. +.. note:: + When running the system in forecast-only mode the Data Assimilation components are not needed and are hence not checked out. ===================== External dependencies @@ -55,7 +56,7 @@ Observation data (OBSPROC/prep) Data **** -Observation data, also known as dump data, is prepared in production and then archived in a global dump archive (GDA) for use by users when running cycled experiment. The GDA (identified as ``$DMPDIR`` in the workflow) is available on supported platforms and the workflow system knows where to find the data. +Observation data, also known as dump data, is prepared in production and then archived in a global dump archive (GDA) for use by users when running cycled experiments. The GDA (identified as ``$DMPDIR`` in the workflow) is available on supported platforms and the workflow system knows where to find the data. * Hera: /scratch1/NCEPDEV/global/glopara/dump * Orion: /work/noaa/rstprod/dump @@ -68,23 +69,25 @@ Global Dump Archive Structure The global dump archive (GDA) mimics the structure of its production source: ``DMPDIR/CDUMP.PDY/[CC/atmos/]FILES`` -The ``CDUMP`` is either gdas, gfs, or rtofs. All three contain production output for each day (``PDY``). The gdas and gfs folders are further broken into cycle (``CC``) and component (atmos). +The ``CDUMP`` is either gdas, gfs, or rtofs. All three contain production output for each day (``PDY``). The gdas and gfs folders are further broken into cycle (``CC``) and component (``atmos``). The GDA also contains special versions of some datasets and experimental data that is being evaluated ahead of implementation into production. The following subfolder suffixes exist: +--------+------------------------------------------------------------------------------------------------------+ -| Suffix | What | -+--------+------------------------------------------------------------------------------------------------------+ -| nr | Non-restricted versions of restricted files in production. | +| SUFFIX | WHAT | ++========+======================================================================================================+ +| nr | Non-restricted versions of restricted files in production. Produced in production. Restriced data is | +| | fully stripped from files. These files remain as is. | +--------+------------------------------------------------------------------------------------------------------+ | ur | Un-restricted versions of restricted files in production. Produced and archived on a 48hrs delay. | +| | Some restricted datasets are unrestricted. Data amounts: restricted > un-restricted > non-restricted | +--------+------------------------------------------------------------------------------------------------------+ | x | Experimental global datasets being evaluated for production. Dates and types vary depending on | | | upcoming global upgrades. | +--------+------------------------------------------------------------------------------------------------------+ -| y | Similar to "x" but only used when there is a duplicate experimental file that is in the x subfolder | -| | with the same name. These files will be different from both the production versions | -| | (if that exists already) and the x versions. This suffix is rarely used. | +| y | Similar to "x" but only used when there is a duplicate experimental file in the x subfolder with the | +| | same name. These files will be different from both the production versions (if that exists already) | +| | and the x versions. This suffix is rarely used. | +--------+------------------------------------------------------------------------------------------------------+ | p | Pre-production copy of full dump dataset, as produced by NCO during final 30-day parallel ahead of | | | implementation. Not always archived. | @@ -94,9 +97,9 @@ The GDA also contains special versions of some datasets and experimental data th Data processing *************** -Upstream of the global-workflow is the collection, quality control, and packaging of observed weather. The handling of that data is done by the OBSPROC group codes and scripts. The global-workflow uses two packages from OBSPROC to run its prep step to prepare observation data for use by the analysis system: +Upstream of the global-workflow is the collection, quality control, and packaging of observed weather. The handling of that data is done by the OBSPROC group codes and scripts. The global-workflow uses two packages from OBSPROC to run its prep step to prepare observation (dump) data for use by the analysis system: 1. https://github.com/NOAA-EMC/obsproc 2. https://github.com/NOAA-EMC/prepobs -Both package versions and locations on support platforms are set in the global-workflow system configs. +Package versions and locations on supported platforms are set in the global-workflow system configs, modulefiles, and version files. diff --git a/docs/source/configure.rst b/docs/source/configure.rst index 5fafddeeee..284297459d 100644 --- a/docs/source/configure.rst +++ b/docs/source/configure.rst @@ -2,34 +2,14 @@ Configure Run ============= -The global-workflow configs contain default settings that match operations. Users wishing to run with different settings should adjust their $EXPDIR configs and then rerun the ``setup_workflow[_fcstonly].py`` script since some configuration settings/switches change the workflow/xml ("Adjusts XML" column value is "YES"). +The global-workflow configs contain switches that change how the system runs. Many defaults are set initially. Users wishing to run with different settings should adjust their $EXPDIR configs and then rerun the ``setup_xml.py`` script since some configuration settings/switches change the workflow/xml ("Adjusts XML" column value is "YES"). +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ | Switch | What | Default | Adjusts XML | More Details | ++================+==============================+===============+=============+===================================================+ +| APP | Model application | ATM | YES | See case block in config.base for options | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| EXP_WARM_START | Is experiment starting warm | .false. | NO | Impacts IAU settings for initial cycle. Can also | -| | (.true.) or cold (.false)? | | | be set when running setup_expy[_fcstonly].py | -| | | | | script with --start flag (e.g. --start warm) | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| RETRO | Use retrospective parallel | NO | NO | Default of NO will tell getic job to pull from | -| | for ICs | | | production tapes. | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| RUN_CCPP | Run CCPP | NO | NO | If YES, set desired suite for CCPP_SUITE | -| | | | | variable [#]_ | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| QUILTING | Use I/O quilting | .true. | NO | If .true. choose OUTPUT_GRID as cubed_sphere_grid | -| | | | | in netcdf or gaussian_grid | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| OUTPUT_GRID | Output grid type | gaussian_grid | NO | If gaussian_grid, set OUTPUT_FILE for nemsio or | -| | | | | netcdf | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| OUTPUT_FILE | Output file type | netcdf | NO | GFSv16 uses netcdf, GFSv15 used nemsio, recommend | -| | | | | netcdf | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| WRITE_DOPOST | Run inline post | .true. | NO | If .true. produces master post output in forecast | -| | | | | job | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| DOIAU | Enable 4DIAU for control | YES | NO | New for GFSv16 | +| DOIAU | Enable 4DIAU for control | YES | NO | Turned off for cold-start first half cycle | | | with 3 increments | | | | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ | DOHYBVAR | Run EnKF | YES | YES | Don't recommend turning off | @@ -37,11 +17,8 @@ The global-workflow configs contain default settings that match operations. User | DONST | Run NSST | YES | NO | If YES, turns on NSST in anal/fcst steps, and | | | | | | turn off rtgsst | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| DO_GLDAS | Run GLDAS to spin up land | YES | YES | Spins up for 84hrs if sflux files not available | -| | ICs | | | | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| DO_WAVE | Runs wave jobs and produce | YES | YES | One-way wave coupling, new for GFSv16 | -| | wave forecast output | | | | +| DO_AWIPS | Run jobs to produce AWIPS | NO | YES | downstream processing, ops only | +| | products | | | | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ | DO_BUFRSND | Run job to produce BUFR | NO | YES | downstream processing | | | sounding products | | | | @@ -49,23 +26,34 @@ The global-workflow configs contain default settings that match operations. User | DO_GEMPAK | Run job to produce GEMPAK | NO | YES | downstream processing, ops only | | | products | | | | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| DO_AWIPS | Run jobs to produce AWIPS | NO | YES | downstream processing, ops only | -| | products | | | | -+----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| WAFSF | Run jobs to produce WAFS | NO | YES | downstream processing, ops only | -| | products | | | | +| DO_GLDAS | Run GLDAS to spin up land | YES | YES | Spins up for 84hrs if sflux files not available | +| | ICs | | | | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ | DO_VRFY | Run vrfy job | NO | YES | Whether to include vrfy job (GSI monitoring, | | | | | | tracker, VSDB, fit2obs) | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ | DO_METP | Run METplus jobs | YES | YES | One cycle spinup | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ -| DO_VSDB | Run VSDB package | YES | NO | Retiring in GFSv17+ | +| EXP_WARM_START | Is experiment starting warm | .false. | NO | Impacts IAU settings for initial cycle. Can also | +| | (.true.) or cold (.false)? | | | be set when running ``setup_expt.py`` script with | +| | | | | the ``--start`` flag (e.g. ``--start warm``) | ++----------------+------------------------------+---------------+-------------+---------------------------------------------------+ +| HPSSARCH | Archive to HPPS | NO | Possibly | Whether to save output to tarballs on HPPS | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ | LOCALARCH | Archive to a local directory | NO | Possibly | Instead of archiving data to HPSS, archive to a | | | | | | local directory, specified by ATARDIR. If | | | | | | LOCALARCH=YES, then HPSSARCH must =NO. Changing | | | | | | HPSSARCH from YES to NO will adjust the XML. | +----------------+------------------------------+---------------+-------------+---------------------------------------------------+ - -.. [#] CCPP will become default soon, variable may be removed. +| QUILTING | Use I/O quilting | .true. | NO | If .true. choose OUTPUT_GRID as cubed_sphere_grid | +| | | | | in netcdf or gaussian_grid | ++----------------+------------------------------+---------------+-------------+---------------------------------------------------+ +| RETRO | Use retrospective parallel | NO | NO | Default of NO will tell getic job to pull from | +| | for ICs | | | production tapes. | ++----------------+------------------------------+---------------+-------------+---------------------------------------------------+ +| WAFSF | Run jobs to produce WAFS | NO | YES | downstream processing, ops only | +| | products | | | | ++----------------+------------------------------+---------------+-------------+---------------------------------------------------+ +| WRITE_DOPOST | Run inline post | .true. | NO | If .true. produces master post output in forecast | +| | | | | job | ++----------------+------------------------------+---------------+-------------+---------------------------------------------------+ diff --git a/docs/source/development.rst b/docs/source/development.rst index 6c2c0ce4fb..6c7711bfe1 100644 --- a/docs/source/development.rst +++ b/docs/source/development.rst @@ -21,7 +21,7 @@ Where to do development? * In authoritative (main) repository: - - Work for the upcoming implementation (who: members of global-workflow-developers team) + - Work for upcoming implementation (who: members of global-workflow-developers team) - Major new features or port work (who: generally code managers and/or members of global-workflow-developers team) * In a fork: @@ -38,7 +38,7 @@ Protected branches The following global-workflow branches are protected by the code management team: * develop (HEAD) -* operations (kept aligned with current production) +* dev/gfs.v16 (kept aligned with current production, as well as ingests bug fixes and updates between release branches) These protected branches require the following to accept changes: @@ -58,12 +58,26 @@ The following steps should be followed in order to make changes to the develop b #. Issue - Open issue to document changes. Reference this issue in commits to your branches (e.g. ``git commit -m "Issue #23 - blah changes for what-not code"``) Click `here `__ to open a new global-workflow issue. #. GitFlow - Follow `GitFlow `_ procedures for development (branch names, forking vs branching, etc.). Read more `here `__ about GitFlow at EMC. #. To fork or not to fork? - If not working within authoritative repository create a fork of the authoritative repository. Read more `here `__ about forking in GitHub. - #. Branch - Create branch in either authoritative repository or fork of authoritative repository. See the `Where to do development? `_ section for how to determine where. Follow Gitflow conventions when creating branch. - #. Development - Perform and test changes in branch. Document work in issue and mention issue number in commit messages to link your work to the issue See `Commit Messages `_ section below. Depending on changes the code manager may request or perform additional pre-commit tests. + #. Branch - Create branch in either authoritative repository or fork of authoritative repository. See the `Where to do development? `_ section for how to determine where. Follow GitFlow conventions when creating branch. + #. Development - Perform and test changes in branch. Document work in issue and mention issue number in commit messages to link your work to the issue. See `Commit Messages `_ section below. Depending on changes the code manager may request or perform additional pre-commit tests. #. Pull request - When ready to merge changes back to develop branch, the lead developer should initiate a pull request (PR) of your branch (either fork or not) into the develop branch. Read `here `__ about pull requests in GitHub. Provide some information about the PR in the proper field, add at least one reviewer to the PR and assign the PR to a code manager. #. Complete - When review and testing is complete the code manager will complete the pull request and subsequent merge/commit. #. Cleanup - When complete the lead developer should delete the branch and close the issue. "Closing keywords" can be used in the PR to automatically close associated issues. +.. _development-tools: + +================= +Development Tools +================= + +See the ``/test`` folder in global-workflow for available development and testing tools. + +---------------- +Comparison Tools +---------------- + +There are several scripts to compare output between two experiments (e.g. control and test). See scripts under ``/test`` folder and read `README` there for information on how to use them. + .. _code-standards: ============== @@ -94,6 +108,8 @@ Commit message standards * The final line of the commit message should include tags to relevant issues (e.g. ``Refs: #217, #300``) Here is the example commit message from the article linked above; it includes descriptions of what would be in each part of the commit message for guidance: + +:: Summarize changes in around 50 characters or less @@ -118,12 +134,12 @@ Here is the example commit message from the article linked above; it includes de vary here If you use an issue tracker, put references to them at the bottom, - like this:: + like this: Resolves: #123 See also: #456, #789 -A detailed commit message is very useful for documenting changes +A detailed commit message is very useful for documenting changes. .. _sync: @@ -131,9 +147,9 @@ A detailed commit message is very useful for documenting changes How to sync fork with the authoritative repository ================================================== -As development in the main authoritative repository moves forward you will need to sync your fork's branches to stay up-to-date. Below is an example of how to sync your fork's copy of a branch with the authoritative repository copy. The branch name for the example will be "feature/new_thing". Click `here `__ for documentation on syncing forks. +As development in the main authoritative repository moves forward you will need to sync your fork branches to stay up-to-date. Below is an example of how to sync your fork copy of a branch with the authoritative repository copy. The branch name for the example will be "feature/new_thing". Click `here `__ for documentation on syncing forks. -1. Clone your fork and checkout branch that needs syncing +1. Clone your fork and checkout branch that needs syncing: :: @@ -141,25 +157,25 @@ As development in the main authoritative repository moves forward you will need cd fork git checkout feature/my_new_thing -2. Add upstream info to your clone so it knows where to merge from. The term "upstream" refers to the authoritative repository from which the fork was create +2. Add upstream info to your clone so it knows where to merge from. The term "upstream" refers to the authoritative repository from which the fork was created. :: git remote add upstream https://github.com/NOAA-EMC/global-workflow.git -3. Fetch upstream information into clone +3. Fetch upstream information into clone: :: git fetch upstream -Later on you can update your fork's remote information by doing the following command +Later on you can update your fork remote information by doing the following command: :: git remote update -4. Merge upstream feature/other_new_thing into your branch +4. Merge upstream ``feature/other_new_thing`` into your branch: :: @@ -167,16 +183,16 @@ Later on you can update your fork's remote information by doing the following co 5. Resolve any conflicts and perform any needed "add"s or "commit"s for conflict resolution. -6. Push the merged copy back up to your fork (origin) +6. Push the merged copy back up to your fork (origin): :: git push origin feature/my_new_thing -7. Note the sync in the GitHub issue tracking your work. - Done! -Moving forward you'll want to perform the "remote update" command regularly to update the metadata for the remote/upstream repository in your fork (e.g. pull in metadata for branches made in auth repo after you forked it):: +Moving forward you'll want to perform the "remote update" command regularly to update the metadata for the remote/upstream repository in your fork (e.g. pull in metadata for branches made in auth repo after you forked it). + +:: git remote update diff --git a/docs/source/errors_faq.rst b/docs/source/errors_faq.rst index 82830d5110..2660a01e60 100644 --- a/docs/source/errors_faq.rst +++ b/docs/source/errors_faq.rst @@ -9,10 +9,7 @@ Error: "ImportError" message when running setup script Example:: $ ./setup_xml.py /path/to/your/experiment/directory - Traceback (most recent call last): - File "./setup_workflow.py", line 32, in - from collections import OrderedDict - ImportError: cannot import name OrderedDict + /usr/bin/env: python3: No such file or directory **Cause:** Missing python module in your environment diff --git a/docs/source/hpc.rst b/docs/source/hpc.rst index a92bc79e1a..7161e2b742 100644 --- a/docs/source/hpc.rst +++ b/docs/source/hpc.rst @@ -33,7 +33,7 @@ NOTE: Only non-restricted data is available on S4. To request rstprod access, do either a and/or b below: -a) If you need restricted data access on WCOSS, fill out form here: +a) If you need restricted data access on WCOSS2, read details about restricted data and fill out form here: https://www.nco.ncep.noaa.gov/sib/restricted_data/restricted_data_sib/ @@ -67,7 +67,7 @@ For any merge with multiple commits, a short synopsis of the merge should appear Version ^^^^^^^ -It is advised to use Git v2+ when available. At the time of writing this documentation the swfault Git clients on the different machines were as noted in the table below. It is recommended that you check the default modules before loading recommneded ones: +It is advised to use Git v2+ when available. At the time of writing this documentation the default Git clients on the different machines were as noted in the table below. It is recommended that you check the default modules before loading recommended ones: +---------+----------+---------------------------------------+ | Machine | Default | Recommended | @@ -85,7 +85,9 @@ It is advised to use Git v2+ when available. At the time of writing this documen Output format ^^^^^^^^^^^^^ -For proper display of Git command output (e.g. git branch and git diff) type the following once on both machines:: +For proper display of Git command output (e.g. git branch and git diff) type the following once per machine: + +:: git config --global core.pager 'less -FRX' diff --git a/docs/source/index.rst b/docs/source/index.rst index 7a73e88c26..6444019de9 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,30 +3,29 @@ Global Workflow ############### -**Global-workflow** is the end-to-end workflow designed to run global configurations of medium range weather forecasting for the UFS weather model. It supports both development and operational implementations. In its current format it supports the Global Forecast System (GFS) and the Gobal Ensemble Forecast System (GEFS) configurations +**Global-workflow** is the end-to-end workflow designed to run global configurations of medium range weather forecasting for the UFS weather model. It supports both development and operational implementations. In its current format it supports the Global Forecast System (GFS) and the Global Ensemble Forecast System (GEFS) configurations ====== Status ====== -* State of develop (HEAD) branch: early GFSv17+ development +* State of develop (HEAD) branch: GFSv17+ development +* State of operations: GFS v16.3.5 `tag: [gfs.v16.3.5] `_ -* State of operations branch: GFS v16.3.5 `tag: [gfs.v16.3.5] `_ - -============== -Code managers: -============== +============= +Code managers +============= * Kate Friedman - @KateFriedman-NOAA / kate.friedman@noaa.gov * Walter Kolczynski - @WalterKolczynski-NOAA / walter.kolczynski@noaa.gov -============== -Announcements: -============== +============= +Announcements +============= General updates: NOAA employees and affiliates can join the gfs-announce distribution list to get updates on the GFS and global-workflow. Contact Kate Friedman (kate.friedman@noaa.gov) and Walter Kolczynski (walter.kolczynski@noaa.gov) to get added to the list or removed from it. -GitHub updates: Users should adjust their "Watch" settings for this repo so they receive notifications as they'd like to. Find the "Watch" or "Unwatch" button towards the top right of this page and click it to adjust how you watch this repo. +GitHub updates: Users should adjust their "Watch" settings for this repo so they receive notifications as they'd like to. Find the "Watch" or "Unwatch" button towards the top right of the `authoritative global-workflow repository page `_ and click it to adjust how you watch the repo. .. toctree:: :numbered: diff --git a/docs/source/init.rst b/docs/source/init.rst index e674260319..8342c46abb 100644 --- a/docs/source/init.rst +++ b/docs/source/init.rst @@ -21,21 +21,27 @@ Resolutions: Supported resolutions in global-workflow: C48, C96, C192, C384, C768 +.. _automated-generation: + ^^^^^^^^^^^^^^^^^^^^ Automated Generation ^^^^^^^^^^^^^^^^^^^^ +.. _cycled: + *********** Cycled mode *********** -Not yet supported. See Manual Generation section below for how to create your ICs yourself (outside of workflow). +Not yet supported. See :ref:`Manual Generation` section below for how to create your ICs yourself (outside of workflow). + +.. _forecastonly-atmonly: ***************************** -Free-forecast mode (atm-only) +Forecast-only mode (atm-only) ***************************** -Free-forecast mode in global workflow includes ``getic`` and ``init`` jobs for the gfs suite. The ``getic`` job pulls inputs for ``chgres_cube`` (init job) or warm start ICs into your ``ROTDIR/COMROT``. The ``init`` job then ingests those files to produce initial conditions for your experiment. +Forecast-only mode in global workflow includes ``getic`` and ``init`` jobs for the gfs suite. The ``getic`` job pulls inputs for ``chgres_cube`` (init job) or warm start ICs into your ``ROTDIR/COMROT``. The ``init`` job then ingests those files to produce initial conditions for your experiment. Users on machines without HPSS access (e.g. Orion) need to perform the ``getic`` step manually and stage inputs for the ``init`` job. The table below lists the needed files for ``init`` and where to place them in your ``ROTDIR``. @@ -54,7 +60,7 @@ Operations/production output location on HPSS: /NCEPPROD/hpssprod/runhistory/rh | | | | | | | gfs.t. ``hh`` z.sfcanl | | | +----------------+---------------------------------+-----------------------------------------------------------------------------+--------------------------------+ -| v14 ops | gfs.t. ``hh`` z.atmanl.nemsio | gpfs_hps_nco_ops_com_gfs_prod_gfs. ``yyyymmddhh`` .anl.tar | gfs. ``yyyymmdd` /`hh`` | +| v14 ops | gfs.t. ``hh`` z.atmanl.nemsio | gpfs_hps_nco_ops_com_gfs_prod_gfs. ``yyyymmddhh`` .anl.tar | gfs. ``yyyymmdd`` /``hh`` | | | | | | | | gfs.t. ``hh`` z.sfcanl.nemsio | | | +----------------+---------------------------------+-----------------------------------------------------------------------------+--------------------------------+ @@ -66,198 +72,220 @@ Operations/production output location on HPSS: /NCEPPROD/hpssprod/runhistory/rh | | | | | | | gfs.t. ``hh`` z.sfcanl.nemsio | | | +----------------+---------------------------------+-----------------------------------------------------------------------------+--------------------------------+ -| v16 ops | gfs.t. ``hh`` z.atmanl.nc | com_gfs_prod_gfs. ``yyyymmdd`` _ ``hh`` .gfs_nca.tar | gfs. ``yyyymmdd`` /``hh``/atmos| +| v16 retro | gfs.t. ``hh`` z.atmanl.nc | gfs_netcdfa.tar* | gfs. ``yyyymmdd`` /``hh``/atmos| | | | | | | | gfs.t. ``hh`` z.sfcanl.nc | | | +----------------+---------------------------------+-----------------------------------------------------------------------------+--------------------------------+ -| v16 retro | gfs.t. ``hh`` z.atmanl.nc | gfs_netcdfa.tar* | gfs. ``yyyymmdd`` /``hh``/atmos| +| v16.0[1] ops | gfs.t. ``hh`` z.atmanl.nc | com_gfs_prod_gfs. ``yyyymmdd`` _ ``hh`` .gfs_nca.tar | gfs. ``yyyymmdd`` /``hh``/atmos| | | | | | | | gfs.t. ``hh`` z.sfcanl.nc | | | +----------------+---------------------------------+-----------------------------------------------------------------------------+--------------------------------+ +| v16.2[3]+ ops | gfs.t. ``hh`` z.atmanl.nc | com_gfs_ ``gfs_ver`` _gfs. ``yyyymmdd`` _ ``hh`` .gfs_nca.tar | gfs. ``yyyymmdd`` /``hh``/atmos| +| | | | | +| | gfs.t. ``hh`` z.sfcanl.nc | | | ++----------------+---------------------------------+-----------------------------------------------------------------------------+--------------------------------+ + +For HPSS path, see retrospective table in :ref:`pre-production parallel section ` below -For HPSS path, see retrospective table in :ref:`pre-production parallel section `: below +.. _forecastonly-coupled: ********************* -Free-forecast coupled +Forecast-only coupled ********************* Coupled initial conditions are currently only generated offline and copied prior to the forecast run. Prototype initial conditions will automatically be used when setting up an experiment as an S2SW app, there is no need to do anything additional. Copies of initial conditions from the prototype runs are currently maintained on Hera, Orion, and WCOSS2. The locations used are determined by ``parm/config/config.coupled_ic``. If you need prototype ICs on another machine, please contact Walter (Walter.Kolczynski@noaa.gov). +.. _manual-generation: + ^^^^^^^^^^^^^^^^^ Manual Generation ^^^^^^^^^^^^^^^^^ NOTE: Initial conditions cannot be generated on S4. These must be generated on another supported platform then pushed to S4. If you do not have access to a supported system or need assistance, please contact David Huber (david.huber@noaa.gov). +.. _coldstarts: + *********** Cold starts *********** The following information is for users needing to generate initial conditions for a cycled experiment that will run at a different resolution or layer amount than the operational GFS (C768C384L127). -The ``chgres_cube`` code is available from the `UFS_UTILS repository `_ on GitHub and can be used to convert GFS ICs to a different resolution or number of layers. Users may clone the develop/HEAD branch or the same version used by global-workflow develop (found in sorc/checkout.sh). The ``chgres_cube`` code/scripts currently support the following GFS inputs: +The ``chgres_cube`` code is available from the `UFS_UTILS repository `_ on GitHub and can be used to convert GFS ICs to a different resolution or number of layers. Users may clone the develop/HEAD branch or the same version used by global-workflow develop (found in ``sorc/checkout.sh``). The ``chgres_cube`` code/scripts currently support the following GFS inputs: * pre-GFSv14 * GFSv14 * GFSv15 * GFSv16 -Clone UFS_UTILS:: +Users can use the copy of UFS_UTILS that is already cloned and built within their global-workflow clone or clone/build it separately: + +Within a built/linked global-workflow clone: + +:: + + cd sorc/ufs_utils.fd/util/gdas_init + +Clone and build separately: + +1. Clone UFS_UTILS: + +:: git clone --recursive https://github.com/NOAA-EMC/UFS_UTILS.git Then switch to a different tag or use the default branch (develop). -Build UFS_UTILS:: +2. Build UFS_UTILS: + +:: sh build_all.sh cd fix sh link_fixdirs.sh emc $MACHINE -where ``$MACHINE`` is ``wcoss2``, ``hera``, ``jet``, or ``orion``. Note: UFS-UTILS builds on Orion but due to the lack of HPSS access on Orion the ``gdas_init`` utility is not supported there. +where ``$MACHINE`` is ``wcoss2``, ``hera``, or ``jet``. -Configure your conversion:: +.. note:: + UFS-UTILS builds on Orion but due to the lack of HPSS access on Orion the ``gdas_init`` utility is not supported there. + +3. Configure your conversion: + +:: cd util/gdas_init vi config Read the doc block at the top of the config and adjust the variables to meet you needs (e.g. ``yy, mm, dd, hh`` for ``SDATE``). -Submit conversion script::` +Most users will want to adjust the following ``config`` settings for the current system design: + +#. EXTRACT_DATA=YES (to pull original ICs to convert off HPSS) +#. RUN_CHGRES=YES (to run chgres_cube on the original ICs pulled off HPSS) +#. LEVS=128 (for the L127 GFS) + +4. Submit conversion script: + +:: ./driver.$MACHINE.sh -where ``$MACHINE`` is currently ``wcoss2``, ``hera`` or ``jet``. Additional options will be available as support for other machines expands. Note: UFS-UTILS builds on Orion but due to lack of HPSS access there is no ``gdas_init`` driver for Orion nor support to pull initial conditions from HPSS for the ``gdas_init`` utility. +where ``$MACHINE`` is currently ``wcoss2``, ``hera`` or ``jet``. Additional options will be available as support for other machines expands. -3 small jobs will be submitted: +.. note:: + UFS-UTILS builds on Orion but due to lack of HPSS access there is no ``gdas_init`` driver for Orion nor support to pull initial conditions from HPSS for the ``gdas_init`` utility. + +Several small jobs will be submitted: - 1 jobs to pull inputs off HPSS - - 2 jobs to run ``chgres_cube`` (1 for deterministic/hires and 1 for each EnKF ensemble member) + - 1 or 2 jobs to run ``chgres_cube`` (1 for deterministic/hires and 1 for each EnKF ensemble member) The chgres jobs will have a dependency on the data-pull jobs and will wait to run until all data-pull jobs have completed. -Check output: +5. Check output: -In the config you will have defined an output folder called ``$OUTDIR``. The converted output will be found there, including the needed abias and radstat initial condition files. The files will be in the needed directory structure for the global-workflow system, therefore a user can move the contents of their ``$OUTDIR`` directly into their ``$ROTDIR/$COMROT``. +In the config you will have defined an output folder called ``$OUTDIR``. The converted output will be found there, including the needed abias and radstat initial condition files (if CDUMP=gdas). The files will be in the needed directory structure for the global-workflow system, therefore a user can move the contents of their ``$OUTDIR`` directly into their ``$ROTDIR/$COMROT``. Please report bugs to George Gayno (george.gayno@noaa.gov) and Kate Friedman (kate.friedman@noaa.gov). +.. _warmstarts-prod: + ***************************** Warm starts (from production) ***************************** -The GFSv15 was implemented into production on June 12th, 2019 at 12z. The GFS was spun up ahead of that cycle and thus production output for the system is available from the 00z cycle (2019061200) and later. Production output tarballs from the prior GFSv14 system are located in the same location on HPSS but have "hps" in the name to represent that it was run on the Cray, where as the GFS now runs in production on the Dell and has "dell1" in the tarball name. +Output and warm start initial conditions from the operational GFS (FV3GFS) are saved on HPSS. Users can pull these warm start initial conditions from tape for their use in running operational resolution experiments. See production output in the following location on HPSS: ``/NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD`` -Example location: - -``/NCEPPROD/hpssprod/runhistory/rh2021/202104/20210420`` - -Example listing for 2021042000 production tarballs:: - - [Kate.Friedman@m72a2 ~]$ hpsstar dir /NCEPPROD/hpssprod/runhistory/rh2021/202104/20210420 | grep gfs | grep _00. | grep -v idx - [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217] - ****************************************************************** - * Welcome to the NESCC High Performance Storage System * - * * - * Current HPSS version: 7.5.3 * - * * - * * - * Please Submit Helpdesk Request to * - * rdhpcs.hpss.help@noaa.gov * - * * - * Announcements: * - ****************************************************************** - Username: Kate.Friedman UID: 2391 Acct: 2391(2391) Copies: 1 COS: 0 Firewall: off [hsi.6.3.0.p1-hgs Thu May 7 09:16:23 UTC 2020] - /NCEPPROD/hpssprod/runhistory/rh2021/202104: - drwxr-xr-x 2 nwprod prod 11776 Apr 19 23:44 20210420 - [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217] - -rw-r----- 1 nwprod rstprod 51268255744 Apr 22 05:29 com_gfs_prod_enkfgdas.20210420_00.enkfgdas.tar - -rw-r--r-- 1 nwprod prod 220121310720 Apr 22 06:42 com_gfs_prod_enkfgdas.20210420_00.enkfgdas_restart_grp1.tar - -rw-r--r-- 1 nwprod prod 220124178944 Apr 22 07:04 com_gfs_prod_enkfgdas.20210420_00.enkfgdas_restart_grp2.tar - -rw-r--r-- 1 nwprod prod 220120305664 Apr 22 07:24 com_gfs_prod_enkfgdas.20210420_00.enkfgdas_restart_grp3.tar - -rw-r--r-- 1 nwprod prod 220116934656 Apr 22 07:38 com_gfs_prod_enkfgdas.20210420_00.enkfgdas_restart_grp4.tar - -rw-r--r-- 1 nwprod prod 220121547776 Apr 22 07:56 com_gfs_prod_enkfgdas.20210420_00.enkfgdas_restart_grp5.tar - -rw-r--r-- 1 nwprod prod 220125794816 Apr 22 08:09 com_gfs_prod_enkfgdas.20210420_00.enkfgdas_restart_grp6.tar - -rw-r--r-- 1 nwprod prod 220117037568 Apr 22 08:23 com_gfs_prod_enkfgdas.20210420_00.enkfgdas_restart_grp7.tar - -rw-r--r-- 1 nwprod prod 220117203968 Apr 22 08:33 com_gfs_prod_enkfgdas.20210420_00.enkfgdas_restart_grp8.tar - -rw-r----- 1 nwprod rstprod 9573153280 Apr 22 02:49 com_gfs_prod_gdas.20210420_00.gdas.tar - -rw-r--r-- 1 nwprod prod 1020249088 Apr 22 02:49 com_gfs_prod_gdas.20210420_00.gdas_flux.tar - -rw-r--r-- 1 nwprod prod 92950728704 Apr 22 03:05 com_gfs_prod_gdas.20210420_00.gdas_nc.tar - -rw-r--r-- 1 nwprod prod 10647806464 Apr 22 02:50 com_gfs_prod_gdas.20210420_00.gdas_pgrb2.tar - -rw-r----- 1 nwprod rstprod 65121796608 Apr 22 02:56 com_gfs_prod_gdas.20210420_00.gdas_restart.tar - -rw-r--r-- 1 nwprod prod 18200814080 Apr 22 03:06 com_gfs_prod_gdas.20210420_00.gdaswave_keep.tar - -rw-r----- 1 nwprod rstprod 13013076992 Apr 22 03:08 com_gfs_prod_gfs.20210420_00.gfs.tar - -rw-r--r-- 1 nwprod prod 62663230976 Apr 22 03:13 com_gfs_prod_gfs.20210420_00.gfs_flux.tar - -rw-r--r-- 1 nwprod prod 127932879360 Apr 22 03:47 com_gfs_prod_gfs.20210420_00.gfs_nca.tar - -rw-r--r-- 1 nwprod prod 138633526272 Apr 22 04:00 com_gfs_prod_gfs.20210420_00.gfs_ncb.tar - -rw-r--r-- 1 nwprod prod 140773240832 Apr 22 03:27 com_gfs_prod_gfs.20210420_00.gfs_pgrb2.tar - -rw-r--r-- 1 nwprod prod 61253672960 Apr 22 03:32 com_gfs_prod_gfs.20210420_00.gfs_pgrb2b.tar - -rw-r--r-- 1 nwprod prod 19702107136 Apr 22 03:34 com_gfs_prod_gfs.20210420_00.gfs_restart.tar - -rw-r--r-- 1 nwprod prod 18617610240 Apr 22 04:02 com_gfs_prod_gfs.20210420_00.gfswave_output.tar - -rw-r--r-- 1 nwprod prod 30737774592 Apr 22 04:05 com_gfs_prod_gfs.20210420_00.gfswave_raw.tar - -The warm starts and other output from production are at C768 deterministic and C384 EnKF. The warm start files must be converted to your desired resolution(s) using ``chgres_cube`` if you wish to run a different resolution. If you are running a C768/C384 experiment you can use them as is. - -.. _fix-netcdf: +Example location for January 2nd 2023: + +``/NCEPPROD/hpssprod/runhistory/rh2023/202301/20230102`` + +Example listing for January 2nd 2023 00z (2023010200) production tarballs: + +:: + + -bash-4.2$ hpsstar dir /NCEPPROD/hpssprod/runhistory/rh2023/202301/20230102 | grep gfs | grep _00. | grep -v idx + [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217] + -rw-r----- 1 nwprod rstprod 34824086016 Jan 4 03:31 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas.tar + -rw-r--r-- 1 nwprod prod 219779890688 Jan 4 04:04 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas_restart_grp1.tar + -rw-r--r-- 1 nwprod prod 219779921408 Jan 4 04:13 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas_restart_grp2.tar + -rw-r--r-- 1 nwprod prod 219775624192 Jan 4 04:23 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas_restart_grp3.tar + -rw-r--r-- 1 nwprod prod 219779726848 Jan 4 04:33 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas_restart_grp4.tar + -rw-r--r-- 1 nwprod prod 219777990656 Jan 4 04:42 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas_restart_grp5.tar + -rw-r--r-- 1 nwprod prod 219780963328 Jan 4 04:52 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas_restart_grp6.tar + -rw-r--r-- 1 nwprod prod 219775471104 Jan 4 05:02 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas_restart_grp7.tar + -rw-r--r-- 1 nwprod prod 219779499008 Jan 4 05:11 com_gfs_v16.3_enkfgdas.20230102_00.enkfgdas_restart_grp8.tar + -rw-r----- 1 nwprod rstprod 2287770624 Jan 4 02:07 com_gfs_v16.3_gdas.20230102_00.gdas.tar + -rw-r--r-- 1 nwprod prod 1026611200 Jan 4 02:07 com_gfs_v16.3_gdas.20230102_00.gdas_flux.tar + -rw-r--r-- 1 nwprod prod 91233038336 Jan 4 02:16 com_gfs_v16.3_gdas.20230102_00.gdas_nc.tar + -rw-r--r-- 1 nwprod prod 10865070592 Jan 4 02:08 com_gfs_v16.3_gdas.20230102_00.gdas_pgrb2.tar + -rw-r----- 1 nwprod rstprod 69913956352 Jan 4 02:11 com_gfs_v16.3_gdas.20230102_00.gdas_restart.tar + -rw-r--r-- 1 nwprod prod 18200814080 Jan 4 02:17 com_gfs_v16.3_gdas.20230102_00.gdaswave_keep.tar + -rw-r--r-- 1 nwprod prod 5493360128 Jan 4 02:18 com_gfs_v16.3_gfs.20230102_00.gfs.tar + -rw-r--r-- 1 nwprod prod 62501531648 Jan 4 02:21 com_gfs_v16.3_gfs.20230102_00.gfs_flux.tar + -rw-r--r-- 1 nwprod prod 121786191360 Jan 4 02:41 com_gfs_v16.3_gfs.20230102_00.gfs_nca.tar + -rw-r--r-- 1 nwprod prod 130729495040 Jan 4 02:48 com_gfs_v16.3_gfs.20230102_00.gfs_ncb.tar + -rw-r--r-- 1 nwprod prod 138344908800 Jan 4 02:29 com_gfs_v16.3_gfs.20230102_00.gfs_pgrb2.tar + -rw-r--r-- 1 nwprod prod 59804635136 Jan 4 02:32 com_gfs_v16.3_gfs.20230102_00.gfs_pgrb2b.tar + -rw-r--r-- 1 nwprod prod 25095460864 Jan 4 02:34 com_gfs_v16.3_gfs.20230102_00.gfs_restart.tar + -rw-r--r-- 1 nwprod prod 21573020160 Jan 4 02:49 com_gfs_v16.3_gfs.20230102_00.gfswave_output.tar + -rw-r--r-- 1 nwprod prod 32850422784 Jan 4 02:51 com_gfs_v16.3_gfs.20230102_00.gfswave_raw.tar + -rw-r----- 1 nwprod rstprod 7419548160 Jan 4 05:15 com_obsproc_v1.1_gfs.20230102_00.obsproc_gfs.tar + +The warm starts and other output from production are at C768 deterministic and C384 EnKF. The warm start files must be converted to your desired resolution(s) using ``chgres_cube`` if you wish to run a different resolution. If you are running a C768C384L127 experiment you can use them as is. -------------------------- -Fix NetCDF checksum issue -------------------------- +------------------------------------------------------------------------------------------ +What files should you pull for starting a new experiment with warm starts from production? +------------------------------------------------------------------------------------------ -Due to a recent change in UFS, the setting to bypass the data verification no longer works, so you may also need an additional offline step to delete the checksum of the NetCDF files for warm start: +That depends on what mode you want to run -- forecast-only or cycled. Whichever mode, navigate to the top of your ``COMROT`` and pull the entirety of the tarball(s) listed below for your mode. The files within the tarball are already in the ``$CDUMP.$PDY/$CYC/$ATMOS`` folder format expected by the system. -On RDHPCS:: +For forecast-only there are two tarballs to pull - module load nco/4.9.3 +1. File #1 (for starting cycle SDATE): -On WCOSS2:: +:: - module load intel/19.1.3.304 - module load netcdf/4.7.4 - module load udunits/2.2.28 - module load gsl/2.7 - module load nco/4.7.9 + /NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD/com_gfs_vGFSVER_gfs.YYYYMMDD_CC.gfs_restart.tar -And then on all platforms:: +...where ``GFSVER`` is the version of the GFS (e.g. "16.3"). - cd $COMROT - for f in $(find ./ -name *tile*.nc); do echo $f; ncatted -a checksum,,d,, $f; done +2. File #2 (for prior cycle GDATE=SDATE-06): ------------------------------------------------------------------------------------------- -What files should you pull for starting a new experiment with warm starts from production? ------------------------------------------------------------------------------------------- +:: -That depends on what mode you want to run -- free-forecast or cycled. Whichever mode navigate to the top of your ``COMROT`` and pull the entirety of the tarball(s) listed below for your mode. The files within the tarball are already in the ``$CDUMP.$PDY/$CYC`` folder format expected by the system. + /NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD/com_gfs_vGFSVER_gdas.YYYYMMDD_CC.gdas_restart.tar -For free-forecast there are two tar balls to pull +...where ``GFSVER`` is the version of the GFS (e.g. "16.3"). - 1. File #1 (for starting cycle SDATE):: - /NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD/com_gfs_prod_gfs.YYYYMMDD_CC.gfs_restart.tar - 2. File #2 (for prior cycle GDATE=SDATE-06):: - /NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD/com_gfs_prod_gdas.YYYYMMDD_CC.gdas_restart.tar +For cycled mode there 18 tarballs to pull (9 for SDATE and 9 for GDATE (SDATE-06)): - For cycled mode there 18 tarballs to pull (9 for SDATE and 9 for GDATE (SDATE-06)):: +:: HPSS path: /NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD/ -Tarballs per cycle:: +Tarballs per cycle: - com_gfs_prod_gdas.YYYYMMDD_CC.gdas_restart.tar - com_gfs_prod_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp1.tar - com_gfs_prod_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp2.tar - com_gfs_prod_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp3.tar - com_gfs_prod_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp4.tar - com_gfs_prod_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp5.tar - com_gfs_prod_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp6.tar - com_gfs_prod_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp7.tar - com_gfs_prod_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp8.tar +:: + + com_gfs_vGFSVER_gdas.YYYYMMDD_CC.gdas_restart.tar + com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp1.tar + com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp2.tar + com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp3.tar + com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp4.tar + com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp5.tar + com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp6.tar + com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp7.tar + com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp8.tar Go to the top of your ``COMROT/ROTDIR`` and pull the contents of all tarballs there. The tarballs already contain the needed directory structure. +.. _warmstarts-preprod-parallels: + ******************************************* Warm starts (from pre-production parallels) ******************************************* @@ -270,7 +298,7 @@ Recent pre-implementation parallel series was for GFS v16 (implemented March 202 * **Where are these tarballs?** See below for the location on HPSS for each v16 pre-implementation parallel. * **What tarballs do I need to grab for my experiment?** Tarballs from two cycles are required. The tarballs are listed below, where $CDATE is your starting cycle and $GDATE is one cycle prior. - - Free-forecast + - Forecast-only + ../$CDATE/gfs_restarta.tar + ../$GDATE/gdas_restartb.tar - Cycled w/EnKF @@ -281,7 +309,7 @@ Recent pre-implementation parallel series was for GFS v16 (implemented March 202 * **Where do I put the warm-start initial conditions?** Extraction should occur right inside your COMROT. You may need to rename the enkf folder (enkf.gdas.$PDY -> enkfgdas.$PDY). -Due to a recent change in the dycore, you may also need an additional offline step to fix the checksum of the NetCDF files for warm start. See the :ref:`fix netcdf checksum section `: above +Due to a recent change in the dycore, you may also need an additional offline step to fix the checksum of the NetCDF files for warm start. See the :ref:`Fix netcdf checksum section `. .. _retrospective: @@ -330,3 +358,42 @@ GFSv15 (Q2FY19) Pre-Implementation Parallel HPSS Locations +---------------------+-----------------+-----------------------------------------------------------+ | 20150503 - 20151130 | fv3q2fy19retro6 | $PREFIX/emc.glopara/WCOSS_DELL_P3/Q2FY19/fv3q2fy19retro6 | +---------------------+-----------------+-----------------------------------------------------------+ + +.. _gfsv17-warmstarts: + +*************************************** +Using pre-GFSv17 warm starts for GFSv17 +*************************************** + +If a user wishes to run a high-res (C768C384L127) GFSv17 experiment with warm starts from the operational GFSv16 (or older) warm starts, they must process the initial condition files before using. See details below in the :ref:`Fix netcdf checksum section `. + +.. _gfsv17-checksum: + +------------------------- +Fix NetCDF checksum issue +------------------------- + +Due to a recent change in UFS, the setting to bypass the data verification no longer works, so you may also need an additional offline step to delete the checksum of the NetCDF files for warm start: + +On RDHPCS: + +:: + + module load nco/4.9.3 + +On WCOSS2: + +:: + + module load intel/19.1.3.304 + module load netcdf/4.7.4 + module load udunits/2.2.28 + module load gsl/2.7 + module load nco/4.7.9 + +And then on all platforms: + +:: + + cd $COMROT + for f in $(find ./ -name *tile*.nc); do echo $f; ncatted -a checksum,,d,, $f; done diff --git a/docs/source/jobs.rst b/docs/source/jobs.rst index 249f305b07..ae7e1cd68a 100644 --- a/docs/source/jobs.rst +++ b/docs/source/jobs.rst @@ -6,13 +6,13 @@ GFS Configuration Schematic flow chart for GFS v16 in operations -The sequence of jobs that are run for an end-to-end (DA+forecast+post processing+verification) GFS configuration using the Global Workflow is shown above. The system utilizes a collection of scripts that perform the tasks for each step. +The sequence of jobs that are run for an end-to-end (analysis+forecast+post processing+verification) GFS configuration using the Global Workflow is shown above. The system utilizes a collection of scripts that perform the tasks for each step. -For any cycle the system consists of two phases -- the gdas phase which provides the initial guess fields, and the gfs phase which creates the initial conditions and forecast of the system. As with the operational system, the gdas runs for each cycle (00, 06, 12, and 18 UTC), however, to save time and space in experiments, the gfs (right side of the diagram) is initially setup to run for only the 00 UTC cycle. (See the "run GFS this cycle?" portion of the diagram) The option to run the GFS for all four cycles is available (see gfs_cyc variable in configuration file). +For any cycle the system consists of two suites -- the "gdas" suite which provides the initial guess fields, and the "gfs" suite which creates the initial conditions and forecast of the system. As with the operational system, the gdas runs for each cycle (00, 06, 12, and 18 UTC), however, to save time and space in experiments, the gfs (right side of the diagram) is initially setup to run for only the 00 UTC cycle (See the "run GFS this cycle?" portion of the diagram). The option to run the GFS for all four cycles is available (see the ``gfs_cyc`` variable in configuration file). An experimental run is different from operations in the following ways: -* Workflow manager: operations utilizes `ecFlow `__, while development currently utilizes `ROCOTO `__. Note, experiments can also be run using ecFlow. +* Workflow manager: operations utilizes `ecFlow `__, while development currently utilizes `ROCOTO `__. Note, experiments can also be run using ecFlow on platforms with ecFlow servers established. * Dump step is not run as it has already been completed during the real-time production runs and dump data is available in the global dump archive on supported machines. @@ -28,8 +28,8 @@ Downstream jobs (e.g. awips, gempak, etc.) are not included in the diagram. Thos Jobs in the GFS Configuration ============================= +-------------------+-----------------------------------------------------------------------------------------------------------------------+ -| Job Name | Purpose | -+-------------------+-----------------------------------------------------------------------------------------------------------------------+ +| JOB NAME | PURPOSE | ++===================+=======================================================================================================================+ | anal | Runs the analysis. 1) Runs the atmospheric analysis (global_gsi) to produce analysis increments; 2) Update surface | | | guess file via global_cycle to create surface analysis on tiles. | +-------------------+-----------------------------------------------------------------------------------------------------------------------+ @@ -44,7 +44,7 @@ Jobs in the GFS Configuration | earcN/eamn | Archival script for EnKF: 1) Write select EnKF output to HPSS; 2) Copy select files to online archive; 3) Clean up | | | EnKF temporary run directories; 4) Remove "old" EnKF files from rotating directory. | +-------------------+-----------------------------------------------------------------------------------------------------------------------+ -| ecenN/ecmn | Recenter ensemble members around hi-res deterministic analysis. Note - GFS v16 recenters ensemble member analysis | +| ecenN/ecmn | Recenter ensemble members around hi-res deterministic analysis. GFS v16 recenters ensemble member analysis. | | | increments. | +-------------------+-----------------------------------------------------------------------------------------------------------------------+ | echgres | Runs chgres on full-resolution forecast for EnKF recentering (ecen). | diff --git a/docs/source/monitor_rocoto.rst b/docs/source/monitor_rocoto.rst index d7126790b3..1b8b2a3836 100644 --- a/docs/source/monitor_rocoto.rst +++ b/docs/source/monitor_rocoto.rst @@ -11,39 +11,56 @@ Using command line You can use Rocoto commands with arguments to check the status of your experiment. -Start or continue a run:: +Start or continue a run: + +:: rocotorun -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -Check the status of the workflow:: +Check the status of the workflow: + +:: rocotostat -d /path/to/workflow/database/file -w /path/to/workflow/xml/file [-c YYYYMMDDCCmm,[YYYYMMDDCCmm,...]] [-t taskname,[taskname,...]] [-s] [-T] -Note: YYYYMMDDCCmm = YearMonthDayCycleMinute ...where mm/Minute is ’00’ for all cycles currently. +.. note:: + YYYYMMDDCCmm = YearMonthDayCycleMinute ...where mm/Minute is ’00’ for all cycles currently. -Check the status of a job:: +Check the status of a job: + +:: rocotocheck -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname -Force a task to run (ignores dependencies - USE CAREFULLY!):: +Force a task to run (ignores dependencies - USE CAREFULLY!): + +:: rocotoboot -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname -Rerun task(s):: +Rerun task(s): + +:: rocotorewind -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname -Set a task to complete (overwrites current state):: + (If job is currently queued or running rocoto will kill the job. Run rocotorun afterwards to fire off rewound task.) + +Set a task to complete (overwrites current state): + +:: rocotocomplete -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname +(Will not kill queued or running job, only update status.) + Several dates and task names may be specified in the same command by adding more -c and -t options. However, lists are not allowed. ^^^^^^^^^^^^^^^^^ Use ROCOTO viewer ^^^^^^^^^^^^^^^^^ -An alternative approach is to use A GUI that was designed to assist with monitoring global workflow experiments that use ROCOTO. It can be found under the ush/rocoto folder in global-workflow. +An alternative approach is to use a GUI that was designed to assist with monitoring global workflow experiments that use ROCOTO. It can be found under the ``workflow`` folder in global-workflow. ***** Usage @@ -53,9 +70,23 @@ Usage ./rocoto_viewer.py -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -Note 1: Terminal/window must be wide enough to display all experiment information columns, viewer will complain if not. +.. note:: + Note 1: Terminal/window must be wide enough to display all experiment information columns, viewer will complain if not. + + Note 2: The viewer requires the full path to the database and xml files if you are not in your EXPDIR when you invoke it. + + Note 3: Only ``TERM=xterm`` is supported. You may wish to create a shell function to switch automatically if you are in a different terminal: + + Bash example: + + :: -Note 2: The viewer requires the full path to the database and xml files if you are not in your EXPDIR when you invoke it. + function rv { + oldterm=${TERM}; + export TERM='xterm'; + ${PATH_TO_VIEWER}/rocoto_viewer.py $@; + export TERM=${oldterm}; + } ********************* What the viewer shows diff --git a/docs/source/run.rst b/docs/source/run.rst index 9c5100dcaf..56728d3282 100644 --- a/docs/source/run.rst +++ b/docs/source/run.rst @@ -2,10 +2,9 @@ Run Global Workflow ################### -Here we will show how you can run an experiment using the Global Workflow. The Global workflow is regularly evolving and the underlying UFS-weather-model that it drives can run many different configurations. So this part of the document will be regularly updated. The workflow as it is configured today can be run as forecast only or cycled (forecast+Data Assimilation). Since cycled mode requires a number of Data Assimilation supporting repositories to be checked out, the instructions for the two modes from initial checkout stage will be slightly different. Apart from this there is a third mode that is rarely used in development mode and is primarily for operational use. This mode switches on specialized post processing needed by the avaiation industry. Since the files associated with this mode are restricted, only few users will have need and/or ability to run in this mode. +Here we will show how you can run an experiment using the Global Workflow. The Global workflow is regularly evolving and the underlying UFS-weather-model that it drives can run many different configurations. So this part of the document will be regularly updated. The workflow as it is configured today can be run as forecast only or cycled (forecast+Data Assimilation). Since cycled mode requires a number of Data Assimilation supporting repositories to be checked out, the instructions for the two modes from initial checkout stage will be slightly different. Apart from this there is a third mode that is rarely used in development mode and is primarily for operational use. This mode switches on specialized post processing needed by the aviation industry. Since the files associated with this mode are restricted, only select users will have need and/or ability to run in this mode. .. toctree:: - :hidden: clone.rst init.rst diff --git a/docs/source/setup.rst b/docs/source/setup.rst index 28ed697b0d..0432f1ff46 100644 --- a/docs/source/setup.rst +++ b/docs/source/setup.rst @@ -4,57 +4,73 @@ Experiment Setup Global workflow uses a set of scripts to help configure and set up the drivers (also referred to as Workflow Manager) that run the end-to-end system. While currently we use a `ROCOTO `__ based system and that is documented here, an `ecFlow `__ based systm is also under development and will be introduced to the Global Workflow when it is mature. To run the setup scripts, you need to make sure to have a copy of ``python3`` with ``numpy`` available. The easiest way to guarantee this is to load python from the `official hpc-stack installation `_ for the machine you are on: -Hera:: +.. list-table:: Python Module Load Commands + :widths: 25 120 + :header-rows: 1 - module use -a /contrib/anaconda/modulefiles - module load anaconda/anaconda3-5.3.1 + * - **MACHINE** + - **COMMAND(S)** + * - Hera + - :: -Orion:: + module use -a /contrib/anaconda/modulefiles + module load anaconda/anaconda3-5.3.1 + * - Orion + - :: - module load python/3.7.5 + module load python/3.7.5 + * - WCOSS2 + - :: -WCOSS2:: + module load python/3.8.6 + * - S4 + - :: - module load python/3.8.6 - -S4:: - - module load miniconda/3.8-s4 + module load miniconda/3.8-s4 If running with Rocoto make sure to have a Rocoto module loaded before running setup scripts: -Hera:: - - module load rocoto/1.3.3 - -Orion:: +.. list-table:: ROCOTO Module Load Commands + :widths: 25 120 + :header-rows: 1 - module load contrib - module load rocoto/1.3.3 + * - **MACHINE** + - **COMMAND(S)** + * - Hera + - :: -WCOSS2:: + module load rocoto/1.3.3 + * - Orion + - :: - module use /apps/ops/test/nco/modulefiles/ - module load core/rocoto/1.3.5 + module load contrib + module load rocoto/1.3.3 + * - WCOSS2 + - :: -S4:: + module use /apps/ops/test/nco/modulefiles/ + module load core/rocoto/1.3.5 + * - S4 + - :: - module load rocoto/1.3.4 + module load rocoto/1.3.4 ^^^^^^^^^^^^^^^^^^^^^^^^ -Free-forecast experiment +Forecast-only experiment ^^^^^^^^^^^^^^^^^^^^^^^^ Scripts that will be used: - * workflow/setup_expt.py - * workflow/setup_xml.py + * ``workflow/setup_expt.py`` + * ``workflow/setup_xml.py`` *************************************** Step 1: Run experiment generator script *************************************** -The following command examples include variables for reference but users should not use environmental variables but explicit values to submit the commands. Exporting variables like EXPDIR to your environment causes an error when the python scripts run. Please explicitly include the argument inputs when running both setup scripts:: +The following command examples include variables for reference but users should not use environmental variables but explicit values to submit the commands. Exporting variables like EXPDIR to your environment causes an error when the python scripts run. Please explicitly include the argument inputs when running both setup scripts: + +:: cd workflow ./setup_expt.py forecast-only --idate $IDATE --edate $EDATE [--app $APP] [--start $START] [--gfs_cyc $GFS_CYC] [--resdet $RESDET] @@ -63,7 +79,7 @@ The following command examples include variables for reference but users should where: * ``forecast-only`` is the first positional argument that instructs the setup script to produce an experiment directory for forecast only experiments. - * $APP is the target application, one of: + * ``$APP`` is the target application, one of: - ATM: atmosphere-only [default] - ATMW: atm-wave @@ -72,33 +88,39 @@ where: - S2SW: atm-ocean-ice-wave - S2SWA: atm-ocean-ice-wave-aerosols - * $START is the start type (warm or cold [default]) - * $IDATE is the initial start date of your run (first cycle CDATE, YYYYMMDDCC) - * $EDATE is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete - * $PSLOT is the name of your experiment [default: test] - * $CONFIGDIR is the path to the /config folder under the copy of the system you're using [default: $TOP_OF_CLONE/parm/config/] - * $RESDET is the FV3 resolution (i.e. 768 for C768) [default: 384] - * $GFS_CYC is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles) - * $COMROT is the path to your experiment output directory. DO NOT include PSLOT folder at end of path, it’ll be built for you. [default: $HOME] - * $EXPDIR is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you. [default: $HOME] - * $ICSDIR is the path to the initial conditions. This is handled differently depending on whether $APP is S2S or not. + * ``$START`` is the start type (warm or cold [default]) + * ``$IDATE`` is the initial start date of your run (first cycle CDATE, YYYYMMDDCC) + * ``$EDATE`` is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete + * ``$PSLOT`` is the name of your experiment [default: test] + * ``$CONFIGDIR`` is the path to the ``/config`` folder under the copy of the system you're using [default: $TOP_OF_CLONE/parm/config/] + * ``$RESDET`` is the FV3 resolution (i.e. 768 for C768) [default: 384] + * ``$GFS_CYC`` is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles) + * ``$COMROT`` is the path to your experiment output directory. DO NOT include PSLOT folder at end of path, it’ll be built for you. [default: $HOME (but do not use default due to limited space in home directories normally, provide a path to a larger scratch space)] + * ``$EXPDIR`` is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you. [default: $HOME] + * ``$ICSDIR`` is the path to the initial conditions. This is handled differently depending on whether ``$APP`` is S2S or not. - - If $APP is ATM or ATMW, this setting is currently ignored - - If $APP is S2S or S2SW, ICs are copied from the central location to this location and the argument is required + - If ``$APP`` is ATM or ATMW, this setting is currently ignored + - If ``$APP`` is S2S or S2SW, ICs are copied from the central location to this location and the argument is required Examples: -Atm-only:: +Atm-only: + +:: cd workflow ./setup_expt.py forecast-only --pslot test --idate 2020010100 --edate 2020010118 --resdet 384 --gfs_cyc 4 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir -Coupled:: +Coupled: + +:: cd workflow ./setup_expt.py forecast-only --app S2SW --pslot coupled_test --idate 2013040100 --edate 2013040100 --resdet 384 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir --icsdir /some_large_disk_area/Joe.Schmo/icsdir -Coupled with aerosols:: +Coupled with aerosols: + +:: cd workflow ./setup_expt.py forecast-only --app S2SWA --pslot coupled_test --idate 2013040100 --edate 2013040100 --resdet 384 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir --icsdir /some_large_disk_area/Joe.Schmo/icsdir @@ -118,26 +140,23 @@ Go to your EXPDIR and check/change the following variables within your config.ba * HPSS_PROJECT (project on HPSS if archiving) * ATARDIR (location on HPSS if archiving) -If you are using cycling, also change these: - - * imp_physics from 8 (Thompson) to 11 (GFDL) - * CCPP_SUITE to FV3_GFS_v16 (or another suite that uses GFDL) [#]_ - -.. [#] This is a temporary measure until cycling mode works with Thompson - Some of those variables will be found within a machine-specific if-block so make sure to change the correct ones for the machine you'll be running on. -Now is also the time to change any other variables/settings you wish to change in config.base or other configs. `Do that now.` Once done making changes to the configs in your EXPDIR go back to your clone to run the second setup script. See :doc: configure.rst for more information on configuring your run. +Now is also the time to change any other variables/settings you wish to change in config.base or other configs. `Do that now.` Once done making changes to the configs in your EXPDIR go back to your clone to run the second setup script. See :doc:configure.rst for more information on configuring your run. ************************************* Step 3: Run workflow generator script ************************************* -This step sets up the files needed by the Workflow Manager/Driver. At this moment only ROCOTO configurations are generated:: +This step sets up the files needed by the Workflow Manager/Driver. At this moment only ROCOTO configurations are generated: + +:: ./setup_xml.py $EXPDIR/$PSLOT -Example:: +Example: + +:: ./setup_xml.py /some_safe_disk_area/Joe.Schmo/expdir/test @@ -153,14 +172,16 @@ Cycled experiment Scripts that will be used: - * workflow/setup_expt.py - * workflow/setup_xml.py + * ``workflow/setup_expt.py`` + * ``workflow/setup_xml.py`` *************************************** Step 1) Run experiment generator script *************************************** -The following command examples include variables for reference but users should not use environmental variables but explicit values to submit the commands. Exporting variables like EXPDIR to your environment causes an error when the python scripts run. Please explicitly include the argument inputs when running both setup scripts:: +The following command examples include variables for reference but users should not use environmental variables but explicit values to submit the commands. Exporting variables like EXPDIR to your environment causes an error when the python scripts run. Please explicitly include the argument inputs when running both setup scripts: + +:: cd workflow ./setup_expt.py cycled --idate $IDATE --edate $EDATE [--app $APP] [--start $START] [--gfs_cyc $GFS_CYC] @@ -170,59 +191,63 @@ The following command examples include variables for reference but users should where: * ``cycled`` is the first positional argument that instructs the setup script to produce an experiment directory for cycled experiments. - * $APP is the target application, one of[#]_: + * ``$APP`` is the target application, one of: - ATM: atmosphere-only [default] - ATMW: atm-wave - * $IDATE is the initial start date of your run (first cycle CDATE, YYYYMMDDCC) - * $EDATE is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete - * $START is the start type (warm or cold [default]) - * $GFS_CYC is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles) - * $RESDET is the FV3 resolution of the deterministic forecast [default: 384] - * $RESENS is the FV3 resolution of the ensemble (EnKF) forecast [default: 192] - * $NENS is the number of ensemble members [default: 20] - * $CDUMP is the starting phase [default: gdas] - * $PSLOT is the name of your experiment [default: test] - * $CONFIGDIR is the path to the /config folder under the copy of the system you're using [default: $TOP_OF_CLONE/parm/config/] - * $COMROT is the path to your experiment output directory. DO NOT include PSLOT folder at end of path, it’ll be built for you. [default: $HOME] - * $EXPDIR is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you. [default: $HOME] - * $ICSDIR is the path to the ICs for your run if generated separately. [default: None] + * ``$IDATE`` is the initial start date of your run (first cycle CDATE, YYYYMMDDCC) + * ``$EDATE`` is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete + * ``$START`` is the start type (warm or cold [default]) + * ``$GFS_CYC`` is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles) + * ``$RESDET`` is the FV3 resolution of the deterministic forecast [default: 384] + * ``$RESENS`` is the FV3 resolution of the ensemble (EnKF) forecast [default: 192] + * ``$NENS`` is the number of ensemble members [default: 20] + * ``$CDUMP`` is the starting phase [default: gdas] + * ``$PSLOT`` is the name of your experiment [default: test] + * ``$CONFIGDIR`` is the path to the config folder under the copy of the system you're using [default: $TOP_OF_CLONE/parm/config/] + * ``$COMROT`` is the path to your experiment output directory. DO NOT include PSLOT folder at end of path, it’ll be built for you. [default: $HOME] + * ``$EXPDIR`` is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you. [default: $HOME] + * ``$ICSDIR`` is the path to the ICs for your run if generated separately. [default: None] .. [#] More Coupled configurations in cycled mode are currently under development and not yet available -Example:: +Example: + +:: cd workflow ./setup_expt.py cycled --pslot test --configdir /home/Joe.Schmo/git/global-workflow/parm/config --idate 2020010100 --edate 2020010118 --comrot /some_large_disk_area/Joe.Schmo/comrot --expdir /some_safe_disk_area/Joe.Schmo/expdir --resdet 384 --resens 192 --nens 80 --gfs_cyc 4 -Example setup_expt.py on WCOSS_C:: +Example ``setup_expt.py`` on Orion: - SURGE-slogin1 > ./setup_expt.py cycled --pslot fv3demo --idate 2017073118 --edate 2017080106 --comrot /gpfs/hps2/ptmp/Joe.Schmo --expdir /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs - SDATE = 2017-07-31 18:00:00 - EDATE = 2017-08-01 06:00:00 - EDITED: /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base as per user input. - DEFAULT: /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base.default is for reference only. - Please verify and delete the default file before proceeding. - SURGE-slogin1 > +:: + + Orion-login-3$ ./setup_expt.py cycled --pslot test --idate 2022010118 --edate 2022010200 --resdet 192 --resens 96 --nens 80 --comrot /work/noaa/stmp/jschmo/comrot --expdir /work/noaa/global/jschmo/expdir + EDITED: /work/noaa/global/jschmo/expdir/test/config.base as per user input. + EDITED: /work/noaa/global/jschmo/expdir/test/config.aeroanl as per user input. + EDITED: /work/noaa/global/jschmo/expdir/test/config.ocnanal as per user input. The message about the config.base.default is telling you that you are free to delete it if you wish but it’s not necessary to remove. Your resulting config.base was generated from config.base.default and the default one is there for your information. -What happens if I run setup_expt.py again for an experiment that already exists?:: +What happens if I run ``setup_expt.py`` again for an experiment that already exists? + +:: + + Orion-login-3$ ./setup_expt.py cycled --pslot test --idate 2022010118 --edate 2022010200 --resdet 192 --resens 96 --nens 80 --comrot /work/noaa/stmp/jschmo/comrot --expdir /work/noaa/global/jschmo/expdir - SURGE-slogin1 > ./setup_expt.py forecast-only --pslot fv3demo --idate 2017073118 - --edate 2017080106 --comrot /gpfs/hps2/ptmp/Joe.Schmo --expdir /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs - COMROT already exists in /gpfs/hps2/ptmp/Joe.Schmo/fv3demo - Do you wish to over-write COMROT [y/N]: y - EXPDIR already exists in /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo - Do you wish to over-write EXPDIR [y/N]: y - SDATE = 2017-07-31 18:00:00 - EDATE = 2017-08-01 06:00:00 - EDITED: /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base as per user input. - DEFAULT: /gpfs/hps3/emc/global/noscrub/Joe.Schmo/para_gfs/fv3demo/config.base.default is for reference only. - Please verify and delete the default file before proceeding. + directory already exists in /work/noaa/stmp/jschmo/comrot/test -Your COMROT and EXPDIR will be deleted and remade. Be careful with this! + Do you wish to over-write [y/N]: y + + directory already exists in /work/noaa/global/jschmo/expdir/test + + Do you wish to over-write [y/N]: y + EDITED: /work/noaa/global/jschmo/expdir/test/config.base as per user input. + EDITED: /work/noaa/global/jschmo/expdir/test/config.aeroanl as per user input. + EDITED: /work/noaa/global/jschmo/expdir/test/config.ocnanal as per user input. + +Your ``COMROT`` and ``EXPDIR`` will be deleted and remade. Be careful with this! **************************************** Step 2: Set user and experiment settings @@ -248,11 +273,15 @@ Now is also the time to change any other variables/settings you wish to change i Step 3: Run workflow generator script ************************************* -This step sets up the files needed by the Workflow Manager/Driver. At this moment only ROCOTO configurations are generated:: +This step sets up the files needed by the Workflow Manager/Driver. At this moment only ROCOTO configurations are generated: + +:: ./setup_xml.py $EXPDIR/$PSLOT -Example:: +Example: + +:: ./setup_xml.py /some_safe_disk_area/Joe.Schmo/expdir/test @@ -260,5 +289,5 @@ Example:: Step 4: Confirm files from setup scripts **************************************** -You will now have a rocoto xml file in your EXPDIR ($PSLOT.xml) and a crontab file generated for your use. Rocoto uses CRON as the scheduler. If you do not have a crontab file you may not have had the rocoto module loaded. To fix this load a rocoto module and then rerun setup_xml.py script again. Follow directions for setting up the rocoto cron on the platform the experiment is going to run on. +You will now have a rocoto xml file in your EXPDIR ($PSLOT.xml) and a crontab file generated for your use. Rocoto uses CRON as the scheduler. If you do not have a crontab file you may not have had the rocoto module loaded. To fix this load a rocoto module and then rerun ``setup_xml.py`` script again. Follow directions for setting up the rocoto cron on the platform the experiment is going to run on. diff --git a/docs/source/start.rst b/docs/source/start.rst index 7775d2db0c..957971e637 100644 --- a/docs/source/start.rst +++ b/docs/source/start.rst @@ -22,7 +22,8 @@ You'll now have a "logs" folder in both your COMROT and EXPDIR. The EXPDIR log f Set up your experiment cron ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Note: Orion currently only supports cron on Orion-login-1. Cron support for other login nodes is coming in the future. +.. note:: + Orion currently only supports cron on Orion-login-1. Cron support for other login nodes is coming in the future. :: @@ -34,7 +35,9 @@ or crontab $PSLOT.crontab -**WARNING: ``crontab $PSLOT.crontab`` command will overwrite existing crontab file on your login node. If running multiple crons recommend editing crontab file with ``crontab -e`` command.** +.. warning:: + + The ``crontab $PSLOT.crontab`` command will overwrite existing crontab file on your login node. If running multiple crons recommend editing crontab file with ``crontab -e`` command. Check your crontab settings:: diff --git a/docs/source/view.rst b/docs/source/view.rst index 744b4525b1..3093755e9a 100644 --- a/docs/source/view.rst +++ b/docs/source/view.rst @@ -5,7 +5,7 @@ View Experiment output The output from your run will be found in the ``COMROT/ROTDIR`` you established. This is also where you placed your initial conditions. Within your ``COMROT`` you will have the following directory structure (based on the type of experiment you run): ^^^^^^^^^^^^^ -Free Forecast +Forecast-only ^^^^^^^^^^^^^ :: @@ -21,7 +21,7 @@ Cycled :: - enkfgdas.YYYYMMDD/CC/atmos/mem###/ <- contains EnKF inputs/outputs for each cycle and each member + enkfgdas.YYYYMMDD/CC/mem###/atmos <- contains EnKF inputs/outputs for each cycle and each member gdas.YYYYMMDD/CC/atmos <- contains deterministic gdas inputs/outputs (atmosphere) gdas.YYYYMMDD/CC/wave <- contains deterministic gdas inputs/outputs (wave) gfs.YYYYMMDD/CC/atmos <- contains deterministic long forecast gfs inputs/outputs (atmosphere) @@ -29,7 +29,9 @@ Cycled logs/ <- logs for each cycle in the run vrfyarch/ <- contains files related to verification and archival -Here is an example ``COMROT`` for a cycled run as it may look several cycles in (note the archival steps remove older cycle folders as the run progresses):: +Here is an example ``COMROT`` for a cycled run as it may look several cycles in (note the archival steps remove older cycle folders as the run progresses): + +:: -bash-4.2$ ll /scratch1/NCEPDEV/stmp4/Joe.Schmo/comrot/testcyc192 total 88