-
Notifications
You must be signed in to change notification settings - Fork 176
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JEDI based Marine DA tasks #1134
JEDI based Marine DA tasks #1134
Conversation
@aerorahul ,@WalterKolczynski-NOAA , @KateFriedman-NOAA :
|
sorc/link_workflow.sh
Outdated
declare -a JEDI_EXE=("fv3jedi_addincrement.x" \ | ||
"fv3jedi_diffstates.x" \ | ||
"fv3jedi_ensvariance.x" \ | ||
"fv3jedi_hofx.x" \ | ||
"fv3jedi_var.x" \ | ||
"fv3jedi_convertincrement.x" \ | ||
"fv3jedi_dirac.x" \ | ||
"fv3jedi_error_covariance_training.x" \ | ||
"fv3jedi_letkf.x" \ | ||
"fv3jedi_convertstate.x" \ | ||
"fv3jedi_eda.x" \ | ||
"fv3jedi_forecast.x" \ | ||
"fv3jedi_plot_field.x" \ | ||
"fv3jedi_data_checker.py" \ | ||
"fv3jedi_enshofx.x" \ | ||
"fv3jedi_hofx_nomodel.x" \ | ||
"fv3jedi_testdata_downloader.py" ) | ||
JEDI_EXE+=("soca_addincrement.x" \ | ||
"soca_enspert.x" \ | ||
"soca_hofx.x" \ | ||
"soca_ensrecenter.x" \ | ||
"soca_convertincrement.x" \ | ||
"soca_ensvariance.x" \ | ||
"soca_letkf.x" \ | ||
"soca_convertstate.x" \ | ||
"soca_error_covariance_training.x" \ | ||
"soca_setcorscales.x" \ | ||
"soca_diffstates.x" \ | ||
"soca_forecast.x" \ | ||
"soca_sqrtvertloc.x" \ | ||
"soca_dirac.x" \ | ||
"soca_gridgen.x" \ | ||
"soca_staticbinit.x" \ | ||
"soca_enshofx.x" \ | ||
"soca_hofx3d.x" \ | ||
"soca_var.x") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are all of these executables used in the workflow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aerorahul , no, only a fraction are used currently but I'm also realizing that some of these will probably never be used. Should I minimize that list to what will eventually be used or to what's currently used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes. We should only link to what we are actively using. There are other EE2 requirements for how they are built and the source code is linked, but I don't have an idea on how to address that issue. For e.g.gsi.x
comes from a directory gsi.fd
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes this is something that we are going to have to work out with NCO, because it probably will be unreasonable to modify JEDI to put every single executable in its own folder when this is seemingly purely an aesthetic choice
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good.
A few comments and a review from the other JEDI developers would be helpful.
if self.app_config.do_jediocnvar: | ||
dep_dict = {'type': 'task', 'name': f'{self.cdump}ocnanalrun'} | ||
dep = rocoto.add_dependency(dep_dict) | ||
dependencies = rocoto.create_dependency(dep=dep) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does gldas
really need to wait for ocean analysis jobs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rather instead the fcst needs to wait for ocnanal
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aerorahul : What @CoryMartin-NOAA said, forecast waiting for ocnanalrun.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes. I understand that dependency (ocnanalrun to complete for fcst to start). Not sure this is going to be the case if gldas
is turned off.
workflow/rocoto/workflow_tasks.py
Outdated
deps.append(rocoto.add_dependency(dep_dict)) | ||
dependencies = rocoto.create_dependency(dep=deps) | ||
|
||
cycledef = self.cdump |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't think you need this. The default for cycledef = cdump
export R2D2_OBS_DUMP=s2s_v1 | ||
export R2D2_OBS_SRC=gdas_marine | ||
export R2D2_OBS_WINDOW=24 # TODO: Check if the R2D2 sampling DB window is still needed | ||
export COMIN_OBS=/scratch2/NCEPDEV/marineda/r2d2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps add a TODO
here to remind us to move this to a common space.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aerorahul : A common space for R2D2 stuff or for the entirety of config.ocnanal?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for R2D2
parm/config/config.ocnanal
Outdated
@@ -0,0 +1,23 @@ | |||
#!/bin/bash -x |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need a -x
here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few comments, mostly looks good to me
jobs/rocoto/ocnanalpost.sh
Outdated
# Source GDASApp modules | ||
module purge | ||
module use "${HOMEgfs}"/sorc/gdas.cd/modulefiles | ||
module load GDAS/hera |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This shouldn't be machine dependent here.
jobs/rocoto/ocnanalprep.sh
Outdated
# Source GDASApp modules | ||
module purge | ||
module use "${HOMEgfs}"/sorc/gdas.cd/modulefiles | ||
module load GDAS/hera |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same comment
jobs/rocoto/ocnanalrun.sh
Outdated
# Source GDASApp modules | ||
module purge | ||
module use "${HOMEgfs}"/sorc/gdas.cd/modulefiles | ||
module load GDAS/hera |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
x3
npes=16 | ||
;; | ||
*) | ||
echo "FATAL: Resolution not supported'" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will be a problem eventually but fine for now
if self.app_config.do_jediocnvar: | ||
dep_dict = {'type': 'task', 'name': f'{self.cdump}ocnanalrun'} | ||
dep = rocoto.add_dependency(dep_dict) | ||
dependencies = rocoto.create_dependency(dep=dep) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rather instead the fcst needs to wait for ocnanal
Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
Thanks for your reviews/suggested commits @aerorahul and @CoryMartin-NOAA . I've addressed your comments and merged your suggestions. |
jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
Outdated
|
||
# Prepare soca static dir | ||
|
||
ln -s "${COMOUT}"/analysis/* . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this in a j-job? @CoryMartin-NOAA
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for putting the burden of explaining what I did on @CoryMartin-NOAA ! So, Cory, what's up with that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aerorahul I remember what the issue was. All the yaml's , static files, backgrounds, obs, etc ... are created/linked/copied in the "prep" step under ${COMOUT}"/analysis/
and the content of that directory is needed by the run step.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@guillaumevernieres follow what I have here for the aero case:
export DATA=${DATA:-${DATAROOT}/aeroanl_${CDATE}}
mkdir -p "${DATA}"
cd "${DATA}" || (echo "${DATA} does not exist. ABORT!"; exit 1)
global-workflow/jobs/JGDAS_GLOBAL_AERO_ANALYSIS_FINALIZE
Lines 48 to 50 in 1536029
export DATA=${DATA:-${DATAROOT}/aeroanl_${CDATE}} | |
mkdir -p "${DATA}" | |
cd "${DATA}" || (echo "${DATA} does not exist. ABORT!"; exit 1) |
@aerorahul , @CoryMartin-NOAA , I just pushed a commit that addresses most of your comments except that I'm still not doing anything in |
cd "${DATAROOT}" || exit 1 | ||
[[ ${KEEPDATA} = "NO" ]] && rm -rf "${DATA}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if you remove this here, you won't be able to use it in the ocnanalrun
job.
I think you need to keep this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, thanks ... I just removed these lines.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
one todo for me, one thing that needs fixed by @guillaumevernieres
# Remove the Temporary working directory | ||
########################################## | ||
cd "${DATAROOT}" || exit 1 | ||
[[ ${KEEPDATA} = "NO" ]] && rm -rf "${DATA}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't want this one either I don't think unless you have nothing to archive/save from this job back to the ROTDIR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@CoryMartin-NOAA , unless I messed up (very likely), the analysis and increment are dumped in COMOUT
(==ROTDIR
?) ... And as I write this, I realize I forgot to do the same for diag
. Is it OK to have symlinks to directories in COMOUT
/ROTDIR
in which we directly write what we want to keep?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for now it is, but 'eventually' it is not. NCO wants no symlinks, and it makes sense from a solid state drive for run / spinning disk for storage perspective. What my plan is/was for the aerosol tasks (see PR#1106) is to copy everything I want to save from $DATA to $COMOUT as part of the finalize job after the run job.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good @CoryMartin-NOAA , I'll do this in a subsequent PR if that's OK with you (I'm just being polite and respectful in exchange for a green check). My next step is to reconcile my cycling branch with the changes here and make it work again, which I'm pretty sure will require some adjustment/bug fixes in the GDASApp and the g-w. Looking forward to it!
sorc/link_workflow.sh
Outdated
"fv3jedi_data_checker.py" \ | ||
"fv3jedi_enshofx.x" \ | ||
"fv3jedi_hofx_nomodel.x" \ | ||
"fv3jedi_testdata_downloader.py" ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note to myself and @RussTreadon-NOAA we will also want to prune this list for fv3jedi executables at a later date
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved pending conflict resolutions
env/HERA.env
Outdated
@@ -87,6 +87,16 @@ | |||
[[ ${NTHREADS_AEROANL} -gt ${nth_max} ]] && export NTHREADS_AEROANL=${nth_max} | |||
export APRUN_AEROANL="${launcher} -n ${npe_aeroanlrun}" | |||
|
|||
elif [ $step = "ocnanalrun" ]; then |
Check notice
Code scanning / shellcheck
Double quote to prevent globbing and word splitting.
env/ORION.env
Outdated
@@ -86,6 +86,16 @@ | |||
[[ ${NTHREADS_AEROANL} -gt ${nth_max} ]] && export NTHREADS_AEROANL=${nth_max} | |||
export APRUN_AEROANL="${launcher} -n ${npe_aeroanlrun}" | |||
|
|||
elif [ $step = "ocnanalrun" ]; then |
Check notice
Code scanning / shellcheck
Prefer putting braces around variable references even when not strictly required.
env/ORION.env
Outdated
@@ -86,6 +86,16 @@ | |||
[[ ${NTHREADS_AEROANL} -gt ${nth_max} ]] && export NTHREADS_AEROANL=${nth_max} | |||
export APRUN_AEROANL="${launcher} -n ${npe_aeroanlrun}" | |||
|
|||
elif [ $step = "ocnanalrun" ]; then |
Check notice
Code scanning / shellcheck
Double quote to prevent globbing and word splitting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Develop has been updated to eliminate all of the existing shellcheck issues in the *.env
files, so this PR will need to address all those present in the new code as well.
Thanks for your review @WalterKolczynski-NOAA , I've addressed your comments. |
* develop: Correct issue in linking final restart files (NOAA-EMC#1285) Remove execute permissions from config files (NOAA-EMC#1281) Make needed updates to run forecast from GEFS (NOAA-EMC#1203) Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259) Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237) Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280) More marine DA j-jobs (NOAA-EMC#1270) Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265) Add new jjob for ocean analysis bmat (NOAA-EMC#1239) Retire ecf/versions in develop (NOAA-EMC#1267) Deploy documentation to RTD (NOAA-EMC#1264) Temporarily disable failing pytest (NOAA-EMC#1263) Remove incorrect/misleading comments in config.base (NOAA-EMC#1261) Add initial Sphinx documentation (NOAA-EMC#1258) Remove nemsio support (NOAA-EMC#1255) Increase wallclock for diag jobs (NOAA-EMC#1216) Use correct resources for GFS gempak (NOAA-EMC#1214) Abstract common j-job tasks (NOAA-EMC#1230) Add missing mkgfsawps.x link (NOAA-EMC#1218) Fix post sounding job (NOAA-EMC#1212) Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240) Use fracoro data for all new UFS applications (NOAA-EMC#1182) Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238) Add more user defined parameters for the marine DA (NOAA-EMC#1235) Update pytests action version and run sequentially (NOAA-EMC#1236) Add utility to compare Fortran namelists (NOAA-EMC#1234) Updates for pygw (NOAA-EMC#1231) Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158) Move member up in directory hierarchy (NOAA-EMC#1201) Enable staging ics for cycled experiments. (NOAA-EMC#1199) Add tests for configuration.py (NOAA-EMC#1192) Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191) define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193) Fix checking for restart files (NOAA-EMC#1186) Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188) Update archive job memory request value for R&Ds (NOAA-EMC#1183) Reorder post so all flux files are generated when running offline (NOAA-EMC#1181) Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179) Add missing jobids in some pre-job scripts (NOAA-EMC#1176) Remove existing directory if it exists when getic runs (NOAA-EMC#1165) Add logging decorator, test and test for yaml_file (NOAA-EMC#1178) fix coding norm check in `hosts.py` (NOAA-EMC#1174) Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172) Support for the GDASApp testing in containers (NOAA-EMC#1151) ATM 3DVAR with and without IAU (NOAA-EMC#1113) Enable checking for python norms and fix violating code (NOAA-EMC#1168) Enforce decimal math in atmos post (NOAA-EMC#1171) Update marine DA j-jobs to new format (NOAA-EMC#1149) Add utility to manipulate files en masse (NOAA-EMC#1166) add action to run pytests (NOAA-EMC#1167) Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162) Add a task base class and basic logger (NOAA-EMC#1160) Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154) move configuration.py to pygw. Use it from there. return AttrDict after sourcing configs (NOAA-EMC#1153) JEDI based Marine DA tasks (NOAA-EMC#1134) Allow customizations based on user/configuration (NOAA-EMC#1146) First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120) enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142) Fix typo in .shellcheckrc Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138) Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132) Address shellcheck warnings in env files (NOAA-EMC#1136) Adds group size and nmem for GEFS (NOAA-EMC#1127) Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133) Convert archive jobs to proper j-jobs (NOAA-EMC#1115) Update C48 forecast to run with one thread (NOAA-EMC#1131) Improved error messages from atmos analysis (NOAA-EMC#1125) Update MODULEPATH for Orion (NOAA-EMC#1126) MPMD variable updates and fix (NOAA-EMC#1124) Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122) Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112) Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111) Correct syntax in remaining sorc scripts (NOAA-EMC#1105) Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104) Add Early Cycle EnKF workflow (NOAA-EMC#1022) Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101) Fixed gfs-utils links (NOAA-EMC#1099) Fix build scripts and bring into compliance (NOAA-EMC#1096) Feature/updates for gdas app (NOAA-EMC#1091) Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094) Resource updates to support WCOSS2 (NOAA-EMC#1070) Set COMPILER in link for detect machine (NOAA-EMC#1092) gfs utils update (NOAA-EMC#1088) GFS-UTILS update for build and ush scripts (NOAA-EMC#1082) Update UFS version to 2022 Oct 19 (NOAA-EMC#1083) Use more cycledefs for task control (NOAA-EMC#1078) removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079) Update UFS to Sept 9 version (NOAA-EMC#1073) Modify default file location for monitor data when using rocoto (NOAA-EMC#1065) Fix companion ocean resolution for C48 (NOAA-EMC#1066) Add trailing slash for gldas topo path (NOAA-EMC#1064) Limit number of CPU for post (NOAA-EMC#1061) Fix eupd trace (NOAA-EMC#1057) Port to S4 (NOAA-EMC#1023) Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049) Add GDAS to the partial build list (NOAA-EMC#1050) Fix group number being treated as octal in gdas arch (NOAA-EMC#1053) Remove trace from link script (NOAA-EMC#1046) Update gfs-utils hash to 3a609ea (NOAA-EMC#1048) Fix link script usage statement (NOAA-EMC#1045) Replace preamble variable commands with functions (NOAA-EMC#1012) Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009) Rename post scripts (NOAA-EMC#1038) Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039) WCOSS2 run support and script/config updates (NOAA-EMC#1030) Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036) initial commit for incoming yaml work (NOAA-EMC#1029) Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026) Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018) Update GSI Monitor for radmon fix Correct shell linter config (NOAA-EMC#1013) Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016) Add shell linter Github action for pull requests (NOAA-EMC#1007) Build updates for WCOSS2 (NOAA-EMC#1002) Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001) Fix preamble id (NOAA-EMC#996) Add missing "atmos" into job dependencies (NOAA-EMC#998) Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992) Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990) Add GSI monitor scripts (NOAA-EMC#969) Fix product generation at some fcst hrs (NOAA-EMC#988) Add initial config files for global aerosol DA (NOAA-EMC#986) Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979) use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970) Externals.cfg was stale and had drifted off (NOAA-EMC#965) Fix post comparison with zero-padded numbers (NOAA-EMC#964)
Description
This PR adds the relevant scripts/steps to exercise the JEDI based marine DA eventually needed for GFSv17. See issue #1072 for more details.
The branch used for this PR was extracted from a "rogue" branch that included changes to the model related scripts to allow for WCDA cycling. The testing was done with the coupled
C48/5deg
atmos/ocean model, using theGSI
for the amos initialization and the JEDI based marine DA system (SOCA) for the ocean initialization.It is probably step 1 of at least 2 steps to get this work to a functional state in develop. I just don't know yet who's responsibility it is to fix the coupled model scripting to make it work in cycled mode.
A few notes:
SOCA
fixed files has yet to be coordinated: An environment variable inconfig.ocnanal
is pointing to/scratch2/NCEPDEV/ocean/Guillaume.Vernieres/data/static/72x35x25/
. This needs to be copied elsewhere.Hera
.Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Checklist