upgrade MERRA2 climatology from 2003-2014 mean to 2014 to 2024 mean #2273

AnningCheng-NOAA · 2024-05-09T13:59:30Z

Commit Queue Requirements:

Fill out all sections of this template.
All sub component pull requests have been reviewed by their code managers.
Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
Commit 'test_changes.list' from previous step

Description:

upgrade MERRA2 climatology from 2003-2014 mean to 2014 to 2024 mean

Commit Message:

* UFSWM - Upgrade MERRA2 climatology from 2003-2014 mean to 2014 to 2024 mean

Priority:

Normal

Git Tracking

UFSWM:

None

Sub component Pull Requests:

None

UFSWM Blocking Dependencies:

None

Changes

Regression Test Changes (Please commit test_changes.list):

PR Updates/Changes Baselines.
All tests using MERRA2 climatology needs new baseline

Input data Changes:

New input data.

Library Changes/Upgrades:

No Updates

Testing Log:

RDHPCS
- Hera

RegressionTests_hera.log

zach1221 · 2024-05-22T12:25:52Z

@AnningCheng-NOAA can you please sync up this PR branch for us? We're looking to begin testing this PR next.

Also, please add a commit message to the template.

zach1221 · 2024-05-22T13:21:16Z

@BrianCurtis-NOAA once the PR is synced up, I'm going to re-run on Hera to confirm which cases are changing, as there are quite a few failures in the previously attached Hera log vs the tests_changes.list.

tests/fv3_conf/control_run.IN

AnningCheng-NOAA · 2024-05-22T15:56:31Z

I do not have the right to access @[INPUTDATA_ROOT. Could anybody ro resolve this issue?

…

On Wed, May 22, 2024 at 10:42 AM Dusan Jovic ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tests/fv3_conf/control_run.IN <#2273 (comment)> : > @@ -133,7 +133,7 @@ fi #merra2 if [ $USE_MERRA2 = .true. ]; then for n in 01 02 03 04 05 06 07 08 09 10 11 12; do - cp @[INPUTDATA_ROOT]/FV3_input_data_INCCN_aeroclim/MERRA2/merra2.aerclim.2003-2014.m${n}.nc aeroclim.m${n}.nc + cp /scratch1/NCEPDEV/global/Anning.Cheng/MERRA2_y14_24/merra2_1423_${n}.nc aeroclim.m${n}.nc The input data should be copied from INPUTDATA_ROOT, not personal directories. — Reply to this email directly, view it on GitHub <#2273 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALQPMIJJ72SHPZTW6LWCVQ3ZDSVF5AVCNFSM6AAAAABHO33YMKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDANZRGQ4TEOJTGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

zach1221 · 2024-05-22T18:35:09Z

RegressionTests_hera.log

BrianCurtis-NOAA · 2024-05-22T20:01:08Z

@zach1221 You're missing two things, one was the label which I took care of and two, addressing any remaining issues in the PR. For the second item, you'll need to let @AnningCheng-NOAA know where the new input data was placed, so they can update their repository with the updated location. Thus this PR is not ready for the commit queue quite yet.

BrianCurtis-NOAA · 2024-05-22T20:03:30Z

@zach1221 Also we'll need an updated test_changes.list

zach1221 · 2024-05-22T20:34:50Z

@zach1221 Also we'll need an updated test_changes.list

Ok, I can add a test change log, I wasn't aware this was my responsibility. I will do this going forward if it's not included.

BrianCurtis-NOAA · 2024-05-23T19:31:42Z

@zach1221 are you using -b test_changes.list in your work for this PR ?

FernandoAndrade-NOAA · 2024-05-23T19:45:09Z

@zach1221 are you using -b test_changes.list in your work for this PR ?

I think he created a temp rt.conf with just the changed tests. Running with -b resulted in rt.sh thinking the tests were not available to run on jet and gaea from my side unless I ran with the full -l rt.conf instead.

BrianCurtis-NOAA · 2024-05-23T19:52:25Z

@zach1221 are you using -b test_changes.list in your work for this PR ?

I think he created a temp rt.conf with just the changed tests. Running with -b resulted in rt.sh thinking the tests were not available to run on jet and gaea from my side unless I ran with the full -l rt.conf instead.

I use it each PR, and this PR caused the issue you describe for me. Would I be correct in thinking the test_changes.list was edited in an editor and not coming from the generated one from rt.sh? I can't seem to find out why this PR is causing the issues, and that is my guess at this point.

FernandoAndrade-NOAA · 2024-05-23T20:02:08Z

@zach1221 are you using -b test_changes.list in your work for this PR ?

I think he created a temp rt.conf with just the changed tests. Running with -b resulted in rt.sh thinking the tests were not available to run on jet and gaea from my side unless I ran with the full -l rt.conf instead.

I use it each PR, and this PR caused the issue you describe for me. Would I be correct in thinking the test_changes.list was edited in an editor and not coming from the generated one from rt.sh? I can't seem to find out why this PR is causing the issues, and that is my guess at this point.

Would manually edited tests into test_changes.list cause an issue with -b's behavior? If so it may be worth seeing if future improvements could allow it to accept the listed tests in any order. It's a bit tricky to diagnose, I'm not getting any real error output other than rt.sh rejecting the specified tests and exiting.

BrianCurtis-NOAA · 2024-05-23T20:08:09Z

@zach1221 are you using -b test_changes.list in your work for this PR ?

I think he created a temp rt.conf with just the changed tests. Running with -b resulted in rt.sh thinking the tests were not available to run on jet and gaea from my side unless I ran with the full -l rt.conf instead.

I use it each PR, and this PR caused the issue you describe for me. Would I be correct in thinking the test_changes.list was edited in an editor and not coming from the generated one from rt.sh? I can't seem to find out why this PR is causing the issues, and that is my guess at this point.

Would manually edited tests into test_changes.list cause an issue with -b's behavior? If so it may be worth seeing if future improvements could allow it to accept the listed tests in any order. It's a bit tricky to diagnose, I'm not getting any real error output other than rt.sh rejecting the specified tests and exiting.

Right, me either. But rt.sh hates that file, somehow.

BrianCurtis-NOAA · 2024-05-23T20:11:25Z

I've compared two test_changes.list and there is a difference:

brian.curtis@alogin01:/lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests> cat -A ../../../BrianCurtis-NOAA/ufs-weather-model/develop/tests/test_changes.list 
cpld_control_p8_mixedmode intel$
cpld_control_gfsv17 intel$
cpld_control_gfsv17_iau intel$
cpld_restart_gfsv17 intel$
cpld_mpi_gfsv17 intel$
cpld_debug_gfsv17 intel$
cpld_control_p8 intel$
cpld_control_p8.v2.sfc intel$
cpld_restart_p8 intel$
cpld_control_qr_p8 intel$
cpld_restart_qr_p8 intel$
cpld_2threads_p8 intel$
cpld_decomp_p8 intel$
cpld_mpi_p8 intel$
cpld_control_ciceC_p8 intel$
cpld_control_c192_p8 intel$
cpld_restart_c192_p8 intel$
cpld_bmark_p8 intel$
cpld_restart_bmark_p8 intel$
cpld_s2sa_p8 intel$
cpld_control_noaero_p8 intel$
cpld_control_nowave_noaero_p8 intel$
cpld_debug_p8 intel$
cpld_debug_noaero_p8 intel$
cpld_control_noaero_p8_agrid intel$
cpld_control_c48 intel$
cpld_control_p8_faster intel$
cpld_control_pdlib_p8 intel$
cpld_restart_pdlib_p8 intel$
cpld_mpi_pdlib_p8 intel$
cpld_debug_pdlib_p8 intel$
control_flake intel$
control_CubedSphereGrid intel$
control_CubedSphereGrid_parallel intel$
control_latlon intel$
control_wrtGauss_netcdf_parallel intel$
control_c48 intel$
control_c48.v2.sfc intel$
control_c192 intel$
control_c384 intel$
control_c384gdas intel$
control_stochy intel$
control_stochy_restart intel$
control_lndp intel$
control_iovr4 intel$
control_iovr5 intel$
control_p8 intel$
control_p8.v2.sfc intel$
control_p8_ugwpv1 intel$
control_restart_p8 intel$
control_noqr_p8 intel$
control_restart_noqr_p8 intel$
control_decomp_p8 intel$
control_2threads_p8 intel$
control_p8_lndp intel$
control_p8_rrtmgp intel$
control_p8_mynn intel$
merra2_thompson intel$
rap_control intel$
rap_decomp intel$
rap_2threads intel$
rap_restart intel$
rap_sfcdiff intel$
rap_sfcdiff_decomp intel$
rap_sfcdiff_restart intel$
hrrr_control intel$
hrrr_control_decomp intel$
hrrr_control_2threads intel$
hrrr_control_restart intel$
rrfs_v1beta intel$
rrfs_v1nssl intel$
rrfs_v1nssl_nohailnoccn intel$
control_csawmg intel$
control_csawmgt intel$
control_ras intel$
control_wam intel$
control_p8_faster intel$
control_CubedSphereGrid_debug intel$
control_wrtGauss_netcdf_parallel_debug intel$
control_stochy_debug intel$
control_lndp_debug intel$
control_csawmg_debug intel$
control_csawmgt_debug intel$
control_ras_debug intel$
control_diag_debug intel$
control_debug_p8 intel$
rap_control_debug intel$
hrrr_control_debug intel$
hrrr_gf_debug intel$
hrrr_c3_debug intel$
rap_unified_drag_suite_debug intel$
rap_diag_debug intel$
rap_cires_ugwp_debug intel$
rap_unified_ugwp_debug intel$
rap_lndp_debug intel$
rap_progcld_thompson_debug intel$
rap_noah_debug intel$
rap_sfcdiff_debug intel$
rap_noah_sfcdiff_cires_ugwp_debug intel$
rrfs_v1beta_debug intel$
rap_clm_lake_debug intel$
rap_flake_debug intel$
gnv1_c96_no_nest_debug intel$
control_wam_debug intel$
rap_control_dyn32_phy32 intel$
hrrr_control_dyn32_phy32 intel$
rap_2threads_dyn32_phy32 intel$
hrrr_control_2threads_dyn32_phy32 intel$
hrrr_control_decomp_dyn32_phy32 intel$
rap_restart_dyn32_phy32 intel$
hrrr_control_restart_dyn32_phy32 intel$
conus13km_control intel$
conus13km_2threads intel$
conus13km_restart_mismatch intel$
rap_control_dyn64_phy32 intel$
rap_control_debug_dyn32_phy32 intel$
hrrr_control_debug_dyn32_phy32 intel$
conus13km_debug intel$
conus13km_debug_qr intel$
conus13km_debug_2threads intel$
conus13km_radar_tten_debug intel$
rap_control_dyn64_phy32_debug intel$
hafs_global_1nest_atm intel$
hafs_global_multiple_4nests_atm intel$
hafs_global_storm_following_1nest_atm intel$
gnv1_nested intel$
control_p8_atmlnd_sbs intel$
control_p8_atmlnd intel$
control_restart_p8_atmlnd intel$
atmwav_control_noaero_p8 intel$
control_atmwav intel$
atmaero_control_p8 intel$
atmaero_control_p8_rad intel$
atmaero_control_p8_rad_micro intel$
control_c48 gnu$
control_stochy gnu$
control_ras gnu$
control_p8 gnu$
control_p8_ugwpv1 gnu$
control_flake gnu$
rap_control gnu$
rap_decomp gnu$
rap_2threads gnu$
rap_restart gnu$
rap_sfcdiff gnu$
rap_sfcdiff_decomp gnu$
rap_sfcdiff_restart gnu$
hrrr_control gnu$
hrrr_control_noqr gnu$
hrrr_control_2threads gnu$
hrrr_control_decomp gnu$
hrrr_control_restart gnu$
hrrr_control_restart_noqr gnu$
rrfs_v1beta gnu$
control_diag_debug gnu$
rap_control_debug gnu$
hrrr_control_debug gnu$
hrrr_gf_debug gnu$
hrrr_c3_debug gnu$
rap_diag_debug gnu$
rap_noah_sfcdiff_cires_ugwp_debug gnu$
rap_progcld_thompson_debug gnu$
rrfs_v1beta_debug gnu$
control_ras_debug gnu$
control_stochy_debug gnu$
control_debug_p8 gnu$
rap_flake_debug gnu$
rap_clm_lake_debug gnu$
gnv1_c96_no_nest_debug gnu$
control_wam_debug gnu$
rap_control_dyn32_phy32 gnu$
hrrr_control_dyn32_phy32 gnu$
rap_2threads_dyn32_phy32 gnu$
hrrr_control_2threads_dyn32_phy32 gnu$
hrrr_control_decomp_dyn32_phy32 gnu$
rap_restart_dyn32_phy32 gnu$
hrrr_control_restart_dyn32_phy32 gnu$
conus13km_control gnu$
conus13km_2threads gnu$
conus13km_restart_mismatch gnu$
rap_control_dyn64_phy32 gnu$
rap_control_debug_dyn32_phy32 gnu$
hrrr_control_debug_dyn32_phy32 gnu$
conus13km_debug gnu$
conus13km_debug_qr gnu$
conus13km_debug_2threads gnu$
conus13km_radar_tten_debug gnu$
rap_control_dyn64_phy32_debug gnu$
cpld_control_p8 gnu$
cpld_control_nowave_noaero_p8 gnu$
cpld_debug_p8 gnu$
cpld_control_pdlib_p8 gnu$
cpld_debug_pdlib_p8 gnu$
brian.curtis@alogin01:/lfs/h1/emc/nems/noscrub/brian.curtis/git/AnningCheng-NOAA/ufs-weather-model/tests> cat -A test_changes.list
cpld_control_p8_mixedmode intel^M$
cpld_control_gfsv17 intel^M$
^M$

I need to know what causes the ^M that the cat -A sees.

BrianCurtis-NOAA · 2024-05-23T20:15:34Z

OK update: ^M is from a Windows editor. I've cleaned that out on my end and it works OK.

@jkbk2004 @zach1221 @FernandoAndrade-NOAA Please take note of this.

…l into mr2_24

FernandoAndrade-NOAA · 2024-05-23T20:32:46Z

OK update: ^M is from a Windows editor. I've cleaned that out on my end and it works OK.

@jkbk2004 @zach1221 @FernandoAndrade-NOAA Please take note of this.

Would this be caused by not specifying LF in a windows editor?

FernandoAndrade-NOAA · 2024-05-23T20:36:45Z

OK update: ^M is from a Windows editor. I've cleaned that out on my end and it works OK.
@jkbk2004 @zach1221 @FernandoAndrade-NOAA Please take note of this.

Would this be caused by not specifying LF in a windows editor?

Never mind, your commit message pretty much confirms that, nice catch!

BrianCurtis-NOAA · 2024-05-24T14:10:38Z

Hey @SamuelTrahanNOAA I'm getting failures on Acorn for IFI tests (control decomp 2threads). I'm only curious if these changes would cause baseline changes in the IFI tests. If yes, no problem and i'll generate new baselines. If no, then we need to figure out why all files in the test are not identical in comparison.

FernandoAndrade-NOAA · 2024-05-24T14:27:39Z

@AnningCheng-NOAA we are getting close to starting the merge process, please be sure to fill out the Commit Message section of the PR template.

jkbk2004 · 2024-05-24T14:41:51Z

Hey @SamuelTrahanNOAA I'm getting failures on Acorn for IFI tests (control decomp 2threads). I'm only curious if these changes would cause baseline changes in the IFI tests. If yes, no problem and i'll generate new baselines. If no, then we need to figure out why all files in the test are not identical in comparison.

https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/tests/regional_ifi_control#L22

BrianCurtis-NOAA · 2024-05-24T15:10:55Z

Hey @SamuelTrahanNOAA I'm getting failures on Acorn for IFI tests (control decomp 2threads). I'm only curious if these changes would cause baseline changes in the IFI tests. If yes, no problem and i'll generate new baselines. If no, then we need to figure out why all files in the test are not identical in comparison.

https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/tests/regional_ifi_control#L22

OK, somehow I thought there were physics changes to this. It makes full sense that the IFI tests are changing here. Thanks.

Use the updated 2013 to 2024 mean MERRA2 climatology instead of 2003 to 2014 mean Depends on #2887 Refs: ufs-community/ufs-weather-model#2272 Refs: ufs-community/ufs-weather-model#2273

AnningCheng-NOAA added 2 commits May 7, 2024 22:07

MERRA2 climatology upgraded to 2014 to 2024 mean

916e7c1

updated MERRA2 climatology to 2014-2024 mean

c29c139

DeniseWorthen mentioned this pull request May 9, 2024

upgrade MERRA2 climatology from 2003-2014 mean to 2014-2024 mean #2272

Closed

jkbk2004 requested review from junwang-noaa and DeniseWorthen May 14, 2024 15:38

zach1221 added the Baseline Updates Current baselines will be updated. label May 22, 2024

jkbk2004 added 2 commits May 22, 2024 13:49

revert hera log: see attached

850c7ac

Merge remote-tracking branch 'upstream/develop' into mr2_24

b439d3d

DusanJovic-NOAA reviewed May 22, 2024

View reviewed changes

tests/fv3_conf/control_run.IN Outdated Show resolved Hide resolved

Update bl_date.conf

869a754

zach1221 added the Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. label May 22, 2024

BrianCurtis-NOAA added the New Input Data Req'd This PR requires new data to be sync across platforms label May 22, 2024

jkbk2004 and others added 11 commits May 22, 2024 16:54

Update control_run.IN

84aee85

Update cpld_control_run.IN

9ee63e2

Update fv3_gocart.IN

a290128

Update lake_control_run.IN

48119f1

Update regional_rrfs_a_run.IN

cd878fa

Update regional_run.IN

9adf5dc

Update test_changes.list

af9c4e9

Update test_changes.list

f8c5637

add orion RT logs: passed

01d18ab

add hercules RT logs: passed

012fc95

add derecho RT logs: passed

d2b009b

jkbk2004 added 2 commits May 23, 2024 08:50

add derecho RT log: passed

939f109

add hera RT log: passed

7739bac

add gaea RT log: passed

f77f8cc

BrianCurtis-NOAA added 2 commits May 23, 2024 20:17

updated test_changes.list with the windows carraige returns removed

e8a4447

Merge branch 'mr2_24' of github.com:AnningCheng-NOAA/ufs-weather-mode…

ac5a2ee

…l into mr2_24

jkbk2004 and others added 2 commits May 24, 2024 11:49

add jet RT log: passed

eb0c662

WCOS2 RT Log

af395d3

Acorn RT Log

982cade

jkbk2004 requested review from BrianCurtis-NOAA and FernandoAndrade-NOAA May 24, 2024 16:36

jkbk2004 approved these changes May 24, 2024

View reviewed changes

FernandoAndrade-NOAA approved these changes May 24, 2024

View reviewed changes

jkbk2004 merged commit 3a48a64 into ufs-community:develop May 24, 2024
3 checks passed

This was referenced Sep 3, 2024

MERRA2 climatology updated from 2003-2014 mean to 2013-2024 mean NOAA-EMC/global-workflow#2887

Closed

Update aerosol climatology to 2013-2024 mean NOAA-EMC/global-workflow#2888

Merged

NickSzapiro-NOAA mentioned this pull request Sep 18, 2024

Updated aerosol climatology not reflected in fix files #2441

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade MERRA2 climatology from 2003-2014 mean to 2014 to 2024 mean #2273

upgrade MERRA2 climatology from 2003-2014 mean to 2014 to 2024 mean #2273

AnningCheng-NOAA commented May 9, 2024 •

edited by FernandoAndrade-NOAA

Loading

zach1221 commented May 22, 2024 •

edited

Loading

zach1221 commented May 22, 2024

AnningCheng-NOAA commented May 22, 2024 via email

zach1221 commented May 22, 2024

BrianCurtis-NOAA commented May 22, 2024

BrianCurtis-NOAA commented May 22, 2024

zach1221 commented May 22, 2024

BrianCurtis-NOAA commented May 23, 2024

FernandoAndrade-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 23, 2024

FernandoAndrade-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 23, 2024

FernandoAndrade-NOAA commented May 23, 2024 •

edited

Loading

FernandoAndrade-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 24, 2024 •

edited

Loading

FernandoAndrade-NOAA commented May 24, 2024

jkbk2004 commented May 24, 2024

BrianCurtis-NOAA commented May 24, 2024

upgrade MERRA2 climatology from 2003-2014 mean to 2014 to 2024 mean #2273

upgrade MERRA2 climatology from 2003-2014 mean to 2014 to 2024 mean #2273

Conversation

AnningCheng-NOAA commented May 9, 2024 • edited by FernandoAndrade-NOAA Loading

Commit Queue Requirements:

Description:

Commit Message:

Priority:

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

Changes

Regression Test Changes (Please commit test_changes.list):

Input data Changes:

Library Changes/Upgrades:

Testing Log:

zach1221 commented May 22, 2024 • edited Loading

zach1221 commented May 22, 2024

AnningCheng-NOAA commented May 22, 2024 via email

zach1221 commented May 22, 2024

BrianCurtis-NOAA commented May 22, 2024

BrianCurtis-NOAA commented May 22, 2024

zach1221 commented May 22, 2024

BrianCurtis-NOAA commented May 23, 2024

FernandoAndrade-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 23, 2024

FernandoAndrade-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 23, 2024

FernandoAndrade-NOAA commented May 23, 2024 • edited Loading

FernandoAndrade-NOAA commented May 23, 2024

BrianCurtis-NOAA commented May 24, 2024 • edited Loading

FernandoAndrade-NOAA commented May 24, 2024

jkbk2004 commented May 24, 2024

BrianCurtis-NOAA commented May 24, 2024

AnningCheng-NOAA commented May 9, 2024 •

edited by FernandoAndrade-NOAA

Loading

zach1221 commented May 22, 2024 •

edited

Loading

FernandoAndrade-NOAA commented May 23, 2024 •

edited

Loading

BrianCurtis-NOAA commented May 24, 2024 •

edited

Loading