Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial stochastic physics implementation #48

Closed
wants to merge 36 commits into from
Closed
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
671c714
Merge pull request #1 from NOAA-EMC/dev/emc
pjpegion May 5, 2020
182ef34
additions for stochastic physics and ePBL perts
May 5, 2020
c2aa2a8
updates from dev/emc
May 5, 2020
3cad1ba
Merge pull request #8 from NOAA-EMC/dev/emc
pjpegion Oct 1, 2020
0a62737
Merge branch 'ocn_stoch' into dev/emc_merge
pjpegion Oct 1, 2020
9896d61
Merge pull request #9 from pjpegion/dev/emc_merge
pjpegion Oct 1, 2020
cd06356
Merge pull request #11 from NOAA-EMC/dev/emc
pjpegion Dec 2, 2020
7de295c
cleanup of code and enhancement of ePBL perts
pjpegion Dec 2, 2020
7212400
Update MOM_diabatic_driver.F90
pjpegion Dec 2, 2020
bd477a9
Update MOM_diabatic_driver.F90
pjpegion Dec 2, 2020
167a62e
Merge pull request #12 from pjpegion/dev/emc
pjpegion Dec 2, 2020
0c15f4c
Update MOM_diabatic_driver.F90
pjpegion Dec 2, 2020
a2a374b
add stochy_restart writing to mom_cap
pjpegion Dec 14, 2020
25ed5ef
additions for stochy restarts
pjpegion Dec 22, 2020
4bd9b9e
clean up debug statements
pjpegion Dec 23, 2020
1dc0f4f
Merge remote-tracking branch 'upstream/dev/emc' into dev/emc
pjpegion Dec 23, 2020
2cba995
Merge branch 'dev/emc' into ocn_stoch
pjpegion Dec 23, 2020
040e1f1
Merge pull request #13 from NOAA-EMC/dev/emc
pjpegion Jan 6, 2021
1d7ffa3
clean up code
pjpegion Jan 6, 2021
6bb9d0b
fix non stochastic ePBL calculation
pjpegion Jan 7, 2021
600ebf9
Merge remote-tracking branch 'upstream/dev/emc' into ocn_stoch
pjpegion Jan 22, 2021
1727d9a
re-write of stochastic code to remove CPP directives
pjpegion Jan 29, 2021
5443f8e
remove blank link in MOM_diagnostics
pjpegion Jan 29, 2021
80f9f44
clean up MOM_domains
pjpegion Jan 29, 2021
85023f8
Merge remote-tracking branch 'upstream/dev/emc' into ocn_stoch
pjpegion Feb 1, 2021
0b99c1f
make stochastics optional
pjpegion Feb 2, 2021
6e3ea1b
correct coupled_driver/ocean_model_MOM.F90 and other cleanup
pjpegion Feb 2, 2021
eb88219
clean up of code for MOM6 coding standards
pjpegion Feb 2, 2021
d984a7e
remove stochastics container
pjpegion Feb 4, 2021
b8d9888
place stochastic array in fluxes container and make SPPT specific arr…
pjpegion Feb 4, 2021
25ed4fc
revert MOM_domains.F90
pjpegion Feb 5, 2021
8afe969
clean up of mom_ocean_model_nuopc.F90
pjpegion Feb 5, 2021
689a73f
remove PE_here from mom_ocean_model_nuopc.F90
pjpegion Feb 5, 2021
a4c0411
Merge remote-tracking branch 'upstream/dev/emc' into ocn_stoch
pjpegion Feb 16, 2021
565e0bb
remove debug statements
pjpegion Feb 26, 2021
61717ee
Merge remote-tracking branch 'origin/dev/emc' into ocn_stoch
pjpegion Feb 26, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions config_src/nuopc_driver/mom_cap.F90
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ module MOM_cap_mod
use NUOPC_Model, only: model_label_SetRunClock => label_SetRunClock
use NUOPC_Model, only: model_label_Finalize => label_Finalize
use NUOPC_Model, only: SetVM
use get_stochy_pattern_mod, only: write_stoch_restart_ocn

implicit none; private

Expand Down Expand Up @@ -1585,6 +1586,16 @@ subroutine ModelAdvance(gcomp, rc)

! write restart file(s)
call ocean_model_restart(ocean_state, restartname=restartname)

DeniseWorthen marked this conversation as resolved.
Show resolved Hide resolved
! write stochastic physics restart file if active
if (ESMF_AlarmIsRinging(stop_alarm, rc=rc)) then
write(restartname,'(A)')"ocn_stoch.res.nc"
else
write(restartname,'(A,I4.4,"-",I2.2,"-",I2.2,"-",I2.2,"-",I2.2,"-",I2.2,A)') &
"ocn_stoch.res.", year, month, day, hour, minute, seconds,".nc"
endif
call ESMF_LogWrite("MOM_cap: Writing restart : "//trim(restartname), ESMF_LOGMSG_INFO)
call write_stoch_restart_ocn('RESTART/'//trim(restartname))
endif

if (is_root_pe()) then
Expand Down
69 changes: 62 additions & 7 deletions config_src/nuopc_driver/mom_ocean_model_nuopc.F90
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ module MOM_ocean_model_nuopc
use MOM_tracer_flow_control, only : call_tracer_register, tracer_flow_control_init
use MOM_tracer_flow_control, only : call_tracer_flux_init
use MOM_unit_scaling, only : unit_scale_type
use MOM_variables, only : surface
use MOM_variables, only : surface, stochastic_pattern
use MOM_verticalGrid, only : verticalGrid_type
use MOM_ice_shelf, only : initialize_ice_shelf, shelf_calc_flux, ice_shelf_CS
use MOM_ice_shelf, only : add_shelf_forces, ice_shelf_end, ice_shelf_save_restart
Expand All @@ -62,6 +62,8 @@ module MOM_ocean_model_nuopc
use MOM_surface_forcing_nuopc, only : convert_IOB_to_forces, ice_ocn_bnd_type_chksum
use MOM_surface_forcing_nuopc, only : ice_ocean_boundary_type, surface_forcing_CS
use MOM_surface_forcing_nuopc, only : forcing_save_restart
use MOM_domains, only : root_PE,PE_here,Get_PElist,num_PEs
use stochastic_physics, only : init_stochastic_physics_ocn, run_stochastic_physics_ocn

#include <MOM_memory.h>

Expand Down Expand Up @@ -174,6 +176,8 @@ module MOM_ocean_model_nuopc
!! steps can span multiple coupled time steps.
logical :: diabatic_first !< If true, apply diabatic and thermodynamic
!! processes before time stepping the dynamics.
logical :: do_sppt !< If true, allocate array for SPPT
logical :: pert_epbl !< If true, allocate arrays for energetic PBL perturbations

real :: eps_omesh !< Max allowable difference between ESMF mesh and MOM6
!! domain coordinates
Expand All @@ -187,6 +191,7 @@ module MOM_ocean_model_nuopc
!! timesteps are taken per thermodynamic step.
type(surface) :: sfc_state !< A structure containing pointers to
!! the ocean surface state fields.
type(stochastic_pattern) :: stochastics !< A structure containing pointers to
type(ocean_grid_type), pointer :: &
grid => NULL() !< A pointer to a grid structure containing metrics
!! and related information.
Expand Down Expand Up @@ -248,6 +253,13 @@ subroutine ocean_model_init(Ocean_sfc, OS, Time_init, Time_in, gas_fields_ocn, i
!! min(HFrz, OBLD), where OBLD is the boundary layer depth.
!! If HFrz <= 0 (default), melt potential will not be computed.
logical :: use_melt_pot!< If true, allocate melt_potential array
! stochastic physics
integer,allocatable :: pelist(:) ! list of pes for this instance of the ocean
integer :: mom_comm ! list of pes for this instance of the ocean
integer :: num_procs ! number of processors to pass to stochastic physics
integer :: iret ! return code from stochastic physics
integer :: me ! my pe
integer :: master ! root pe

! This include declares and sets the variable "version".
#include "version_variable.h"
Expand Down Expand Up @@ -416,6 +428,39 @@ subroutine ocean_model_init(Ocean_sfc, OS, Time_init, Time_in, gas_fields_ocn, i

endif

! get number of processors and PE list for stocasthci physics initialization
call get_param(param_file, mdl, "DO_SPPT", OS%do_sppt, &
"If true, then stochastically perturb the thermodynamic "//&
"tendemcies of T,S, amd h. Amplitude and correlations are "//&
"controlled by the nam_stoch namelist in the UFS model only.", &
default=.false.)
call get_param(param_file, mdl, "PERT_EPBL", OS%pert_epbl, &
"If true, then stochastically perturb the kinetic energy "//&
"production and dissipation terms. Amplitude and correlations are "//&
"controlled by the nam_stoch namelist in the UFS model only.", &
default=.false.)
if (OS%do_sppt .OR. OS%pert_epbl) then
num_procs=num_PEs()
allocate(pelist(num_procs))
call Get_PElist(pelist,commID = mom_comm)
me=PE_here()
master=root_PE()

call init_stochastic_physics_ocn(OS%dt_therm,OS%grid%geoLonT,OS%grid%geoLatT,OS%grid%ied-OS%grid%isd+1,OS%grid%jed-OS%grid%jsd+1,OS%grid%ke,&
OS%pert_epbl,OS%do_sppt,master,mom_comm,iret)
if (iret/=0) then
write(6,*) 'call to init_stochastic_physics_ocn failed'
call MOM_error(FATAL, "stochastic physics in enambled in MOM6 but "// &
"not activated in stochastic_physics namelist ")
return
endif

if (OS%do_sppt) allocate(OS%stochastics%sppt_wts(OS%grid%isd:OS%grid%ied, OS%grid%jsd:OS%grid%jed))
if (OS%pert_epbl) then
allocate(OS%stochastics%t_rp1(OS%grid%isd:OS%grid%ied, OS%grid%jsd:OS%grid%jed))
allocate(OS%stochastics%t_rp2(OS%grid%isd:OS%grid%ied, OS%grid%jsd:OS%grid%jed))
endif
endif
call close_param_file(param_file)
call diag_mediator_close_registration(OS%diag)

Expand Down Expand Up @@ -585,17 +630,23 @@ subroutine update_ocean_model(Ice_ocean_boundary, OS, Ocean_sfc, &
call disable_averaging(OS%diag)
Master_time = OS%Time ; Time1 = OS%Time

! update stochastic physics patterns before running next time-step
if (OS%do_sppt .OR. OS%pert_epbl ) then
call run_stochastic_physics_ocn(OS%stochastics%sppt_wts, OS%stochastics%t_rp1, OS%stochastics%t_rp2)
endif

if (OS%offline_tracer_mode) then
call step_offline(OS%forces, OS%fluxes, OS%sfc_state, Time1, dt_coupling, OS%MOM_CSp)
elseif ((.not.do_thermo) .or. (.not.do_dyn)) then
! The call sequence is being orchestrated from outside of update_ocean_model.
call step_MOM(OS%forces, OS%fluxes, OS%sfc_state, Time1, dt_coupling, OS%MOM_CSp, &
Waves=OS%Waves, do_dynamics=do_thermo, do_thermodynamics=do_dyn, &
reset_therm=Ocn_fluxes_used)
reset_therm=Ocn_fluxes_used, stochastics=OS%stochastics)
!### What to do with these? , start_cycle=(n==1), end_cycle=.false., cycle_length=dt_coupling)

elseif (OS%single_step_call) then
call step_MOM(OS%forces, OS%fluxes, OS%sfc_state, Time1, dt_coupling, OS%MOM_CSp, Waves=OS%Waves)
call step_MOM(OS%forces, OS%fluxes, OS%sfc_state, Time1, dt_coupling, OS%MOM_CSp, Waves=OS%Waves, &
stochastics=OS%stochastics)
else
n_max = 1 ; if (dt_coupling > OS%dt) n_max = ceiling(dt_coupling/OS%dt - 0.001)
dt_dyn = dt_coupling / real(n_max)
Expand All @@ -620,16 +671,19 @@ subroutine update_ocean_model(Ice_ocean_boundary, OS, Ocean_sfc, &
dtdia = dt_dyn*min(nts,n_max-(n-1))
call step_MOM(OS%forces, OS%fluxes, OS%sfc_state, Time2, dtdia, OS%MOM_CSp, &
Waves=OS%Waves, do_dynamics=.false., do_thermodynamics=.true., &
start_cycle=(n==1), end_cycle=.false., cycle_length=dt_coupling)
start_cycle=(n==1), end_cycle=.false., cycle_length=dt_coupling, &
stochastics=OS%stochastics)
endif

call step_MOM(OS%forces, OS%fluxes, OS%sfc_state, Time2, dt_dyn, OS%MOM_CSp, &
Waves=OS%Waves, do_dynamics=.true., do_thermodynamics=.false., &
start_cycle=.false., end_cycle=(n==n_max), cycle_length=dt_coupling)
start_cycle=.false., end_cycle=(n==n_max), cycle_length=dt_coupling, &
stochastics=OS%stochastics)
else
call step_MOM(OS%forces, OS%fluxes, OS%sfc_state, Time2, dt_dyn, OS%MOM_CSp, &
Waves=OS%Waves, do_dynamics=.true., do_thermodynamics=.false., &
start_cycle=(n==1), end_cycle=.false., cycle_length=dt_coupling)
start_cycle=(n==1), end_cycle=.false., cycle_length=dt_coupling, &
stochastics=OS%stochastics)

step_thermo = .false.
if (thermo_does_span_coupling) then
Expand All @@ -646,7 +700,8 @@ subroutine update_ocean_model(Ice_ocean_boundary, OS, Ocean_sfc, &
Time2 = Time2 - set_time(int(floor((dtdia - dt_dyn) + 0.5)))
call step_MOM(OS%forces, OS%fluxes, OS%sfc_state, Time2, dtdia, OS%MOM_CSp, &
Waves=OS%Waves, do_dynamics=.false., do_thermodynamics=.true., &
start_cycle=.false., end_cycle=(n==n_max), cycle_length=dt_coupling)
start_cycle=.false., end_cycle=(n==n_max), cycle_length=dt_coupling, &
stochastics=OS%stochastics)
endif
endif

Expand Down
17 changes: 11 additions & 6 deletions src/core/MOM.F90
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ module MOM
use MOM_variables, only : surface, allocate_surface_state, deallocate_surface_state
use MOM_variables, only : thermo_var_ptrs, vertvisc_type
use MOM_variables, only : accel_diag_ptrs, cont_diag_ptrs, ocean_internal_state
use MOM_variables, only : rotate_surface_state
use MOM_variables, only : rotate_surface_state, stochastic_pattern
use MOM_verticalGrid, only : verticalGrid_type, verticalGridInit, verticalGridEnd
use MOM_verticalGrid, only : fix_restart_scaling
use MOM_verticalGrid, only : get_thickness_units, get_flux_units, get_tr_flux_units
Expand Down Expand Up @@ -418,7 +418,7 @@ module MOM
!! occur inside of diabatic.
subroutine step_MOM(forces_in, fluxes_in, sfc_state, Time_start, time_int_in, CS, &
Waves, do_dynamics, do_thermodynamics, start_cycle, &
end_cycle, cycle_length, reset_therm)
end_cycle, cycle_length, reset_therm, stochastics)
type(mech_forcing), target, intent(inout) :: forces_in !< A structure with the driving mechanical forces
type(forcing), target, intent(inout) :: fluxes_in !< A structure with pointers to themodynamic,
!! tracer and mass exchange forcing fields
Expand All @@ -443,6 +443,7 @@ subroutine step_MOM(forces_in, fluxes_in, sfc_state, Time_start, time_int_in, CS
logical, optional, intent(in) :: reset_therm !< This indicates whether the running sums of
!! thermodynamic quantities should be reset.
!! If missing, this is like start_cycle.
type(stochastic_pattern), optional, intent(in) :: stochastics !< random patternss for stochastics

! local variables
type(ocean_grid_type), pointer :: G => NULL() ! pointer to a structure containing
Expand Down Expand Up @@ -703,7 +704,8 @@ subroutine step_MOM(forces_in, fluxes_in, sfc_state, Time_start, time_int_in, CS

! Apply diabatic forcing, do mixing, and regrid.
call step_MOM_thermo(CS, G, GV, US, u, v, h, CS%tv, fluxes, dtdia, &
end_time_thermo, .true., Waves=Waves)
end_time_thermo, .true., Waves=Waves, &
stochastics=stochastics)
CS%time_in_thermo_cycle = CS%time_in_thermo_cycle + dtdia

! The diabatic processes are now ahead of the dynamics by dtdia.
Expand Down Expand Up @@ -801,7 +803,8 @@ subroutine step_MOM(forces_in, fluxes_in, sfc_state, Time_start, time_int_in, CS

! Apply diabatic forcing, do mixing, and regrid.
call step_MOM_thermo(CS, G, GV, US, u, v, h, CS%tv, fluxes, dtdia, &
Time_local, .false., Waves=Waves)
Time_local, .false., Waves=Waves, &
stochastics=stochastics)
CS%time_in_thermo_cycle = CS%time_in_thermo_cycle + dtdia

if ((CS%t_dyn_rel_thermo==0.0) .and. .not.do_dyn) then
Expand Down Expand Up @@ -1209,7 +1212,7 @@ end subroutine step_MOM_tracer_dyn
!> MOM_step_thermo orchestrates the thermodynamic time stepping and vertical
!! remapping, via calls to diabatic (or adiabatic) and ALE_main.
subroutine step_MOM_thermo(CS, G, GV, US, u, v, h, tv, fluxes, dtdia, &
Time_end_thermo, update_BBL, Waves)
Time_end_thermo, update_BBL, Waves, stochastics)
type(MOM_control_struct), intent(inout) :: CS !< Master MOM control structure
type(ocean_grid_type), intent(inout) :: G !< ocean grid structure
type(verticalGrid_type), intent(inout) :: GV !< ocean vertical grid structure
Expand All @@ -1222,6 +1225,7 @@ subroutine step_MOM_thermo(CS, G, GV, US, u, v, h, tv, fluxes, dtdia, &
intent(inout) :: h !< layer thickness [H ~> m or kg m-2]
type(thermo_var_ptrs), intent(inout) :: tv !< A structure pointing to various thermodynamic variables
type(forcing), intent(inout) :: fluxes !< pointers to forcing fields
type(stochastic_pattern), intent(in) :: stochastics !< surface ocean state
pjpegion marked this conversation as resolved.
Show resolved Hide resolved
real, intent(in) :: dtdia !< The time interval over which to advance [T ~> s]
type(time_type), intent(in) :: Time_end_thermo !< End of averaging interval for thermo diags
logical, intent(in) :: update_BBL !< If true, calculate the bottom boundary layer properties.
Expand Down Expand Up @@ -1284,7 +1288,8 @@ subroutine step_MOM_thermo(CS, G, GV, US, u, v, h, tv, fluxes, dtdia, &
call cpu_clock_begin(id_clock_diabatic)

call diabatic(u, v, h, tv, CS%Hml, fluxes, CS%visc, CS%ADp, CS%CDp, dtdia, &
Time_end_thermo, G, GV, US, CS%diabatic_CSp, OBC=CS%OBC, Waves=Waves)
Time_end_thermo, G, GV, US, CS%diabatic_CSp, OBC=CS%OBC, Waves=Waves, &
stochastics=stochastics)
fluxes%fluxes_used = .true.

if (showCallTree) call callTree_waypoint("finished diabatic (step_MOM_thermo)")
Expand Down
5 changes: 5 additions & 0 deletions src/core/MOM_variables.F90
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,11 @@ module MOM_variables

!> Container for information about the summed layer transports
!! and how they will vary as the barotropic velocity is changed.
type, public :: stochastic_pattern
real, allocatable :: sppt_wts(:,:) !< Random pattern for ocean SPPT
real, allocatable :: t_rp1(:,:) !< Random pattern for K.E. generation
real, allocatable :: t_rp2(:,:) !< Random pattern for K.E. dissipation
end type stochastic_pattern
type, public :: BT_cont_type
real, allocatable :: FA_u_EE(:,:) !< The effective open face area for zonal barotropic transport
!! drawing from locations far to the east [H L ~> m2 or kg m-1].
Expand Down
5 changes: 3 additions & 2 deletions src/framework/MOM_domains.F90
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ module MOM_domains

! This file is part of MOM6. See LICENSE.md for the license.


use MOM_array_transform, only : rotate_array
use MOM_coms, only : PE_here, root_PE, num_PEs, MOM_infra_init, MOM_infra_end
use MOM_coms, only : PE_here, root_PE, num_PEs, MOM_infra_init, MOM_infra_end, Get_PElist
pjpegion marked this conversation as resolved.
Show resolved Hide resolved
use MOM_coms, only : broadcast, sum_across_PEs, min_across_PEs, max_across_PEs
use MOM_cpu_clock, only : cpu_clock_begin, cpu_clock_end
use MOM_error_handler, only : MOM_error, MOM_mesg, NOTE, WARNING, FATAL, is_root_pe
Expand Down Expand Up @@ -37,7 +38,7 @@ module MOM_domains

public :: MOM_domains_init, MOM_infra_init, MOM_infra_end, get_domain_extent, get_domain_extent_dsamp2
public :: MOM_define_domain, MOM_define_io_domain, clone_MOM_domain
public :: pass_var, pass_vector, PE_here, root_PE, num_PEs
public :: pass_var, pass_vector, PE_here, root_PE, num_PEs, Get_PElist
pjpegion marked this conversation as resolved.
Show resolved Hide resolved
public :: pass_var_start, pass_var_complete, fill_symmetric_edges, broadcast
public :: pass_vector_start, pass_vector_complete
public :: global_field_sum, sum_across_PEs, min_across_PEs, max_across_PEs
Expand Down
Loading