Compare timings of Nonhydrostatic and Hydrostatic models

[francispoulin]

I have an unstable jet set up with exactly the same parameters using the Nonhydrostatic and HydrostaticFreeSurface models. For the latter I am using an ImplicitFreeSurface.

When we do a 3D simulation with about 200^3 we find that the timings differ qualitatively from GPUs to CPUs.

On a cluster, with GPU, we find that the HydrostaticFreeSurface model runs about 20 times faster.

On my laptop, with a CPU, we find the HydrostaticFreeSurface model runs about 2 times slower.

I found the slower hydrostatic model surprising and wanted to ask if this is suprising to others too?

I suspect it would be faster using an ExplicitFreeSurface and I can try that but wanted to get peoples reactions before I try that.

[glwagner]

Can you give more details about the free surface formulation? There are a few options: split-explicit, implicit free surface with FFTs, implicit free surface with a matrix-based preconditioner and preconditioned conjugate gradient solver, explicit free surface. It also probably matters how many tracers you are using, closure, buoyancy, etc.

[francispoulin]

Here is a trimmed down version of the hydrostatic code. Nothing fancy going on.

using Oceananigans
using Oceananigans.Units
using Statistics, Random
using CUDA, Adapt

CUDA.allowscalar(false)

const f     = 0.864e-4 
const fₕ    = 0.0 
const N²    = (3.7e-3)^2 
const ν     = 0.36/2.2e5
const Umax  = 0.36
const Lx     = 10000
const Ly     = 30000
const Lⱼ    = 2000
const Lz    = 1000
const D     = 200
const z0    = -Lz/2

grid = RectilinearGrid(
    CPU(),
    size=(1, 200, 100),
    x= (-Lx, Lx),
    y = (-Ly, Ly),
    z = (-Lz, 0),
    topology=(Periodic, Bounded, Bounded)
    )

b̄(x, y, z) = N² * z + 2*f*Umax*Lⱼ/D^2 * (tanh(y/Lⱼ)) * (z-z0) * exp(-(z-z0)^2/D^2)
ū(x, y, z) = Umax / cosh(y/Lⱼ)^2 * exp(-(z-z0)^2/D^2)

Random.seed!(25)
uᵢ(x, y, z) = ū(x, y, z)
bᵢ(x, y, z) = b̄(x, y, z) + 1e-6*rand()

model = HydrostaticFreeSurfaceModel(; grid = grid,
    free_surface = ImplicitFreeSurface(),
        coriolis = FPlane(f = f),
        buoyancy = BuoyancyTracer(),
        tracers = (:b),
        momentum_advection = WENO(grid=grid),
        tracer_advection = WENO(grid=grid),
        closure = VerticalScalarDiffusivity(ν=ν, κ=ν))

set!(model, u = uᵢ, b = bᵢ)

run!(simulation)

The nonhydrostatic solver is almost exactly the same except for the model definitio is replaced by these lines.

model = NonhydrostaticModel(; grid,
        coriolis = FPlane(f = f),
        buoyancy = BuoyancyTracer(),
        tracers = (:b),
        advection = WENO(grid=grid),
        closure = VerticalScalarDiffusivity(ν=ν, κ=ν))

[glwagner]

This

free_surface = ImplicitFreeSurface()

uses the default --- what is that default?

[francispoulin]

I checked and see it's FFTImplicitFreeSurfaceSolver.

[glwagner]

Since you're using FFTs, you may want to check the difference between using a number like 256^3 rather than 200^3

[nguyenquangchien]

Yes and before the last line of code, we should expect something like
simulation = Simulation(model, Δt = 1minutes, stop_time = 1hour)