Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(0.91.14) Extend Lagrangian advection to immersed grids and add tests #3765

Merged
merged 24 commits into from
Sep 21, 2024
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
4b72263
extend lagrangian advection to all AbstractGrids
tomchor Sep 6, 2024
2334603
cleaner syntax
tomchor Sep 6, 2024
25d02a6
bugfix for immersed grids with flat dimensions
tomchor Sep 6, 2024
c37be12
add tests with immersed boundary grid
tomchor Sep 6, 2024
5e28d4b
oops, wrong topology
tomchor Sep 6, 2024
9634e51
convert immersed function to boolean output
tomchor Sep 6, 2024
abff491
simplify
tomchor Sep 9, 2024
e858e54
Merge branch 'main' into tc/ibg-tracking
tomchor Sep 9, 2024
97b197f
clean up logic
tomchor Sep 9, 2024
54b1dcf
Merge branch 'tc/ibg-tracking' of github.com:CliMA/Oceananigans.jl in…
tomchor Sep 9, 2024
8bcdef1
bump patch version
tomchor Sep 9, 2024
81ff38f
added docstring
tomchor Sep 9, 2024
6ac566a
Merge branch 'main' into tc/ibg-tracking
tomchor Sep 10, 2024
c786eaa
Merge branch 'main' into tc/ibg-tracking
tomchor Sep 11, 2024
31bec4e
correct indices
tomchor Sep 16, 2024
28171f6
Merge branch 'tc/ibg-tracking' of github.com:CliMA/Oceananigans.jl in…
tomchor Sep 16, 2024
1fd6833
add LatLonGrid to tests
tomchor Sep 18, 2024
17a5f89
Merge branch 'main' into tc/ibg-tracking
tomchor Sep 18, 2024
9635d03
write speed differently to try and avoid using too much parameter space
tomchor Sep 18, 2024
a06faf1
use only horizontal speed
tomchor Sep 18, 2024
f590acb
fix docs and convert code to doctest
tomchor Sep 18, 2024
65e76c0
add output to doctest
tomchor Sep 19, 2024
0cfe076
add filtering
tomchor Sep 19, 2024
2c53428
Merge branch 'main' into tc/ibg-tracking
tomchor Sep 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name = "Oceananigans"
uuid = "9e8cae18-63c1-5223-a75c-80ca9d6e9a09"
authors = ["Climate Modeling Alliance and contributors"]
version = "0.91.13"
version = "0.91.14"

[deps]
Adapt = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,13 @@ Do nothing on Flat dimensions.
const f = Face()
const c = Center()

"""
immersed_boundary_topology(grid_topology)

Unless `Flat`, immersed boundaries are treated as `Bounded` regardless of underlying grid topology.
"""
immersed_boundary_topology(grid_topology) = ifelse(grid_topology == Flat, Flat(), Bounded())

tomchor marked this conversation as resolved.
Show resolved Hide resolved
"""
bounce_immersed_particle((x, y, z), grid, restitution, previous_particle_indices)

Expand All @@ -48,27 +55,34 @@ bouncing the particle off the immersed boundary with a coefficient or `restituti
X = flattened_node((x, y, z), ibg)

# Determine current particle cell
fi, fj, fk = fractional_indices(X, ibg.underlying_grid, (c, c, c))
fi, fj, fk = fractional_indices(X, ibg.underlying_grid, c, c, c)
i, j, k = truncate_fractional_indices(fi, fj, fk)

if immersed_cell(i, j, k, ibg)
# Determine whether particle was _previously_ in a non-immersed cell
i⁻, j⁻, k⁻ = previous_particle_indices
# Determine whether particle was _previously_ in a non-immersed cell
i⁻, j⁻, k⁻ = previous_particle_indices

if !immersed_cell(i⁻, j⁻, k⁻, ibg)
# Left-right bounds of the previous, non-immersed cell
xᴿ, yᴿ, zᴿ = node(i⁻+1, j⁻+1, k⁻+1, ibg, f, f, f)
xᴸ, yᴸ, zᴸ = node(i⁻, j⁻, k⁻, ibg, f, f, f)
# Left bounds of the previous cell
xᴿ = xnode(i⁻ + 1, j, k, ibg, f, f, f)
yᴿ = ynode(i, j⁻ + 1, k, ibg, f, f, f)
zᴿ = znode(i, j, k⁻ + 1, ibg, f, f, f)

Cʳ = restitution
x⁺ = enforce_boundary_conditions(Bounded(), x, xᴸ, xᴿ, Cʳ)
y⁺ = enforce_boundary_conditions(Bounded(), y, yᴸ, yᴿ, Cʳ)
z⁺ = enforce_boundary_conditions(Bounded(), z, zᴸ, zᴿ, Cʳ)
# Right bounds of the previous cell
xᴸ = xnode(i⁻, j, k, ibg, f, f, f)
yᴸ = ynode(i, j⁻, k, ibg, f, f, f)
zᴸ = znode(i, j, k⁻, ibg, f, f, f)
Copy link
Collaborator

@simone-silvestri simone-silvestri Sep 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite different from how it was written before.
For example it might be that:
xnode(i⁻, j, k, ibg, f, f, f) != node(i⁻, j⁻, k⁻, ibg, f, f, f)[1]
(not necessarily for a rectilinear grid but possibly for a curvilinear one)
Was there a bug in the previous implementation?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there was a bug with the previous implementation and it needed re-writing. Specifically, it was failing for Flat dimensions iirc. I'm not familiar with how this is implemented in curvilinear coords, and I don't remember lagrangian tracking being done in them before. Can we keep the code as it currently is in the PR?

Copy link
Collaborator

@simone-silvestri simone-silvestri Sep 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, I understand why node(i⁻+1, j⁻+1, k⁻+1, ibg, f, f, f) would fail, but node(i⁻, j⁻, k⁻, ibg, f, f, f) should exist also for Flat topologies. Can we do this:

xᴸ, yᴸ, zᴸ = node(i⁻, j⁻, k⁻, ibg, f, f, f)

and keep the right boundaries as written in this PR?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous code fails not because of the +1 index in the flat dimension, but because it doesn't return anything for the flat dimension:

julia> grid_base = RectilinearGrid(topology = (Periodic, Flat, Periodic), size=(5, 5), extent=(1, 1))
5×1×5 RectilinearGrid{Float64, Periodic, Flat, Periodic} on CPU with 3×0×3 halo
├── Periodic x ∈ [0.0, 1.0)  regularly spaced with Δx=0.2
├── Flat y                   
└── Periodic z ∈ [-1.0, 0.0) regularly spaced with Δz=0.2

julia> node(2, 2, 2, grid_base, Center(), Center(), Center())
(0.3, -0.7)

So in these cases the statement xᴸ, yᴸ, zᴸ = node(i⁻, j⁻, k⁻, ibg, f, f, f) fails with

  Got exception outside of a @test
  BoundsError: attempt to access Tuple{Float64, Float64} at index [3]
  Stacktrace:
    [1] indexed_iterate
      @ ./tuple.jl:92 [inlined]
    [2] bounce_immersed_particle
      @ ~/repos/Oceananigans.jl2/src/Models/LagrangianParticleTracking/lagrangian_particle_advection.jl:75 [inlined]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, if I understand your comment correctly, we can just replace

    xᴿ = xnode(i⁻ + 1, j, k, ibg, f, f, f)
    yᴿ = ynode(i, j⁻ + 1, k, ibg, f, f, f)
    zᴿ = znode(i, j, k⁻ + 1, ibg, f, f, f)

with

    xᴿ = xnode(i⁻ + 1, j⁻ + 1, k⁻ + 1, ibg, f, f, f)
    yᴿ = ynode(i⁻ + 1, j⁻ + 1, k⁻ + 1, ibg, f, f, f)
    zᴿ = znode(i⁻ + 1, j⁻ + 1, k⁻ + 1, ibg, f, f, f)

I think that should be equivalent. Note that node() is just a shorthand for (xnode, ynode, znode)

@inline _node(i, j, k, grid, ℓx, ℓy, ℓz) = (ξnode(i, j, k, grid, ℓx, ℓy, ℓz),
ηnode(i, j, k, grid, ℓx, ℓy, ℓz),
rnode(i, j, k, grid, ℓx, ℓy, ℓz))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to use ξnode than node[1] which is a hack and not as easy to understand either.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, I think there are at least 2 separate issues being discussed here.

  1. The indices to be used when calling the location for the right indices. The code before this PR was node(i⁻+1, j⁻+1, k⁻+1, ibg, f, f, f), which I believe should translate to ξnode(i⁻+1, j⁻+1, k⁻+1, ibg, f, f, f) and the same for η and r, with the same indices (that's how the PR actually is). @simone-silvestri seems to think that the correct call should be (as per a code suggestion below)
    xᴿ = xnode(i⁻ + 1, j⁻, k⁻, ibg, f, f, f)
    yᴿ = ynode(i⁻, j⁻ + 1, k⁻, ibg, f, f, f)
    zᴿ = znode(i⁻, j⁻, k⁻ + 1, ibg, f, f, f)

or perhaps with ξ, η and r, but importantly here is that the indices are different. @simone-silvestri, could you elaborate?

  1. @glwagner is suggesting using ξnode (η and r) instead of xnode (and y and z). This is how it was in the code before this PR, but I think we actually want xnode and ynode since the advection is performed with velocities in length/time, no?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what's correct. Seems like there is a missing test... but maybe @simone-silvestri can say. We did use this code on the sphere I thought.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I am a bit confused which node should be the correct ``right'' node. I guess if there are tests now and the i⁻+1, j⁻+1, k⁻+1, ibg, f, f, f signature works, then the implementation is correct

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be fair, tests works regardless of which of the two signatures we're considering here, so whatever the difference is, the tests don't pick up on it. I think I'll keep the algorithm as close to as it was before this PR as possible. This includes using the i⁻+1, j⁻+1, k⁻+1, ibg, f, f, f signature. As for the xnode versus ξnode debate, I actually realized the existing code used xnode for the non-immersed part of the algorithm and ξnode for the immersed grid. Likely the latter was done in error, so I'll keep everything as xnode for now. We can revisit this is the future if we find evidence that this isn't the proper way to do things.


end
end
Cʳ = restitution
tx, ty, tz = map(immersed_boundary_topology, topology(ibg))
xb⁺ = enforce_boundary_conditions(tx, x, xᴸ, xᴿ, Cʳ)
yb⁺ = enforce_boundary_conditions(ty, y, yᴸ, yᴿ, Cʳ)
zb⁺ = enforce_boundary_conditions(tz, z, zᴸ, zᴿ, Cʳ)

return x⁺, y⁺, z⁺
immersed = immersed_cell(i, j, k, ibg)
x⁺ = ifelse(immersed, xb⁺, x)
y⁺ = ifelse(immersed, yb⁺, y)
z⁺ = ifelse(immersed, zb⁺, z)

return (x⁺, y⁺, z⁺)
end

"""
Expand Down Expand Up @@ -134,7 +148,7 @@ given `velocities`, time-step `Δt, and coefficient of `restitution`.
z⁺ = enforce_boundary_conditions(tz, z⁺, zᴸ, zᴿ, Cʳ)
if grid isa ImmersedBoundaryGrid
previous_particle_indices = current_particle_indices # particle has been advected
x⁺, y⁺, z⁺ = bounce_immersed_particle((x⁺, y⁺, z⁺), grid, Cʳ, previous_particle_indices)
(x⁺, y⁺, z⁺) = bounce_immersed_particle((x⁺, y⁺, z⁺), grid, Cʳ, previous_particle_indices)
end

return (x⁺, y⁺, z⁺)
Expand All @@ -145,11 +159,13 @@ end
# * Sphere metric for `LatitudeLongitudeGrid` and geographic coordinates
@inline x_metric(i, j, grid::RectilinearGrid) = 1
@inline x_metric(i, j, grid::LatitudeLongitudeGrid{FT}) where FT = @inbounds 1 / (grid.radius * hack_cosd(grid.φᵃᶜᵃ[j])) * FT(360 / 2π)
@inline x_metric(i, j, grid::ImmersedBoundaryGrid) = x_metric(i, j, grid.underlying_grid)

@inline y_metric(i, j, grid::RectilinearGrid) = 1
@inline y_metric(i, j, grid::LatitudeLongitudeGrid{FT}) where FT = 1 / grid.radius * FT(360 / 2π)
@inline y_metric(i, j, grid::ImmersedBoundaryGrid) = y_metric(i, j, grid.underlying_grid)

@kernel function _advect_particles!(particles, restitution, grid::AbstractUnderlyingGrid, Δt, velocities)
@kernel function _advect_particles!(particles, restitution, grid::AbstractGrid, Δt, velocities)
p = @index(Global)

@inbounds begin
Expand Down
19 changes: 19 additions & 0 deletions test/test_lagrangian_particle_tracking.jl
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,19 @@ lagrangian_particle_test_grid(arch, ::Periodic, z) =
lagrangian_particle_test_grid(arch, ::Flat, z) =
RectilinearGrid(arch; topology=(Periodic, Flat, Bounded), size=(5, 5), x=(-1, 1), z)

lagrangian_particle_test_grid_expanded(arch, ::Periodic, z) =
RectilinearGrid(arch; topology=(Periodic, Periodic, Bounded), size=(5, 5, 5), x=(-1, 1), y=(-1, 1), z = 2 .*z)
lagrangian_particle_test_grid_expanded(arch, ::Flat, z) =
RectilinearGrid(arch; topology=(Periodic, Flat, Bounded), size=(5, 5), x=(-1, 1), z = 2 .*z)
tomchor marked this conversation as resolved.
Show resolved Hide resolved

function lagrangian_particle_test_immersed_grid(arch, y_topo, z)
underlying_grid = lagrangian_particle_test_grid_expanded(arch, y_topo, z)
z_immersed_boundary(x, z) = ifelse(z < -1, true, ifelse(z > 1, true, false))
z_immersed_boundary(x, y, z) = z_immersed_boundary(x, z)
GFB = GridFittedBoundary(z_immersed_boundary)
return ImmersedBoundaryGrid(underlying_grid, GFB)
end

@testset "Lagrangian particle tracking" begin
timesteppers = (:QuasiAdamsBashforth2, :RungeKutta3)
y_topologies = (Periodic(), Flat())
Expand All @@ -278,5 +291,11 @@ lagrangian_particle_test_grid(arch, ::Flat, z) =
@info " Testing Lagrangian particle tracking [$(typeof(arch)), $timestepper] with y $(typeof(y_topo)) on vertically $z_grid_type grid ..."
grid = lagrangian_particle_test_grid(arch, y_topo, z)
run_simple_particle_tracking_tests(grid, timestepper)

if z isa NTuple{2} # Test immersed regular grids
@info " Testing Lagrangian particle tracking [$(typeof(arch)), $timestepper] with y $(typeof(y_topo)) on vertically $z_grid_type immersed grid ..."
grid = lagrangian_particle_test_immersed_grid(arch, y_topo, z)
run_simple_particle_tracking_tests(grid, timestepper)
end
end
end