Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow statically eliding many JET failures in Krylov/KrylovKit.jl #2597

Closed
charleskawczynski opened this issue Jan 29, 2024 · 8 comments
Closed

Allow statically eliding many JET failures in Krylov/KrylovKit.jl #2597

charleskawczynski opened this issue Jan 29, 2024 · 8 comments
Assignees
@charleskawczynski @dennisYatunin
Labels
Performance monitoring 🚀 🔍

Comments

@charleskawczynski
Copy link
Member

I just tried to run the JET script on the PR where we upgrade some dependencies. The printing is so horribly bad that I can only fit 3 lines on the terminal (only VI seems to be able to handle opening the log) and it's therefore very difficult to diagnose any issues.

I think the best course of action is:

  • Fix the printing
  • Reduce noise of other issues
  • Reduce extraneous issues
  • Add a JET test (if CI permits) to prevent future regressions. I think we used to have tests, but they were very expensive. We should revisit adding them back in now that compile times are better.
  • Diagnose the actual inference failure to upgrade our dependencies.

The log can be downloaded from this build, and here is a snippet of a fraction of a line:

k;t=1706383471492^G  ^[[35m│^[[39m^[[34m│^[[39m^[[33m│^[[39m^[[32m│^[[39m^[[35m│^[[39m^[[34m│^[[39m^[[33m│^[[39m^[[32m│^[[39m^[[35m│^[[39m^[[34m│^[[39m^[[33m┌ ^[[39m^[[0m^[[1mkwcall^[[22m^[[0m^[[1m(^[[22m::^[[0m@NamedTuple^[[90m{krylovdim::Int64, maxiter::Int64, tol::Float64}^[[39m, ::^[[0mtypeof(KrylovKit.eigs     olve), ^[[90mf^[[39m::^[[0mClimaCore.MatrixFields.var"#213#214"^[[90m{ClimaCore.MatrixFields.StationaryIterativeSolve{false, ClimaCore.MatrixFields.BlockArrowheadSchurComplementPreconditioner{ClimaCore.MatrixFields.MainDiagonalPreconditioner, ClimaCore.MatrixFields.BlockLowerTriangularSolve{Tuple{ClimaCore.MatrixFi     elds.FieldName{(:c, :uₕ)}}, ClimaCore.MatrixFields.BlockDiagonalSolve, ClimaCore.MatrixFields.BlockDiagonalSolve}}, @NamedTuple{krylovdim::Int64, maxiter::Int64, tol::Float64}}, @NamedTuple{P_cache::@NamedTuple{P::ClimaCore.MatrixFields.FieldMatrix{ClimaCore.MatrixFields.FieldMatrixKeys{Tuple{Tuple{ClimaCore.Matrix     Fields.FieldName{(:c, :uₕ)}, ClimaCore.MatrixFields.FieldName{(:c, :uₕ)}}, Tuple{ClimaCore.MatrixFields.FieldName{(:f, :u₃)}, ClimaCore.MatrixFields.FieldName{(:c, :uₕ)}}, Tuple{ClimaCore.MatrixFields.FieldName{(:f, :u₃)}, ClimaCore.MatrixFields.FieldName{(:f, :u₃)}}}, ClimaCore.MatrixFields.FieldNameTreeNode{Clima     Core.MatrixFields.FieldName{()}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c,)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρ)}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ)}, Tuple{ClimaCore.M     atrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data, 1)}},      ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data, 2)}}}}}}}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρe_tot)}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρq_tot)}}}}, ClimaCore.Matrix     Fields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f,)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaC     ore.MatrixFields.FieldName{(:f, :u₃, :components, :data)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components, :data, 1)}}}}}}}}}}}}}, Tuple{LinearAlgebra.UniformScaling{Float32}, ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{ClimaCore.MatrixFields.BidiagonalM     atrixRow{ClimaCore.Geometry.AxisTensor{Float32, 2, Tuple{ClimaCore.Geometry.CovariantAxis{(3,)}, ClimaCore.Geometry.ContravariantAxis{(1, 2)}}, StaticArraysCore.SMatrix{1, 2, Float32, 2}}}, 4, Array{Float32, 5}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.G     rids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, Vector{Tuple{Int64, Int64,      Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCor     e.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IJFH{Float32, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCor     e.Geometry.UVVector{Float32}}, 4, Array{Float32, 3}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Sy     mbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, Matrix{Float32}}     }, ClimaCore.Grids.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, Array{Float32, 5}}}, ClimaCore.Grids.CellFace}}, ClimaCore.Fields.     Field{ClimaCore.DataLayouts.VIJFH{ClimaCore.MatrixFields.TridiagonalMatrixRow{ClimaCore.Geometry.AxisTensor{Float32, 2, Tuple{ClimaCore.Geometry.CovariantAxis{(3,)}, ClimaCore.Geometry.ContravariantAxis{(3,)}}, StaticArraysCore.SMatrix{1, 1, Float32, 1}}}, 4, Array{Float32, 5}}, ClimaCore.Spaces.ExtrudedFiniteDiffe     renceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vect     or{CartesianIndex{3}}, Array{Int64, 3}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GL     L{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IJFH{Float32, 4, Array{Float32, 4}}, ClimaCore.D     ataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, Array{Float32, 3}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.     Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Flo     at32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, Matrix{Float32}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4,      Array{Float32, 5}}}, ClimaCore.Grids.CellFace}}}}, b::ClimaCore.MatrixFields.FieldVectorView{ClimaCore.MatrixFields.FieldVectorKeys{Tuple{ClimaCore.MatrixFields.FieldName{(:f,)}, ClimaCore.MatrixFields.FieldName{(:c, :uₕ)}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{()}, Tuple{Clima     Core.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c,)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρ)}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.Matr     ixFields.FieldName{(:c, :uₕ, :components)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data, 1)}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{Cli     maCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data, 2)}}}}}}}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρe_tot)}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρq_tot)}}}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFie     lds.FieldName{(:f,)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :compon     ents, :data)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components, :data, 1)}}}}}}}}}}}}}, Tuple{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{@NamedTuple{u₃::ClimaCore.Geometry.Covariant3Vector{Float32}}, 4, Array{Float32, 5}}, ClimaCore.Spaces.ExtrudedFinite     DifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap},      Vector{CartesianIndex{3}}, Array{Int64, 3}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratur     es.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IJFH{Float32, 4, Array{Float32, 4}}, ClimaC     ore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, Array{Float32, 3}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{Clima     Core.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}     , Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, Matrix{Float32}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}     }, 4, Array{Float32, 5}}}, ClimaCore.Grids.CellFace}}, ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.Covariant12Vector{Float32}, 4, Array{Float32, 5}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.     Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{In     t64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry     .LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IJFH{Float32, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, Arr     ay{Float32, 3}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geom     etry.ZPoint{Float32}}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, Matrix{Float32}}}, ClimaCore.Grids.Flat, ClimaCore.Ge     ometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, Array{Float32, 5}}}, ClimaCore.Grids.CellCenter}}}}, x::ClimaCore.MatrixFields.FieldVectorView{ClimaCor     e.MatrixFields.FieldVectorKeys{Tuple{ClimaCore.MatrixFields.FieldName{(:c, :uₕ)}, ClimaCore.MatrixFields.FieldName{(:f, :u₃)}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{()}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c,)}, Tuple{ClimaCore.Matri     xFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρ)}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCo     re.MatrixFields.FieldName{(:c, :uₕ, :components, :data)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data, 1)}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data, 2)}}}}}}}}, ClimaCore.MatrixFields.Fie     ldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρe_tot)}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρq_tot)}}}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f,)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldN     ame{(:f, :u₃)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components, :data)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:f     , :u₃, :components, :data, 1)}}}}}}}}}}}}}, Tuple{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.Covariant12Vector{Float32}, 4, Array{Float32, 5}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topol     ogies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64,      Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.Loca     lGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IJFH{Float32, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, Array{Fl     oat32, 3}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.     ZPoint{Float32}}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, Matrix{Float32}}}, ClimaCore.Grids.Flat, ClimaCore.Geometr     y.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, Array{Float32, 5}}}, ClimaCore.Grids.CellCenter}}, ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{ClimaC     ore.Geometry.Covariant3Vector{Float32}, 4, Array{Float32, 5}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.Equiangula     rCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}},      Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32,      4}}, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IJFH{Float32, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, Array{Float32, 3}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{Clima     Comms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, Clim     aCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaCore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, Matrix{Float32}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3),      ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, Array{Float32, 5}}}, ClimaCore.Grids.CellFace}}}}, cache::@NamedTuple{cache₁::ClimaCore.MatrixFields.FieldVectorView{ClimaCore.MatrixFields.FieldVectorKeys{Tuple{ClimaCore.MatrixFields.FieldName{(:c, :uₕ)}}, ClimaCor     e.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{()}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c,)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρ)}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.F     ieldName{(:c, :uₕ)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldNam     e{(:c, :uₕ, :components, :data, 1)}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :uₕ, :components, :data, 2)}}}}}}}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:c, :ρe_tot)}}, ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldNa     me{(:c, :ρq_tot)}}}}, ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f,)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components)}, Tuple{ClimaCore     .MatrixFields.FieldNameTreeNode{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components, :data)}, Tuple{ClimaCore.MatrixFields.FieldNameTreeLeaf{ClimaCore.MatrixFields.FieldName{(:f, :u₃, :components, :data, 1)}}}}}}}}}}}}}, Tuple{ClimaCore.Fields.Field{ClimaCore.DataLayouts.VIJFH{Tuple{}, 4, Array{Float32, 5}}, Cli     maCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Grids.ExtrudedFiniteDifferenceGrid{ClimaCore.Grids.SpectralElementGrid2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Me     shes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, I     nt64}}}, ClimaCore.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, Array{Float32, 4}}, ClimaCore.DataLayouts.IJFH{Float32, 4     , Array{Float32, 4}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, Array{Float32, 3}}, @NamedTuple{}}, ClimaCore.Grids.FiniteDifferenceGrid{ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CPUSingleThreaded}, ClimaCo     re.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, @NamedTuple{bottom::Int64, top::Int64}}, ClimaCore.Geometry.CartesianGlobalGeometry, ClimaCore.DataLayouts.VF{ClimaCore.Geometry.LocalGeometry{(3,), ClimaC     ore.Geometry.ZPoint{Float32}, Float32, StaticArraysCore.SMatrix{1, 1, Float32, 1}}, Matrix{Float32}}}, ClimaCore.Grids.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}

Unfortunately, the other thing I've noticed is that the number of JET failures has ballooned from October 25, 2023:

Dict{Any, Any} with 3 entries:
--
  | "limiters_func!" => 4
  | "set_precomputed_quantities!" => 9
  | "step!" => 16

to

Dict{Any, Any} with 7 entries:
--
  | "tracer_hyperdiffusion_tendency!" => 30
  | "limited_tendency!" => 30
  | "limiters_func!" => 2
  | "remaining_tendency!" => 30
  | "set_precomputed_quantities!" => 7
  | "step!" => 1069
  | "hyperdiffusion_tendency!" => 30

Since most of these are coming in through step!, and the other functions have a relatively low number of inference failures (but we should track these down, too), and diagnostics is not well inferred, these are likely coming from diagnostics.

Previously, all of our callbacks had toggles to turn on / off. The diagnostics don't appear to have this option, @Sbozzolo, can you please update the diagnostics so that we can disable them for performance analysis? Otherwise the amount of noise is prohibitively expensive.

I'll see what we can do on the printing side, and I'll see if there's a way we can hook in tests, to prevent further regressions.
@charleskawczynski charleskawczynski self-assigned this Jan 29, 2024
@Sbozzolo
Copy link
Member

Sbozzolo commented Jan 29, 2024 via email

@charleskawczynski
Copy link
Member Author

charleskawczynski commented Jan 29, 2024 •
edited
Loading

You will still end up with one callback (orchestrate_diagnostics).

Yes, this what I'm referring to. Can you please add an option to remove it?

@Sbozzolo
Copy link
Member

You will still end up with one callback (orchestrate_diagnostics).

Yes, this what I'm referring to. Can you please add an option to remove it?

Yes, I can add that if there are no diagnostics, this callback is not added.

@charleskawczynski
Copy link
Member Author

charleskawczynski commented Jan 29, 2024 •
edited
Loading

Ok, thank you, please prioritize this since we cannot update Thermodynamics, SurfaceFluxes, or CloudMicrophysics until this is fixed. And please add me for review.

@Sbozzolo Sbozzolo mentioned this issue Jan 29, 2024
Closed
@charleskawczynski
Copy link
Member Author

Today I opened a PR in JET.jl: aviatesk/JET.jl#601, which attempts to fix the issue. Hopefully this gets fixed soon, because it's basically impossible to use the master branch in production (since the buildkite logs overflow).

In the mean time, I'm trying to JET through our existing issues on my fork: https://github.com/charleskawczynski/JET.jl/tree/ck/depth_limited_type_printing (in case anyone else needs to try to use JET on ClimaAtmos).

@juliasloan25 juliasloan25 mentioned this issue Jan 30, 2024
Merged
@charleskawczynski charleskawczynski changed the title Fix JET failures to allow upgrading dependencies Allow statically eliding many JET failures in Krylov/KrylovKit.jl Feb 9, 2024
@charleskawczynski
Copy link
Member Author

charleskawczynski commented Feb 9, 2024 •
edited
Loading

We've updated the dependencies, but the JET script still dumps tons of data. @dennisYatunin, alright if I assign you to fix this issue? Ultimately, it would be nice if we can reduce the number of JET failures observed in the jet_report_nfailures script?

This will help in reducing the noise when looking at jobs that don't use the Krylov pieces.

I can update the JET script to skip over the thermodynamics errors (resulting from KernelAbstractions.@print)

@dennisYatunin
Copy link
Member

Yes, I can try to make it so that the Krylov stuff doesn't get compiled when we don't need it for debugging.

@charleskawczynski
Copy link
Member Author

Closed by CliMA/ClimaCore.jl#1617

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@charleskawczynski charleskawczynski

@dennisYatunin dennisYatunin

Labels
Performance monitoring 🚀 🔍
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@charleskawczynski @dennisYatunin @Sbozzolo