CUDA.CUFFTError Error when trying the "Running your first model" example

[raphaelouillon]

Hi all, I just changed my GPU to an RTX 4090 to run larger simulations and leverage the 24gb of memory. Unfortunately even the simplest test case (the "Running your first model" example from the readme) returns an error. I am putting the complete error message below, but I was wondering if anyone had encountered such an issue with more recent GPUs? It seems to be an error related to CUDA (I have NVIDIA drivers 530.41.03 installed, with CUDA Version 12.1, running Julia 1.8.5 with Oceananigans 0.79.6).

ERROR: CUFFTError: driver or internal cuFFT library error (code 5, CUFFT_INTERNAL_ERROR)
Stacktrace:
  [1] throw_api_error(res::CUDA.CUFFT.cufftResult_t)
    @ CUDA.CUFFT ~/.julia/packages/CUDA/BbliS/lib/cufft/error.jl:64
  [2] macro expansion
    @ ~/.julia/packages/CUDA/BbliS/lib/cufft/error.jl:77 [inlined]
  [3] cufftMakePlan2d(plan::Int32, nx::Int64, ny::Int64, type::CUDA.CUFFT.cufftType_t, workSize::Base.RefValue{UInt64})
    @ CUDA.CUFFT ~/.julia/packages/CUDA/BbliS/lib/utils/call.jl:26
  [4] create_plan(xtype::CUDA.CUFFT.cufftType_t, xdims::Tuple{Int64, Int64, Int64}, region::Vector{Int64})
    @ CUDA.CUFFT ~/.julia/packages/CUDA/BbliS/lib/cufft/fft.jl:156
  [5] plan_fft!
    @ ~/.julia/packages/CUDA/BbliS/lib/cufft/fft.jl:291 [inlined]
  [6] plan_forward_transform
    @ ~/.julia/packages/Oceananigans/Zg9Bd/src/Solvers/plan_transforms.jl:43 [inlined]
  [7] plan_transforms(grid::RectilinearGrid{Float64, Periodic, Periodic, Flat, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, GPU}, storage::CUDA.CuArray{ComplexF64, 3, CUDA.Mem.DeviceBuffer}, planner_flag::UInt32)
    @ Oceananigans.Solvers ~/.julia/packages/Oceananigans/Zg9Bd/src/Solvers/plan_transforms.jl:131
  [8] Oceananigans.Solvers.FFTBasedPoissonSolver(grid::RectilinearGrid{Float64, Periodic, Periodic, Flat, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, GPU}, planner_flag::UInt32)
    @ Oceananigans.Solvers ~/.julia/packages/Oceananigans/Zg9Bd/src/Solvers/fft_based_poisson_solver.jl:66
  [9] FFTBasedPoissonSolver
    @ ~/.julia/packages/Oceananigans/Zg9Bd/src/Solvers/fft_based_poisson_solver.jl:52 [inlined]
 [10] PressureSolver(arch::GPU, grid::RectilinearGrid{Float64, Periodic, Periodic, Flat, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, GPU})
    @ Oceananigans.Models.NonhydrostaticModels ~/.julia/packages/Oceananigans/Zg9Bd/src/Models/NonhydrostaticModels/NonhydrostaticModels.jl:23
 [11] NonhydrostaticModel(; grid::RectilinearGrid{Float64, Periodic, Periodic, Flat, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, GPU}, clock::Clock{Float64}, advection::WENO{3, Float64, Nothing, Nothing, Nothing, true, Nothing, WENO{2, Float64, Nothing, Nothing, Nothing, true, Nothing, UpwindBiased{1, Float64, Nothing, Nothing, Nothing, Nothing, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}, Centered{2, Float64, Nothing, Nothing, Nothing, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, buoyancy::Nothing, coriolis::Nothing, stokes_drift::Nothing, forcing::NamedTuple{(), Tuple{}}, closure::Nothing, boundary_conditions::NamedTuple{(), Tuple{}}, tracers::Tuple{}, timestepper::Symbol, background_fields::NamedTuple{(), Tuple{}}, particles::Nothing, biogeochemistry::Nothing, velocities::Nothing, pressures::Nothing, diffusivity_fields::Nothing, pressure_solver::Nothing, immersed_boundary::Nothing, auxiliary_fields::NamedTuple{(), Tuple{}})
    @ Oceananigans.Models.NonhydrostaticModels ~/.julia/packages/Oceananigans/Zg9Bd/src/Models/NonhydrostaticModels/nonhydrostatic_model.jl:180
 [12] top-level scope
    @ REPL[10]:1
 [13] top-level scope
    @ ~/.julia/packages/CUDA/BbliS/src/initialization.jl:52

[glwagner]

Can you run non-Oceananigans code using CuArrays on this GPU?

[glwagner]

I believe going to CUDA 4.0 requires either updating AMGX or removing it. @navidcy might know

[glwagner]

#3017 passed @raphaelouillon so you can try unrestricting CUDA locally

[navidcy]

#3017 does not update CUDA to v4. It only updates the compat entry so one would be in position to install CUDA v4. So the test passing in that particular PR don't mean much.

[raphaelouillon]

@glwagner @navidcy I modified the Oceananigans.jl/Project.toml file locally to include CUDA 4.0, and was able to run my test case! Hopefully that means things are compatible with CUDA v4, at least with the parts of the code I am using.

[navidcy]

We should be updating Oceananigans to use CUDA v4 soon. E.g., see #2924

[navidcy]

Can you type

(@v1.8) pkg> st

to see what packages you have installed in your general registry? One of them might be creating the conflict.

[raphaelouillon]

@navidcy for the purpose of this test I only have CUDA and Oceananigans:

(oceananigans_simulations) pkg> st
Status `~/Documents/oceananigans_simulations/Project.toml`
⌅ [052768ef] CUDA v3.13.1
  [9e8cae18] Oceananigans v0.79.6
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated`

[navidcy]

oh OK, ... I was wondering if some other package there was imposing CUDA version restrictions.

[navidcy]

Is this still an issue @raphaelouillon? Since Oceananigans v0.81.1 we use CUDA v4.