Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'context is destroyed' exception after using CuFFT. #971

Closed
ebprj21 opened this issue Mar 16, 2023 · 2 comments · Fixed by #972
Closed

'context is destroyed' exception after using CuFFT. #971

ebprj21 opened this issue Mar 16, 2023 · 2 comments · Fixed by #972

Comments

@ebprj21
Copy link

ebprj21 commented Mar 16, 2023

I am using your masterpiece very well.
If I had the ability, I would be able to help with anything, but in reality, I only write issue like this.

ILGPU's CuFFT example code is as follows:
(\Samples\AlgorithmsCuFFT\Program.cs)

static void DoForwardPlan(
    CudaAccelerator accelerator,
    CuFFT cufft,
    Complex[] input,
    out Complex[] output)
{
    using (var stream = accelerator.CreateStream() as CudaStream)
    {
        using (var inputBuffer = accelerator.Allocate1D(input))
        {
            using (var outputBuffer = accelerator.Allocate1D<Complex>(input.Length))
            {
                CuFFTException.ThrowIfFailed(
                    cufft.Plan1D(
                        out var plan,
                        input.Length,
                        CuFFTType.CUFFT_Z2Z,
                        batch: 1));
                using (plan)
                {
                    plan.SetStream(stream);
                    CuFFTException.ThrowIfFailed(
                        plan.ExecZ2Z(
                            inputBuffer.View.BaseView,
                            outputBuffer.View.BaseView,
                            CuFFTDirection.FORWARD));

                    output = outputBuffer.GetAsArray1D(stream);
                }
                WorkaroundKnownIssue(accelerator, cufft.API);

                Console.WriteLine("Output Values:");
                for (var i = 0; i < output.Length; i++)
                    Console.WriteLine($"  [{i}] = {output[i]}");
            }
        }
    }
}

In the above code, when 'plan' is Disposed, Context is also destroyed.
Therefore, when outputBuffer is Disposing, 'context is destroyed' exception is thrown.

The issue essentially looks like this.
inducer/pycuda#356

I hope you will consider this issue seriously. please.

ps) BTW, the new features in CUDA v12 are interesting.
CUDA Context-Independent Module Loading

@MoFtZ
Copy link
Collaborator

MoFtZ commented Mar 16, 2023

hi @ebprj21. Thanks for reporting this issue.

This looks like an issue with the CUDA SDK. Notice the call to WorkaroundKnownIssue. The sample code limits the workaround to specific versions of the CUDA SDK. The workaround will restore the original CUDA context. If you force the workaround to always run, it should fix your issue.

The CUDA Release Notes mention that in v11.2 and v11.3, there was a known issue:
cuFFT planning and plan estimation functions may not restore correct context affecting CUDA driver API applications.
https://docs.nvidia.com/cuda/archive/11.4.0/cuda-toolkit-release-notes/index.html

Do you know which version of the Cuda SDK you are using? It looks like the bug is there on other SDK versions.

@ebprj21
Copy link
Author

ebprj21 commented Mar 16, 2023

Your assistance was greatly appreciated.

I tested it on two systems. CUDA v11.6, v11.8.
Different version than you mention, but both cause problems.

I guess I have to coding in a way that always call 'WorkaroundKnownIssue' inevitably.

Thank you very much.

ps) I will look forward when ILGPU can use LibDevice in the CUDA v12 environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants