cuDNN version mismatch for TensorFlow 2.5.0

[verdurin]

According to https://www.tensorflow.org/install/gpu#software_requirements TF 2.5.0 , cuDNN 8.1 is required.
We have version 8.0.4.30 for CUDA 11.1 in fosscuda/2020b.

We have also seen some problems with our install of TF 2.5.0, which we suspect are related to this version mismatch:

2021-08-17 07:37:18.488694: F ./tensorflow/core/kernels/random_op_gpu.h:244] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), key, counter, gen, data, size, dist) status: Internal: no kernel image is available for execution on the device

cuDNN 8.1.1.33 is available for download, albeit for CUDA 11.2. Technically, it's in the part of the NVIDIA archive labelled 11.0, 11.1, 11.2 but the explicit version in the tarball is 11.2.

In summary, we would like to try TF 2.5.0 with cuDNN 8.1 in fosscuda/2020b, which is based on CUDA11.1, while the cuDNNrelease appears to be intended forCUDA 11.2`.

[Flamefire]

no kernel image is available for execution on the device

This usually means that the GPUs compute capability is not included in the binaries. Make sure that TF is build with the correct cuda-compute-capabilities options for your GPUs