PyTorch 1.3 inference output is different from PyTorch 1.2

[shijianjian]

🐛 Bug

I tested with EfficientNet-b0, b3, which were trained around one week ago with promising results with PyTorch 1.2, CUDA 10.0. After I upgraded PyTorch from 1.2 to 1.3, CUDA 10.0 to 10.1, the inferencing using the exact model and weights gave different results. I am not sure is it expected or a bug in 1.3.

To Reproduce

Steps to reproduce the behavior:

1. Inferencing with EfficientNet-b0 on PyTorch 1.2, CUDA 10.0.
2. Inferencing with EfficientNet-b0 on PyTorch 1.3 CUDA 10.1.
3. Comparison.

I got AUC of my model improved or decrease in +/- 0.1 or so.

Expected behavior

It shall be all identical among runs.

Environment

Collecting environment information...
PyTorch version: 1.3.0
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration:
GPU 0: Tesla V100-DGXS-32GB
GPU 1: Tesla V100-DGXS-32GB
GPU 2: Tesla V100-DGXS-32GB
GPU 3: Tesla V100-DGXS-32GB

Nvidia driver version: 418.87.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2

Versions of relevant libraries:
[pip] efficientnet-pytorch==0.4.0
[pip] numpy==1.16.4
[pip] pytorch-ignite==0.2.1
[pip] pytorch-lightning==0.4.9
[pip] robust-loss-pytorch==0.0.2
[pip] torch==1.3.0
[pip] torch-dct==0.1.5
[pip] torchsummary==1.5.1
[pip] torchvision==0.4.1
[conda] blas 1.0 mkl
[conda] efficientnet-pytorch 0.4.0 pypi_0 pypi
[conda] mkl 2019.4 243
[conda] mkl-service 2.0.2 py37h7b6447c_0
[conda] mkl_fft 1.0.14 py37ha843d7b_0
[conda] mkl_random 1.0.2 py37hd81dba3_0
[conda] pytorch-ignite 0.2.1 pypi_0 pypi
[conda] pytorch-lightning 0.4.9 pypi_0 pypi
[conda] robust-loss-pytorch 0.0.2 pypi_0 pypi
[conda] torch 1.3.0 pypi_0 pypi
[conda] torch-dct 0.1.5 pypi_0 pypi
[conda] torchsummary 1.5.1 pypi_0 pypi
[conda] torchvision 0.4.1 pypi_0 pypi

[farleylai]

We also noticed a similar performance reduction by -0.2% with Kinetics400 in PyTorch 1.3 compared with PyTorch 1.1.

The cudnn.benchmark is set to False while cudnn.deterministic is set to True.
A specific seed is also set to to both Python random, numpy and torch, torch.cuda.
The same results can be reproduced once switched back to 1.1.
However, PyTorch 1.3 produces different results during training and inferences.

Update

It is confirmed that the major performance drop is due to the fix of batchnorm weights (GAMMA) in PyTorch 1.2. The difference between 1.2 and 1.3 exists but does not seem significant (+0.04%).

[t-vi]

There is no guarantee that the same code will produce the exact same result in the "deterministic computation" sense, neither between PyTorch versions (nor between CuDNN/CUDA, I think).
These differences come from addition order sensitivity of floats and when something changes the e.g. the thread layout taken for a given configuration, this is expected to change. Giving a guarantee about this if at all possible, would preclude optimizations (e.g. like we did for norm operations #28201).
That said, it is always possible that you found a bug. Are the differences in network output small or large? Can you pinpoint to a given operation?

[shijianjian]

I tested with:

1. PyTorch 1.2 with CUDA10.1
2. PyTorch 1.3 with CUDA10.1

They produced identical results. I suspect it might be something with either CUDA or cudnn.

[t-vi]

Thank you for reporting back. If the difference stems from discrepancies within numerical precision, I am afraid there is little we can do, so I am closing this bug.

[francoishernandez]

Hey guys,

We also have some strange behaviors with OpenNMT-py when updating to pytorch 1.3.

Pytorch 1.2 - CUDA 10.2
The same inference command outputs exactly the same each time.

Pytorch 1.3 - CUDA 10.2
Running the same translate command several times does not give consistent output.
(Note: on CPU, inference runs are consistent.)

This is easily reproducible with the available pretrained models.