Single-Node Multi-GPU Training Stuck

[andrewssobral]

Hello everyone!

I am trying to launch a single-node multi-gpu training script, but i don't get any warning/error message, and the script is stuck for long time, nothing occurs....screenshot below:

The script was launched in a multi-gpu node (4 GPUs Tesla K80), as you can see below:

nvidia-smi info header:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06    Driver Version: 450.36.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+

When the script is "running" , I have the following behavior in my nvtop:

I waited for several minutes (around 30min), and nothing happens, I still have the following output:

Please see below my source code:

import os
import torch
import torch.nn.functional as F
import pytorch_lightning as pl
import multiprocessing

from torch import nn
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader, random_split
from torchvision import transforms
from pytorch_lightning.metrics.functional import accuracy

# Download MNIST
# For new versions of TorchVision
# wget -O MNIST.tar.gz https://activeeon-public.s3.eu-west-2.amazonaws.com/datasets/MNIST.new.tar.gz
# For old versions of TorchVision (<= 1.0.1*)
# wget -O MNIST.tar.gz https://activeeon-public.s3.eu-west-2.amazonaws.com/datasets/MNIST.old.tar.gz
# tar -zxvf MNIST.tar.gz
import wget
import tarfile
if not os.path.exists("MNIST"):
    wget.download("https://activeeon-public.s3.eu-west-2.amazonaws.com/datasets/MNIST.new.tar.gz", "MNIST.tar.gz")
    tar = tarfile.open("MNIST.tar.gz", "r:gz")
    tar.extractall()
    tar.close()


class LitMNIST(pl.LightningModule):
    def __init__(self, data_dir='./', hidden_size=64, learning_rate=2e-4):
        super().__init__()
        # Set our init args as class attributes
        self.data_dir = data_dir
        self.hidden_size = hidden_size
        self.learning_rate = learning_rate
        # Hardcode some dataset specific attributes
        self.num_classes = 10
        self.dims = (1, 28, 28)
        channels, width, height = self.dims
        self.transform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,))
        ])
        # Define PyTorch model
        self.model = nn.Sequential(
            nn.Flatten(),
            nn.Linear(channels * width * height, hidden_size),
            nn.ReLU(),
            nn.Dropout(0.1),
            nn.Linear(hidden_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(0.1),
            nn.Linear(hidden_size, self.num_classes)
        )

    def forward(self, x):
        x = self.model(x)
        return F.log_softmax(x, dim=1)

    def training_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        loss = F.nll_loss(logits, y)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        loss = F.nll_loss(logits, y)
        preds = torch.argmax(logits, dim=1)
        acc = accuracy(preds, y)

        # Calling self.log will surface up scalars for you in TensorBoard
        # https://pytorch-lightning.readthedocs.io/en/1.1.2/multi_gpu.html#synchronize-validation-and-test-logging
        self.log('val_loss', loss, prog_bar=True, on_step=True, on_epoch=True, sync_dist=True)
        self.log('val_acc', acc, prog_bar=True)
        return loss

    def test_step(self, batch, batch_idx):
        # Here we just reuse the validation_step for testing
        return self.validation_step(batch, batch_idx)

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)
        return optimizer

    def prepare_data(self):
        # download
        MNIST(self.data_dir, train=True, download=False)
        MNIST(self.data_dir, train=False, download=False)

    def setup(self, stage=None):
        # Assign train/val datasets for use in dataloaders
        if stage == 'fit' or stage is None:
            mnist_full = MNIST(self.data_dir, train=True, transform=self.transform)
            self.mnist_train, self.mnist_val = random_split(mnist_full, [55000, 5000])
        # Assign test dataset for use in dataloader(s)
        if stage == 'test' or stage is None:
            self.mnist_test = MNIST(self.data_dir, train=False, transform=self.transform)

    def train_dataloader(self):
        if torch.cuda.is_available():
            return DataLoader(self.mnist_train, batch_size=32, num_workers=multiprocessing.cpu_count(), pin_memory=True)
        else:
            return DataLoader(self.mnist_train, batch_size=32, num_workers=multiprocessing.cpu_count())

    def val_dataloader(self):
        if torch.cuda.is_available():
            return DataLoader(self.mnist_val, batch_size=32, num_workers=multiprocessing.cpu_count(), pin_memory=True)
        else:
            return DataLoader(self.mnist_val, batch_size=32, num_workers=multiprocessing.cpu_count())

    def test_dataloader(self):
        if torch.cuda.is_available():
            return DataLoader(self.mnist_test, batch_size=32, num_workers=multiprocessing.cpu_count(), pin_memory=True)
        else:
            return DataLoader(self.mnist_test, batch_size=32, num_workers=multiprocessing.cpu_count())


# Build CNN
model = LitMNIST()

# Use GPUs if they are available
gpus = None
if torch.cuda.is_available():
    gpus = -1 # or torch.cuda.device_count()

# Train for 10 epochs
# Data Parallel (accelerator='dp') (multiple-gpus, 1 machine)
# https://pytorch-lightning.readthedocs.io/en/1.1.2/multi_gpu.html#distributed-modes
trainer = pl.Trainer(gpus=gpus, accelerator='dp', max_epochs=10, progress_bar_refresh_rate=20)
trainer.fit(model)

I'm using the following setup ($ pip freeze):

absl-py==0.12.0
aiohttp==3.7.4.post0
async-timeout==3.0.1
attrs==20.3.0
cachetools==4.2.1
certifi==2020.12.5
chardet==4.0.0
dataclasses==0.8
fsspec==0.8.7
future==0.18.2
google-auth==1.27.1
google-auth-oauthlib==0.4.3
grpcio==1.36.1
idna==2.10
idna-ssl==1.1.0
importlib-metadata==3.7.2
Markdown==3.3.4
multidict==5.1.0
numpy==1.19.5
oauthlib==3.1.0
Pillow==8.1.2
protobuf==3.15.6
py4j==0.10.9.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pytorch-lightning==1.2.3
PyYAML==5.3.1
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.7.2
six==1.15.0
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.0
torch==1.8.0
torchaudio==0.8.0
torchvision==0.9.0
tqdm==4.59.0
typing-extensions==3.7.4.3
urllib3==1.26.3
Werkzeug==1.0.1
wget==3.2
yarl==1.6.3
zipp==3.4.1

Someone knows what's happening ? Something wrong in the source code? Thanks in advance!

[andrewssobral]

I saw the following warning:

So i tested with ddp instead of dp, but i still stuck in the following screen:

The GPUs are at 100%:

But nothing happens....what it can be?

[andrewssobral]

Finally it works when I use Horovod + Gloo interface

[andrewssobral]

Hello @AlessandroW,

I solved it by using the Gloo interface instead of NCCL, here's the latest version of my source code:
https://gist.github.com/andrewssobral/090dcab34308bdd1ed75e5f2f6b4a1d0

I am using a NVIDIA K80 GPU Cluster and the NCCL is not working on this environment (i don't know why yet), so I just switched from NCCL to Gloo and now everything is working well...

If you are using DistributedDataParallel the only thing you need to do is:

$ export PL_TORCH_DISTRIBUTED_BACKEND=gloo
$ python pytorch_lightning_demo.py --accelerator ddp --gpus 2 --max_epochs 3 --model_checkpoint_enabled

If you are using Horovod:

$ horovodrun --gloo ....

Please let me know if it works for you

[AlessandroW]

Hey @andrewssobral ,

thanks for responding so quickly. A minute ago I stumbled upon this paragraph in the pl docs:

By default, Lightning will select the nccl backend over gloo when running on GPUs.
Lightning exposes an environment variable PL_TORCH_DISTRIBUTED_BACKEND for the user to change the backend.
PL_TORCH_DISTRIBUTED_BACKEND=gloo python train.py ...

Setting this variable fixed the issue for me as well, like you said.

I think others are having similar problems: #4612. However, in my case I have the same problem with DataParallel in the PyTorch example script. So it's probably a PyTorch bug, not Lightning's fault.

Thanks again!

[andrewssobral]

Great, thank you @AlessandroW !

[awaelchli]

Hi, just wanted to add on to it in case someone finds this discussion..

You mentinod that you have the gpus pinned at 100% and ddp does not initialize.
I strongly believe this is a problem with the iommu group, see here: pytorch/pytorch#1637
using gloo solves your problem yes, but if you run on gpu you probably want nccl ...

[aspenstarss]

It works for me.

[kingmbc]

Thank you for your effort! I also tried your 'horovod' code and succeeded, but I wanted to use 'ddp'.

And I finally found this, #4471 (comment)

'rank_zero_only=True' in self.log() function to solve this issue.

[nmichlo]

Having a similar issue. Even using the example MNIST code on the home page README but changing to gpus=2.

[jhschwartz]

@andrewssobral I'm having pretty much the same exact issue that you had. I'm wondering if you were running on a SLURM HPC? And if so, how did you install horovod?

[awaelchli]

I recommend moving the code

if not os.path.exists("MNIST"):
    wget.download("https://activeeon-public.s3.eu-west-2.amazonaws.com/datasets/MNIST.new.tar.gz", "MNIST.tar.gz")
    tar = tarfile.open("MNIST.tar.gz", "r:gz")
    tar.extractall()
    tar.close()

to the prepare_data method. Otherwise you run the risk of race condition or corrupted files as multiple workers attempt to download these files.