Testing in dp mode uses only one of the GPUs

[Ir1d]

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

Run a test without training

Code sample

Modified from the conference-seed repo

trainer = Trainer(
            gpus="-1",
            distributed_backend='dp',
        )
trainer.test(model)

Expected behavior

Environment

• pl version: 0.6.0
• PyTorch Version (e.g., 1.0): 1.2
• OS (e.g., Linux): Ubuntu
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.6
• CUDA/cuDNN version: 10.1
• GPU models and configuration:
• Any other relevant information:

Additional context

[williamFalcon]

ummm yeah, that's a bug. it should run via dp. @Ir1d want to submit a PR?
@PyTorchLightning/core-contributors any one else experience this?

[Ir1d]

@williamFalcon I tried wrapping the model in LightningDataParallel like the training loop did, but it tells me that LightningDataParallel object doen't have a test_step. How do I debug this?

[williamFalcon]

you wouldn’t wrap it yourself ever haha.
the trainer does the wrapping for you.

the trainer needs to be modified to run the test on the correct method when done this way

[Ir1d]

I was trying to wrap it in evaluate in pytorch_lightning/trainer/evaluation_loop.py . Do you have any idea where to wrap this func?

[Ir1d]

Anyway, we've find one possible workround here:
After defining a torch model, and before sending it into PL model, wrap it with nn.dataparallel.

[williamFalcon]

evaluate is private... you're not meant to call it directly.

call .test()

[williamFalcon]

lightning does the wrapping by itself...

the fact that this doesn't work, is a bug.

model = MyLightningModule.load_from_checkpoint(...)
trainer = Trainer(
            gpus="-1",
            distributed_backend='dp',
        )
trainer.test(model)

The bug needs to be addressed correctly.

It's weird because we have tests for this... double check that this is really not working for you.

[Borda]

evaluate is private... you're not meant to call it directly.
call .test()

so let's rename it starting with _ to be clear that it is private from it name

[Ir1d]

I was calling .test and its not working

[Borda]

@neggert may you have look at this multi GPU issue?

[edenlightning]

@neggert ping :)

[williamFalcon]

looking at this with next sprint

[williamFalcon]

fixed! (0.8.5)