test_auto3dseg_ensemble #6155

wyli · 2023-03-16T06:59:58Z

PolynomialLR is new in torch 1.13 which breaks tests with earlier versions of pytorch

[2023-03-16T00:35:16.877Z] current epoch: 2 current mean dice: 0.4376 best mean dice: 0.4376 at epoch 2
[2023-03-16T00:35:16.877Z] train completed, best_metric: 0.4376 at epoch: 2
[2023-03-16T00:35:16.877Z] 2023-03-16 00:35:14,819 - INFO - The keys num_warmup_epochs cannot be found in the /tmp/tmpn5i1ty3o/workdir/dints_0/configs/hyper_parameters.yaml for training. Skipped overriding key num_warmup_epochs.
[2023-03-16T00:35:16.877Z] 2023-03-16 00:35:14,819 - INFO - The keys use_pretrain cannot be found in the /tmp/tmpn5i1ty3o/workdir/dints_0/configs/hyper_parameters.yaml for training. Skipped overriding key use_pretrain.
[2023-03-16T00:35:16.877Z] 2023-03-16 00:35:14,819 - INFO - The keys pretrained_path cannot be found in the /tmp/tmpn5i1ty3o/workdir/dints_0/configs/hyper_parameters.yaml for training. Skipped overriding key pretrained_path.
[2023-03-16T00:35:16.877Z] 2023-03-16 00:35:14,820 - INFO - Launching: python /tmp/tmpn5i1ty3o/workdir/dints_0/scripts/train.py run --config_file='/tmp/tmpn5i1ty3o/workdir/dints_0/configs/transforms_validate.yaml','/tmp/tmpn5i1ty3o/workdir/dints_0/configs/transforms_train.yaml','/tmp/tmpn5i1ty3o/workdir/dints_0/configs/network.yaml','/tmp/tmpn5i1ty3o/workdir/dints_0/configs/hyper_parameters.yaml','/tmp/tmpn5i1ty3o/workdir/dints_0/configs/transforms_infer.yaml','/tmp/tmpn5i1ty3o/workdir/dints_0/configs/hyper_parameters_search.yaml','/tmp/tmpn5i1ty3o/workdir/dints_0/configs/network_search.yaml' --training#num_images_per_batch=2 --training#num_epochs=2 --training#num_epochs_per_validation=1 --training#determ=True
[2023-03-16T00:35:24.956Z] monai.transforms.io.dictionary LoadImaged.__init__:image_only: Current default value of argument `image_only=False` has been deprecated since version 1.1. It will be changed to `image_only=True` in version 1.3.
[2023-03-16T00:35:24.958Z] [info] number of GPUs: 1
[2023-03-16T00:35:24.958Z] [info] world_size: 1
[2023-03-16T00:35:24.958Z] train_files: 8
[2023-03-16T00:35:24.958Z] val_files: 4
[2023-03-16T00:35:24.958Z] 2023-03-16 00:35:19.387838 - Length of input patch is recommended to be a multiple of 32.
[2023-03-16T00:35:24.958Z] num_epochs 2
[2023-03-16T00:35:24.958Z] num_epochs_per_validation 1
[2023-03-16T00:35:24.958Z] Traceback (most recent call last):
[2023-03-16T00:35:24.958Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/bundle/config_item.py", line 292, in instantiate
[2023-03-16T00:35:24.958Z]     return instantiate(modname, mode, **args)
[2023-03-16T00:35:24.958Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/utils/module.py", line 246, in instantiate
[2023-03-16T00:35:24.958Z]     raise ModuleNotFoundError(f"Cannot locate class or function path: '{__path}'.")
[2023-03-16T00:35:24.958Z] ModuleNotFoundError: Cannot locate class or function path: 'torch.optim.lr_scheduler.PolynomialLR'.
[2023-03-16T00:35:24.958Z] 
[2023-03-16T00:35:24.958Z] The above exception was the direct cause of the following exception:
[2023-03-16T00:35:24.958Z] 
[2023-03-16T00:35:24.958Z] Traceback (most recent call last):
[2023-03-16T00:35:24.958Z]   File "/tmp/tmpn5i1ty3o/workdir/dints_0/scripts/train.py", line 479, in <module>
[2023-03-16T00:35:24.958Z]     fire.Fire()
[2023-03-16T00:35:24.958Z]   File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
[2023-03-16T00:35:24.958Z]     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[2023-03-16T00:35:24.958Z]   File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire
[2023-03-16T00:35:24.958Z]     component, remaining_args = _CallAndUpdateTrace(
[2023-03-16T00:35:24.958Z]   File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
[2023-03-16T00:35:24.958Z]     component = fn(*varargs, **kwargs)
[2023-03-16T00:35:24.958Z]   File "/tmp/tmpn5i1ty3o/workdir/dints_0/scripts/train.py", line 214, in run
[2023-03-16T00:35:24.958Z]     lr_scheduler = lr_scheduler_part.instantiate(optimizer=optimizer)
[2023-03-16T00:35:24.958Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/bundle/config_item.py", line 294, in instantiate
[2023-03-16T00:35:24.958Z]     raise RuntimeError(f"Failed to instantiate {self}.") from e
[2023-03-16T00:35:24.958Z] RuntimeError: Failed to instantiate {'_target_': 'torch.optim.lr_scheduler.PolynomialLR', 'optimizer': {'_target_': 'torch.optim.SGD', 'lr': 0.2, 'momentum': 0.9, 'weight_decay': 4e-05}, 'power': 0.5, 'total_iters': 3}.
[2023-03-16T00:35:25.212Z] 
[2023-03-16T00:35:25.212Z] EFinished test: test_ensemble (tests.test_auto3dseg_ensemble.TestEnsembleBuilder) (76.7s)

tests/test_integration_autorunner.py
tests/test_auto3dseg_ensemble.py

issue introduced by af46d7b @mingxin-zheng @dongyang0122 could you please have a look soon? skip the test or update the algo

The text was updated successfully, but these errors were encountered:

wyli · 2023-03-16T07:04:42Z

the same test might have some memory issue as well

current epoch: 1 current mean dice: 0.3736 best mean dice: 0.3736 at epoch 1
----------
epoch 2/2
learning rate is set to 0.0125
Traceback (most recent call last):
  File "/tmp/tmpox_opph9/workdir/dints_0/scripts/search.py", line 647, in <module>
    fire.Fire()
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/tmp/tmpox_opph9/workdir/dints_0/scripts/search.py", line 331, in run
    scaler.step(optimizer)
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py", line 370, in step
    retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py", line 290, in _maybe_opt_step
    retval = optimizer.step(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 69, in wrapper
    return wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/optimizer.py", line 280, in wrapper
    out = func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/optimizer.py", line 33, in _use_grad
    ret = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/sgd.py", line 76, in step
    sgd(params_with_grad,
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/sgd.py", line 222, in sgd
    func(params,
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/sgd.py", line 291, in _multi_tensor_sgd
    device_grads = torch._foreach_add(device_grads, device_params, alpha=weight_decay)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.76 GiB total capacity; 2.02 GiB already allocated; 15.00 MiB free; 2.06 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 42535 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 42534) of binary: /opt/conda/bin/python
Traceback (most recent call last):
  File "/opt/conda/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/tmp/tmpox_opph9/workdir/dints_0/scripts/search.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-03-16_05:18:53
  host      : 957ff6064347
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 42534)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

EFinished test: test_ensemble (tests.test_auto3dseg_ensemble.TestEnsembleBuilder) (58.7s)

mingxin-zheng · 2023-03-16T08:04:42Z

Need to discuss with @dongyang0122 .

I am inclined to update the algorithm templates. What do you think @dongyang0122 ?

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

dongyang0122 · 2023-03-16T14:59:44Z

Which version of PyTorch should we use for developing to pass the unit test?

mingxin-zheng · 2023-03-16T15:13:33Z

We are using PyTorch 1.8 - 1.13. I think the PyTorch 1.8-1.12 tests are optional during the PR.

mingxin-zheng · 2023-03-16T15:15:30Z

Is it viable to detect the version of PyTorch in dints Auto3DSeg templates, and fall back to the previous scheduler if it is not causing too much of performance decrease? I'm also okay to skip the tests for now, but it will mean PyTorch 1.12 users will not be able to use Auto3DSeg.

dongyang0122 · 2023-03-16T15:40:53Z

I may need to re-write the scheduler logic manually to avoid the version issue.

wyli · 2023-03-16T15:42:41Z

ok, let's keep this ticket open and I'll make a workaround to skip the tests for torch <=1.12 to unblock the release candidate

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

wyli · 2023-03-16T18:35:13Z

the related tutorials https://github.com/Project-MONAI/tutorials are also broken

./auto3dseg/notebooks/ensemble_byoc.ipynb
./auto3dseg/notebooks/auto3dseg_hello_world.ipynb
./auto3dseg/notebooks/auto3dseg_autorunner_ref_api.ipynb

mingxin-zheng · 2023-03-18T14:22:15Z

Hi @dongyang0122 @wyli ,
I followed @dongyang0122 's thought partially to fix this issue in research-contribution:
Project-MONAI/research-contributions#204

The main issue there was that the class script is from PyTorch. So I include the full license in the script for legal.

wyli added a commit that referenced this issue Mar 16, 2023

workaround #6155

3a29933

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

wyli added a commit to wyli/MONAI that referenced this issue Mar 16, 2023

workaround Project-MONAI#6155

dd31ea2

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

mingxin-zheng mentioned this issue Mar 18, 2023

PolynomialLR is only supported after PyTorch 1.13, which limits Auto3DSeg applications Project-MONAI/research-contributions#203

Closed

wyli mentioned this issue Mar 21, 2023

Update ALGO_HASH to make Auto3DSeg dints support older version of PyT… #6175

Merged

1 task

wyli closed this as completed in #6175 Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_auto3dseg_ensemble #6155

test_auto3dseg_ensemble #6155

wyli commented Mar 16, 2023 •

edited

Loading

wyli commented Mar 16, 2023

mingxin-zheng commented Mar 16, 2023 •

edited

Loading

dongyang0122 commented Mar 16, 2023

mingxin-zheng commented Mar 16, 2023

mingxin-zheng commented Mar 16, 2023

dongyang0122 commented Mar 16, 2023

wyli commented Mar 16, 2023

wyli commented Mar 16, 2023

mingxin-zheng commented Mar 18, 2023 •

edited

Loading

test_auto3dseg_ensemble #6155

test_auto3dseg_ensemble #6155

Comments

wyli commented Mar 16, 2023 • edited Loading

wyli commented Mar 16, 2023

mingxin-zheng commented Mar 16, 2023 • edited Loading

dongyang0122 commented Mar 16, 2023

mingxin-zheng commented Mar 16, 2023

mingxin-zheng commented Mar 16, 2023

dongyang0122 commented Mar 16, 2023

wyli commented Mar 16, 2023

wyli commented Mar 16, 2023

mingxin-zheng commented Mar 18, 2023 • edited Loading

wyli commented Mar 16, 2023 •

edited

Loading

mingxin-zheng commented Mar 16, 2023 •

edited

Loading

mingxin-zheng commented Mar 18, 2023 •

edited

Loading