Auto3dSeg Swin-Unet Shared Memory issues

[peterhessey]

When trying to run the AutoRunner in AzureML I am running into lack of shared memory errors during the Swin-Unet training stage. The node has 448GB of shared memory which I thought would be more than sufficient, especially as I am only using a single image per batch. Interestingly I don't run into this error when running on my local cluster with the same GPUs + RAM available.

Any ideas on the causes and potential fixes for this issue?

Stack trace for reference:

2022-12-15 11:09:38,980 - INFO - Launching: torchrun --nnodes=1 --nproc_per_node=4 /mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/scripts/train.py run --config_file='/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/transforms_validate.yaml','/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/network.yaml','/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/hyper_parameters.yaml','/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/transforms_infer.yaml','/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/transforms_train.yaml' --num_iterations=8 --num_iterations_per_validation=4 --num_images_per_batch=1 --num_epochs=2 --num_warmup_iterations=4

./outputs/lung/dints_0 does not contain /mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/dints_0/algo_object.pkl.Now override the Algo output_path with: /mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/dints_0
./outputs/lung/swinunetr_0 does not contain /mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/algo_object.pkl.Now override the Algo output_path with: /mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0
./outputs/lung/segresnet2d_0 does not contain /mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/segresnet2d_0/algo_object.pkl.Now override the Algo output_path with: /mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/segresnet2d_0
Traceback (most recent call last):
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 185, in _run_cmd
    normal_out = subprocess.run(cmd.split(), env=ps_environ, check=True, capture_output=True)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['torchrun', '--nnodes=1', '--nproc_per_node=4', '/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/scripts/train.py', 'run', "--config_file='/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/transforms_validate.yaml','/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/network.yaml','/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/hyper_parameters.yaml','/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/transforms_infer.yaml','/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/configs/transforms_train.yaml'", '--num_iterations=8', '--num_iterations_per_validation=4', '--num_images_per_batch=1', '--num_epochs=2', '--num_warmup_iterations=4']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "lung_autoseg_specific_architectures.py", line 98, in <module>
    main(run_info)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "lung_autoseg_specific_architectures.py", line 93, in main
    runner.run()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/monai/apps/auto3dseg/auto_runner.py", line 586, in run
    self._train_algo_in_sequence(history)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/monai/apps/auto3dseg/auto_runner.py", line 488, in _train_algo_in_sequence
    algo.train(self.train_params)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 202, in train
    return self._run_cmd(cmd, devices_info)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 190, in _run_cmd
    raise RuntimeError(f"subprocess call error {e.returncode}: {errors}, {output}") from e
RuntimeError: subprocess call error 1: b'WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
\x00ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
\x00Traceback (most recent call last):
  File "/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/scripts/train.py", line 409, in <module>
    fire.Fire()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire
    target=component.__name__)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/scripts/train.py", line 300, in run
    for val_data in val_loader:
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1316, in _next_data
    idx, data = self._get_data()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1282, in _get_data
    success, data = self._try_get_data()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1120, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 305, in rebuild_storage_fd
    fd = df.detach()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 498, in Client
    answer_challenge(c, authkey)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 741, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1120, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 305, in rebuild_storage_fd
    fd = df.detach()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 498, in Client
    answer_challenge(c, authkey)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 741, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/scripts/train.py", line 409, in <module>
    fire.Fire()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire
    target=component.__name__)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/scripts/train.py", line 300, in run
    for val_data in val_loader:
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1316, in _next_data
    idx, data = self._get_data()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1282, in _get_data
    success, data = self._try_get_data()
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1120, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 4595) is killed by signal: Bus error. It is possible that dataloader\'s workers are out of shared memory. Please try to raise your shared memory limit.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 4620) is killed by signal: Bus error. It is possible that dataloader\'s workers are out of shared memory. Please try to raise your shared memory limit.
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4248 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4250 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 4247) of binary: /azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/bin/python
Traceback (most recent call last):
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/distributed/run.py", line 762, in main
    run(args)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/distributed/run.py", line 756, in run
    )(*cmd_args)
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/azureml-envs/azureml_7330a11b6ce5e1af87f67443c6076ab5/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 248, in launch_agent
    failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/mnt/azureml/cr/j/ac8eb57c886249a0970c97ee42214639/exe/wd/outputs/lung/swinunetr_0/scripts/train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2022-12-15_11:10:35
  host      : bb3cc6c16c0a48ad9d7dfb85f2cefd83000008.internal.cloudapp.net
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 4249)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-12-15_11:10:35
  host      : bb3cc6c16c0a48ad9d7dfb85f2cefd83000008.internal.cloudapp.net
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 4247)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
', b'[info] number of GPUs: 4
[info] number of GPUs: 4
[info] number of GPUs: 4
2022-12-15 11:09:45,551 - Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-15 11:09:45,553 - Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-15 11:09:45,563 - Added key: store_based_barrier_key:1 to store for rank: 0
[info] number of GPUs: 4
2022-12-15 11:09:45,658 - Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-15 11:09:45,659 - Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
[info] world_size: 4
2022-12-15 11:09:45,663 - Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
[info] world_size: 4
train_files: 9
2022-12-15 11:09:45,665 - Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-15 11:09:45,665 - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
[info] world_size: [info] world_size:4
 4
val_files: 3
train_files: 9
val_files: 3
train_files: 9
train_files: 9
val_files: 3
val_files: 3
num_epochs 2
num_epochs_per_validation 1
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4247 [0] NCCL INFO Bootstrap : Using eth0:10.0.0.4<0>
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4247 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4247 [0] NCCL INFO cudaDriverVersion 11040
NCCL version 2.14.3+cuda11.7
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4248 [1] NCCL INFO cudaDriverVersion 11040
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4249 [2] NCCL INFO cudaDriverVersion 11040
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4250 [3] NCCL INFO cudaDriverVersion 11040
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO NET/Socket : Using [0]eth0:10.0.0.4<0>
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Using network Socket
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4248 [1] NCCL INFO Bootstrap : Using eth0:10.0.0.4<0>
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4248 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4249 [2] NCCL INFO Bootstrap : Using eth0:10.0.0.4<0>
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4249 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4250 [3] NCCL INFO Bootstrap : Using eth0:10.0.0.4<0>
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4250 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO NET/Socket : Using [0]eth0:10.0.0.4<0>
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Using network Socket
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO NET/Socket : Using [0]eth0:10.0.0.4<0>
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Using network Socket
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO NET/Socket : Using [0]eth0:10.0.0.4<0>
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Using network Socket
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Channel 00/02 :    0   1   2   3
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Channel 01/02 :    0   1   2   3
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Channel 00 : 1[b24600000] -> 2[d33f00000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Channel 00 : 0[964300000] -> 1[b24600000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Channel 00 : 3[f76b00000] -> 0[964300000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Channel 00 : 2[d33f00000] -> 3[f76b00000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Channel 01 : 1[b24600000] -> 2[d33f00000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Channel 01 : 0[964300000] -> 1[b24600000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Channel 01 : 3[f76b00000] -> 0[964300000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Channel 01 : 2[d33f00000] -> 3[f76b00000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Connected all rings
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Connected all rings
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Connected all rings
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Connected all rings
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Channel 00 : 3[f76b00000] -> 2[d33f00000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Channel 01 : 3[f76b00000] -> 2[d33f00000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Channel 00 : 1[b24600000] -> 0[964300000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Channel 00 : 2[d33f00000] -> 1[b24600000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Channel 01 : 1[b24600000] -> 0[964300000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Channel 01 : 2[d33f00000] -> 1[b24600000] via SHM/direct/direct
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO Connected all trees
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO Connected all trees
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO Connected all trees
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO Connected all trees
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4250:4352 [3] NCCL INFO comm 0x56315e63b980 rank 3 nranks 4 cudaDev 3 busId f76b00000 - Init COMPLETE
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4349 [0] NCCL INFO comm 0x55fd19ed0fc0 rank 0 nranks 4 cudaDev 0 busId 964300000 - Init COMPLETE
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4248:4350 [1] NCCL INFO comm 0x5619dd9403c0 rank 1 nranks 4 cudaDev 1 busId b24600000 - Init COMPLETE
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4351 [2] NCCL INFO comm 0x55d36f866ea0 rank 2 nranks 4 cudaDev 2 busId d33f00000 - Init COMPLETE
[info] training from scratch
[info] training from scratch
[info] training from scratch[info] training from scratch

[info] amp enabled
----------
epoch 1/2
learning rate is set to 0.0001
[2022-12-15 11:10:23] 1/9, train_loss: 2.5514
2022-12-15 11:10:23,686 - Reducer buckets have been rebuilt in this iteration.
2022-12-15 11:10:23,686 - Reducer buckets have been rebuilt in this iteration.
2022-12-15 11:10:23,686 - Reducer buckets have been rebuilt in this iteration.
2022-12-15 11:10:23,687 - Reducer buckets have been rebuilt in this iteration.
[2022-12-15 11:10:24] 2/9, train_loss: 2.6897
[2022-12-15 11:10:25] 3/9, train_loss: 2.3405
[2022-12-15 11:10:26] 4/9, train_loss: 2.3721
[2022-12-15 11:10:27] 5/9, train_loss: 2.1750
[2022-12-15 11:10:28] 6/9, train_loss: 2.3879
[2022-12-15 11:10:29] 7/9, train_loss: 2.1925
[2022-12-15 11:10:30] 8/9, train_loss: 2.1194
[2022-12-15 11:10:31] 9/9, train_loss: 2.1066
epoch 1 average loss: 2.3342, best mean dice: -1.0000 at epoch -1
2022-12-15 11:10:31,644 - > collate dict key "image" out of 4 keys
2022-12-15 11:10:31,645 - > collate dict key "image" out of 4 keys
2022-12-15 11:10:31,655 - >> collate/stack a list of tensors
2022-12-15 11:10:31,657 - >> collate/stack a list of tensors
2022-12-15 11:10:31,695 - > collate dict key "label" out of 4 keys
2022-12-15 11:10:31,702 - >> collate/stack a list of tensors
2022-12-15 11:10:31,705 - > collate dict key "label" out of 4 keys
2022-12-15 11:10:31,713 - >> collate/stack a list of tensors
2022-12-15 11:10:31,714 - > collate dict key "image_meta_dict" out of 4 keys
2022-12-15 11:10:31,718 - >> collate dict key "sizeof_hdr" out of 43 keys
2022-12-15 11:10:31,718 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,718 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,718 - >> collate dict key "extents" out of 43 keys
2022-12-15 11:10:31,719 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,719 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,719 - >> collate dict key "session_error" out of 43 keys
2022-12-15 11:10:31,719 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,719 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,719 - >> collate dict key "dim_info" out of 43 keys
2022-12-15 11:10:31,719 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,720 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,720 - >> collate dict key "dim" out of 43 keys
2022-12-15 11:10:31,720 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,720 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,720 - >> collate dict key "intent_p1" out of 43 keys
2022-12-15 11:10:31,720 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,721 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,721 - >> collate dict key "intent_p2" out of 43 keys
2022-12-15 11:10:31,721 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,721 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,721 - >> collate dict key "intent_p3" out of 43 keys
2022-12-15 11:10:31,721 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,722 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,722 - >> collate dict key "intent_code" out of 43 keys
2022-12-15 11:10:31,722 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,722 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,722 - >> collate dict key "datatype" out of 43 keys
2022-12-15 11:10:31,722 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,722 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,723 - >> collate dict key "bitpix" out of 43 keys
2022-12-15 11:10:31,723 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,723 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,723 - >> collate dict key "slice_start" out of 43 keys
2022-12-15 11:10:31,723 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,723 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,723 - >> collate dict key "pixdim" out of 43 keys
2022-12-15 11:10:31,724 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,724 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,724 - >> collate dict key "vox_offset" out of 43 keys
2022-12-15 11:10:31,724 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,725 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,725 - >> collate dict key "scl_slope" out of 43 keys
2022-12-15 11:10:31,725 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,725 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,725 - >> collate dict key "scl_inter" out of 43 keys
2022-12-15 11:10:31,725 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,726 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,726 - >> collate dict key "slice_end" out of 43 keys
2022-12-15 11:10:31,726 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,726 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,726 - >> collate dict key "slice_code" out of 43 keys
2022-12-15 11:10:31,726 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,726 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,727 - >> collate dict key "xyzt_units" out of 43 keys
2022-12-15 11:10:31,727 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,727 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,727 - >> collate dict key "cal_max" out of 43 keys
2022-12-15 11:10:31,727 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,727 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,727 - >> collate dict key "cal_min" out of 43 keys
2022-12-15 11:10:31,727 - > collate dict key "image_meta_dict" out of 4 keys
2022-12-15 11:10:31,728 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,728 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,728 - >> collate dict key "slice_duration" out of 43 keys
2022-12-15 11:10:31,728 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,728 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,728 - >> collate dict key "toffset" out of 43 keys
2022-12-15 11:10:31,729 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,729 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,729 - >> collate dict key "glmax" out of 43 keys
2022-12-15 11:10:31,729 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,729 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,729 - >> collate dict key "glmin" out of 43 keys
2022-12-15 11:10:31,729 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,730 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,730 - >> collate dict key "qform_code" out of 43 keys
2022-12-15 11:10:31,730 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,730 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,730 - >> collate dict key "sform_code" out of 43 keys
2022-12-15 11:10:31,730 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,730 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,731 - >> collate dict key "quatern_b" out of 43 keys
2022-12-15 11:10:31,731 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,731 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,731 - >> collate dict key "quatern_c" out of 43 keys
2022-12-15 11:10:31,731 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,731 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,731 - >> collate dict key "quatern_d" out of 43 keys
2022-12-15 11:10:31,732 - >> collate dict key "sizeof_hdr" out of 43 keys
2022-12-15 11:10:31,732 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,732 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,732 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,732 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,732 - >> collate dict key "extents" out of 43 keys
2022-12-15 11:10:31,732 - >> collate dict key "qoffset_x" out of 43 keys
2022-12-15 11:10:31,732 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,732 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,732 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,732 - >> collate dict key "session_error" out of 43 keys
2022-12-15 11:10:31,733 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,733 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,733 - >> collate dict key "qoffset_y" out of 43 keys
2022-12-15 11:10:31,733 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,733 - >> collate dict key "dim_info" out of 43 keys
2022-12-15 11:10:31,733 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,733 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,733 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,733 - >> collate dict key "dim" out of 43 keys
2022-12-15 11:10:31,733 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,733 - >> collate dict key "qoffset_z" out of 43 keys
2022-12-15 11:10:31,733 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,734 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,734 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,734 - >> collate dict key "intent_p1" out of 43 keys
2022-12-15 11:10:31,734 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,734 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,734 - >> collate dict key "srow_x" out of 43 keys
2022-12-15 11:10:31,734 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,734 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,734 - >> collate dict key "intent_p2" out of 43 keys
2022-12-15 11:10:31,734 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,734 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,735 - >> collate dict key "srow_y" out of 43 keys
2022-12-15 11:10:31,735 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,735 - >> collate dict key "intent_p3" out of 43 keys
2022-12-15 11:10:31,735 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,735 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,735 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,735 - >> collate dict key "srow_z" out of 43 keys
2022-12-15 11:10:31,735 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,735 - >> collate dict key "intent_code" out of 43 keys
2022-12-15 11:10:31,735 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,735 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,736 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,736 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,736 - >> collate dict key "affine" out of 43 keys
2022-12-15 11:10:31,736 - >> collate dict key "datatype" out of 43 keys
2022-12-15 11:10:31,736 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,736 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,736 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,736 - >> collate dict key "bitpix" out of 43 keys
2022-12-15 11:10:31,736 - >> collate dict key "original_affine" out of 43 keys
2022-12-15 11:10:31,736 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,736 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,736 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,737 - >> collate dict key "slice_start" out of 43 keys
2022-12-15 11:10:31,737 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,737 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,737 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,737 - >> collate dict key "pixdim" out of 43 keys
2022-12-15 11:10:31,737 - >> collate dict key "as_closest_canonical" out of 43 keys
2022-12-15 11:10:31,737 - >> collate dict key "spatial_shape" out of 43 keys
2022-12-15 11:10:31,737 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,737 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,737 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,738 - >> collate dict key "space" out of 43 keys
2022-12-15 11:10:31,738 - >> collate dict key "original_channel_dim" out of 43 keys
2022-12-15 11:10:31,738 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,738 - >> collate dict key "filename_or_obj" out of 43 keys
2022-12-15 11:10:31,738 - >> collate dict key "vox_offset" out of 43 keys
2022-12-15 11:10:31,738 - > collate dict key "label_meta_dict" out of 4 keys
2022-12-15 11:10:31,738 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,738 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,738 - >> collate dict key "scl_slope" out of 43 keys
2022-12-15 11:10:31,738 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,739 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,739 - >> collate dict key "scl_inter" out of 43 keys
2022-12-15 11:10:31,739 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,739 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,739 - >> collate dict key "slice_end" out of 43 keys
2022-12-15 11:10:31,740 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,740 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,740 - >> collate dict key "slice_code" out of 43 keys
2022-12-15 11:10:31,740 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,740 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,740 - >> collate dict key "xyzt_units" out of 43 keys
2022-12-15 11:10:31,740 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,740 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,741 - >> collate dict key "cal_max" out of 43 keys
2022-12-15 11:10:31,741 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,741 - >> collate dict key "sizeof_hdr" out of 43 keys
2022-12-15 11:10:31,741 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,741 - >> collate dict key "cal_min" out of 43 keys
2022-12-15 11:10:31,741 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,741 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,741 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,741 - >> collate dict key "extents" out of 43 keys
2022-12-15 11:10:31,741 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,742 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,742 - >> collate dict key "slice_duration" out of 43 keys
2022-12-15 11:10:31,742 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,742 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,742 - >> collate dict key "session_error" out of 43 keys
2022-12-15 11:10:31,742 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,742 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,742 - >> collate dict key "toffset" out of 43 keys
2022-12-15 11:10:31,742 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,742 - >> collate dict key "dim_info" out of 43 keys
2022-12-15 11:10:31,742 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,742 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,742 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,743 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,743 - >> collate dict key "glmax" out of 43 keys
2022-12-15 11:10:31,743 - >> collate dict key "dim" out of 43 keys
2022-12-15 11:10:31,743 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,743 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,743 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,743 - >> collate dict key "glmin" out of 43 keys
2022-12-15 11:10:31,743 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,743 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,743 - >> collate dict key "intent_p1" out of 43 keys
2022-12-15 11:10:31,743 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,743 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,743 - >> collate dict key "qform_code" out of 43 keys
2022-12-15 11:10:31,743 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,744 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,744 - >> collate dict key "intent_p2" out of 43 keys
2022-12-15 11:10:31,744 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,744 - >> collate dict key "sform_code" out of 43 keys
2022-12-15 11:10:31,744 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,744 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,744 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,744 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,744 - >> collate dict key "intent_p3" out of 43 keys
2022-12-15 11:10:31,744 - >> collate dict key "quatern_b" out of 43 keys
2022-12-15 11:10:31,744 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,744 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,745 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,745 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,745 - >> collate dict key "quatern_c" out of 43 keys
2022-12-15 11:10:31,745 - >> collate dict key "intent_code" out of 43 keys
2022-12-15 11:10:31,745 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,745 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,745 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,745 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,745 - >> collate dict key "datatype" out of 43 keys
2022-12-15 11:10:31,745 - >> collate dict key "quatern_d" out of 43 keys
2022-12-15 11:10:31,745 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,745 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,745 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,745 - >> collate dict key "bitpix" out of 43 keys
2022-12-15 11:10:31,746 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,746 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,746 - >> collate dict key "qoffset_x" out of 43 keys
2022-12-15 11:10:31,746 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,746 - >> collate dict key "slice_start" out of 43 keys
2022-12-15 11:10:31,746 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,746 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,746 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,746 - >> collate dict key "pixdim" out of 43 keys
2022-12-15 11:10:31,746 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,746 - >> collate dict key "qoffset_y" out of 43 keys
2022-12-15 11:10:31,747 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,747 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,747 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,747 - >> collate dict key "vox_offset" out of 43 keys
2022-12-15 11:10:31,747 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,747 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,747 - >> collate dict key "qoffset_z" out of 43 keys
2022-12-15 11:10:31,747 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,747 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,748 - >> collate dict key "scl_slope" out of 43 keys
2022-12-15 11:10:31,748 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,748 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,748 - >> collate dict key "srow_x" out of 43 keys
2022-12-15 11:10:31,748 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,748 - >> collate dict key "scl_inter" out of 43 keys
2022-12-15 11:10:31,748 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,748 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,748 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,748 - >> collate dict key "srow_y" out of 43 keys
2022-12-15 11:10:31,748 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,749 - >> collate dict key "slice_end" out of 43 keys
2022-12-15 11:10:31,749 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,749 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,749 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,749 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,749 - >> collate dict key "srow_z" out of 43 keys
2022-12-15 11:10:31,749 - >> collate dict key "slice_code" out of 43 keys
2022-12-15 11:10:31,749 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,749 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,749 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,749 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,749 - >> collate dict key "xyzt_units" out of 43 keys
2022-12-15 11:10:31,749 - >> collate dict key "affine" out of 43 keys
2022-12-15 11:10:31,750 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,750 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,750 - >> collate dict key "cal_max" out of 43 keys
2022-12-15 11:10:31,750 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,750 - >> collate dict key "original_affine" out of 43 keys
2022-12-15 11:10:31,750 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,750 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,750 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,750 - >> collate dict key "cal_min" out of 43 keys
2022-12-15 11:10:31,750 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,751 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,751 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,751 - >> collate dict key "slice_duration" out of 43 keys
2022-12-15 11:10:31,751 - >> collate dict key "as_closest_canonical" out of 43 keys
2022-12-15 11:10:31,751 - >> collate dict key "spatial_shape" out of 43 keys
2022-12-15 11:10:31,751 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,751 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,751 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,751 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,751 - >> collate dict key "toffset" out of 43 keys
2022-12-15 11:10:31,751 - >> collate dict key "space" out of 43 keys
2022-12-15 11:10:31,751 - >> collate dict key "original_channel_dim" out of 43 keys
2022-12-15 11:10:31,751 - >> collate dict key "filename_or_obj" out of 43 keys
2022-12-15 11:10:31,751 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,752 - > collate dict key "label_meta_dict" out of 4 keys
2022-12-15 11:10:31,752 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,752 - >> collate dict key "glmax" out of 43 keys
2022-12-15 11:10:31,752 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,752 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,752 - >> collate dict key "glmin" out of 43 keys
2022-12-15 11:10:31,752 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,753 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,753 - >> collate dict key "qform_code" out of 43 keys
2022-12-15 11:10:31,753 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,753 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,753 - >> collate dict key "sform_code" out of 43 keys
2022-12-15 11:10:31,753 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,753 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,753 - >> collate dict key "quatern_b" out of 43 keys
2022-12-15 11:10:31,754 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,754 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,754 - >> collate dict key "quatern_c" out of 43 keys
2022-12-15 11:10:31,754 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,754 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,754 - >> collate dict key "quatern_d" out of 43 keys
2022-12-15 11:10:31,755 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,755 - >> collate dict key "sizeof_hdr" out of 43 keys
2022-12-15 11:10:31,755 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,755 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,755 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,755 - >> collate dict key "qoffset_x" out of 43 keys
2022-12-15 11:10:31,755 - >> collate dict key "extents" out of 43 keys
2022-12-15 11:10:31,755 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,755 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,755 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,756 - >> collate dict key "session_error" out of 43 keys
2022-12-15 11:10:31,756 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,756 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,756 - >> collate dict key "qoffset_y" out of 43 keys
2022-12-15 11:10:31,756 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,756 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,756 - >> collate dict key "dim_info" out of 43 keys
2022-12-15 11:10:31,756 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,756 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,756 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,756 - >> collate dict key "qoffset_z" out of 43 keys
2022-12-15 11:10:31,756 - >> collate dict key "dim" out of 43 keys
2022-12-15 11:10:31,756 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,757 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,757 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,757 - >> collate dict key "intent_p1" out of 43 keys
2022-12-15 11:10:31,757 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,757 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,757 - >> collate dict key "srow_x" out of 43 keys
2022-12-15 11:10:31,757 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,757 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,757 - >> collate dict key "intent_p2" out of 43 keys
2022-12-15 11:10:31,757 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,757 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,757 - >> collate dict key "srow_y" out of 43 keys
2022-12-15 11:10:31,758 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,758 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,758 - >> collate dict key "intent_p3" out of 43 keys
2022-12-15 11:10:31,758 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,758 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,758 - >> collate dict key "srow_z" out of 43 keys
2022-12-15 11:10:31,758 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,758 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,758 - >> collate dict key "intent_code" out of 43 keys
2022-12-15 11:10:31,758 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,759 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,759 - >> collate dict key "affine" out of 43 keys
2022-12-15 11:10:31,759 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,759 - >> collate dict key "datatype" out of 43 keys
2022-12-15 11:10:31,759 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,759 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,759 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,759 - >> collate dict key "bitpix" out of 43 keys
2022-12-15 11:10:31,759 - >> collate dict key "original_affine" out of 43 keys
2022-12-15 11:10:31,759 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,759 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,760 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,760 - >> collate dict key "slice_start" out of 43 keys
2022-12-15 11:10:31,760 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,760 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,760 - >> collate dict key "pixdim" out of 43 keys
2022-12-15 11:10:31,760 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,760 - >> collate dict key "as_closest_canonical" out of 43 keys
2022-12-15 11:10:31,760 - >> collate dict key "spatial_shape" out of 43 keys
2022-12-15 11:10:31,760 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,760 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,761 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,761 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,761 - >> collate dict key "space" out of 43 keys
2022-12-15 11:10:31,761 - >> collate dict key "original_channel_dim" out of 43 keys
2022-12-15 11:10:31,761 - >> collate dict key "vox_offset" out of 43 keys
2022-12-15 11:10:31,761 - >> collate dict key "filename_or_obj" out of 43 keys
2022-12-15 11:10:31,761 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,761 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,761 - >> collate dict key "scl_slope" out of 43 keys
2022-12-15 11:10:31,761 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,762 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,762 - >> collate dict key "scl_inter" out of 43 keys
2022-12-15 11:10:31,762 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,762 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,762 - >> collate dict key "slice_end" out of 43 keys
2022-12-15 11:10:31,762 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,763 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,763 - >> collate dict key "slice_code" out of 43 keys
2022-12-15 11:10:31,763 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,763 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,763 - >> collate dict key "xyzt_units" out of 43 keys
2022-12-15 11:10:31,763 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,763 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,764 - >> collate dict key "cal_max" out of 43 keys
2022-12-15 11:10:31,764 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,764 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,764 - >> collate dict key "cal_min" out of 43 keys
2022-12-15 11:10:31,764 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,764 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,765 - >> collate dict key "slice_duration" out of 43 keys
2022-12-15 11:10:31,765 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,765 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,765 - >> collate dict key "toffset" out of 43 keys
2022-12-15 11:10:31,765 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,765 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,765 - >> collate dict key "glmax" out of 43 keys
2022-12-15 11:10:31,766 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,766 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,766 - >> collate dict key "glmin" out of 43 keys
2022-12-15 11:10:31,766 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,766 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,766 - >> collate dict key "qform_code" out of 43 keys
2022-12-15 11:10:31,766 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,767 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,767 - >> collate dict key "sform_code" out of 43 keys
2022-12-15 11:10:31,767 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,767 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,767 - >> collate dict key "quatern_b" out of 43 keys
2022-12-15 11:10:31,767 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,767 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,768 - >> collate dict key "quatern_c" out of 43 keys
2022-12-15 11:10:31,768 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,768 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,768 - >> collate dict key "quatern_d" out of 43 keys
2022-12-15 11:10:31,768 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,769 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,769 - >> collate dict key "qoffset_x" out of 43 keys
2022-12-15 11:10:31,769 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,769 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,769 - >> collate dict key "qoffset_y" out of 43 keys
2022-12-15 11:10:31,769 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,770 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,770 - >> collate dict key "qoffset_z" out of 43 keys
2022-12-15 11:10:31,770 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,770 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,771 - >> collate dict key "srow_x" out of 43 keys
2022-12-15 11:10:31,771 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,771 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,771 - >> collate dict key "srow_y" out of 43 keys
2022-12-15 11:10:31,771 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,772 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,772 - >> collate dict key "srow_z" out of 43 keys
2022-12-15 11:10:31,772 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,772 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,772 - >> collate dict key "affine" out of 43 keys
2022-12-15 11:10:31,773 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,773 - >> collate dict key "original_affine" out of 43 keys
2022-12-15 11:10:31,773 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,773 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,774 - >> collate dict key "as_closest_canonical" out of 43 keys
2022-12-15 11:10:31,774 - >> collate dict key "spatial_shape" out of 43 keys
2022-12-15 11:10:31,774 - >>> collate/stack a list of numpy arrays
2022-12-15 11:10:31,774 - >>> collate/stack a list of tensors
2022-12-15 11:10:31,774 - >> collate dict key "space" out of 43 keys
2022-12-15 11:10:31,774 - >> collate dict key "original_channel_dim" out of 43 keys
2022-12-15 11:10:31,774 - >> collate dict key "filename_or_obj" out of 43 keys
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4355 [0] NCCL INFO [Service thread] Connection closed by localRank 0
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4247:4247 [0] NCCL INFO comm 0x55fd19ed0fc0 rank 0 nranks 4 cudaDev 0 busId 964300000 - Abort COMPLETE
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4354 [2] NCCL INFO [Service thread] Connection closed by localRank 2
bb3cc6c16c0a48ad9d7dfb85f2cefd83000008:4249:4249 [2] NCCL INFO comm 0x55d36f866ea0 rank 2 nranks 4 cudaDev 2 busId d33f00000 - Abort COMPLETE
'

[wyli]

if this is within a docker container, perhaps you can set --shm-size or --ipc=host

[peterhessey]

The shm size is already set to the maximum available memory (448GB). Should have specified that in the original post. Would that really be insufficient amounts of memory?

[wyli]

it shouldn't be a real OOM... is it possible to also try --ipc=host?

[peterhessey]

Unfortunately I don't think that's a parameter that is exposed for AzureML runs, I'll do some more investigation today though

[Nic-Ma]

I agree with @wyli , it looks like number of processes for the PyTorch DataLoader is limited in the docker, better to set --ipc=host.
Or maybe you can try num_workers=0 or num_workers=1.
Some related PyTorch docstring: https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L775

Thanks.

[peterhessey]

Thanks guys, the responses are much appreciated. Is it possible to control the number of DataLoader workers with AutoRunner.set_training_params()? Or will I need to manually configure each algorithm for this?