Bundles Update #1493

ericspod · 2023-08-24T23:49:13Z

Fixes #1457, #1452.

Description

This adds tutorials going through the process of creating bundles in their entirety.

Notebooks to add:

Very simple intro to bundles and the file structure
Simple network bundle that actually does something with training
Bundle wrapping pre-existing code and network definitions which would cover non-MONAI code
Bundle wrapping non-Pytorch code, maybe?

Checks

Avoid including large-size files in the PR.
Clean up long text outputs from code cells in the notebook.
For security purposes, please check the contents and remove any sensitive info such as user names and private key.
Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the ./figure folder
Notebook runs automatically ./runner.sh -t <path to .ipynb file>

review-notebook-app · 2023-08-24T23:49:17Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

for more information, see https://pre-commit.ci

DCO Remediation Commit for Eric Kerfoot <eric.kerfoot@kcl.ac.uk> I, Eric Kerfoot <eric.kerfoot@kcl.ac.uk>, hereby add my Signed-off-by to this commit: db2f488 I, Eric Kerfoot <eric.kerfoot@kcl.ac.uk>, hereby add my Signed-off-by to this commit: 04d581f I, Eric Kerfoot <eric.kerfoot@kcl.ac.uk>, hereby add my Signed-off-by to this commit: ba19596 I, Eric Kerfoot <eric.kerfoot@kcl.ac.uk>, hereby add my Signed-off-by to this commit: 54583ba Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

for more information, see https://pre-commit.ci

DCO Remediation Commit for Eric Kerfoot <eric.kerfoot@kcl.ac.uk> I, Eric Kerfoot <eric.kerfoot@kcl.ac.uk>, hereby add my Signed-off-by to this commit: b82a5f7 I, Eric Kerfoot <eric.kerfoot@kcl.ac.uk>, hereby add my Signed-off-by to this commit: a2b12eb Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

wyli

thanks, the tutorials look nice, cc @mingxin-zheng @Nic-Ma is it possible to skip the cell format requirements? I feel the format is defined primarily for pure python notebooks, in this PR, many cells demonstrate CLI features.

bundle/02_mednist_classification.ipynb

ericspod · 2023-09-08T12:16:02Z

The examples use the command tree to show the files in a bundle, is this not present in the CI environment? That might explain the errors with the notebook tests.

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> I had to set MKL_SERVICE_FORCE_INTEL=1 to get the tests to run, maybe this is an issue on the CI system as well.

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

ericspod · 2023-09-12T12:25:02Z

I had to mangle the third notebook to get an output about the command line error that was encountered. The command fails with:

Traceback (most recent call last):
...
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py", line 247, in _lazy_init
    torch._C._cuda_init()
RuntimeError: The NVIDIA driver on your system is too old (found version 10010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org/ to install a PyTorch version that has been compiled with your version of the CUDA driver.

The top line of nvidia-smi for the containing running the tests is Driver Version: 418.126.02 CUDA Version: 11.6 however when I run the same image locally I get Driver Version: 530.30.02 CUDA Version: 12.1.

wyli · 2023-09-13T08:53:38Z

thanks, let me try to ask about upgrading the driver version. also is it possible to remove the related component that is causing the issue?

ericspod · 2023-09-13T10:20:04Z

I think the component is the network being moved onto CUDA so I could just train in software but that's not demonstrating proper practice in a clear way. The network is just densenet121. I would have thought this issue would come up elsewhere but it may be related to spawning a subprocess in notebook 3 to train.

vikashg

This is great @ericspod. Thanks.

vikashg · 2023-09-13T14:58:48Z

As I am working through the examples I got this error

(monai-test) gupta@DIGI1036734:~/disk/Tools/CodeReview/monai_bundle$ python -m monai.bundle run train     --bundle_root "$BUNDLE"     --logging_file "$BUNDLE/configs/logging.conf"     --meta_file "$BUNDLE/configs/metadata.json"     --config_file "['$BUNDLE/configs/common.yaml','$BUNDLE/configs/train.yaml']"     --max_epochs 2 
2023-09-13 10:55:39,642 - INFO - --- input summary of monai.bundle.scripts.run ---
2023-09-13 10:55:39,642 - INFO - > run_id: 'train'
2023-09-13 10:55:39,642 - INFO - > meta_file: './MedNISTClassifier_v2/configs/metadata.json'
2023-09-13 10:55:39,642 - INFO - > config_file: ['./MedNISTClassifier_v2/configs/common.yaml',
 './MedNISTClassifier_v2/configs/train.yaml']
2023-09-13 10:55:39,642 - INFO - > logging_file: './MedNISTClassifier_v2/configs/logging.conf'
2023-09-13 10:55:39,642 - INFO - > bundle_root: './MedNISTClassifier_v2'
2023-09-13 10:55:39,642 - INFO - > max_epochs: 2
2023-09-13 10:55:39,642 - INFO - ---


2023-09-13 10:55:39,643 - INFO - Setting logging properties based on config: ./MedNISTClassifier_v2/configs/logging.conf.
Traceback (most recent call last):
  File "/home/gupta/miniconda3/envs/monai-test/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/gupta/miniconda3/envs/monai-test/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/gupta/disk/Tools/CodeReview/MONAI/monai/bundle/__main__.py", line 30, in <module>
    fire.Fire()
  File "/home/gupta/miniconda3/envs/monai-test/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/gupta/miniconda3/envs/monai-test/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/gupta/miniconda3/envs/monai-test/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/gupta/disk/Tools/CodeReview/MONAI/monai/bundle/scripts.py", line 684, in run
    workflow = ConfigWorkflow(
  File "/home/gupta/disk/Tools/CodeReview/MONAI/monai/bundle/workflows.py", line 221, in __init__
    self.parser.read_config(f=config_file)
  File "/home/gupta/disk/Tools/CodeReview/MONAI/monai/bundle/config_parser.py", line 320, in read_config
    content.update(self.load_config_files(f, **kwargs))
  File "/home/gupta/disk/Tools/CodeReview/MONAI/monai/bundle/config_parser.py", line 424, in load_config_files
    for k, v in (cls.load_config_file(i, **kwargs)).items():
  File "/home/gupta/disk/Tools/CodeReview/MONAI/monai/bundle/config_parser.py", line 405, in load_config_file
    return yaml.load(f, CheckKeyDuplicatesYamlLoader, **kwargs)  # type: ignore[no-any-return]
  File "/home/gupta/disk/Tools/CodeReview/MONAI/monai/utils/module.py", line 436, in __getattr__
    raise self._exception
  File "/home/gupta/disk/Tools/CodeReview/MONAI/monai/utils/module.py", line 395, in optional_import
    pkg = __import__(module)  # top level module
monai.utils.module.OptionalImportError: import yaml (No module named 'yaml').

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

I installed yaml using conda install yaml but the error persists.

ericspod · 2023-09-13T15:07:36Z

The bundles need fire and yaml to run (those using JSON only need fire). I could put that in the first cell of the notebooks like other notebooks do though the requirements are for the programs being run and not the notebook itself.

vikashg · 2023-09-13T15:17:43Z

Ya I have both fire and yaml installed. Let me try again

ericspod · 2023-09-13T15:21:06Z

The package is pyyaml.

vikashg · 2023-09-13T15:52:18Z

Its working now. Thanks

…hurt Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

vikashg

LGTM

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Fixes Project-MONAI#1457, Project-MONAI#1452. ### Description This adds tutorials going through the process of creating bundles in their entirety. Notebooks to add: - [x] Very simple intro to bundles and the file structure - [x] Simple network bundle that actually does something with training - [x] Bundle wrapping pre-existing code and network definitions which would cover non-MONAI code - [ ] Bundle wrapping non-Pytorch code, maybe? ### Checks  - [x] Avoid including large-size files in the PR. - [x] Clean up long text outputs from code cells in the notebook. - [x] For security purposes, please check the contents and remove any sensitive info such as user names and private key. - [x] Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the `./figure` folder - [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>` --------- Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

ericspod added 4 commits July 14, 2023 15:31

Starting on introductory bundle tutorials

db2f488

Update

04d581f

Adding classification notebook tutorial

ba19596

Updated organisation of files

54583ba

pre-commit-ci bot and others added 3 commits August 24, 2023 23:50

[pre-commit.ci] auto fixes from pre-commit.com hooks

2dec291

for more information, see https://pre-commit.ci

Merge branch 'main' into bundles_update

fd7d290

vikashg self-assigned this Aug 25, 2023

ericspod and others added 8 commits August 30, 2023 17:02

Adding third notebook

0000e1e

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Last notebook started

b82a5f7

Updates, need to finalise content of notebook 4 to be useful

a2b12eb

[pre-commit.ci] auto fixes from pre-commit.com hooks

4ff059b

for more information, see https://pre-commit.ci

Merge branch 'main' into bundles_update

b9cbb47

Updates to last notebook for now

decd1b1

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Merge branch 'main' into bundles_update

56e5a83

ericspod marked this pull request as ready for review September 7, 2023 16:48

ericspod requested review from wyli, Nic-Ma, aylward, MMelQin and KumoLiu September 7, 2023 16:48

ericspod added 2 commits September 7, 2023 17:52

README Update

a8ce90a

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Formatting

b00fa49

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

ericspod requested a review from vikashg September 7, 2023 17:16

wyli approved these changes Sep 7, 2023

View reviewed changes

wyli requested a review from mingxin-zheng September 7, 2023 20:42

KumoLiu reviewed Sep 8, 2023

View reviewed changes

bundle/02_mednist_classification.ipynb Show resolved Hide resolved

ericspod and others added 12 commits September 8, 2023 13:16

Merge branch 'main' into bundles_update

b32d6bf

Move Code Into Markdown

ace252a

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Trying notebook fix

6721823

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Fix for notebook

2607f65

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Attempting to Fix Papermill Problem

83ccc0e

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> I had to set MKL_SERVICE_FORCE_INTEL=1 to get the tests to run, maybe this is an issue on the CI system as well.

Test Change

05440ec

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

tree Not Present in CI Environment?

b5331dd

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Without tree?

63f078c

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Final Try

68f42e0

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Further fixes

92e7594

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Merge branch 'main' into bundles_update

ca0f86b

Need to modify notebook to capture output

f94d594

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

vikashg approved these changes Sep 13, 2023

View reviewed changes

Adding expected 'setup' cells, they aren't strictly needed but won't …

6e00fe9

…hurt Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

vikashg approved these changes Sep 19, 2023

View reviewed changes

Let's just ignore these notebooks for now

6a3082b

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

ericspod enabled auto-merge (squash) September 19, 2023 15:45

ericspod merged commit 0b2638b into Project-MONAI:main Sep 19, 2023

ericspod deleted the bundles_update branch September 19, 2023 16:10

ericspod mentioned this pull request Sep 22, 2023

Simple example of adapting non-MONAI Pytorch code to bundles #1452

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundles Update #1493

Bundles Update #1493

ericspod commented Aug 24, 2023 •

edited

Loading

review-notebook-app bot commented Aug 24, 2023

wyli left a comment

ericspod commented Sep 8, 2023

ericspod commented Sep 12, 2023

wyli commented Sep 13, 2023

ericspod commented Sep 13, 2023

vikashg left a comment

vikashg commented Sep 13, 2023

ericspod commented Sep 13, 2023

vikashg commented Sep 13, 2023

ericspod commented Sep 13, 2023

vikashg commented Sep 13, 2023

vikashg left a comment

Bundles Update #1493

Bundles Update #1493

Conversation

ericspod commented Aug 24, 2023 • edited Loading

Description

Checks

review-notebook-app bot commented Aug 24, 2023

wyli left a comment

Choose a reason for hiding this comment

ericspod commented Sep 8, 2023

ericspod commented Sep 12, 2023

wyli commented Sep 13, 2023

ericspod commented Sep 13, 2023

vikashg left a comment

Choose a reason for hiding this comment

vikashg commented Sep 13, 2023

ericspod commented Sep 13, 2023

vikashg commented Sep 13, 2023

ericspod commented Sep 13, 2023

vikashg commented Sep 13, 2023

vikashg left a comment

Choose a reason for hiding this comment

ericspod commented Aug 24, 2023 •

edited

Loading