Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DFP Integrated Training Updates (Stress Testing/Benchmarks) #924

Merged
merged 42 commits into from
May 3, 2023
Merged

DFP Integrated Training Updates (Stress Testing/Benchmarks) #924

merged 42 commits into from
May 3, 2023

Conversation

bsuryadevara
Copy link
Contributor

@bsuryadevara bsuryadevara commented May 1, 2023 •
edited
Loading

Description

  • Updated DFP integrated training benchmarks
  • Updated DFP integrated training modules to avoid out of memory error for training and inference tasks
  • Updated docs and added structural diagrams for better understanding of the workflow
  • Added monitor module implementation
  • Separated batch (file) and streaming (kafka) pipeline execution files
  • Re-organised notebooks to notebooks folder
  • Added new control message definitions
  • Added notebooks for DFP integrated training (Duo and Azure) with workflow diagrams and documentation.
  • Ensured all (DFP stages and modules) the notebooks running successfully
  • Addressed most of the flake8 warnings

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@bsuryadevara
updated benchmarks to support integrated training feature
a1795bb
@bsuryadevara
add benchmarks data path to gitignore
bfe8173
@bsuryadevara
updated README.md with dfp benchmarks
b82ba2d
@bsuryadevara
Update README.md
5f1b580
@bsuryadevara
Update README.md
da3d414
@bsuryadevara
added dfp integrated training notebooks
89f58a5
@bsuryadevara
updated duo integrated trianing notebook
032b634
@bsuryadevara
updated duo integrated trianing notebook
2e01437
@bsuryadevara
added dfp integrated training notebooks
e995b80
@bsuryadevara
added dfp integrated training notebooks
52cd335
@bsuryadevara
added dfp integrated training notebooks
804c044
@bsuryadevara
added dfp integrated training notebooks
a935454
@bsuryadevara
added dfp integrated training notebooks
a70a4e0
@bsuryadevara
Merge remote-tracking branch 'upstream/branch-23.07' into 890-fea-end…
5c8f674 
…-to-end-example-of-dfp-with-integrated-traning
@bsuryadevara
updates to dfp integrated pipeline
ed3f224
@bsuryadevara
updates to dfp integrated pipeline
97f83fe
@bsuryadevara
updates to dfp integrated pipeline
3d71aef
@bsuryadevara
updates to dfp integrated pipeline
acb1277
@bsuryadevara
trivial changes to monitor description
e0f3c96
@bsuryadevara
restructured dfp examples images folder
4bb5caf
@bsuryadevara
Merge branch 'nv-morpheus:branch-23.07' into 890-fea-end-to-end-examp…
84a1c11 
…le-of-dfp-with-integrated-traning
@bsuryadevara
flake8 format correction
9c997cc
@bsuryadevara
updates to comments
751152d
@bsuryadevara bsuryadevara added enhancement Additional functionality added to an existing feature non-breaking Non-breaking change improvement Improvement to existing functionality 2 - In Progress labels May 1, 2023
@bsuryadevara bsuryadevara requested review from a team as code owners May 1, 2023 15:08
@bsuryadevara bsuryadevara self-assigned this May 1, 2023
@bsuryadevara bsuryadevara linked an issue May 1, 2023 that may be closed by this pull request
[FEA]: DFP Stress/Benchmark testing Updates #890
Closed
2 tasks
mdemoret-nv
mdemoret-nv approved these changes May 1, 2023
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving once comments are fixed and @dagardner-nv has reviewed the docs changes we can merge.

docs/source/developer_guide/guides/10_modular_pipeline_digital_fingerprinting.md Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/benchmarks/test_bench_e2e_dfp_pipeline.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_inference_pipe.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_inference_pipe.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_monitor.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/stages/dfp_postprocessing_stage.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/utils/dfp_arg_parser.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/utils/dfp_arg_parser.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/utils/monitor_utils.py Outdated Show resolved Hide resolved
morpheus/utils/module_utils.py Show resolved Hide resolved
dagardner-nv
dagardner-nv requested changes May 1, 2023
View reviewed changes
docs/source/developer_guide/guides/10_modular_pipeline_digital_fingerprinting.md Show resolved Hide resolved
docs/source/developer_guide/guides/10_modular_pipeline_digital_fingerprinting.md Show resolved Hide resolved
docs/source/developer_guide/guides/10_modular_pipeline_digital_fingerprinting.md Outdated Show resolved Hide resolved
docs/source/modules/examples/digital_fingerprinting/dfp_deployment.md Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/benchmarks/README.md Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_inference_pipe.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_inference_pipe.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_inference_pipe.py Outdated Show resolved Hide resolved
examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_training_pipe.py Outdated Show resolved Hide resolved
.../digital_fingerprinting/production/morpheus/images/dfp_integrated_training_file_pipeline.jpg Outdated Show resolved Hide resolved
@bsuryadevara bsuryadevara marked this pull request as draft May 1, 2023 20:11
bsuryadevara and others added 10 commits May 1, 2023 15:23
@bsuryadevara @dagardner-nv
Update examples/digital_fingerprinting/production/morpheus/dfp/module…
d1ab698 
…s/dfp_inference_pipe.py

Co-authored-by: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
@bsuryadevara @dagardner-nv
Update examples/digital_fingerprinting/production/morpheus/dfp/module…
55590d5 
…s/dfp_training_pipe.py

Co-authored-by: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
@bsuryadevara @dagardner-nv
Update examples/digital_fingerprinting/production/morpheus/dfp/module…
4fd1d1b 
…s/dfp_inference_pipe.py

Co-authored-by: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
@bsuryadevara @dagardner-nv
Update docs/source/developer_guide/guides/10_modular_pipeline_digital…
99e2c52 
…_fingerprinting.md

Co-authored-by: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
@bsuryadevara @dagardner-nv
Update docs/source/modules/examples/digital_fingerprinting/dfp_deploy…
e9e93ae 
…ment.md

Co-authored-by: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
@bsuryadevara @dagardner-nv
Update examples/digital_fingerprinting/production/morpheus/benchmarks…
ec350c4 
…/README.md

Co-authored-by: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
@bsuryadevara
Merge remote-tracking branch 'upstream/branch-23.07' into dfp-stress-…
fdbc9c8 
…testing-updates
@bsuryadevara
updated monitor stage and docs
3a73f4c
@bsuryadevara
Merge branch 'dfp-stress-testing-updates' of github.com:bsuryadevara/…
8e2f4ca 
…Morpheus into dfp-stress-testing-updates
@bsuryadevara
Merge remote-tracking branch 'upstream/branch-23.07' into dfp-stress-…
7420377 
…testing-updates
@bsuryadevara bsuryadevara marked this pull request as ready for review May 3, 2023 06:28
@bsuryadevara @dagardner-nv
Update examples/digital_fingerprinting/production/morpheus/benchmarks…
3d45aff 
…/README.md

Co-authored-by: David Gardner <96306125+dagardner-nv@users.noreply.github.com>
@bsuryadevara
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit e103a58 into nv-morpheus:branch-23.07 May 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pdmack pdmack pdmack left review comments

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

@dagardner-nv dagardner-nv dagardner-nv approved these changes

@drobison00 drobison00 Awaiting requested review from drobison00

Assignees

@bsuryadevara bsuryadevara

Labels
enhancement Additional functionality added to an existing feature improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA]: DFP Stress/Benchmark testing Updates
4 participants
@bsuryadevara @pdmack @mdemoret-nv @dagardner-nv