Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create tests for examples with custom stages #885

Merged
merged 90 commits into from
Apr 28, 2023

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Apr 13, 2023

Description

This PR creates at least one test for each example containing custom stages.
This PR currently only covers those examples which do not require additional packages.
Part of #849.

  • Moves the bert vocabulary files to morpheus/data dir, no longer requiring them to be fetched from LFS and making them available to unittests.
  • Fixes type hints and remove a redundant method in examples/log_parsing/inference.py
  • Remove redundant copies of bert-base-cased-hash.txt and bert-base-uncased-hash.txt files, replacing them with symlinks to the files in the morpheus/data` dir fixes [FEA]: Remove redundant bert hash files #850
  • Explicitly set encoding='UTF-8' in examples/log_parsing/postprocessing.py as a work-around for issue [BUG]: cupy, and by extension cudf changes the system's preferred encoding to ANSI_X3.4-1968 #859
  • Add py::kw_only to Python bindings for TensorMemory and sublasses to ensure parity with Python impls.
  • Set repr=False for the tensors field of TensorMemory avoids bug when printing due to the fact that we assign the value to self._tensors
  • Seed cupy's random number generator in manual_seed method.
  • Fix usage of reload_modules fixture, requesting a reload of multiple modules should be done with @pytest.mark.reload_modules([mod1, mod2]) not calling reload_modules twice.
  • New test data in tests/tests_data/log_parsing is based upon the first 5 rows of data from models/datasets/validation-data/log-parsing-validation-data-input.csv

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

…ed out here since the original is large and not part of the dataset pulled by the test stage in CI
…ased-hash.txt to the version in the morpheus/data dir
…with just the id2label actually used by LogParsingPostProcessingStage
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few suggestions:

  • Make the tests/examples directory better mirror the examples directory
    • Since there are no python files in the root of examples, there shouldnt be any tests in the root of tests/examples
    • Move any of the tests/examples/test_XXX.py file in the root of tests/examples into a folder to match examples/XXX
    • i.e. tests/examples/test_pass_thru.py -> tests/examples/developer_guide/test_pass_thru.py
  • Make the tests/tests_data/examples directory better mirror the examples directory
    • Same as the previous comment. Things like tests/tests_data/examples/abp_pcap.jsonlines should be moved to tests/tests_data/examples/abp_pcap_detection/abp_pcap.jsonlines

tests/examples/log_parsing/conftest.py Show resolved Hide resolved
tests/examples/test_pass_thru.py Outdated Show resolved Hide resolved
tests/examples/test_recipient_features_stage.py Outdated Show resolved Hide resolved
tests/examples/test_abp_pcap_preprocessing.py Show resolved Hide resolved
tests/utils/dataset_manager.py Outdated Show resolved Hide resolved
tests/utils/dataset_manager.py Show resolved Hide resolved
@mdemoret-nv
Copy link
Contributor

/merge

@rapids-bot rapids-bot bot merged commit 18fcbce into nv-morpheus:branch-23.07 Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[FEA]: Remove redundant bert hash files
2 participants