[FDS-2491] Integration tests for Schematic API Test plan #1512

BryanFauble · 2024-10-03T16:51:13Z

Problem:

Running through the test plan is a largely manual process: https://sagebionetworks.jira.com/wiki/spaces/SCHEM/pages/3055779846/Schematic+API+test+plan
We do not have integration tests covering this functionality, and those tests we do have do not verify everything that the manual test would test for.
Some items cannot be verified in code and still need manual verification. Those items will be marked as such.

Solution:

Creating integration tests starting with manifest generation.
Updating the verification of the tests to assert as much information about the generated excel/google sheets as we can.
Writing out the excel file to a directory where files can be manually checked for the portions that cannot be verified in code.

Testing:

I verified that while running the Schematic API server locally I could run these tests.

TODO:

Mark any tests with pytest mark that require manual verification
Create some sort of feature flag that is only set when running the tests manually (IE: If tests are running in the CI pipeline, skip writing files out for manual verification)
Structure the tests to allow using a flask server passed in as a pytest fixture, OR, the schematic API server running locally, in dev, or in staging.
Document (Likely in confluence) the tests that need manual verification along with what needs verification - In addition to the docstring of the tests themselves.

…ong with verification

tests/integration/test_manifest_generation.py

tests/integration/test_manifest_submission.py

…ping on manual verification mode

tests/integration/test_manifest_generation.py

* adds initial verbose testing implementation * cleans up new manifest generation tests * updates new tests for different servers * adds conditional manual testing * fixes URL

…c-api-test-plan

…n' of https://github.com/Sage-Bionetworks/schematic into fds-2491-programatic-checking-of-schematic-api-test-plan

… coverage

…n' of https://github.com/Sage-Bionetworks/schematic into fds-2491-programatic-checking-of-schematic-api-test-plan

…a deployed API instance

…c-api-test-plan

tests/integration/test_manifest_submission.py

…c-api-test-plan

schematic/utils/general.py

…c-api-test-plan

sonarqubecloud · 2024-11-05T01:24:30Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

thomasyu888

🔥 LGTM! Incredible work here

tests/integration/test_manifest_generation.py

Integration tests for Schematic API Test plan

* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Update pyproject.toml --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> Co-authored-by: Jenny V Medina <jenny.medina@sagebase.org> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> Co-authored-by: Lingling <55448354+linglp@users.noreply.github.com> Co-authored-by: GiaJordan <gianna.jordan@sagebase.org>

* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Include schematic_api for now (#1547) * update toml version to 24.11.1 --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> Co-authored-by: Jenny V Medina <jenny.medina@sagebase.org> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> Co-authored-by: Lingling <55448354+linglp@users.noreply.github.com> Co-authored-by: GiaJordan <gianna.jordan@sagebase.org>

* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Include schematic_api for now (#1547) * update toml version to 24.11.1 (#1548) * [SCHEMATIC-193] Support exporting telemetry data from GH integration test runs (#1550) * Support exporting telemetry data from GH run via access token retrieved via oauth2 * [SCHEMATIC-30, SCHEMATIC-200] Add version to click cli / use pathlib.Path module for checking cache size (#1542) * Add version to click cli * Add version * Run black * Reformat * Fix * Update schematic/schemas/data_model_parser.py * Add test for check_synapse_cache_size * Reformat * Fix tests * Remove unused parameter * Install all-extras for now * Make otel flash non-optional * Update dockerfile * Add dependencies as non-optional * Update pyproject toml * Fix trivy issue * Add service version * Run black * Move all utils.general tests into separate folder * Use pre-commit * Add updates to contribution doc * Fix * Add service version to log provider --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * [SCHEMATIC-212] Prevent traces from being combined (#1552) * Set instance id in github CI run, uninstrument flask auto during integration test run * [SCHEMATIC-163] Catch error when manifest is generated and existing one doesn't have `entityId` (#1551) * adds error handling * adds unit tests for _get_file_entityIds * updates error message * adds entityid check to parent func * updates docstring * [SCHEMATIC-183] Use paths from file view for manifest generation (#1529) source manifest file paths from synapse fileviews at generation * [SCHEMATIC-214] Wrap pandas functions to support not including `None` with the NA values argument (#1553) * Wrap pandas functions to support not including `None` with the NA values argument * Ignore types * pylint issues * ordering of ignore * Add to integration test to cover none in a manifest * Add additional test for manifest * [SCHEMATIC-210] Add attribute to nones data model (#1555) Update example_test_nones.model.csv component and add new invalid manifest with nones * first commit * ran black * add test for validateModelManifest * [SCHEMATIC-214] change data model and component (#1556) * add valid values to Patient attributes * update data model * add test manifests * update test for new model * update test for new valid value * change test to use new manifests * remove uneeded test file * revert file * revert file * change tests to use new manifests * remove uneeded manifests * ran black * add tests back in * ran black * revert manifest * Split up valid and errored test as separate testing functions * Remove unused import --------- Co-authored-by: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrewelamb@gmail.com> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> * incremented packge version number * Update publish.yml * Update test.yml * Update api_test.yml * Update pdoc.yml * Update version.py * updates publish.yml (#1558) (#1561) Co-authored-by: Brad Macdonald <52762200+BWMac@users.noreply.github.com> --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> Co-authored-by: Jenny V Medina <jenny.medina@sagebase.org> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> Co-authored-by: Lingling <55448354+linglp@users.noreply.github.com> Co-authored-by: GiaJordan <gianna.jordan@sagebase.org> Co-authored-by: Brad Macdonald <52762200+BWMac@users.noreply.github.com> Co-authored-by: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com>

BryanFauble added 2 commits October 3, 2024 09:46

Create test for single excel and google sheets manifest generation al…

8e70d94

…ong with verification

Correct manual step comment

ea4847d

BWMac reviewed Oct 3, 2024

View reviewed changes

tests/integration/test_manifest_generation.py Show resolved Hide resolved

add integration test for submitting record base manifest

e70c298

BryanFauble commented Oct 3, 2024

View reviewed changes

tests/integration/test_manifest_submission.py Outdated Show resolved Hide resolved

BryanFauble commented Oct 3, 2024

View reviewed changes

tests/integration/test_manifest_submission.py Outdated Show resolved Hide resolved

BryanFauble commented Oct 3, 2024

View reviewed changes

tests/integration/test_manifest_submission.py Outdated Show resolved Hide resolved

linglp and others added 2 commits October 3, 2024 18:08

add another submission test

af0b680

Add option to run tests locally, or for deployed instance, allow flip…

3d4dfb8

…ping on manual verification mode

BryanFauble commented Oct 4, 2024

View reviewed changes

tests/integration/test_manifest_generation.py Outdated Show resolved Hide resolved

BWMac and others added 10 commits October 4, 2024 12:32

Adds Manifest Generation Tests to Common Testing Branch (#1515)

423c78e

* adds initial verbose testing implementation * cleans up new manifest generation tests * updates new tests for different servers * adds conditional manual testing * fixes URL

Merge branch 'develop' into fds-2491-programatic-checking-of-schemati…

85a9811

…c-api-test-plan

add manifest file

5d4d308

Merge branch 'fds-2491-programatic-checking-of-schematic-api-test-pla…

66996c9

…n' of https://github.com/Sage-Bionetworks/schematic into fds-2491-programatic-checking-of-schematic-api-test-plan

move download_location fixture to conftest

2c1598c

Update for unique download location

9fc7fd1

Move configuration flags to conftest, Add more to docstrings for test…

be16ffa

… coverage

Add more steps to docstring

2a717f0

add docstring

6a8aae8

Merge branch 'fds-2491-programatic-checking-of-schematic-api-test-pla…

23c9c02

…n' of https://github.com/Sage-Bionetworks/schematic into fds-2491-programatic-checking-of-schematic-api-test-plan

jaymedina mentioned this pull request Oct 15, 2024

[FDS-2500] Add Integration Tests for: Manifest Validation #1516

Merged

2 tasks

BryanFauble and others added 4 commits October 15, 2024 14:15

Add code that allows us to reuse integration tests for local API, or …

21d95f5

…a deployed API instance

Merge branch 'develop' into fds-2491-programatic-checking-of-schemati…

d75fb13

…c-api-test-plan

add more submission tests after resolve merge conflict

c86e48d

add new mock file

8945837

linglp reviewed Oct 15, 2024

View reviewed changes

tests/integration/test_manifest_submission.py Show resolved Hide resolved

linglp added 2 commits October 16, 2024 11:16

add file based manifest table test

fc8d936

revert previous changes

57b6496

BryanFauble mentioned this pull request Oct 16, 2024

deploy current develop to aws to allow faster testing Sage-Bionetworks/schematic-infra#153

Merged

remove id in test manifest

fecb778

BryanFauble added 3 commits October 17, 2024 09:07

Bump to 10 retries on tests

9486ec6

Merge branch 'develop' into fds-2491-programatic-checking-of-schemati…

faf8ae2

…c-api-test-plan

Patch for missing fixture

aa38a21

thomasyu888 reviewed Oct 17, 2024

View reviewed changes

schematic/utils/general.py Outdated Show resolved Hide resolved

BryanFauble added 2 commits October 24, 2024 15:43

Merge branch 'develop' into fds-2491-programatic-checking-of-schemati…

3d64c2e

…c-api-test-plan

Force a re-query of the fileview to occur when no results are found

b8121e4

BryanFauble requested a review from jaymedina as a code owner October 31, 2024 20:44

BryanFauble and others added 5 commits November 1, 2024 10:22

Updating integration test to create unique dataset for test runs

a977c34

remove non-used constants

f3ec726

Correct folder prefix and print to cli

33986c0

adds file and entities tests

1a1973c

Cleanup of changes

1a95e73

BryanFauble requested a review from a team as a code owner November 4, 2024 17:55

BryanFauble added 4 commits November 4, 2024 10:15

Force new build

7880584

Merge branch 'develop' into fds-2491-programatic-checking-of-schemati…

6004a36

…c-api-test-plan

Run flakey tests with single process instead of with xdist

ee3c067

Remove xdist

6990234

thomasyu888 approved these changes Nov 5, 2024

View reviewed changes

andrewelamb reviewed Nov 5, 2024

View reviewed changes

tests/integration/test_manifest_generation.py Show resolved Hide resolved

andrewelamb self-requested a review November 5, 2024 16:53

andrewelamb approved these changes Nov 5, 2024

View reviewed changes

BWMac approved these changes Nov 5, 2024

View reviewed changes

BryanFauble merged commit 4bbbb68 into develop Nov 5, 2024
7 checks passed

BryanFauble deleted the fds-2491-programatic-checking-of-schematic-api-test-plan branch November 5, 2024 21:32

GiaJordan pushed a commit that referenced this pull request Nov 8, 2024

[FDS-2491] Integration tests for Schematic API Test plan (#1512)

256403c

Integration tests for Schematic API Test plan

andrewelamb mentioned this pull request Dec 16, 2024

[SCHEMATIC-192] V24.12.1 into Main (#1562) #1563

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FDS-2491] Integration tests for Schematic API Test plan #1512

[FDS-2491] Integration tests for Schematic API Test plan #1512

BryanFauble commented Oct 3, 2024 •

edited

Loading

sonarqubecloud bot commented Nov 5, 2024

thomasyu888 left a comment

[FDS-2491] Integration tests for Schematic API Test plan #1512

[FDS-2491] Integration tests for Schematic API Test plan #1512

Conversation

BryanFauble commented Oct 3, 2024 • edited Loading

sonarqubecloud bot commented Nov 5, 2024

Quality Gate passed

thomasyu888 left a comment

Choose a reason for hiding this comment

BryanFauble commented Oct 3, 2024 •

edited

Loading