[FDS-2525] Authenticated export of telemetry data #1527

BryanFauble · 2024-10-29T19:37:53Z

Problem:

When creating telemetry data where an open-telemetry collector is not available to perform an oauth2 token exchange the only way to attach an auth token on outbound requests was to define a static header an environment variables.

Solution:

Adding the requests-oauth2client library that will handle the oauth2 token exchange with Auth0 for us if the environment variables are set.
Attach the auth mechanism on the requests.Session() object that the opentelemetry exporter uses to send data out.
This is the first portion of https://sagebionetworks.jira.com/browse/FDS-2525 - Which allows developers to export telemetry data locally. It also verifies that all we need to do for FDS-2525 is set a few environment variables and allow schematic running within AWS to export it's data to the OTEL collector, which will then forward the data to the DPE K8s cluster.

Testing:

I ran this code locally with the environment variables set and confirmed that both traces AND log were exported and viewable within SigNoz on the sandbox kubernetes cluster.
I also verified I could point this at the https://dev.sagedpe.org URL too

TODO:

Remove hacky code around verify=False for the HTTP POST logic when non-self signed certs can be used.

…s-2386-synapse-refactoring

…enticated export of telemetry data

… local

schematic/__init__.py

thomasyu888

🔥 LGTM! Thanks for doing this. Make sense to me, I'll leave it to at least one other developer to review this.

GiaJordan · 2024-11-07T18:56:54Z

@BryanFauble The changes look good to me!
I just see that a couple tests are failing and there are a couple merge conflicts.

… version" This reverts commit 2e3b3cf.

This reverts commit a096639.

…docker

sonarqubecloud · 2024-11-13T01:53:08Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
45.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

* Authenticated export of telemetry data, updating to HTTP otel library

* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Update pyproject.toml --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> Co-authored-by: Jenny V Medina <jenny.medina@sagebase.org> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> Co-authored-by: Lingling <55448354+linglp@users.noreply.github.com> Co-authored-by: GiaJordan <gianna.jordan@sagebase.org>

* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Include schematic_api for now (#1547) * update toml version to 24.11.1 --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> Co-authored-by: Jenny V Medina <jenny.medina@sagebase.org> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> Co-authored-by: Lingling <55448354+linglp@users.noreply.github.com> Co-authored-by: GiaJordan <gianna.jordan@sagebase.org>

* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Include schematic_api for now (#1547) * update toml version to 24.11.1 (#1548) * [SCHEMATIC-193] Support exporting telemetry data from GH integration test runs (#1550) * Support exporting telemetry data from GH run via access token retrieved via oauth2 * [SCHEMATIC-30, SCHEMATIC-200] Add version to click cli / use pathlib.Path module for checking cache size (#1542) * Add version to click cli * Add version * Run black * Reformat * Fix * Update schematic/schemas/data_model_parser.py * Add test for check_synapse_cache_size * Reformat * Fix tests * Remove unused parameter * Install all-extras for now * Make otel flash non-optional * Update dockerfile * Add dependencies as non-optional * Update pyproject toml * Fix trivy issue * Add service version * Run black * Move all utils.general tests into separate folder * Use pre-commit * Add updates to contribution doc * Fix * Add service version to log provider --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * [SCHEMATIC-212] Prevent traces from being combined (#1552) * Set instance id in github CI run, uninstrument flask auto during integration test run * [SCHEMATIC-163] Catch error when manifest is generated and existing one doesn't have `entityId` (#1551) * adds error handling * adds unit tests for _get_file_entityIds * updates error message * adds entityid check to parent func * updates docstring * [SCHEMATIC-183] Use paths from file view for manifest generation (#1529) source manifest file paths from synapse fileviews at generation * [SCHEMATIC-214] Wrap pandas functions to support not including `None` with the NA values argument (#1553) * Wrap pandas functions to support not including `None` with the NA values argument * Ignore types * pylint issues * ordering of ignore * Add to integration test to cover none in a manifest * Add additional test for manifest * [SCHEMATIC-210] Add attribute to nones data model (#1555) Update example_test_nones.model.csv component and add new invalid manifest with nones * first commit * ran black * add test for validateModelManifest * [SCHEMATIC-214] change data model and component (#1556) * add valid values to Patient attributes * update data model * add test manifests * update test for new model * update test for new valid value * change test to use new manifests * remove uneeded test file * revert file * revert file * change tests to use new manifests * remove uneeded manifests * ran black * add tests back in * ran black * revert manifest * Split up valid and errored test as separate testing functions * Remove unused import --------- Co-authored-by: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrewelamb@gmail.com> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> * incremented packge version number * Update publish.yml * Update test.yml * Update api_test.yml * Update pdoc.yml * Update version.py * updates publish.yml (#1558) (#1561) Co-authored-by: Brad Macdonald <52762200+BWMac@users.noreply.github.com> --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> Co-authored-by: Jenny V Medina <jenny.medina@sagebase.org> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> Co-authored-by: Lingling <55448354+linglp@users.noreply.github.com> Co-authored-by: GiaJordan <gianna.jordan@sagebase.org> Co-authored-by: Brad Macdonald <52762200+BWMac@users.noreply.github.com> Co-authored-by: Gianna Jordan <61707471+GiaJordan@users.noreply.github.com>

BryanFauble added 30 commits September 17, 2024 13:22

Adding changes for synapse entity tracking

b369ef4

Create unique test suites for each run

30def7e

Remove extra try-catch

093e179

Remove todo

05e936c

Remove un-needed code

57b8408

mypy flagged issues

2ef297f

mypy flagged issues

476b627

pylint patching

9b08554

pylint issue resolution

e4eff7a

Remove py3.10 only feature

6226efa

Merge branch 'develop' into fds-2386-synapse-refactoring

6b77396

Patching for issues

906a298

Merge remote-tracking branch 'origin/develop-fix-gh-workflow' into fd…

a7fab56

…s-2386-synapse-refactoring

Bug patching

fe9b27e

Merge branch 'develop' into fds-2386-synapse-refactoring

b445853

Adding tests for io_utils changes

f10640e

Log tests to CLI (Temp)

38eb3ec

Create dir if not exsists

51bc8fc

Merge branch 'develop' into fds-2386-synapse-refactoring

058eed4

Don't log the cli

14fdc96

Update to released version of synapse python client

cd60d3c

Add docstrings

662461d

Code review feedback

ff02757

Don't build data docs if not logging for debug

f84fad0

assertion that entity_type is not None

8cc688b

Attach package version

a92a27f

Include the file size of the manifest used in the operation

6489df4

Move version of package to Resource definition

07afa48

Move to grabbing owner_id and exporting everything over GRPC lib

8f10eaf

Add examples

3a01976

BryanFauble added 3 commits October 24, 2024 15:39

increment synapseclient version

248d8aa

Add requests-oauth2client to supported oauth2 token exchange for auth…

6a9fc2f

…enticated export of telemetry data

Allow setting static exporter headers and add how to export data from…

4de4b24

… local

Base automatically changed from attach-additional-telemetry-data to develop November 1, 2024 19:01

BryanFauble added 2 commits November 4, 2024 15:01

Remove hacky code due to self-signed SSL certs

0ab29dc

Add more comments to readme

8fe6d51

BryanFauble commented Nov 4, 2024

View reviewed changes

schematic/__init__.py Show resolved Hide resolved

BryanFauble commented Nov 4, 2024

View reviewed changes

schematic/__init__.py Show resolved Hide resolved

BryanFauble commented Nov 4, 2024

View reviewed changes

schematic/__init__.py Show resolved Hide resolved

Merge branch 'develop' into fds-2525-authenticated-exports

7c06d42

BryanFauble marked this pull request as ready for review November 4, 2024 23:21

BryanFauble requested a review from a team as a code owner November 4, 2024 23:21

thomasyu888 approved these changes Nov 5, 2024

View reviewed changes

BryanFauble added 7 commits November 12, 2024 11:50

Correct version grabbing

a096639

Merge branch 'develop' into fds-2525-authenticated-exports

adff2f1

Add pip install to dockerfile to support pulling self package version

2e3b3cf

Revert "Add pip install to dockerfile to support pulling self package…

0b6a50d

… version" This reverts commit 2e3b3cf.

Revert "Correct version grabbing"

30af57e

This reverts commit a096639.

Pull back setting service version as I could not get this working in …

b718f85

…docker

Merge branch 'develop' into fds-2525-authenticated-exports

5f8dc9f

GiaJordan approved these changes Nov 13, 2024

View reviewed changes

BryanFauble merged commit 778bf54 into develop Nov 13, 2024
7 checks passed

BryanFauble deleted the fds-2525-authenticated-exports branch November 13, 2024 16:35

GiaJordan pushed a commit that referenced this pull request Nov 13, 2024

[FDS-2525] Authenticated export of telemetry data (#1527)

65fb55d

* Authenticated export of telemetry data, updating to HTTP otel library

andrewelamb mentioned this pull request Dec 16, 2024

[SCHEMATIC-192] V24.12.1 into Main (#1562) #1563

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FDS-2525] Authenticated export of telemetry data #1527

[FDS-2525] Authenticated export of telemetry data #1527

BryanFauble commented Oct 29, 2024 •

edited

Loading

thomasyu888 left a comment

GiaJordan commented Nov 7, 2024

sonarqubecloud bot commented Nov 13, 2024

[FDS-2525] Authenticated export of telemetry data #1527

[FDS-2525] Authenticated export of telemetry data #1527

Conversation

BryanFauble commented Oct 29, 2024 • edited Loading

thomasyu888 left a comment

Choose a reason for hiding this comment

GiaJordan commented Nov 7, 2024

sonarqubecloud bot commented Nov 13, 2024

Quality Gate passed

BryanFauble commented Oct 29, 2024 •

edited

Loading