Skip to content

Commit 046160a

Browse files
authored
Merge pull request #42 from Sage-Bionetworks/gx-data-validation
(WIP) Add GX data validation for synapse data warehouse
2 parents 9567cfa + ce46a3d commit 046160a

20 files changed

+2181
-0
lines changed
File renamed without changes.

main.tf _tf_archive/main.tf

File renamed without changes.

roles.tf _tf_archive/roles.tf

File renamed without changes.

secrets.tf _tf_archive/secrets.tf

File renamed without changes.

users.tf _tf_archive/users.tf

File renamed without changes.
File renamed without changes.

data_validation/README.md

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Great Expectations
2+
3+
Follow instructions here: https://docs.greatexpectations.io/docs/oss/tutorials/quickstart/ to get started.
4+
5+
pip install sqlalchemy python-dotenv snowflake-sqlalchemy boto3 botocore
6+
7+
1. Be sure to export and create the GX data source environmental variable. Follow these [steps](https://docs.greatexpectations.io/docs/oss/guides/connecting_to_your_data/fluent/database/connect_sql_source_data?sql-database-type=snowflake) to set it up:
8+
9+
```
10+
export GX_SNOWFLAKE_CONNECTION="snowflake://<USER_NAME>:<PASSWORD>@<ACCOUNT_NAME_OR_LOCATOR>/<DATABASE_NAME>/<SCHEMA_NAME>?warehouse=<WAREHOUSE_NAME>&role=<ROLE_NAME>"
11+
```
12+
13+
2. Run validation `python validation.py`. Be sure to log into jumpcloud (dpe aws account) and export your credentials in the terminal.

0 commit comments

Comments
 (0)