Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(WIP) Add GX data validation for synapse data warehouse #42

Merged
merged 15 commits into from
Sep 16, 2024
Merged

(WIP) Add GX data validation for synapse data warehouse #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
eca21c4
Add in GX validation logic
thomasyu888 May 18, 2024
80c24fb
Add in GX
thomasyu888 May 18, 2024
fbc0786
Add node validation values
thomasyu888 May 18, 2024
bec5a5b
Add in expectation suites
thomasyu888 May 18, 2024
cb59327
Move files into tf archive
thomasyu888 May 18, 2024
d67e3a3
Add s3 site
thomasyu888 May 18, 2024
b21cd50
Update readme
thomasyu888 May 18, 2024
695814a
Add in more expectation
thomasyu888 May 19, 2024
3c55432
Add in portal query datasource
thomasyu888 May 19, 2024
1061593
Patch custom expectation
thomasyu888 May 19, 2024
418a8ad
Add in custom expectation and support snowSQL
thomasyu888 May 19, 2024
768e653
Add in study validation
thomasyu888 May 20, 2024
6724834
Run linter
thomasyu888 May 20, 2024
5ddc176
Add validation rules
thomasyu888 May 20, 2024
ce46a3d
Add in validation rules
thomasyu888 May 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 databases.tf → _tf_archive/databases.tf
View file
Open in desktop
File renamed without changes.
0 main.tf → _tf_archive/main.tf
View file
Open in desktop
File renamed without changes.
0 roles.tf → _tf_archive/roles.tf
View file
Open in desktop
File renamed without changes.
0 secrets.tf → _tf_archive/secrets.tf
View file
Open in desktop
File renamed without changes.
0 users.tf → _tf_archive/users.tf
View file
Open in desktop
File renamed without changes.
0 warehouses.tf → _tf_archive/warehouses.tf
View file
Open in desktop
File renamed without changes.
13 changes: 13 additions & 0 deletions data_validation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Great Expectations

Follow instructions here: https://docs.greatexpectations.io/docs/oss/tutorials/quickstart/ to get started.

pip install sqlalchemy python-dotenv snowflake-sqlalchemy boto3 botocore

1. Be sure to export and create the GX data source environmental variable. Follow these [steps](https://docs.greatexpectations.io/docs/oss/guides/connecting_to_your_data/fluent/database/connect_sql_source_data?sql-database-type=snowflake) to set it up:

```
export GX_SNOWFLAKE_CONNECTION="snowflake://<USER_NAME>:<PASSWORD>@<ACCOUNT_NAME_OR_LOCATOR>/<DATABASE_NAME>/<SCHEMA_NAME>?warehouse=<WAREHOUSE_NAME>&role=<ROLE_NAME>"
```

2. Run validation `python validation.py`. Be sure to log into jumpcloud (dpe aws account) and export your credentials in the terminal.
Loading