Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data validation Phase 1 #1239

Merged
merged 36 commits into from
Dec 20, 2022
Merged

Data validation Phase 1 #1239

merged 36 commits into from
Dec 20, 2022

Conversation

utkarsharma2
Copy link
Collaborator

@utkarsharma2 utkarsharma2 commented Nov 13, 2022 •
edited
Loading

Description

What is the current behavior?

Implements Phase 1 of the data validation story:
Introduces operator ColumnCheckOperator to validate Column on Dataframe and Astro Table

@utkarsharma2 utkarsharma2 marked this pull request as draft November 13, 2022 21:20
@codecov
Copy link

codecov bot commented Dec 7, 2022 •
edited
Loading

Codecov Report

Base: 92.90% // Head: 93.54% // Increases project coverage by +0.63% 🎉

Coverage data is based on head (ec40143) compared to base (fa654a6).
Patch coverage: 88.65% of modified lines in pull request are covered.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1239      +/-   ##
==========================================
+ Coverage   92.90%   93.54%   +0.63%     
==========================================
  Files          61       82      +21     
  Lines        3215     3981     +766     
  Branches      382      400      +18     
==========================================
+ Hits         2987     3724     +737     
- Misses        151      176      +25     
- Partials       77       81       +4
Impacted Files Coverage Δ
.../operators/data_validations/ColumnCheckOperator.py 86.30% <86.30%> (ø)
...sql/operators/data_validations/SQLCheckOperator.py 94.73% <94.73%> (ø)
python-sdk/src/astro/databases/__init__.py 100.00% <100.00%> (ø)
python-sdk/src/astro/databases/base.py 95.98% <100.00%> (ø)
python-sdk/src/astro/databases/google/bigquery.py 90.04% <100.00%> (ø)
python-sdk/src/astro/sql/__init__.py 95.83% <100.00%> (+0.37%) ⬆️
sql-cli/sql_cli/templates/render_dag.py.jinja2 100.00% <0.00%> (ø)
sql-cli/sql_cli/configuration.py 100.00% <0.00%> (ø)
sql-cli/sql_cli/__main__.py 100.00% <0.00%> (ø)
sql-cli/sql_cli/exceptions.py 100.00% <0.00%> (ø)
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@utkarsharma2 utkarsharma2 marked this pull request as ready for review December 8, 2022 06:54
@utkarsharma2
Override GoogleBaseHook with BigqueryHook (#1442)
1220883 
# Description
## What is the current behavior?
Because of below issue:
```
airflow.exceptions.AirflowException: You are trying to use `common-sql` with GoogleBaseHook, but its provider does not support it. Please upgrade the provider to a version that supports `common-sql`. The hook class should be a subclass of `airflow.providers.common.sql.hooks.sql.DbApiHook`. Got GoogleBaseHook Hook with class hierarchy: [<class 'airflow.providers.google.common.hooks.base_google.GoogleBaseHook'>, <class 'airflow.hooks.base.BaseHook'>, <class 'airflow.utils.log.logging_mixin.LoggingMixin'>, <class 'object'>]
```
We are using a work around and using Bigquey Hook
@utkarsharma2 utkarsharma2 mentioned this pull request Dec 20, 2022
Closed
@utkarsharma2 utkarsharma2 requested a review from kaxil as a code owner December 20, 2022 10:42
utkarsharma2 and others added 12 commits December 20, 2022 16:31
@utkarsharma2
Update conn_id
3905153
@utkarsharma2
Change return type
a0e6773
@utkarsharma2
Updated hook
686a09d
@utkarsharma2 @feluelle
Update python-sdk/src/astro/sql/operators/data_validations/ColumnChec…
2b91128 
…kOperator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
@utkarsharma2 @feluelle
Update python-sdk/src/astro/sql/operators/data_validations/ColumnChec…
ec3b4fd 
…kOperator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
@utkarsharma2 @feluelle
Update python-sdk/src/astro/sql/operators/data_validations/ColumnChec…
cd828d7 
…kOperator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
@utkarsharma2 @feluelle
Update python-sdk/src/astro/sql/operators/data_validations/ColumnChec…
d1d5c28 
…kOperator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
@utkarsharma2
Revert changes to check since nunique() don't count None
a0543ad
@utkarsharma2
Fix Deep Source
764e08a
@utkarsharma2
Merge branch 'main' into DataValidation
c25a736
@utkarsharma2
Refactored code to remove duplication
9673af7
@utkarsharma2
Refactored code
5864a07
@utkarsharma2 utkarsharma2 merged commit 00ff092 into main Dec 20, 2022
@utkarsharma2 utkarsharma2 deleted the DataValidation branch December 20, 2022 14:34
@utkarsharma2 utkarsharma2 mentioned this pull request Jan 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@feluelle feluelle feluelle approved these changes

@dimberman dimberman Awaiting requested review from dimberman dimberman is a code owner

@tatiana tatiana Awaiting requested review from tatiana tatiana is a code owner

@sunank200 sunank200 Awaiting requested review from sunank200 sunank200 is a code owner

@pankajastro pankajastro Awaiting requested review from pankajastro pankajastro is a code owner

@pankajkoti pankajkoti Awaiting requested review from pankajkoti

@kaxil kaxil Awaiting requested review from kaxil

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@utkarsharma2 @feluelle