Experimental library as of February 2021! The Great Expectations core team maintains this provider in an experimental state and does not guarantee ongoing support yet.
An Airflow operator for Great Expectations, a Python library for testing and validating data.
Pre-requisites: An environment running great_expectations and apache-airflow, of course.
pip install airflow-provider-great-expectations
In order to run the BigQueryOperator, you will also need to install the relevant dependencies: pybigquery and apache-airflow-providers-google
Great Expectations Operator: A base operator for Great Expectations. Import into your DAG via:
from great_expectations_provider.operators.great_expectations import GreatExpectationsOperator
Great Expectations BigQuery Operator: An operator for Great Expectations that provides some pre-set parameters for a BigQuery Datasource and Expectation, Validation, and Data Docs stores in Google Cloud Storage. The operator can also be configured to send email on validation failure. See the docstrings in the class for more configuration options. Import into your DAG via:
from great_expectations_provider.operators.great_expectations_bigquery import GreatExpectationsBigQueryOperator
See the examples directory for an example DAG with some sample tasks that demonstrate operator functionality. The example DAG file contains a comment with instructions on how to run the examples.
**This operator is in very early stages of development! Feel free to submit issues, PRs, or ping the current author (Sam Bail) in the Great Expectations Slack for feedback. Thanks to Pete DeJoy and the Astronomer.io team for the support.