feat(ingest): add support for parsing lineage for redash connector (#…

…3267) Co-authored-by: tibrahim <taufiq.ibrahim@bizzy.co.id>
datahub-project · Oct 6, 2021 · ed38168 · ed38168
1 parent 9cb71d9
commit ed38168
Show file tree

Hide file tree

Showing 5 changed files with 396 additions and 62 deletions.
diff --git a/metadata-ingestion/examples/recipes/redash_to_file.yml b/metadata-ingestion/examples/recipes/redash_to_file.yml
@@ -18,6 +18,7 @@ source:
     #     - ^denied chart.*
     #   allow:
     #     - .*allowed chart.*
+    # parse_table_names_from_sql: false
 
 sink:
   type: "file"

diff --git a/metadata-ingestion/setup.py b/metadata-ingestion/setup.py
@@ -107,7 +107,7 @@ def get_long_description():
     "okta": {"okta~=1.7.0"},
     "oracle": sql_common | {"cx_Oracle"},
     "postgres": sql_common | {"psycopg2-binary", "GeoAlchemy2"},
-    "redash": {"redash-toolbelt"},
+    "redash": {"redash-toolbelt", "sql-metadata"},
     "redshift": sql_common | {"sqlalchemy-redshift", "psycopg2-binary", "GeoAlchemy2"},
     "redshift-usage": sql_common
     | {"sqlalchemy-redshift", "psycopg2-binary", "GeoAlchemy2"},

diff --git a/metadata-ingestion/source_docs/redash.md b/metadata-ingestion/source_docs/redash.md
@@ -11,6 +11,7 @@ To install this plugin, run `pip install 'acryl-datahub[redash]'`.
 This plugin extracts the following:
 
 - Redash dashboards and queries/visualization
+- Redash chart table lineages (disabled by default)
 
 ## Quickstart recipe
 
@@ -38,24 +39,31 @@ source:
     #     - ^denied chart.*
     #   allow:
     #     - .*allowed chart.*
+    # parse_table_names_from_sql: false
 ```
 
 ## Config details
 
 Note that a `.` is used to denote nested fields in the YAML recipe.
 
-| Field                      | Required | Default                | Description                                                      |
-| -------------------------- | -------- | ---------------------- | ---------------------------------------------------------------- |
-| `connect_uri`              | ✅       | http://localhost:5000/ | Redash base URL.                                                 |
-| `api_key`                  | ✅       |                        | Redash user API key.                                             |
-| `api_page_limit`           |          | `None`                 | Limit on ingested dashboards and charts API pagination.          |
-| `skip_draft`               |          | `true`                 | Only ingest published dashboards and charts.                     |
-| `dashboard_patterns.allow` |          |                        | List of regex patterns for dashboards to include in ingestion.   |
-| `dashboard_patterns.deny`  |          |                        | List of regex patterns for dashboards to exclude from ingestion. |
-| `chart_patterns.allow`     |          |                        | List of regex patterns for charts to include in ingestion.       |
-| `chart_patterns.deny`      |          |                        | List of regex patterns for charts to exclude from ingestion.     |
-| `env`                      |          | `"PROD"`               | Environment to use in namespace when constructing URNs.          |
-
+| Field                         | Required | Default                | Description                                                      |
+| ----------------------------- | -------- | ---------------------- | ---------------------------------------------------------------- |
+| `connect_uri`                 | ✅       | http://localhost:5000/ | Redash base URL.                                                 |
+| `api_key`                     | ✅       |                        | Redash user API key.                                             |
+| `api_page_limit`              |          | `None`                 | Limit on ingested dashboards and charts API pagination.          |
+| `skip_draft`                  |          | `true`                 | Only ingest published dashboards and charts.                     |
+| `dashboard_patterns.allow`    |          |                        | List of regex patterns for dashboards to include in ingestion.   |
+| `dashboard_patterns.deny`     |          |                        | List of regex patterns for dashboards to exclude from ingestion. |
+| `chart_patterns.allow`        |          |                        | List of regex patterns for charts to include in ingestion.       |
+| `chart_patterns.deny`         |          |                        | List of regex patterns for charts to exclude from ingestion.     |
+| `env`                         |          | `"PROD"`               | Environment to use in namespace when constructing URNs.          |
+| `parse_table_names_from_sql`  |          | `false`                | See note below.                                                  |
+
+Note! The integration can use an SQL parser to try to parse the tables the chart depends on. This parsing is disabled by default, 
+but can be enabled by setting `parse_table_names_from_sql: true`.  The default parser is based on the [`sql-metadata`](https://pypi.org/project/sql-metadata/) package. 
+As this package doesn't officially support all the SQL dialects that Redash supports, the result might not be correct. You can, however, implement a
+custom parser and take it into use by setting the `sql_parser` configuration value. A custom SQL parser must inherit from `datahub.utilities.sql_parser.SQLParser`
+and must be made available to Datahub by ,for example, installing it. The configuration then needs to be set to `module_name.ClassName` of the parser.
 
 ## Compatibility