Add source code facet for operators #1537
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rajaths010494
changed the title
Codecov ReportBase: 94.01% // Head: 94.00% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #1537 +/- ##
==========================================
- Coverage 94.01% 94.00% -0.02%
==========================================
Files 89 89
Lines 4347 4372 +25
Branches 428 432 +4
==========================================
+ Hits 4087 4110 +23
Misses 178 178
- Partials 82 84 +2
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
10 tasks
pankajastro
reviewed
Jan 6, 2023
pankajastro
force-pushed
the
source_code_facet
branch
2 times, most recently
from
6dda086 to
997522d
Compare
pankajastro
requested review from
dimberman,
tatiana,
utkarsharma2,
sunank200,
pankajkoti and
feluelle
as code owners
pankajastro
force-pushed
the
source_code_facet
branch
from
62625a8 to
d461dc8
Compare
pankajkoti
reviewed
Jan 9, 2023
sunank200
requested changes
Jan 9, 2023
There was a problem hiding this comment.
We need the following before merging this PR:
- Screenshot of DAG with lineage values on Marquez populating for operators for databases SQLite, google bigquery and Snowflake.
- The documentation changes for details around
OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE. - Issue link on Openlineage if sourceCodeFacet is not visible on Marquez UI.
- Screenshot of DAG with lineage values on astro-cloud populating for operators for databases SQLite, google bigquery and Snowflake.
|
Snowflake transform and dataframe |
|
BigQuery |
|
Sqlite |
|
Opened an issue in. Openlineage for |
rajaths010494
force-pushed
the
source_code_facet
branch
from
ff946ce to
7c3161e
Compare
JDarDagran
reviewed
Jan 9, 2023
sunank200
approved these changes
Jan 9, 2023
|
Merging it. please let we have any further improvement suggestion we will take care of that in a separate PR |
utkarsharma2
pushed a commit
that referenced
this pull request
Jan 17, 2023
**Please describe the feature you'd like to see** closes: #1467 The python transform should also expose its content with the sourceCode facet, similar to the sql facet. Example: [link](https://github.com/OpenLineage/OpenLineage/blob/3090ced24604c95716dacd667c2cff52bf438aba/integration/airflow/openlineage/airflow/extractors/python_extractor.py#L31) [Action item: Create an issue on astro-sdk] Include the transformation python code that the transformations were running in the OpenLineage events so that they showed up in the Info tab For demo purposes I hard-coded both of these in a custom openlineage-airflow fork like: ``` code = inspect.getsource(task.python_callable) job_facet = {"sql": SqlJobFacet(query=code), "sourceCodeLocation": SourceCodeLocationJobFacet("git", "https://github.com/astronomer/astro-days-chicago/blob/9cca4e166d73106e903f3d9f32af334d7b5560a3/dags/airflow_ecosystem.py")} ``` ^ the code would probably be cleaner as {"sourceCode": SourceCodeJobFacet("python", code)} I stuffed it in sql only due to the lack of: astronomer/astro#2150 which is a temporary limitation **Describe the solution you'd like** - Add source code facet for operators for open lineage integrations Currently source code facet is done for base decorator operators. Co-authored-by: Pankaj <pankaj.singh@astronomer.io> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please describe the feature you'd like to see
closes: #1467
The python transform should also expose its content with the sourceCode facet, similar to the sql facet. Example: link [Action item: Create an issue on astro-sdk]
Include the transformation python code that the transformations were running in the OpenLineage events so that they showed up in the Info tab
For demo purposes I hard-coded both of these in a custom openlineage-airflow fork like:
^ the code would probably be cleaner as {"sourceCode": SourceCodeJobFacet("python", code)} I stuffed it in sql only due to the lack of: https://github.com/astronomer/astro/issues/2150 which is a temporary limitation
Describe the solution you'd like
Currently source code facet is done for base decorator operators.