Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix MetastoreHivePartitionSensor failing due to duplicate aliases #45001

Merged
merged 3 commits into from
Dec 18, 2024

Conversation

CYarros10
Copy link
Contributor

@CYarros10 CYarros10 commented Dec 17, 2024

MetastoreHivePartitionSensor is not working.

The query it uses to list partitions

query = f"""
                SELECT *
                FROM PARTITIONS
                INNER JOIN TBLS
                ON PARTITIONS.TBL_ID = TBLS.TBL_ID
                WHERE
                    TBLS.TBL_NAME = '{table}'"""

fails, due to Duplicate Aliases (https://cloud.google.com/spanner/docs/reference/standard-sql/query-syntax#duplicate_aliases)

This query fixes the issue:

query = f"""
            SELECT PARTITIONS.*, TBLS.TBL_TYPE, TBLS.TBL_NAME
            FROM PARTITIONS
            INNER JOIN TBLS ON PARTITIONS.TBL_ID = TBLS.TBL_ID
            WHERE TBLS.TBL_NAME = '{table}'"""

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Dec 17, 2024
@potiuk potiuk merged commit 553ddf3 into apache:main Dec 18, 2024
64 checks passed
LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this pull request Jan 5, 2025
…ache#45001)

* fix MetastoreHivePartitionSensor with updated query to avoid duplicate aliases

* Update TestMetastoreHivePartitionSensor

* Fix MetastoreHivePartitionSensor pytests
agupta01 pushed a commit to agupta01/airflow that referenced this pull request Jan 6, 2025
…ache#45001)

* fix MetastoreHivePartitionSensor with updated query to avoid duplicate aliases

* Update TestMetastoreHivePartitionSensor

* Fix MetastoreHivePartitionSensor pytests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants