Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Spark version detection for automatic Deequ maven coordinate definition #114

Merged
merged 2 commits into from
Dec 30, 2022

Conversation

lecardozo
Copy link
Contributor

Issue #, if available:
#111

Description of changes:
As discussed in #111, we are reverting the Spark version detection method for Deequ's maven coordinate definition, as the current solution is broken.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Copy link
Contributor

@chenliu0831 chenliu0831 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I also heard some complain on how this broken people because Python is setup in a quite strange fashion on Glue/EMR.

@chenliu0831 chenliu0831 merged commit 1704002 into awslabs:master Dec 30, 2022
@chenliu0831
Copy link
Contributor

chenliu0831 commented Dec 30, 2022

@lecardozo thanks for the contribution - but I noticed the CI was actually broken, can you take a look? Do we need to add the env var back?

@lecardozo
Copy link
Contributor Author

Yeah, I just noticed a small typo on the tests! I'll open a new PR with a quick fix 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants