Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(glue): --conf parameter is no longer a reserved keyword for glue jobs #23673

Merged
merged 3 commits into from
Jan 18, 2023

Conversation

moomindani
Copy link
Contributor

AWS Glue has changed its public description for the job parameter --conf.

The latest description:
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

Controls Spark config parameters. It is for advanced use cases.

According to this change, AWS CDK no longer needs to block the usage of this parameter.

This PR removes the validation for the parameter --conf.

Background

The parameter --conf is expected to use several Spark capabilities. One of the typical usage is with native data lake table format support.
https://aws.amazon.com/about-aws/whats-new/2022/11/aws-glue-apache-spark-native-data-lake-frameworks-apache-hudi-iceberg-delta-lake/

Public sample notebooks (e.g. https://github.com/aws-samples/aws-glue-samples/blob/master/examples/notebooks/native_hudi_sql.ipynb) use the parameter --conf.


All Submissions:

Adding new Construct Runtime Dependencies:

  • This PR adds new construct runtime dependencies following the process described here

New Features

  • Have you added the new feature to an integration test?
    • Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Jan 13, 2023

@aws-cdk-automation aws-cdk-automation requested a review from a team January 13, 2023 08:38
@github-actions github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK p2 labels Jan 13, 2023
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

@aws-cdk-automation aws-cdk-automation dismissed their stale review January 13, 2023 12:34

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

@TheRealAmazonKendra TheRealAmazonKendra changed the title fix(aws-glue): allow --conf parameter for glue jobs fix(glue): --conf parameter is no longer a reserved keyword for glue jobs Jan 18, 2023
@mergify
Copy link
Contributor

mergify bot commented Jan 18, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 409671d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 3d0f4ba into aws:main Jan 18, 2023
@mergify
Copy link
Contributor

mergify bot commented Jan 18, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK p2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants