Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lakeformation: Fix various bugs including SELECT permission issues #20108

Merged
merged 24 commits into from
Jul 8, 2021
Merged

lakeformation: Fix various bugs including SELECT permission issues #20108

merged 24 commits into from
Jul 8, 2021

Conversation

YakDriver
Copy link
Member

@YakDriver YakDriver commented Jul 8, 2021 •
edited
Loading

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #20047
Closes #20048
Relates #19817

Problems Fixed

  • State not refreshing properly when using SELECT by itself in permissions involving tables with columns/tables and wildcards (AWS translates SELECT permissions from one type on input to another on output) - see details below
  • Timeout and error when deleting IAP permissions
  • Acceptance tests not able to run under STS credentials

SELECT with wildcard

  • The resource to which a SELECT permission pertains can be different than other permissions.
  • Example, SELECT on a table_with_columns where wildcard = true is equivalent to SELECT on a table. If you input one or the other, you may get a different version back. The AWS provider did not previously handle both possibilities.
  • Example, SELECT on a table_with_columns where wildcard = true and name = "ALL_TABLES" is equivalent to SELECT on a table with wildcard = true. The AWS provider did not previously handle both possibilities.
  • This was discovered because of the incompatibility between IAM_ALLOWED_PRINCIPALS and individual permissions. Having IAM_ALLOWED_PRINCIPALS in addition to individual permissions changes the permissions. SELECT on a column becomes SELECT on wildcard, which the AWS provider did not handle correctly when refreshing state.
  • However, it still remains that IAM_ALLOWED_PRINCIPALS is not compatible with individual permissions. These changes will allow the provider to error more gracefully in that situation but not overcome the inherent incompatibility.

Output from acceptance testing (us-west-2):

--- PASS: TestAccAWSLakeFormation_serial (816.03s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable (245.57s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/iamAllowed (46.62s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/implicit (24.22s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/multipleRoles (26.05s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectOnly (24.98s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectPlus (25.03s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/basic (25.42s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/selectOnly (25.43s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/selectPlus (25.32s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardNoSelect (22.50s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns (160.48s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/basic (60.02s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/implicit (23.92s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardExcludedColumns (25.65s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectOnly (25.59s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectPlus (25.30s)
    --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings (43.87s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/basic (10.46s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/dataSource (11.97s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/disappears (10.88s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/withoutCatalogId (10.56s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic (230.00s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/basic (29.83s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/database (22.25s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/databaseIAMAllowed (43.26s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/databaseMultiple (22.56s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/dataLocation (27.14s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/disappears (84.96s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource (136.12s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/basic (23.38s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/database (27.02s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/dataLocation (29.17s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/table (29.89s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/tableWithColumns (26.67s)

@github-actions github-actions bot added documentation Introduces or discusses updates to documentation. service/lakeformation Issues and PRs that pertain to the lakeformation service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. size/XL Managed by automation to categorize the size of a PR. labels Jul 8, 2021
@YakDriver YakDriver added this to the v3.49.0 milestone Jul 8, 2021
maryelizbeth
maryelizbeth reviewed Jul 8, 2021
View reviewed changes
Copy link
Contributor

@maryelizbeth maryelizbeth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some suggestions to the docs to enhance the flow when reading through; not essential so feel free to discard!

website/docs/r/lakeformation_permissions.html.markdown Outdated Show resolved Hide resolved
website/docs/r/lakeformation_permissions.html.markdown Outdated Show resolved Hide resolved
website/docs/r/lakeformation_permissions.html.markdown Outdated Show resolved Hide resolved
YakDriver and others added 4 commits July 8, 2021 15:16
@YakDriver @maryelizbeth
docs/r/lakeformation_permissions: Soften language
306bd03 
Co-authored-by: Mary Elizabeth <mary.cutrali@gmail.com>
@YakDriver @maryelizbeth
docs/r/lakeformation_permissions: Clarify
ddab58c 
Co-authored-by: Mary Elizabeth <mary.cutrali@gmail.com>
@YakDriver @maryelizbeth
docs/r/lakeformation_permissions: Clean up language
fb040dd 
Co-authored-by: Mary Elizabeth <mary.cutrali@gmail.com>
@YakDriver
r/lakeformation_permissions: Errant newline
ad016d2
@YakDriver YakDriver changed the title lakeformation: Fix various bugs lakeformation: Fix various bugs including SELECT permission issues Jul 8, 2021
ewbankkit
ewbankkit approved these changes Jul 8, 2021
View reviewed changes
Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀.

Commercial
% make testacc TEST=./aws TESTARGS='-run=TestAccAWSLakeFormation_serial'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSLakeFormation_serial -timeout 180m
=== RUN   TestAccAWSLakeFormation_serial
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings/basic
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings/dataSource
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings/disappears
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings/withoutCatalogId
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/disappears
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/basic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/database
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/databaseIAMAllowed
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/databaseMultiple
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/dataLocation
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/basic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/database
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/dataLocation
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/table
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/tableWithColumns
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/multipleRoles
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/selectOnly
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/selectPlus
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectOnly
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectPlus
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/basic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/implicit
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/wildcardNoSelect
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/iamAllowed
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/basic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/implicit
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardExcludedColumns
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectOnly
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectPlus
--- PASS: TestAccAWSLakeFormation_serial (937.37s)
    --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings (47.04s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/basic (12.18s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/dataSource (12.18s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/disappears (11.32s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/withoutCatalogId (11.36s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic (242.74s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/disappears (86.25s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/basic (33.59s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/database (24.93s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/databaseIAMAllowed (44.04s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/databaseMultiple (24.12s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/dataLocation (29.81s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource (140.78s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/basic (23.75s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/database (30.81s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/dataLocation (27.30s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/table (31.73s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/tableWithColumns (27.19s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable (334.87s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/multipleRoles (103.87s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/selectOnly (26.45s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/selectPlus (26.98s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectOnly (26.09s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectPlus (26.30s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/basic (28.35s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/implicit (26.36s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardNoSelect (23.25s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/iamAllowed (47.23s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns (171.95s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/basic (65.78s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/implicit (25.62s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardExcludedColumns (27.78s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectOnly (26.76s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectPlus (26.00s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	941.355s
GovCloud
% make testacc TEST=./aws TESTARGS='-run=TestAccAWSLakeFormation_serial'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSLakeFormation_serial -timeout 180m
=== RUN   TestAccAWSLakeFormation_serial
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings/basic
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings/dataSource
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings/disappears
=== RUN   TestAccAWSLakeFormation_serial/DataLakeSettings/withoutCatalogId
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/basic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/database
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/databaseIAMAllowed
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/databaseMultiple
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/dataLocation
=== RUN   TestAccAWSLakeFormation_serial/PermissionsBasic/disappears
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/dataLocation
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/table
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/tableWithColumns
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/basic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsDataSource/database
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/implicit
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/selectOnly
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/selectPlus
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectOnly
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectPlus
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/basic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/iamAllowed
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/multipleRoles
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTable/wildcardNoSelect
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/basic
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/implicit
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardExcludedColumns
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectOnly
=== RUN   TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectPlus
--- PASS: TestAccAWSLakeFormation_serial (878.63s)
    --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings (64.18s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/basic (17.19s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/dataSource (15.97s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/disappears (15.70s)
        --- PASS: TestAccAWSLakeFormation_serial/DataLakeSettings/withoutCatalogId (15.32s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic (245.12s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/basic (25.42s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/database (25.42s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/databaseIAMAllowed (47.88s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/databaseMultiple (30.63s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/dataLocation (30.40s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsBasic/disappears (85.37s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource (136.04s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/dataLocation (30.17s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/table (28.96s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/tableWithColumns (24.16s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/basic (24.89s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsDataSource/database (27.86s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable (255.33s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/implicit (26.19s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/selectOnly (25.91s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/selectPlus (25.47s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectOnly (25.25s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardSelectPlus (25.32s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/basic (27.15s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/iamAllowed (49.50s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/multipleRoles (26.26s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTable/wildcardNoSelect (24.28s)
    --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns (177.97s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/basic (70.54s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/implicit (30.45s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardExcludedColumns (25.77s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectOnly (25.61s)
        --- PASS: TestAccAWSLakeFormation_serial/PermissionsTableWithColumns/wildcardSelectPlus (25.59s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	881.703s

@YakDriver YakDriver merged commit f268ddf into main Jul 8, 2021
@YakDriver YakDriver deleted the b-lakeformation-iam-allowed branch July 8, 2021 21:20
github-actions bot pushed a commit that referenced this pull request Jul 8, 2021
Update CHANGELOG.md for #20108
7271ccb
@github-actions
Copy link

github-actions bot commented Jul 8, 2021

This functionality has been released in v3.49.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

github-actions bot commented Aug 8, 2021

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@maryelizbeth maryelizbeth maryelizbeth left review comments

@ewbankkit ewbankkit ewbankkit approved these changes

Assignees
No one assigned
Labels
documentation Introduces or discusses updates to documentation. service/lakeformation Issues and PRs that pertain to the lakeformation service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v3.49.0
3 participants
@YakDriver @maryelizbeth @ewbankkit