Refactor role grants to use new system with grant_privileges_to_role

[ian-r-rose]

This updates the terraform role grants for our Snowflake setup to use the new grant_privileges_to_role. Other resources for declaring role grants have been deprecated and consolidated into this new one.

Despite the churn, I like this change. The number of role grant resources for assembling the "access roles" has gone down from 177 (!) to 54. This is because each access role grant (e.g., read access on RAW) now takes a list of privileges, rather than needing to create one grant per privilege. So we can grant both SELECT and INSERT to a read-write role at the same time rather than separately. This is both simpler and closer to how the idiomatic RBAC in SQL would look.

Of course, I have yet to test this, which I plan to do in our dev account.

Fixes #143.

[AeriShan-ODI]

This looks good to me. But I'm not sure how I would test it other than "validate"

[ian-r-rose]

The main thing I'm worried about is the OWNERSHIP grants, which have some special behavior and aren't as easily revoked: Snowflake-Labs/terraform-provider-snowflake#1942

[ian-r-rose]

Upon poking at this with a _TEST environment, I don't think this refactoring is ready to go in. As per the linked discussions, OWNERSHIP grants are a little different, and there is still some implementation happening on the snowflake side to handle them appropriately (specifically, the grant_ownership_to_role resource needs to be created)

[AeriShan-ODI]

How do you run terraform with the _TEST environment?

[ian-r-rose]

How do you run terraform with the _TEST environment?

Sorry, that was insufficiently clear. I created an entirely new version of our whole architecture with the suffix _TEST (i.e., RAW_TEST, TRANSFORM_TEST, etc) in main, and then checked out this branch and tried to migrate it. Most things went okay, but the inability to properly migrate OWNERSHIP was seemingly a blocker.

I've since deleted all the _TEST infrastructure

[jkarpen]

Per @ian-r-rose this is no longer blocked, he plans to make it a high priority item to complete when he returns from leave in sprint 2024-20.

[ian-r-rose]

@ram-kishore-odi this is now ready for review, I can go into some detail this afternoon.

Unfortunately, this migration needs to be done in two steps, since I need to use the last version of the terraform provider that had the deprecated resources (0.92) in order to destroy them properly. I can make a (much simpler) follow-up PR after this one is in.

[ram-kishore-odi]

Hi @ian-r-rose

Please see my comments below -

• Overall the code changes for refactoring/migration of the deprecated functionality looks alright to me
• Since the code mainly involves to migrating to the newer version of the snowflake provider and changes to OWNERSHIP grants, I cannot think of any better ways of doing it.
• The official terraform documentation suggests that this provider is still in the experimental phase, So I think it may be good to explore other options like Snowflake CLI (SnowSQL) or Pulumi. This may simply the overall process

As an additional note the following compares open source pulumi with terraform and shows the advantages - https://www.pulumi.com/docs/iac/concepts/vs/terraform/

[ian-r-rose]

I agree that it's not ideal to rely on a pre-1.0 version of a terraform provider, but I'm not really aware of alternatives that are more stable (the SnowSQL CLI doesn't really do declarative state management, and I don't think that there are stable pulumi providers?).

Some alternatives that I'd love to explore include schemachange, permifrost, or titan. But those have other drawbacks (schemachange doesn't do roles, permifrost and titan are both third-party and not stable yet).

Of those alternatives, Titan seems quite promising, but is also very new

[ram-kishore-odi]

hi @ian-r-rose. Please see my additional notes below -

Yes your argument is very valid and I agree.

I was referring SnowSQL more from ease of managing and roles and permissions by leveraging SQL commands and scripts but not from a declarative state management perspective. Pulumi helps with declarative statement but I agree it is hard to tell if Pulumi snowflake provider is more stable than terraform snowflake provider. In general being open source and more flexible compared to terraform it is becoming more and more popular.

Secondly I also felt that looking at the documentation, AWS snowflake provider seems to be a good choice compared to others and the last major release was in April of last year. But since we are using terraform to be cloud-agnostic, I know we may not want to use it. But if our snowflake, dbtCloud and FiveTran are all running on AWS (assumption), then in the future considering cloud formation templates for full scale automation may not be a bad option I think. Just expressing my thought.

Thank you for sharing the other options you are thinking about. I would like to explore them more