Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SNOW-198] Create ownership database roles for data warehouse #104

Merged
merged 4 commits into from
Jan 25, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,17 @@ jobs:
-w compute_xsmall \
-d METADATA

- name: future grants
shell: bash
run: |
schemachange \
-f admin/future_grants \
-a $SNOWFLAKE_ACCOUNT \
-u $SNOWFLAKE_USER \
-r SECURITYADMIN \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just logged onto the DPE_SERVICE account and noticed it has SYSADMIN access but no SECURITYADMIN access, so I feel compelled to ask: Are we sure the $SNOWFLAKE_ACCOUNT we use for these schemachange operations has access to SECURITYADMIN role? I don't see this role used elsewhere so asking just in case.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Looks like we've never had to use SECURITYADMIN to manage grants before. Everything has been done as the owner of an object granting privileges upon that object. I'll open another PR.

-w compute_xsmall \
-d METADATA

schemachange_synapse_data_warehouse_prod:
runs-on: ubuntu-22.04
if: github.ref == 'refs/heads/main'
Expand Down
13 changes: 12 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,17 @@ graph TD;
C-->D[Gold - Snowflake];
```

### Roles and grants

Privileges within the data warehouse are managed according to the design outlined in the design doc [here](https://sagebionetworks.jira.com/wiki/spaces/DPE/pages/3829006353/Synapse+Data+Warehouse+Role+Hierarchy#Role-Hierarchy). The aspect of privilege management most relevant to contributors is that if you are introducing a new object type to a namespace (for example, an instance of an `EXTERNAL TABLE` into a schema which doesn't yet have any `EXTERNAL TABLE` objects), _you_ are responsible for setting up privilege management of that object type, following the already established model. More specifically, responsibilities entail:

* Granting ownership of this object type to the relevant `*_ALL_ADMIN` database role. See [here](https://github.com/Sage-Bionetworks/snowflake/pull/104/files#diff-d23e4b75e1cfeaf9ee3bf2f274210e62505eb30c14fdc86f985cf625d04de928R513-R542) for an example.
* Creating the necessary object-type-specific database role(s) and granting them appropriate privileges upon the object type. The general rule here is to create a single database role `{database}.{schema}_{object_type}_READ` (see [here](https://docs.snowflake.com/en/sql-reference/sql/grant-privilege) for a complete list of object types) and grant that role read-only privileges so that non-owners are able to see the object but not modify it. Rarely, there may be additional use-cases, or this object type might not have mere read-type privileges that can be even be granted upon it (the PROCEDURE object, a.k.a. stored procedures, being a common example), which may necessitate additional database roles or differently assigned privileges. See [here](https://github.com/Sage-Bionetworks/snowflake/pull/107/files#diff-d23e4b75e1cfeaf9ee3bf2f274210e62505eb30c14fdc86f985cf625d04de928R520-R539) for an example of read-type privileges being granted to object-type-specific database roles.
* Granting ownership and usage of the above database role(s) to the appropriate aggregator database role(s). See [here](https://github.com/Sage-Bionetworks/snowflake/pull/107/files#diff-a2c8db7044dc7f329368172c8b620bd8b08a2ade6a8ee2fe9629bf4b924286eaR4-R26) for an example of the creation and granting of ownership and usage privileges on object-type-specific database roles.
* Granting appropriate privileges on future objects (via future grants) to admin/analyst/developer database roles. See [here](https://github.com/Sage-Bionetworks/snowflake/pull/107/files#diff-eda3321a2afcec5ae10699543d6a5bf24617fc8eca196ede71f94322d29a5ee5R3-R22) for an example of granting future read-type privileges to their respective object-type-specific database roles. A similar process can be followed for granting ownership privileges on future objects to the appropriate `*_ALL_ADMIN` database role.

Fortunately, because of the rigorous privilege management performed during the initial deployment of an object-type, there is unlikely to be any privilege management required for deploying object-types which already exist within a namespace. Each user archetype – represented by the namespace-specific admin, developer, and analyst account roles – will inherit their archetype-tailored privileges for that object-type from preexisting database roles. One exception that could potentially occur is if your object has special privacy considerations, such as a field in a table which should only be visible to specific users. These objects will need their privileges handled on a case-by-case basis, although the general practice of granting privileges to a namespace-specific database role still applies.

## RECOVER (PoC)

The RECOVER data is processed via AWS and is compressed to parquet datasets. The parquet datasets are then ingested into snowflake for easy querying and validation.
Expand All @@ -58,4 +69,4 @@ WIP

## Visualizing with Streamlit

Users can customize their own data visualization dashboards from the data available on Snowflake by using the [streamlit-snowflake-template](https://github.com/Sage-Bionetworks/streamlit-snowflake-template). This is a template repository that leverages [Streamlit](https://streamlit.io/) to create and deploy internal applications for Synapse-derived data insight and analysis.
Users can customize their own data visualization dashboards from the data available on Snowflake by using the [streamlit-snowflake-template](https://github.com/Sage-Bionetworks/streamlit-snowflake-template). This is a template repository that leverages [Streamlit](https://streamlit.io/) to create and deploy internal applications for Synapse-derived data insight and analysis.
97 changes: 97 additions & 0 deletions admin/future_grants/V1.2.0__data_warehouse_ownership.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---- SYNAPSE_DATA_WAREHOUSE ----
-- SYNAPSE
GRANT OWNERSHIP
ON FUTURE TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE DYNAMIC TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE STAGES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE VIEWS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE TASKS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN;

-- SYNAPSE_RAW
GRANT OWNERSHIP
ON FUTURE TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE STAGES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE STREAMS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE VIEWS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE TASKS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN;

-- SCHEMACHANGE
GRANT OWNERSHIP
ON FUTURE TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SCHEMACHANGE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SCHEMACHANGE_ALL_ADMIN;

---- SYNAPSE_DATA_WAREHOUSE_DEV ----
-- SYNAPSE
GRANT OWNERSHIP
ON FUTURE TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE DYNAMIC TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE STAGES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE VIEWS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE TASKS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN;

-- SYNAPSE_RAW
GRANT OWNERSHIP
ON FUTURE TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE STAGES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE STREAMS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN;
GRANT OWNERSHIP
ON FUTURE TASKS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN;

-- SCHEMACHANGE
GRANT OWNERSHIP
ON FUTURE TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SCHEMACHANGE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SCHEMACHANGE_ALL_ADMIN;
160 changes: 159 additions & 1 deletion admin/grants.sql
Original file line number Diff line number Diff line change
Expand Up @@ -508,4 +508,162 @@ GRANT USAGE
TO ROLE SECURITYADMIN;
GRANT SELECT, INSERT
ON TABLE METADATA.SCHEMACHANGE.CHANGE_HISTORY
TO ROLE SECURITYADMIN;
TO ROLE SECURITYADMIN;

---- RBAC reconfiguration of data warehouse ----
-- The following grants transfer ownership of current and future
-- data warehouse objects from SYSADMIN to each namespace's respective
-- `*_ALL_ADMIN` database role.

---- SYNAPSE_DATA_WAREHOUSE ----
GRANT OWNERSHIP
ON DATABASE SYNAPSE_DATA_WAREHOUSE
TO ROLE SYNAPSE_DATA_WAREHOUSE_ADMIN
COPY CURRENT GRANTS;

-- SYNAPSE
GRANT OWNERSHIP
ON SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL DYNAMIC TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL STAGES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL VIEWS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TASKS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;

-- SYNAPSE_RAW
GRANT OWNERSHIP
ON SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL STAGES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL STREAMS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL VIEWS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TASKS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;

-- SCHEMACHANGE
GRANT OWNERSHIP
ON SCHEMA SYNAPSE_DATA_WAREHOUSE.SCHEMACHANGE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SCHEMACHANGE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE.SCHEMACHANGE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE.SCHEMACHANGE_ALL_ADMIN
COPY CURRENT GRANTS;

---- SYNAPSE_DATA_WAREHOUSE_DEV ----
GRANT OWNERSHIP
ON DATABASE SYNAPSE_DATA_WAREHOUSE_DEV
TO ROLE SYNAPSE_DATA_WAREHOUSE_DEV_ADMIN
COPY CURRENT GRANTS;

-- SYNAPSE
GRANT OWNERSHIP
ON SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL DYNAMIC TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL STAGES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL VIEWS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TASKS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_ALL_ADMIN
COPY CURRENT GRANTS;

-- SYNAPSE_RAW
GRANT OWNERSHIP
ON SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL STAGES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL STREAMS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TASKS
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SYNAPSE_RAW_ALL_ADMIN
COPY CURRENT GRANTS;

-- SCHEMACHANGE
GRANT OWNERSHIP
ON SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SCHEMACHANGE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SCHEMACHANGE_ALL_ADMIN
COPY CURRENT GRANTS;
GRANT OWNERSHIP
ON ALL TABLES
IN SCHEMA SYNAPSE_DATA_WAREHOUSE_DEV.SCHEMACHANGE
TO DATABASE ROLE SYNAPSE_DATA_WAREHOUSE_DEV.SCHEMACHANGE_ALL_ADMIN
COPY CURRENT GRANTS;
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
USE DATABASE {{ database_name }}; --noqa: JJ01,PRS,TMP

-- Create database roles which will own the respective namespace's objects
CREATE OR REPLACE DATABASE ROLE SYNAPSE_ALL_ADMIN;
CREATE OR REPLACE DATABASE ROLE SYNAPSE_RAW_ALL_ADMIN;
CREATE OR REPLACE DATABASE ROLE SCHEMACHANGE_ALL_ADMIN;

-- Grant ownership of the database roles to the database admin
GRANT OWNERSHIP
ON DATABASE ROLE SYNAPSE_ALL_ADMIN
TO ROLE {{ database_name }}_ADMIN; --noqa: JJ01,PRS,TMP
GRANT OWNERSHIP
ON DATABASE ROLE SYNAPSE_RAW_ALL_ADMIN
TO ROLE {{ database_name }}_ADMIN; --noqa: JJ01,PRS,TMP
GRANT OWNERSHIP
ON DATABASE ROLE SCHEMACHANGE_ALL_ADMIN
TO ROLE {{ database_name }}_ADMIN; --noqa: JJ01,PRS,TMP

-- Grant database roles to account roles
GRANT DATABASE ROLE SYNAPSE_ALL_ADMIN
TO ROLE {{ database_name }}_ADMIN; --noqa: JJ01,PRS,TMP
GRANT DATABASE ROLE SYNAPSE_RAW_ALL_ADMIN
TO ROLE {{ database_name }}_ADMIN; --noqa: JJ01,PRS,TMP
GRANT DATABASE ROLE SCHEMACHANGE_ALL_ADMIN
TO ROLE {{ database_name }}_ADMIN; --noqa: JJ01,PRS,TMP