Revise control flow for Migrate Groups with more resiliency

[zpappa]

Background

This issue attempts to capture the proposed control flow for group migration. This control flow should be idempotently repeatable, transacted at various states, and provide different reporting points to the user.

Related Issues

#343
#342
#344

Entry Points

The entry point should be easily runnable via CLI and take the workspace id as a default parameter

databricks labs ucx migrate-groups --workspace-id=12132312323

Optional Parameters

Parameter	Comment
--use-workspace-scoped-group-names	Instruct tool to automatically create account groups with the workspace name prepended
--skip-groups	Do not create groups
--skip-object-permissions	Do not re-apply object permissions on workspace objects
--skip-table-acls	Do not translate table ACLs to UC
--persist-manifest	Keep the manifest files for inspection, will be deleted if the command is run again successfully without this
--dry-run	Do not create groups, output instead the proposed changes

Control Flow Overview

• This command and all scenarios should run idempotently
• This command should assume:
  • The assessment step has been run
  • That a metastore has been assigned
  • Should provide acceptable specific error output otherwise
• This command will operate in 3 distinct phases, each with its own temporary state management
  • Workspace Group Migration
  • Workspace Object Permission Migration
  • Workspace Table ACL Migration

This entire control flow should operate with some temporary state management between actions so that operations can be continued or the control flow can be retried in the case of an error or interruption.

Persisted Data Structures

External Workspace Groups Requiring SCIM Configuration #343

Field	Comment
workspace_id	the workspace id in question, int
local_group_name	the name of the group in the given workspace, string
user_name	the name of the user in the group, string
new_account_group_name	What is the new name of the account group I need to create, string

Duplicated workspace groups with different members #344

Field	Comment
workspace_id	the workspace id in question, int
duplicated_group_name	the name of the group in the given workspace, string
members_as_list_in_account_console	comma-separated members from the account console, string
members_as_list_in_workspace	comma-separated members from the account console, string

Temporarily Persisted Data Structures

object_type	object_id	migrated	failures
TABLES	hive_metastore.foo.bar	1	[]
TABLES	hive_metastore.foo.baz	0	[]
TABLES	hive_metastore.foo.boz	0	["Unsupported SerDe formant: OpenCSV"]
GRANT_SELECT	hive_metastore.foo.baz:group_1	0	[]
GRANT_MODIFY	hive_metastore.foo.baz:group_2	0	[]
CLUSTER_PERMISSIONS	2984-dsfjlkskd-2393	0	[]
CLUSTER_PERMISSIONS	2984-dsfjlkskd-2394	1	[]
CLUSTER_PERMISSIONS	2984-dsfjlkskd-2394	1	["uses storage SPN"]

Phase 1 - Workspace Group Migration

• Check to see if the skip groups flag has been set

  • Continue to the next phase if it is set to True

• Check to see if the Workspace Group Migration Manifest exists

  • Otherwise, perform these steps:
    • Gather all groups and memberships from Workspace $inventory
      • Order Workspace $inventory groups as they are retrieved, and add group names to Workspace Group Migration Manifest
    • Gather all groups and memberships from the Account console $inventory

• For each group in Workspace $inventory as ordered by your Group Upgrade Manifest where not migrated

  1. See if the group exists in the Account console (use the flag for workspace-scoped group names to modify the name before evaluating)
  2. If the group exists, and if the members are the same,
     • Do nothing
  3. If the group does exist, and if the members are not the same, and you are using workspace-scoped group names
     • Move to an error queue and continue
  4. If the group is an external group
     • **Add it to an inventory table of migrated groups see Notify account/scim administrators about workspace groups not present as Databricks Account group #342
     • All users should be added, one per row, so the same group will appear multiple times.
  5. If the workspace local group contains groups
     • For each group in the workspace local group
       • Re-evaluate from step 1 Note this would be a recursive action.
       • As an implementation suggestion, you skip this logic and perform the ordering before you create the manifest
         • A group's children must be created before the group itself is created.
  6. If the group does not exist
     • Create it
     • Populate it with the same members from the UI
     • Set the owner of the group to be the workspace admin/s
  7. Manage state in memory, commit to manifest file in the case of an exception

• Delete the Workspace Group Migration Manifest when the operation is complete.

Phase 2 - Workspace Object Permission Migration

• Check to see if the skip workspace acls flag has been set
  • Continue to the next phase if it is set to True
• Check to see if the Workspace Object Permissions Upgrade Manifest exists
  • Otherwise, perform these steps:
    • Gather all groups and memberships from the Account console group $inventory
      • This should include group->new_group
    • Gather all workspace object ACLs from $inventory
    • Create the Workspace Object Permission Upgrade Manifest with all migrated values set to False.
• For each workspace object as ordered by your Object Permission Upgrade Manifest where not migrated
  1) Map object ACLs that have groups from group->new_group as per your $inventory
  2) Manage state in memory, commit to manifest file in the case of an exception
  • Delete the Workspace Object Permission Upgrade Manifest when the operation is complete

Phase 3 - Workspace Table ACL Migration

• Check to see if the skip table acls flag has been set
  • End execution if true
• Check to see if the Workspace Table ACL Migration Manifest exists
  • Otherwise, perform these steps:
    • Gather all groups and memberships from the Account console group $inventory
      • This should include group->new_group
    • Gather all workspace table ACLs from $inventory
    • Create the Workspace Table ACL Upgrade Manifest with all migrated values set to False.
• For each acl statement as ordered by your Object Permission Upgrade Manifest where not migrated
  1) Parse the ACL statement, tokenize into principal, securable, permission
  2) Map the principal if it is a group (will not be an email address or a service principal guid)
  3) Add to a string builder to generate a SQL notebook to run all grant commands, make sure to revoke the existing privilege
  4) Manage state in memory, commit to manifest file in the case of an exception,
  5) Generate a SQL notebook/query from the manifest/memory, run the notebook/query
  • Delete the Workspace Table ACL Migration Manifest Manifest when the operation is complete

[zpappa]

@nfx Here is a design overview for Group/Object Permission/ACL migration

We should agree on functionality, and operation here and then create follow-up issues to get alignment on what already exists.

[nfx]

@zpappa This issue is too long. Create a PR and copy the contents into a markdown file in docs folder and consolidate it with the other group migration docs from there, because this issue mixes current state and desired state. Also don't be that prescriptive on dashboards/internal persistence structures.

Let's take it from there.

[nfx]

@zpappa is committed to split of this to multiple different issues and close this one.

[zpappa]

@nfx I will leave this open and link back to it so implementors have the full context when working on their issue

[nfx]

there has to be only one view:

object_type	object_id	migrated	failures
TABLES	hive_metastore.foo.bar	1	[]
TABLES	hive_metastore.foo.baz	0	[]
TABLES	hive_metastore.foo.boz	0	["Unsupported SerDe formant: OpenCSV"]
GRANT_SELECT	hive_metastore.foo.baz:group_1	0	[]
GRANT_MODIFY	hive_metastore.foo.baz:group_2	0	[]
CLUSTER_PERMISSIONS	2984-dsfjlkskd-2393	0	[]
CLUSTER_PERMISSIONS	2984-dsfjlkskd-2394	1	[]
CLUSTER_PERMISSIONS	2984-dsfjlkskd-2394	1	["uses storage SPN"]

and number of persistend structures has to get to the very minimum

[FastLee]

@nfx @zpappa may I suggest to combine the assessment step for group migration with the group migration and take it out of assessment. It takes a really long time to run and is not a necessary step before the actual table migration.

[nfx]

We might separate it, but isn't it the goal to prepare all the data before any other steps? We need to list notebooks for scanning their contents anyway.

[nfx]

scanned through requirements - generating a notebook must not be an option (this is solutions), we have predefined workflows (this is design).

[nfx]

new permission migration api makes this issue irrelevant.