[2DC] AlterReplication API to add / remove tables to replicate

[ndeodhar]

After replication is setup between 2 universes, if new tables are added to the universes, we need an API to add the new tables to the list of tables being replicated.
Similarly, we need an ability to stop replicating certain tables.

[ajcaldera1]

Similar to the proposal for creating replicates with the qualifiers (ycql_table, ycql_keyspace, ysql_table, ysql_schema and ysql_database) in #2421, here are some thoughts on what this API should look like.

The new alter_universe_replication subcommand should have 2 additional possible arguments (see #2965 for the other) for the adding and dropping of replicates: add_replicate and delete_replicate. A case could be made for drop_replicate instead of delete_replicate, but since we used delete_universe_replication I think we should remain consistent.

1. Add a list of tables to replicate (lowest primitive)
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> add_replicate <tablet_id>[, tablet_id2, ..., tablet_idN ]
2. Remove a list of tablet to replicate
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> delete_replicate <tablet_id>[, tablet_id2, ..., tablet_idN]
3. Add a list of YCQL tables to replicate:
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> add_replicate <ycql_table:keyspace.table_name>[,ycql_table:keyspace.table_name2, ..., ycql_table:keyspace.table_nameN]
4. Add a list of YCQL keyspaces to replicate:
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> add_replicate <ycql_keyspace:keyspace>[,ycql_keyspace:keyspace2,...,ycql_keyspace:keyspaceN]
5. Add a list of YSQL tables to replicate:
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> add_replicate <ysql_table:database.schema.table_name>[,ysql_table:database.schema.table_name2, ..., ysql_table:database.schema.table_nameN]
6. Add a list of YSQL schemas to replicate:
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> add_replicate <ysql_schema:database.schema>[,ysql_schema:database.schema2,...,ysql_schema:database.schemaN]
7. Add a list of YSQL database to replicate:
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> add_replicate <ysql_database:database>[,ysql_database:database2,...,ysql_database:databaseN]
8. Remove a list of YCQL tables to replicate:
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> delete_replicate <ycql_table:keyspace.table_name>[,ycql_table:keyspace.table_name2, ..., ycql_table:keyspace.table_nameN]
9. Remove a list of YCQL keyspaces to replicate:
   yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> delete_replicate <ycql_keyspace:keyspace>[,ycql_keyspace:keyspace2,...,ycql_keyspace:keyspaceN]
10. Remove a list of YSQL tables to replicate:
    yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> delete_replicate <ysql_table:database.schema.table_name>[,ysql_table:database.schema.table_name2, ..., ysql_table:database.schema.table_nameN]
11. Remove a list of YSQL schemas to replicate:
    yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> delete_replicate <ysql_schema:database.schema>[,ysql_schema:database.schema2,...,ysql_schema:database.schemaN]
12. Remove a list of YSQL databases to replicate:
    yb-admin -master_addresses master1,master2,master3 alter_universe_replication <target_uuid> delete_replicate <ysql_database:database>[,ysql_database:database2,...,ysql_database:databaseN]

The subcommands add_replicate and delete_replicate should be able to handle different combinations of the qualifiers: ycql_table,ycql_keyspace,ysql_table,ysql_schema and ysql_database in addition to the default of unqualified tablet_id.

[nspiegelberg]

@ajcaldera1 : thoughts on your proposal

1. I don't think we'll need "master_addresses" after the first command unless we're altering them
2. I'm not sure it's safe to remove at the granularity, since it would be really easy to miss a tablet and have incomplete data. I'm trying to understand the use case that makes it worth the downside. Are you thinking about doing a remove/add for a set of tablets that's fallen behind because of a misbehaving tablet server on the consumer? Is that an issue in practice?
3. Is there an existing API that handles all the qualifiers already? If you have one, I can try to plug into that abstraction. If not, I think we should put that as a separate task for yugaware/API.

[ajcaldera1]

@nspiegelberg:

1. The references to master_addresses are illustrative for later documentation purposes and there for completeness of example. As far as I know, all yb-admin commands require -master_addresses unless you happen to be on the leader or in an RF1 environment.
2. I'm referring to the tablet_uuid which is how we are currently addressing replicates. It is not my intent to go to the sub-tablet identifier.
3. I can't think of one off hand, and YW isn't always going to be in play to coordinate these activities, so I think the better place is to build the logic into yb-admin which has access to the catalog information.

We'll also have to think about how this interacts with the new co-location feature. There a tablet_uuid could potentially refer to an entire PG database.

[nspiegelberg]

@ajcaldera1 : thanks for the clarification on #1 and #2. I think, for #3, I understand your mode of thinking but I think I'd need to scope out the effort and, if non-trivial, deprioritize it to after CDC GA MustHave Tasks are complete.