[DocDB][DBscope] Could not find CDC stream: stream_id in tserver

[agsh-yb]

Jira Link: DB-14784

Description

Version: 2024.1.4.0-b108
Test: test_xcluster_db_scope_all_in_one

Error:

AssertionError: Failed to add to replication: ["Error running add_namespace_to_xcluster_replication: Invalid argument (yb/master/xrepl_catalog_manager.cc:5787): Alter for CDC producer currently running: replication_group_id: "67d226e3-1c5d-4e9c-b667-2cf5472a4458_rep" producer_table_ids_to_add: "0000401a00003000800000000000401f" producer_table_ids_to_add: "0000401a00003000800000000000401a" producer_table_ids_to_add: "0000401a000030008000000000004100" producer_table_ids_to_add: "0000401a000030008000000000004015" producer_bootstrap_ids_to_add: "fafc5de42a549a983343a03c7036d9ff" producer_bootstrap_ids_to_add: "884ee6314450c7baa64eda0ba2f9a704" producer_bootstrap_ids_to_add: "a693710d63595287b74f601f7881f6e9" producer_bootstrap_ids_to_add: "84cbfdd7a21ff1a3254bc69c0a5d14f1" universe_uuid: "ea2f4e74-3376-4f84-aa0c-aa06c374784a" producer_namespace_to_add { id: "0000401a000030008000000000000000" name: "yb_db_dr_rand_cca5653_1" database_type: YQL_DATABASE_PGSQL } (master error 29)\n", "Usage: /home/yugabyte/tserver/bin/yb-admin add_namespace_to_xcluster_replication <replication_group_id> <namespace_name> <target_master_addresses>\n"]

In tserver logs:

Already running schema version 0 got alter request for version 0
I0107 20:20:48.603577 45909 client_master_rpc.cc:77] 0x000016db5a303920 -> GetCDCStream: Failed, got resp error: Not found (yb/master/xrepl_catalog_manager.cc:3005): Could not find CDC stream: stream_id: "f8e3da5f4704648d3a41be13ff43e409" (master error 3)
I0107 20:20:49.306546 46755 rate_limiter.cc:136] CDC Service rate limiter status:
available_bytes: 828.31k
queue_size: 0
total_bytes_requested_per_second: 3.67k
total_request_per_second: 189
unthrottled_request_per_second: 189
throttled_request_per_second: 0.000
time_since_refresh: 1.007s

Steps test performed:

Steps:
1. Create multiple database( non-colocated, colocated )
2. Create tables for each database
3. Setup replication
4. Start cycle:
    a. Start parallel nemesis on both source and target
    b. Create new databases (non-colocated/colocated)
    c. Create tables/indexes (both sides)
    d. Add databases to the replication(random)
    e. Remove database from replicaton (random)
    f. Drop replication
    g. Stop nemesis
    h. Create replication

attached logs in jira

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.