[YCQL] Table stuck in the keyspace after deletion

[smalyshev]

I created a new keyspace and then created table sessions in it, with a couple of indexes. After that, I tried to delete it, however something strange happened: table is stuck in half-deleted state and I can not neither use nor delete it. This is what I get:

Desc tables sees it:

> desc tables;

sessions

I can describe it:

> desc table sessions;

CREATE TABLE test_space.sessions (
    session text,
    ts timestamp,
    PRIMARY KEY (session, ts)
) WITH CLUSTERING ORDER BY (ts ASC)
    AND default_time_to_live = 0;

If I try to drop it, it says it doesn't exist:

> drop table sessions;

InvalidRequest: Error from server: code=2200 [Invalid query] message="Object Not Found. The object does not exist: table_name: "sessions"
namespace {
  name: "test_space"
}

drop table sessions;
           ^^^^^^^^
 (error -301)"

If I try to query it, it doesn't exist:

> select * from sessions;
InvalidRequest: Error from server: code=2200 [Invalid query] message="Object Not Found
select * from sessions;
              ^^^^^^^^
 (error -301)"

If I try to create the same table again, it creates it!

> create table sessions (
                 ...     session text,
                 ...     ts timestamp,
                 ...     PRIMARY KEY (session, ts)
                 ... ) WITH CLUSTERING ORDER BY (ts ASC)
                 ...     AND default_time_to_live = 0;

No error! But the keys in table description are duplicated now:

> desc table sessions;
CREATE TABLE test_space.sessions (
    session text,
    ts timestamp,
    PRIMARY KEY ((session, session), ts, ts)
) WITH CLUSTERING ORDER BY (ts ASC, ts ASC)
    AND default_time_to_live = 0;

And I can query and drop this duplicate table now. But once I dropped it, it's back to the phantom table now. Also, can not drop the keyspace:

> drop keyspace test_space;
ServerError: Server Error. Cannot delete namespace which has table: sessions [id=de3137bc9950419ea82c77af2583e395]: namespace {
  name: "test_space"
}

drop keyspace test_space;
              ^^^^^^^^^^^^^
 (error -2)

[smalyshev]

If I query the system_schema.tables I get:

 keyspace_name | table_name | bloom_filter_fp_chance | caching | cdc  | comment | compaction | compression | crc_check_chance | dclocal_read_repair_chance | default_time_to_live | extensions | flags        | gc_grace_seconds | id                                   | max_index_interval | memtable_flush_period_in_ms | min_index_interval | read_repair_chance | speculative_retry | transactions
---------------+------------+------------------------+---------+------+---------+------------+-------------+------------------+----------------------------+----------------------+------------+--------------+------------------+--------------------------------------+--------------------+-----------------------------+--------------------+--------------------+-------------------+---------------------
    test_space |   sessions |                   null |    null | null |    null |       null |        null |             null |                       null |                    0 |       null | {'compound'} |             null | 95e38325-af77-2ca8-9e41-5099bc3731de |               null |                        null |               null |               null |              null | {'enabled': 'true'}

Interestingly enough, in another keyspace where I had deleted tables I also have duplicate records with the same name (but different ids). Is this normal?

[OlegLoginov]

It's correct to have tables with the same names in different keyspaces.

@smalyshev, you are talking about duplicate records:

in another keyspace where I had deleted tables I also have duplicate records with the same name (but different ids). Is this normal?

If there are a few table records with the same name (and different ids) in the same keyspace - that looks strange. Could you please provide a log how do you see that?

[smalyshev]

For example:

cqlsh:system_schema> select * from tables where keyspace_name='test';

 keyspace_name | table_name | bloom_filter_fp_chance | caching | cdc  | comment | compaction | compression | crc_check_chance | dclocal_read_repair_chance | default_time_to_live | extensions | flags        | gc_grace_seconds | id                                   | max_index_interval | memtable_flush_period_in_ms | min_index_interval | read_repair_chance | speculative_retry | transactions
---------------+------------+------------------------+---------+------+---------+------------+-------------+------------------+----------------------------+----------------------+------------+--------------+------------------+--------------------------------------+--------------------+-----------------------------+--------------------+--------------------+-------------------+----------------------
          test |   sessions |                   null |    null | null |    null |       null |        null |             null |                       null |                    0 |       null | {'compound'} |             null | c5108580-e2f3-5597-0c44-d3ae6a1d0340 |               null |                        null |               null |               null |              null | {'enabled': 'false'}
          test |   sessions |                   null |    null | null |    null |       null |        null |             null |                       null |                    0 |       null | {'compound'} |             null | 1d960be8-4790-55a6-f149-cab8aaaecfff |               null |                        null |               null |               null |              null | {'enabled': 'false'}
          test |   ip_stats |                   null |    null | null |    null |       null |        null |             null |                       null |                    0 |       null | {'compound'} |             null | 9a562c29-f56c-15a0-b845-1e741fd65115 |               null |                        null |               null |               null |              null | {'enabled': 'false'}

As you can see, two tables named sessions. I could reproduce the original problem in this keyspace too now - after I tried to delete these tables, this keyspace is stuck too - tables can't be deleted and can't be used. So now I have two keyspaces like that.

[ndeodhar]

The problem is in CreateTable. Consider this scenario:

create table foo; // This creates table with ID 100
drop table foo;
create table foo; // This creates table with ID 200
create index foo_idx on foo;

In this example, when foo with ID 100 is dropped, we set the table's state as DELETING and start deleting tablets. For large clusters with large number of tablets, deleting all tablets can take up to a minute. Once all tablets are deleted, we mark the table as DELETED.
Now, it's possible to create a new table foo while the older table is DELETING but not yet DELETED. In the next step, when we create an index on foo, cql server does a lookup to find the indexed (base) table. Here, it finds foo with ID 100 in its cache and sends a request to master to create an index on foo(id=100).
Master goes ahead and creates the index and alters the base table to add the index table to its list of indexes. This causes the table state to change to ALTERING and finally RUNNING once the alter command runs!

There are 2 fixes needed:

1. CreateTable should ensure that indexed table is not in deleting or deleted state.
2. AlterTable should ensure that table being altered in not in deleting or deleted state.

[smalyshev]

The scenario above sounds pretty close to what I did - I had a table for which I've forgotten to create some fields and indexes, so I've deleted it and started to recreate it immediately (I wasn't aware of the complications above then), so I think it matches what is described above.

[OlegLoginov]

Reproduced the main issue - stuck keyspace:

    // Create test table.
    session.execute("create table test_drop (h1 int primary key, " + // TS-1
                    "c1 int, c2 int, c3 int, c4 int, c5 int) " +
                    "with transactions = {'enabled' : true};");
    // Create test indexes.
    session.execute("create index i1 on test_drop (c1);");           // TS-2
    session.execute("create index i2 on test_drop (c2);");           // TS-3
    session.execute("select * from test_drop;");                     // TS-1
    // Drop test table.
    session.execute("drop table test_drop;");                        // TS-2
    // Create test table again.
    session.execute("create table test_drop (h1 int primary key, " + // TS-3
                    "c1 int, a2 int, a3 int, a4 int, a5 int) " +
                    "with transactions = {'enabled' : true};");
    // Create index.
    session.execute("create index i6 on test_drop (c1);");           // TS-1
    session.execute("drop table test_drop;");                        // TS-2
    session.execute("drop keyspace test_ks;");                       // TS-3

Error:

	at org.yb.cql.TestIndex.testDropTableTimeout(TestIndex.java:296)
Caused by: com.datastax.driver.core.exceptions.ServerError: 
An unexpected error occurred server side on /127.204.240.183:9042: Server Error. Cannot delete namespace which has index: i6 [id=36f277befd6243fea6361a9c037e827e]: namespace {
  name: "test_ks"
}
drop keyspace test_ks;
              ^^^^^^^
 (ql error -2)

[ERROR] testDropTableTimeout(org.yb.cql.TestIndex)  Time elapsed: 7.446 s  <<< ERROR!
com.datastax.driver.core.exceptions.InvalidQueryException: 
Object Not Found. The object does not exist: table_name: "test_drop"
namespace {
  name: "test_ks"
  database_type: YQL_DATABASE_CQL
}
DROP TABLE test_ks.test_drop;
                   ^^^^^^^^^
 (ql error -301)

[OlegLoginov]

Due to race conditions between TServers the test result can reference to the wrong Index (above) OR to the table:

Caused by: com.datastax.driver.core.exceptions.ServerError: 
An unexpected error occurred server side on /127.159.161.216:9042: Server Error. Cannot delete namespace which has table: test_drop [id=32c6deec55254066895075c62355e462]: namespace {
  name: "test_ks"
}

drop keyspace test_ks;
              ^^^^^^^
 (ql error -2)

[ERROR] testDropTableTimeout(org.yb.cql.TestIndex)  Time elapsed: 11.544 s  <<< ERROR!
com.datastax.driver.core.exceptions.InvalidQueryException: 
Object Not Found. The object does not exist: table_name: "test_drop"
namespace {
  name: "test_ks"
  database_type: YQL_DATABASE_CQL
}

DROP TABLE test_ks.test_drop;
                   ^^^^^^^^^
 (ql error -301)

In this case we have race between (slow) DeleteTable from TS-1 and DeleteNamespace from TS-2.

[OlegLoginov]

Mentioned above unexpected 'Object Not Found' error as result of DROP TABLE - is tracked by this: #3133

[OlegLoginov]

Fixed by the commit above.