ErrSnapshotSchemaNotFound or ErrSchemaStorageGCed after migrating tables

[liuzix]

Bug Report

This issue is part of the problem described in #1056

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   Run a changefeed on three CDC nodes, with five tables and a total QPS around 5000. The changefeed stops with ErrSnapshotSchemaNotFound or ErrSchemaStorageGCed, after scaling in and scaling out the CDC cluster.

2. What did you expect to see?
   That the changefeed runs normally.

3. What did you see instead?
   The errors ErrSnapshotSchemaNotFound or ErrSchemaStorageGCed.

4. Versions of the cluster
   All components are in v4.0.8.

Cause Analysis

This problem is caused by bugs in handling table migrations. Since stopping a table is not immediate, the checkpoint will proceed a bit after the checkpoint at which the owner started the migration. But the owner has recorded the checkpoint prematurely in replica-info.startTs. When the table is restarted on another capture that previously has not run this changefeed, the SchemaStorage does not have information about the Schemas before the current checkpoint, but the table pullers fetch data from replica-info.startTs, which is before the current checkpoint. Hence the errors.

[liuzix]

My thoughts on how to fix: I'm wondering if we can just let the global checkpoint override the startTs of a table, i.e. we always start the table puller with the current global checkpoint as the starting point. @amyangfei @leoppro

[amyangfei]

My thoughts on how to fix: I'm wondering if we can just let the global checkpoint override the startTs of a table, i.e. we always start the table puller with the current global checkpoint as the starting point. @amyangfei @leoppro

Maybe this bug can be fixed via following ways

1. as you have described, use the max(startTs, globalCheckpointTs to start puller
2. or change the logic of schema snapshot, such as amending missing schema version, which is more complex.

IMO the first solution is all right, we could add an integration test and have a try @liuzix