Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

syncer(dm): fix different output format for operate-schema get (#5824) #5884

Closed
wants to merge 7 commits into from
Closed

syncer(dm): fix different output format for operate-schema get (#5824) #5884

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0996588
This is an automated cherry-pick of #5824
ForwardStar Jun 15, 2022
0fdd386
Fix cherry pick problems
ForwardStar Jun 23, 2022
7227b21
Merge branch 'release-5.4' into cherry-pick-5824-to-release-5.4
ForwardStar Jun 24, 2022
85640c4
Fix unit test errors
ForwardStar Jun 24, 2022
f38420f
Merge branch 'cherry-pick-5824-to-release-5.4' of https://github.com/…
ForwardStar Jun 24, 2022
96bb072
Fix unit test error
ForwardStar Jun 24, 2022
87f9290
Solve linter-test failed problem
ForwardStar Jun 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 38 additions & 13 deletions dm/syncer/schema.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
"bytes"
"context"
"encoding/json"
"fmt"
"strings"

"github.com/pingcap/tidb-tools/pkg/filter"
Expand All @@ -27,7 +28,7 @@ import (
"github.com/pingcap/tidb/parser/format"
"github.com/pingcap/tidb/parser/model"
"github.com/pingcap/tiflow/dm/pkg/utils"
"github.com/pingcap/tiflow/dm/syncer/dbconn"
"github.com/pingcap/tiflow/pkg/quotes"
"go.uber.org/zap"

"github.com/pingcap/tiflow/dm/dm/config"
Expand All @@ -50,29 +51,53 @@ func (s *Syncer) OperateSchema(ctx context.Context, req *pb.OperateWorkerSchemaR
for i, schema := range allSchema {
schemaList[i] = schema.Name.String()
}
schemaListJSON, err := json.Marshal(schemaList)
if err != nil {
return "", terror.ErrSchemaTrackerMarshalJSON.Delegate(err, schemaList)
schemaListJSON, err2 := json.Marshal(schemaList)
if err2 != nil {
return "", terror.ErrSchemaTrackerMarshalJSON.Delegate(err2, schemaList)
}
return string(schemaListJSON), err
return string(schemaListJSON), err2
case pb.SchemaOp_ListTable:
tables, err := s.schemaTracker.ListSchemaTables(req.Database)
if err != nil {
return "", err
tables, err2 := s.schemaTracker.ListSchemaTables(req.Database)
if err2 != nil {
return "", err2
}
tableListJSON, err := json.Marshal(tables)
if err != nil {
return "", terror.ErrSchemaTrackerMarshalJSON.Delegate(err, tables)
tableListJSON, err2 := json.Marshal(tables)
if err2 != nil {
return "", terror.ErrSchemaTrackerMarshalJSON.Delegate(err2, tables)
}
return string(tableListJSON), err
return string(tableListJSON), err2
case pb.SchemaOp_GetSchema:
// when task is paused, schemaTracker is closed. We get the table structure from checkpoint.
ti := s.checkpoint.GetTableInfo(req.Database, req.Table)
if ti == nil {
s.tctx.L().Info("table schema is not in checkpoint, fetch from downstream",
zap.String("table", sourceTable.String()))
targetTable := s.route(sourceTable)
return dbconn.GetTableCreateSQL(s.tctx.WithContext(ctx), s.downstreamTrackConn, targetTable.String())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need to remove the dbconn.GetTableCreateSQL() calling and replace it with SHOW CREATE TABLE calling. But rather:

			result, err2 := dbconn.GetTableCreateSQL(s.tctx.WithContext(ctx), s.downstreamTrackConn, targetTable.String())
			result = strings.Replace(result, fmt.Sprintf("CREATE TABLE %s", quotes.QuoteName(targetTable.Name)), fmt.Sprintf("CREATE TABLE %s", quotes.QuoteName(sourceTable.Name)), 1)
			return utils.CreateTableSQLToOneRow(result), err2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dbconn.GetTableCreateSQL() is not included in the old-version branch. Moreover, the committed logic is not equal to dbconn.GetTableCreateSQL() since conn.QuerySQL() between the versions are of different APIs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Can we reuse (*schema.Tracker).GetCreateTable() ? In my opinion, this PR mainly changes the table name from target table name to source table name, and changes the sql into a single line ( not in this version ). Maybe :

targetTable := s.route(sourceTable)
result, err2 := s.schemaTracker.GetCreateTable(ctx, targetTable)
result = strings.Replace(result, fmt.Sprintf("CREATE TABLE %s", quotes.QuoteName(targetTable.Name)), fmt.Sprintf("CREATE TABLE %s", quotes.QuoteName(sourceTable.Name)), 1)
return result, err2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I am not sure about this. In my opinion, getting the table from schema tracker is an incorrect version. Getting schema can only be operated when task is paused. However, when the task is paused, the schema tracker is down and nothing would return. I have tried to build the original version and observed this phenomenon.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Seems PR #5350 fixed this problem. Does that PR also need to be cherry-picked into 5.4 ? @lance6716

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done by #5415

// Get table create SQL
tableID := targetTable.String()
querySQL := fmt.Sprintf("SHOW CREATE TABLE %s", tableID)
var table, createStr string

rows, err2 := s.downstreamTrackConn.QuerySQL(s.tctx.WithContext(ctx), querySQL)
if err2 != nil {
return "", err2
}

defer rows.Close()
if rows.Next() {
if scanErr := rows.Scan(&table, &createStr); scanErr != nil {
return "", terror.DBErrorAdapt(scanErr, terror.ErrDBDriverError)
}
} else {
return "", terror.ErrSyncerDownstreamTableNotFound.Generate(tableID)
}

if err = rows.Close(); err != nil {
return "", terror.DBErrorAdapt(rows.Err(), terror.ErrDBDriverError)
}

result := strings.Replace(createStr, fmt.Sprintf("CREATE TABLE %s", quotes.QuoteName(targetTable.Name)), fmt.Sprintf("CREATE TABLE %s", quotes.QuoteName(sourceTable.Name)), 1)
return utils.CreateTableSQLToOneRow(result), err2
}

result := bytes.NewBuffer(make([]byte, 0, 512))
Expand Down
46 changes: 46 additions & 0 deletions dm/syncer/syncer_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1024,6 +1024,52 @@ func (s *testSyncerSuite) TestRun(c *C) {
testJobs.RUnlock()

cancel()
<-resultCh // wait for the process to finish

// test OperateSchema starts
ctx, cancel = context.WithCancel(context.Background())

syncer.sessCtx = utils.NewSessionCtx(map[string]string{"time_zone": "UTC"})
sourceSchemaFromCheckPoint, err := syncer.OperateSchema(ctx, &pb.OperateWorkerSchemaRequest{Op: pb.SchemaOp_GetSchema, Database: "test_1", Table: "t_1"})
c.Assert(err, IsNil)

syncer.tableRouter, err = router.NewTableRouter(false, nil)
c.Assert(err, IsNil)
c.Assert(syncer.tableRouter.AddRule(&router.TableRule{
SchemaPattern: "test_1",
TablePattern: "t_1",
TargetSchema: "test_1",
TargetTable: "t_2",
}), IsNil)

syncer.checkpoint.(*RemoteCheckPoint).points = make(map[string]map[string]*binlogPoint)

showTableResultString := "CREATE TABLE `t_2` (\n" +
" `id` int(11) NOT NULL,\n" +
" `name` varchar(24) DEFAULT NULL,\n" +
" PRIMARY KEY (`id`) /*T![clustered_index] NONCLUSTERED */,\n" +
" KEY `index1` (`name`)\n" +
") ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin"

mock.ExpectQuery("SHOW CREATE TABLE " + "`test_1`.`t_2`").WillReturnRows(
sqlmock.NewRows([]string{"Table", "Create Table"}).
AddRow("t_2", showTableResultString))

sourceSchemaFromDownstream, err := syncer.OperateSchema(ctx, &pb.OperateWorkerSchemaRequest{Op: pb.SchemaOp_GetSchema, Database: "test_1", Table: "t_1"})
c.Assert(err, IsNil)

sourceSchemaExpected := "CREATE TABLE `t_1` (" +
" `id` int(11) NOT NULL," +
" `name` varchar(24) DEFAULT NULL," +
" PRIMARY KEY (`id`) /*T![clustered_index] NONCLUSTERED */," +
" KEY `index1` (`name`)" +
") ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin"
c.Assert(sourceSchemaFromCheckPoint, Equals, sourceSchemaExpected)
c.Assert(sourceSchemaFromDownstream, Equals, sourceSchemaExpected)

cancel()
// test OperateSchema ends

syncer.Close()
c.Assert(syncer.isClosed(), IsTrue)

Expand Down