tls(dm): support init tls without client kay/certs #5393

Ehco1996 · 2022-05-11T08:36:38Z

What problem does this PR solve?

Issue Number: close #5391 #5335

What is changed and how it works?

update tidb version and ajust new API of FieldType see more in *: Refactor FieldType structure tidb#33569
use tidb/util to init tls connection

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

start a source with tls and enable relay

source-id: mysql-replica-01-tls
flavor: ""
enable-gtid: true
enable-relay: true
relay-binlog-name: ""
relay-binlog-gtid: ""
from:
  host: localhost
  user: dm_tls_test
  password: 123456
  port: 3306
  security:
    ssl-ca: "/Users/ehco/Library/Application\ Support/com.tinyapp.DBngin/Engines/mysql/5E575342-71D5-409E-9080-E72881FD0008/ca.pem"
checker:
  check-enable: false
  backoff-rollback: 5m
  backoff-max: 5m

purge:
  interval: 10
  expires: 1
  remain-space: 15

start relay for this source

❯ dmctl query-status -s mysql-replica-01-tls
{
    "result": true,
    "msg": "",
    "sources": [
        {
            "result": true,
            "msg": "no sub task started",
            "sourceStatus": {
                "source": "mysql-replica-01-tls",
                "worker": "worker1",
                "result": null,
                "relayStatus": {
                    "masterBinlog": "(mysql-bin.000001, 10636)",
                    "masterBinlogGtid": "cd3cf208-edc0-11eb-852c-d674c270ea47:1-45",
                    "relaySubDir": "cd3cf208-edc0-11eb-852c-d674c270ea47.000001",
                    "relayBinlog": "(mysql-bin.000001, 10636)",
                    "relayBinlogGtid": "cd3cf208-edc0-11eb-852c-d674c270ea47:1-45",
                    "relayCatchUpMaster": true,
                    "stage": "Running",
                    "result": null
                }
            },
            "subTaskStatus": [
            ]
        }
    ]

3 check if the connection from dm-worker is using tls

mysql> show processlist;
+------+-------------+-----------------+------+------------------+------+---------------------------------------------------------------+------------------+
| Id   | User        | Host            | db   | Command          | Time | State                                                         | Info             |
+------+-------------+-----------------+------+------------------+------+---------------------------------------------------------------+------------------+
| 1051 | dm_tls_test | localhost:60272 | NULL | Binlog Dump GTID |   80 | Master has sent all binlog to slave; waiting for more updates | NULL             |
| 1055 | root        | localhost:60310 | NULL | Query            |    0 | starting                                                      | show processlist |
+------+-------------+-----------------+------+------------------+------+---------------------------------------------------------------+------------------+
2 rows in set (0.00 sec)



mysql> SELECT * FROM performance_schema.threads WHERE processlist_id = 1051\G
*************************** 1. row ***************************
          THREAD_ID: 1076
               NAME: thread/sql/one_connection
               TYPE: FOREGROUND
     PROCESSLIST_ID: 1051
   PROCESSLIST_USER: dm_tls_test
   PROCESSLIST_HOST: localhost
     PROCESSLIST_DB: NULL
PROCESSLIST_COMMAND: Binlog Dump GTID
   PROCESSLIST_TIME: 156
  PROCESSLIST_STATE: Master has sent all binlog to slave; waiting for more updates
   PROCESSLIST_INFO: NULL
   PARENT_THREAD_ID: NULL
               ROLE: NULL
       INSTRUMENTED: YES
            HISTORY: YES
    CONNECTION_TYPE: SSL/TLS
       THREAD_OS_ID: 12477417
1 row in set (0.01 sec)

check MySQL tls version

mysql> SHOW GLOBAL VARIABLES LIKE 'tls_version';
+---------------+---------------+
| Variable_name | Value         |
+---------------+---------------+
| tls_version   | TLSv1,TLSv1.1 |
+---------------+---------------+
1 row in set (0.00 sec)

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update key monitor metrics in both TiCDC document and official document

Release note

 `fix a bug that can't establish a tls connection when only ssl-ca is set`.

ti-chi-bot · 2022-05-11T08:36:40Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

GMHDBJD
liuzix

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Ehco1996 · 2022-05-12T02:40:35Z

also need update https://github.com/PingCAP-QE/go-sqlsmith

Ehco1996 · 2022-05-17T07:20:46Z

/run-all-tests

Ehco1996 · 2022-05-28T08:13:30Z

/run-all-tests

Ehco1996 · 2022-05-28T08:27:20Z

dm/tests/_utils/wait_process_exit

@@ -4,7 +4,7 @@
 process=$1

 WAIT_COUNT=0
-while [ $WAIT_COUNT -lt 30 ]; do
+while [ $WAIT_COUNT -lt 120 ]; do


[2022/05/28 10:27:52.614 +08:00] [INFO] [main.go:84] ["got signal to exit"] [signal=hangup] [2022/05/28 10:27:52.614 +08:00] [INFO] [server.go:281] ["closing server"] [2022/05/28 10:27:52.614 +08:00] [INFO] [election.go:325] ["current member is not the leader"] [component=election] ["current member"="{\"id\":\"master1\",\"addr\":\"127.0.0.1:8261\"}"] [leader="{\"id\":\"master3\",\"addr\":\"127.0.0.1:8461\"}"] [2022/05/28 10:27:52.614 +08:00] [WARN] [election.go:359] ["ignore notify the leader's information because context canceled"] [component=election] ["current member"="{\"id\":\"master1\",\"addr\":\"127.0.0.1:8261\"}"] [leader="{\"id\":\"master3\",\"addr\":\"127.0.0.1:8461\"}"] [2022/05/28 10:27:52.615 +08:00] [INFO] [election.go:190] ["election is closing"] [component=election] ["current member"="{\"id\":\"master1\",\"addr\":\"127.0.0.1:8261\"}"] [2022/05/28 10:27:52.616 +08:00] [DEBUG] [watch.go:191] ["failed to receive watch request from gRPC stream"] [component="embed etcd"] [error="rpc error: code = Canceled desc = context canceled"] [2022/05/28 10:27:52.620 +08:00] [DEBUG] [store.go:1133] ["found command name" [2022/05/28 10:28:52.783 +08:00] [DEBUG] [peer_status.go:76] ["peer deactivated again"] [component="embed etcd"] [peer-id=7315b5b7a5c31dc] [error="failed to dial 7315b5b7a5c31dc on stream MsgApp v2 (dial tcp 127.0.0.1:8292: connect: connection refused)"] [2022/05/28 10:28:52.852 +08:00] [DEBUG] [interceptor.go:182] ["request stats"] [component="embed etcd"] ["start time"=2022/05/28 10:28:49.130 +08:00] ["time spent"=3.722337167s] [remote=127.0.0.1:50042] ["response type"=/etcdserverpb.Lease/LeaseRevoke] ["request count"=-1] ["request size"=-1] ["response count"=-1] ["response size"=-1] ["request content"=] [2022/05/28 10:28:52.853 +08:00] [ERROR] [election.go:205] ["fail to close etcd session"] [component=election] [lease=152419835867964164] [error="context deadline exceeded"] [2022/05/28 10:28:52.853 +08:00] [INFO] [election.go:198] ["election is closed"] [component=election] ["current member"="{\"id\":\"master1\",\"addr\":\"127.0.0.1:8261\"}"] [2022/05/28 10:28:52.854 +08:00] [INFO] [etcd.go:368] ["closing etcd server"] [component="embed etcd"] [name=master1] [data-dir=default.master1] [advertise-peer-urls="[https://127.0.0.1:8291]"] [advertise-client-urls="[https://127.0.0.1:8261]"] [2022/05/28 10:28:52.856 +08:00] [INFO] [server.go:1433] ["skipped leadership transfer; local server is not leader"] [component="embed etcd"] [local-member-id=8bbb35d21267021d] [current-leader-member-id=0] [2022/05/28 10:28:52.856 +08:00] [INFO] [peer.go:330] ["stopping remote peer"] [component="embed etcd"] [remote-peer-id=5e5ef2613797764c] [2022/05/28 10:28:52.857 +08:00] [INFO] [stream.go:294] ["stopped TCP streaming connection with remote peer"] [component="embed etcd"] [stream-writer-type="stream MsgApp v2"] [remote-peer-id=5e5ef2613797764c] [2022/05/28 10:28:52.857 +08:00] [INFO] [stream.go:294] ["stopped TCP streaming connection with remote peer"] [component="embed etcd"] [stream-writer-type="stream Message"] [remote-peer-id=5e5ef2613797764c] [2022/05/28 10:28:52.857 +08:00] [INFO] [pipeline.go:85] ["stopped HTTP pipelining with remote peer"] [component="embed etcd"] [local-member-id=8bbb35d21267021d] [remote-peer-id=5e5ef2613797764c] [2022/05/28 10:28:52.857 +08:00] [INFO] [stream.go:442] ["stopped stream reader with remote peer"] [component="embed etcd"] [stream-reader-type="stream MsgApp v2"] [local-member-id=8bbb35d21267021d] [remote-peer-id=5e5ef2613797764c] [2022/05/28 10:28:52.858 +08:00] [INFO] [stream.go:442] ["stopped stream reader with remote peer"] [component="embed etcd"] [stream-reader-type="stream Message"] [local-member-id=8bbb35d21267021d] [remote-peer-id=5e5ef2613797764c] [2022/05/28 10:28:52.858 +08:00] [INFO] [peer.go:335] ["stopped remote peer"] [component="embed etcd"] [remote-peer-id=5e5ef2613797764c] [2022/05/28 10:28:52.858 +08:00] [INFO] [peer.go:330] ["stopping remote peer"] [component="embed etcd"] [remote-peer-id=7315b5b7a5c31dc] [2022/05/28 10:28:52.858 +08:00] [INFO] [stream.go:294] ["stopped TCP streaming connection with remote peer"] [component="embed etcd"] [stream-writer-type="stream MsgApp v2"] [remote-peer-id=7315b5b7a5c31dc] [2022/05/28 10:28:52.858 +08:00] [INFO] [stream.go:294] ["stopped TCP streaming connection with remote peer"] [component="embed etcd"] [stream-writer-type="stream Message"] [remote-peer-id=7315b5b7a5c31dc] [2022/05/28 10:28:52.858 +08:00] [INFO] [pipeline.go:85] ["stopped HTTP pipelining with remote peer"] [component="embed etcd"] [local-member-id=8bbb35d21267021d] [remote-peer-id=7315b5b7a5c31dc] [2022/05/28 10:28:52.859 +08:00] [INFO] [stream.go:442] ["stopped stream reader with remote peer"] [component="embed etcd"] [stream-reader-type="stream MsgApp v2"] [local-member-id=8bbb35d21267021d] [remote-peer-id=7315b5b7a5c31dc] [2022/05/28 10:28:52.859 +08:00] [INFO] [stream.go:442] ["stopped stream reader with remote peer"] [component="embed etcd"] [stream-reader-type="stream Message"] [local-member-id=8bbb35d21267021d] [remote-peer-id=7315b5b7a5c31dc] [2022/05/28 10:28:52.859 +08:00] [INFO] [peer.go:335] ["stopped remote peer"] [component="embed etcd"] [remote-peer-id=7315b5b7a5c31dc] [2022/05/28 10:28:52.921 +08:00] [INFO] [etcd.go:563] ["stopping serving peer traffic"] [component="embed etcd"] [address=127.0.0.1:8291] [2022/05/28 10:28:52.923 +08:00] [INFO] [etcd.go:568] ["stopped serving peer traffic"] [component="embed etcd"] [address=127.0.0.1:8291] [2022/05/28 10:28:52.923 +08:00] [INFO] [etcd.go:370] ["closed etcd server"] [component="embed etcd"] [name=master1] [data-dir=default.master1] [advertise-peer-urls="[https://127.0.0.1:8291]"] [advertise-client-urls="[https://127.0.0.1:8261]"] [2022/05/28 10:28:52.923 +08:00] [INFO] [server.go:283] ["server closed"] [2022/05/28 10:28:52.923 +08:00] [INFO] [main.go:91] ["dm-master exit"]

30s is not enough for kill dm-master

From the logs, we can see that it may take up to 60s to fully exit dm-master

OK we can take a look later

Ehco1996 · 2022-05-28T08:36:28Z

/run-dm-integration-tests

Ehco1996 · 2022-05-28T08:59:57Z

/run-dm-integration-tests

Ehco1996 · 2022-05-28T09:16:38Z

/run-dm-integration-tests

Ehco1996 · 2022-05-28T09:28:47Z

/run-dm-integration-tests

Ehco1996 · 2022-05-28T09:49:52Z

/run-dm-integration-tests

GMHDBJD

LGTM

codecov-commenter · 2022-05-28T10:33:51Z

Codecov Report

Merging #5393 (5c4f38d) into master (fcea4d5) will increase coverage by 0.4811%.
The diff coverage is 59.5103%.

Flag	Coverage Δ
cdc	`62.0498% <62.2047%> (+0.9016%)`	⬆️
dm	`52.0489% <48.9164%> (+0.0062%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #5393        +/-   ##
================================================
+ Coverage   56.0768%   56.5579%   +0.4811%     
================================================
  Files           535        671       +136     
  Lines         70143      78431      +8288     
================================================
+ Hits          39334      44359      +5025     
- Misses        27078      29914      +2836     
- Partials       3731       4158       +427

lance6716

lgtm

lance6716 · 2022-05-28T11:00:53Z

dm/tests/_utils/wait_process_exit

@@ -4,7 +4,7 @@
 process=$1

 WAIT_COUNT=0
-while [ $WAIT_COUNT -lt 30 ]; do
+while [ $WAIT_COUNT -lt 120 ]; do


OK we can take a look later

Ehco1996 · 2022-05-28T11:41:31Z

/merge

ti-chi-bot · 2022-05-28T11:41:34Z

This pull request has been accepted and is ready to merge.

Commit hash: 5c4f38d

Ehco1996 · 2022-05-28T12:05:07Z

/run-dm-integration-tests

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2022-05-28T12:33:26Z

In response to a cherrypick label: new pull request created: #5626.

ref #5335, close #5391

overvenus · 2022-05-30T05:23:34Z

go.mod

+	github.com/pingcap/tidb v1.1.0-beta.0.20220528045048-5495dc6c4360
+	github.com/pingcap/tidb-tools v6.0.1-0.20220516050036-b3ea358e374a+incompatible
+	github.com/pingcap/tidb/parser v0.0.0-20220528045048-5495dc6c4360
+	github.com/prometheus/client_golang v1.12.2


Conflicts to #5494 ?

Need check：

Check If we have used following three metric. If true, need modify the grafna rules in this pr.

go_gc_heap_allocs_by_size_bytes_total -> go_gc_heap_allocs_by_size_bytes, go_gc_heap_frees_by_size_bytes_total -> go_gc_heap_allocs_by_size_bytes go_gc_pauses_seconds_total -> go_gc_pauses_seconds.

Check if we use NewGoCollection(). If true, add GoRuntimeMemStatsCollection | GoRuntimeMetricsCollection in this pr.

save work

602e14c

Ehco1996 force-pushed the ehco-update-tidb branch from 391bf71 to 602e14c Compare May 11, 2022 08:40

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 11, 2022

Ehco1996 added 3 commits May 11, 2022 16:53

finish dm

7b0e807

save

390c7e0

save work

15f5ec5

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed do-not-merge/needs-triage-completed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 11, 2022

finish cdc

29eaa66

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 12, 2022

Ehco1996 added 5 commits May 17, 2022 10:39

Merge branch 'master' into ehco-update-tidb

dcd1040

update mod

bf1d7b6

fix mod

6d562b2

fix a test

b2ec760

fix more

665e2dd

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 17, 2022

Ehco1996 added 2 commits May 23, 2022 10:37

Merge branch 'master' into ehco-update-tidb

cb913eb

fix mode

2f67751

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 23, 2022

Merge branch 'master' into ehco-update-tidb

08a01b4

incr wait process exit

5c4f38d

Ehco1996 commented May 28, 2022

View reviewed changes

GMHDBJD approved these changes May 28, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 28, 2022

liuzix approved these changes May 28, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 28, 2022

lance6716 reviewed May 28, 2022

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 28, 2022

Ehco1996 added the needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. label May 28, 2022

ti-chi-bot merged commit 3faaeb6 into pingcap:master May 28, 2022

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request May 28, 2022

This is an automated cherry-pick of pingcap#5393

4698403

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request May 28, 2022

tls(dm): support init tls without client key/certs (#5393) #5626

Merged

ti-chi-bot added a commit that referenced this pull request May 29, 2022

tls(dm): support init tls without client key/certs (#5393) (#5626)

8b7808f

ref #5335, close #5391

Ehco1996 mentioned this pull request May 30, 2022

regression: Support TLS connections when only ssl-ca is set #5335

Closed

overvenus reviewed May 30, 2022

View reviewed changes

This was referenced May 30, 2022

Revert prometheus client_golang from from v1.12.2 to v1.11.0 #5494

Closed

metrics(*): fix for promtheus_client breaking change #5643

Merged

ti-chi-bot mentioned this pull request May 30, 2022

metrics(*): fix for promtheus_client breaking change (#5643) #5653

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tls(dm): support init tls without client kay/certs #5393

tls(dm): support init tls without client kay/certs #5393

Ehco1996 commented May 11, 2022 •

edited

Loading

ti-chi-bot commented May 11, 2022 •

edited

Loading

Ehco1996 commented May 12, 2022

Ehco1996 commented May 17, 2022

Ehco1996 commented May 28, 2022

Ehco1996 May 28, 2022 •

edited

Loading

lance6716 May 28, 2022

Ehco1996 commented May 28, 2022

Ehco1996 commented May 28, 2022

Ehco1996 commented May 28, 2022

Ehco1996 commented May 28, 2022

Ehco1996 commented May 28, 2022

GMHDBJD left a comment

codecov-commenter commented May 28, 2022 •

edited

Loading

lance6716 left a comment

lance6716 May 28, 2022

Ehco1996 commented May 28, 2022

ti-chi-bot commented May 28, 2022

Ehco1996 commented May 28, 2022

ti-chi-bot commented May 28, 2022

overvenus May 30, 2022

maxshuang May 30, 2022 •

edited

Loading

tls(dm): support init tls without client kay/certs #5393

tls(dm): support init tls without client kay/certs #5393

Conversation

Ehco1996 commented May 11, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented May 11, 2022 • edited Loading

Ehco1996 commented May 12, 2022

Ehco1996 commented May 17, 2022

Ehco1996 commented May 28, 2022

Ehco1996 May 28, 2022 • edited Loading

Choose a reason for hiding this comment

lance6716 May 28, 2022

Choose a reason for hiding this comment

Ehco1996 commented May 28, 2022

Ehco1996 commented May 28, 2022

Ehco1996 commented May 28, 2022

Ehco1996 commented May 28, 2022

Ehco1996 commented May 28, 2022

GMHDBJD left a comment

Choose a reason for hiding this comment

codecov-commenter commented May 28, 2022 • edited Loading

Codecov Report

lance6716 left a comment

Choose a reason for hiding this comment

lance6716 May 28, 2022

Choose a reason for hiding this comment

Ehco1996 commented May 28, 2022

ti-chi-bot commented May 28, 2022

Ehco1996 commented May 28, 2022

ti-chi-bot commented May 28, 2022

overvenus May 30, 2022

Choose a reason for hiding this comment

maxshuang May 30, 2022 • edited Loading

Choose a reason for hiding this comment

Ehco1996 commented May 11, 2022 •

edited

Loading

ti-chi-bot commented May 11, 2022 •

edited

Loading

Ehco1996 May 28, 2022 •

edited

Loading

codecov-commenter commented May 28, 2022 •

edited

Loading

maxshuang May 30, 2022 •

edited

Loading