pkg/sink(ticdc): iterate all Kafka configs to support KOP (#8893)

[ti-chi-bot]

This is an automated cherry-pick of #8893

What problem does this PR solve?

Issue Number: close #8892

What is changed and how it works?

Iterate all Kafka configs to support KOP.

Check List

Tests

• Manual test(see this doc)

Questions

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

No

Release note

 Iterate all config items to support KOP when getting the broker's configs

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[wuhuizuo]

/test verify

maybe it's a bug in prow, the status is passed, but no link reported.

[ti-chi-bot]

@wuhuizuo: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

• /test verify

Use /test all to run all jobs.

In response to this:

/test verify

maybe it's a bug in prow, the status is passed, but no link reported.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wuhuizuo]

/test verify

[Rustin170506]

Tested locally:

1. created success

➜  bin git:(pr/ti-chi-bot/8904) ✗ ./cdc cli changefeed create --sink-uri="kafka://127.0.0.1:9092/topic1?max-message-bytes=1048576&protocol=open-protocol&kafka-version=2.0.0&replication-factor=1" --config=changefeed.toml
[2023/05/23 17:53:51.355 +08:00] [WARN] [kafka.go:434] ["topic's `max.message.bytes` less than the `max-message-bytes`,use topic's `max.message.bytes` to initialize the Kafka producer"] [max.message.bytes=1000012] [max-message-bytes=1048576]
[2023/05/23 17:53:51.355 +08:00] [WARN] [kafka.go:443] ["topic already exist, TiCDC will not create the topic"] [topic=topic1] [detail="{\"NumPartitions\":1,\"ReplicationFactor\":1,\"ReplicaAssignment\":{\"0\":[1001]},\"ConfigEntries\":{\"segment.bytes\":\"1073741824\"}}"]
Create changefeed successfully!
ID: afd0b7d5-41dc-4abe-be57-24b21965cfb1
Info: {"upstream-id":0,"sink-uri":"kafka://127.0.0.1:9092/topic1?max-message-bytes=1048576\u0026protocol=open-protocol\u0026kafka-version=2.0.0\u0026replication-factor=1","opts":{},"create-time":"2023-05-23T17:53:51.097951839+08:00","start-ts":441669551664398337,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"case-sensitive":true,"enable-old-value":true,"force-replicate":false,"check-gc-safe-point":true,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null},"mounter":{"worker-num":16},"sink":{"transaction-atomicity":"","protocol":"open-protocol","dispatchers":[{"matcher":["test.*","test2.*"],"dispatcher":"","partition":"ts","topic":""}],"column-selectors":null,"schema-registry":"","encoder-concurrency":16,"kafka-config":{"sasl-mechanism":"OAUTHBEARER","sasl-oauth-client-id":"producer-kafka","sasl-oauth-client-secret":"cHJvZHVjZXIta2Fma2E=","sasl-oauth-token-url":"http://127.0.0.1:4444/oauth2/token","sasl-oauth-scopes":["producer.kafka"],"sasl-oauth-grant-type":"client_credentials"}},"cyclic-replication":{"enable":false,"replica-id":0,"filter-replica-ids":null,"id-buckets":0,"sync-ddl":false},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"storage":""}},"state":"normal","error":null,"sync-point-enabled":false,"sync-point-interval":600000000000,"creator-version":"v6.1.0-master-dirty"}

2. get the message from consumer:

{ topic: 'topic1',
  value: '\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000',
  offset: 426,
  partition: 0,
  highWaterOffset: 427,
  key:
   '\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u001f{"ts":441669557497102337,"t":3}',
  timestamp: 2023-05-23T09:54:15.196Z }

[Rustin170506]

/run-all-tests

[purelind]

/run-dm-compatibility-tests
/run-dm-integration-tests

[Rustin170506]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 21ea26d

[Rustin170506]

/run-all-tests

[Rustin170506]

/run-integration-test
/run-kafka-integration-test

[Rustin170506]

/run-kafka-integration-test

[ti-chi-bot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by release team.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick, it must first be approved by the collaborators.

AFTER it has been approved by collaborators, please ping the release team in a comment to request a cherry pick review.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.