Cherry pick bugfix commits for Release 2.4.2 #2591

GOODBOY008 · 2023-10-30T03:41:20Z

Cherry pick bugfix commits for Release 2.4.2.

GOODBOY008 · 2023-10-30T03:44:39Z

@Jiabao-Sun Becasuse the first commit depends on the under three commits, so I add the under three commit to 2.4.2.

leonardBang

Thanks @GOODBOY008 for the great work, I left some comments about API change commits, I think we should only introduce the bugfix commits for bugfix version.

leonardBang · 2023-10-30T05:20:37Z

...rverica/cdc/connectors/mongodb/table/MongoDBConnectorFullChangelogDeserializationSchema.java

+ * produces full changelog mode structure {@link RowData}.
+ */
+@PublicEvolving
+public class MongoDBConnectorFullChangelogDeserializationSchema


This is a feature with public API change, I'm not sure we can pick this or not, I tend to skip this one

Thanks @leonardBang, it makes sense to me.
The supports of MongoDB 6.0 will indeed affect the API, but it will maintain compatibility with the old version under the default configuration. I tend to skip this one as well.

cc @GOODBOY008.

leonardBang · 2023-10-30T05:22:15Z

...es-cdc/src/main/java/com/ververica/cdc/connectors/postgres/source/PostgresSourceBuilder.java

+     * <p>See more
+     * https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished.
+     */
+    public PostgresSourceBuilder<T> closeIdleReaders(boolean closeIdleReaders) {


This should be a API change feature?

There is no API change, here is something omitted here at 2.3

leonardBang · 2023-10-30T05:24:10Z

flink-connector-vitess-cdc/src/main/java/com/ververica/cdc/connectors/vitess/VitessSource.java

-        /** VTCtld server config. */
-        public Builder<T> vtctldConfig(VtctldConfig vtctldConfig) {
-            this.vtctldConfig = vtctldConfig;
+        /**


Looks like we pick some features

These changes belong to [vitess] Migrate connector properties to debezium 1.9.7 (#2260). It's ok to merge this commit.

I understand your concern and drop this commit.

GOODBOY008 · 2023-10-30T07:05:48Z

@Jiabao-Sun Becasuse the first commit depends on the under three commits, so I add the under three commit to 2.4.2.

@leonardBang About mongodb hotfix, I added under three commits because upper bugfix depend on them. I confirmed with @Jiabao-Sun , he think it's ok to add this three commits.

leonardBang · 2023-10-30T07:12:37Z

@GOODBOY008 @Jiabao-Sun Keep API compatible is pretty important, we need ensure that all users who use Flink CDC 2.4.0& 2.4.1 can bump to Flink CDC 2.4.2 without any SQL/code change, I'm not sure this PR would break this constraint or not, looks like this PR would break this?

…xx = database, schema or table) (apache#2212) (cherry picked from commit ba5fcbc)

(cherry picked from commit 0714688)

This closes apache#2334. (cherry picked from commit 883b684)

…m for Oracle CDC Connector This closes apache#2325. Co-authored-by: skylines <34996528+gtk96@users.noreply.github.com> (cherry picked from commit 65bdbfd)

(cherry picked from commit 6c2fe08)

(cherry picked from commit bca60a4)

(cherry picked from commit 0b9ca59)

This closes apache#2345. (cherry picked from commit 61b9a2a)

…ailedSource (apache#2384) * [hotfix][mysql] fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource * [hotfix][mysql] fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource (cherry picked from commit b8f468e)

This closes #apache#2362. Co-authored-by: TJX2014 <tangjinxin@deepexi.com> (cherry picked from commit 094fff4)

…pleTest#testConsumingAllEvents (apache#2364) (cherry picked from commit a3ddc48)

apache#2388) Co-authored-by: DESKTOP-TQDONF3\狗狗jian <zhangjian@adream.org> (cherry picked from commit f7df47e)

…ner reuse bug (apache#2394) (cherry picked from commit 90ed963)

…tikv disk full` failure (apache#2402) (cherry picked from commit 8098c82)

… tables (apache#2160) (cherry picked from commit 92aca18)

(cherry picked from commit 420fe11)

… parsing MySQL DDL (apache#2418) (cherry picked from commit 8e7378d)

* [sqlserver] Fix backfill stream task hang * [sqlserver] Fix backfill stream task hang (cherry picked from commit 6fa02ef)

…#2369) Co-authored-by: pgm-rookie <669768751@qq.com> (cherry picked from commit 54df7e9)

* [oracle] Correct the naming error * Fix comments * fix name error (cherry picked from commit 8feb51b)

…able name with dot (apache#2443) (cherry picked from commit 7801ac9)

…pache#2399) (cherry picked from commit 84e97df)

… calculation This closes apache#2421. Co-authored-by: Leonard Xu <leonard@apache.org> (cherry picked from commit dddab4d)

(cherry picked from commit 07db169)

(cherry picked from commit 7172695)

…stream split (apache#2425) (cherry picked from commit 82e3053)

(cherry picked from commit 928ccf1)

GOODBOY008 · 2023-10-30T10:09:50Z

@Jiabao-Sun I just pick the two commits, PTAL.

) (cherry picked from commit aec2a91)

lvyanquan · 2023-10-31T02:40:25Z

LGTM, +1

leonardBang · 2023-10-31T07:40:17Z

The CI failed even I triggered retry, @GOODBOY008 @lvyanquan could you take a look ?

GOODBOY008 · 2023-10-31T07:47:27Z

The CI failed even I triggered retry, @GOODBOY008 @lvyanquan could you take a look ?

Yes,I am trying to find out reason.

(cherry picked from commit 2f79b24)

(cherry picked from commit 6f9a064)

leonardBang

Thank all for this PR, especially thanks @GOODBOY008 for preparing the PR and
troubleshooting the CI failure, LGTM

GOODBOY008 requested review from Jiabao-Sun and leonardBang October 30, 2023 03:47

leonardBang reviewed Oct 30, 2023

View reviewed changes

GOODBOY008 force-pushed the release-2.4 branch from d239f72 to 473e749 Compare October 30, 2023 07:17

senlizishi and others added 23 commits October 30, 2023 15:39

[hotfix] update dbz option name 'xx. whitelist' to 'xx.include.list'(…

8352936

…xx = database, schema or table) (apache#2212) (cherry picked from commit ba5fcbc)

[build] Add vcs.xml for IDEA (apache#2064)

a4ba02a

(cherry picked from commit 0714688)

[mongodb] [tests] Loosen E2E timeout limitations

b1b0825

This closes apache#2334. (cherry picked from commit 883b684)

[docs][oracle] Add an example of Incremental Snapshot based DataStrea…

f6f13c1

…m for Oracle CDC Connector This closes apache#2325. Co-authored-by: skylines <34996528+gtk96@users.noreply.github.com> (cherry picked from commit 65bdbfd)

[doc] fix errors in PostgresParallelSourceExample (apache#2324)

60483e8

(cherry picked from commit 6c2fe08)

[polish] Add type parameter for MySqlRecords(apache#2300)

4ea4aa0

(cherry picked from commit bca60a4)

[mongodb][hotfix] Fix SourceRecord range check (apache#2343)

980f1b1

(cherry picked from commit 0b9ca59)

[db2] [tests] Fix unstable Db2ConnectorITCase due to cleanup failure

6855aec

This closes apache#2345. (cherry picked from commit 61b9a2a)

[hotfix][mysql] Catch underlying throwable in snapshot reading.

9f4e65f

This closes #apache#2362. Co-authored-by: TJX2014 <tangjinxin@deepexi.com> (cherry picked from commit 094fff4)

[test][mysql] Provide the server-time-zone setting in MySqlSourceExam…

76f9f39

…pleTest#testConsumingAllEvents (apache#2364) (cherry picked from commit a3ddc48)

[mysql] quota the field names when executing mysql query (apache#2381) (

2476638

apache#2388) Co-authored-by: DESKTOP-TQDONF3\狗狗jian <zhangjian@adream.org> (cherry picked from commit f7df47e)

[test][postgres] Close postgres containers after tests and fix contai…

70a2bec

…ner reuse bug (apache#2394) (cherry picked from commit 90ed963)

[test] Fix TiDBE2eITCase by updating the container version to avoid `…

cf390f9

…tikv disk full` failure (apache#2402) (cherry picked from commit 8098c82)

[mysql] Filter databases that do not need to be read when discovering…

0c2cbd0

… tables (apache#2160) (cherry picked from commit 92aca18)

[Doc] Use redo log instead of binlog in Oracle document (apache#2408)

06a521e

(cherry picked from commit 420fe11)

[hotfix][mysql] Trim default value for BIGINT and SMALLINT types when…

dbd33e6

… parsing MySQL DDL (apache#2418) (cherry picked from commit 8e7378d)

[hotfix][sqlserver] Fix backfill stream task hang (apache#2374)

a6e4380

* [sqlserver] Fix backfill stream task hang * [sqlserver] Fix backfill stream task hang (cherry picked from commit 6fa02ef)

[sqlserver] Add table filter to speed up SqlServerSchema read (apache…

0fe1cbd

…#2369) Co-authored-by: pgm-rookie <669768751@qq.com> (cherry picked from commit 54df7e9)

[oracle] Correct the naming error (apache#2405)

29816f9

* [oracle] Correct the naming error * Fix comments * fix name error (cherry picked from commit 8feb51b)

[cdc-base] Fix parsing error when serializing and deserializing the t…

b6d87d4

…able name with dot (apache#2443) (cherry picked from commit 7801ac9)

[mysql] Add finished unack splits to state for the MysqlSourceReader (a…

7d5e197

…pache#2399) (cherry picked from commit 84e97df)

[hotfix][mysql] Keep assigned splits in order to fix wrong meta group…

f6173f2

… calculation This closes apache#2421. Co-authored-by: Leonard Xu <leonard@apache.org> (cherry picked from commit dddab4d)

wuzhenhua01 and others added 4 commits October 30, 2023 15:40

[postgres] Close idle readers when snapshot finished (apache#2400)

cdc5990

(cherry picked from commit 07db169)

[cdc-base] Fix TM hangs caused by uncaught exception (apache#2511)

929f853

(cherry picked from commit 7172695)

[postgres] Close jdbc connection after creating replication slot for …

039b2ab

…stream split (apache#2425) (cherry picked from commit 82e3053)

[postgres] Not drop replication slot for stream split (apache#2436)

31b7b79

(cherry picked from commit 928ccf1)

GOODBOY008 force-pushed the release-2.4 branch from 473e749 to 31b7b79 Compare October 30, 2023 07:41

[e2e] Remove db docker image after test cases are executed. (apache#2508

92e642f

) (cherry picked from commit aec2a91)

GOODBOY008 force-pushed the release-2.4 branch from e850811 to 92e642f Compare October 30, 2023 10:28

mergify bot mentioned this pull request Oct 30, 2023

[TEST] [WIP] reproduce oracle issue #2167

Closed

GOODBOY008 added 3 commits October 31, 2023 16:57

[hotfix] Fix vitess docker image tag

e02cac6

[sqlserver] Add support bigint type as table split column (apache#2416)

0ba8626

(cherry picked from commit 2f79b24)

[sqlserver] Fix get latest lsn (apache#2551)

e8fca81

(cherry picked from commit 6f9a064)

leonardBang approved these changes Oct 31, 2023

View reviewed changes

leonardBang merged commit 80cd840 into apache:release-2.4 Oct 31, 2023

GOODBOY008 deleted the release-2.4 branch January 11, 2024 15:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry pick bugfix commits for Release 2.4.2 #2591

Cherry pick bugfix commits for Release 2.4.2 #2591

GOODBOY008 commented Oct 30, 2023

GOODBOY008 commented Oct 30, 2023

leonardBang left a comment

leonardBang Oct 30, 2023

Jiabao-Sun Oct 30, 2023

Jiabao-Sun Oct 30, 2023

leonardBang Oct 30, 2023

Jiabao-Sun Oct 30, 2023

leonardBang Oct 30, 2023

GOODBOY008 Oct 30, 2023 •

edited

Loading

GOODBOY008 Oct 30, 2023

GOODBOY008 commented Oct 30, 2023

leonardBang commented Oct 30, 2023 •

edited

Loading

GOODBOY008 commented Oct 30, 2023

lvyanquan commented Oct 31, 2023

leonardBang commented Oct 31, 2023

GOODBOY008 commented Oct 31, 2023

leonardBang left a comment

Cherry pick bugfix commits for Release 2.4.2 #2591

Cherry pick bugfix commits for Release 2.4.2 #2591

Conversation

GOODBOY008 commented Oct 30, 2023

GOODBOY008 commented Oct 30, 2023

leonardBang left a comment

Choose a reason for hiding this comment

leonardBang Oct 30, 2023

Choose a reason for hiding this comment

Jiabao-Sun Oct 30, 2023

Choose a reason for hiding this comment

Jiabao-Sun Oct 30, 2023

Choose a reason for hiding this comment

leonardBang Oct 30, 2023

Choose a reason for hiding this comment

Jiabao-Sun Oct 30, 2023

Choose a reason for hiding this comment

leonardBang Oct 30, 2023

Choose a reason for hiding this comment

GOODBOY008 Oct 30, 2023 • edited Loading

Choose a reason for hiding this comment

GOODBOY008 Oct 30, 2023

Choose a reason for hiding this comment

GOODBOY008 commented Oct 30, 2023

leonardBang commented Oct 30, 2023 • edited Loading

GOODBOY008 commented Oct 30, 2023

lvyanquan commented Oct 31, 2023

leonardBang commented Oct 31, 2023

GOODBOY008 commented Oct 31, 2023

leonardBang left a comment

Choose a reason for hiding this comment

GOODBOY008 Oct 30, 2023 •

edited

Loading

leonardBang commented Oct 30, 2023 •

edited

Loading