[FLINK-34613][cdc] Support recover from a specific savepoint file #2959

lvyanquan · 2024-01-03T12:59:47Z

This closes #2940.
Use fromSavepoint to be consistent with Flink cli. Some codes are inspired from CliFrontendParser.

lvyanquan · 2024-01-03T13:03:22Z

Jazzylol · 2024-01-30T11:25:28Z

@lvyanquan 您好，我使用您的PR打了包，结果报错，错误如下 Exception in thread "main" com.ververica.cdc.shaded.com.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: --fromSavepoint
at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.handleUnknownToken(DefaultParser.java:561)
at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.handleLongOptionWithoutEqual(DefaultParser.java:383)
at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.handleLongOption(DefaultParser.java:334)
at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.handleToken(DefaultParser.java:540)
at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.parse(DefaultParser.java:712)
at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.parse(DefaultParser.java:679)
at com.ververica.cdc.shaded.com.apache.commons.cli.DefaultParser.parse(DefaultParser.java:660)
at com.ververica.cdc.cli.CliFrontend.main(CliFrontend.java:50)

执行的命令 bash /opt/flink-cdc-3.0.0/bin/flink-cdc.sh /opt/flink-cdc-3.0.0/conf/mysql-to-starrocks.yaml --fromSavepoint file:/opt/flink-storage/savepoint/savepoint-f414a3-f2540c5684d0

请问是有什么需要注意的地方吗？

lvyanquan · 2024-01-31T06:55:20Z

@Jazzylol Have you updated lib/flink-cdc-dist-3.0-SNAPSHOT.jar? I have tested again and run correctly.

leonardBang

Thanks @lvyanquan for the contribution, I left some comments

leonardBang · 2024-02-29T09:23:42Z

flink-cdc-cli/src/main/java/com/ververica/cdc/cli/CliFrontendOptions.java

+                    .build();
+
+    public static final Option SAVEPOINT_RESTORE_MODE =
+            Option.builder("rm")


rm is a dangerous abbreviation in CS which usually means remove, please avoid use this abbreviation

leonardBang · 2024-02-29T09:25:59Z

flink-cdc-cli/src/main/java/com/ververica/cdc/cli/CliFrontendOptions.java

+
+    public static final Option SAVEPOINT_RESTORE_MODE =
+            Option.builder("rm")
+                    .longOpt("restoreMode")


Suggested change

.longOpt("restoreMode")

.longOpt("savepointRestoreMode")

leonardBang · 2024-02-29T09:28:19Z

flink-cdc-cli/src/test/java/com/ververica/cdc/cli/CliFrontendTest.java

@@ -137,6 +156,16 @@ private String globalPipelineConfig() throws Exception {
                    + "                               CDC pipelines\n"
                    + "    -h,--help                  Display help message\n"
                    + "       --jar <arg>             JARs to be submitted together with the pipeline\n"
+                    + "    -rm,--restoreMode <arg>    Defines how should we restore from the given\n"


Could we consider other savepoint settings like allowNonRestoredState?

lvyanquan · 2024-03-02T05:29:41Z

Addressed it.

leonardBang

Thanks @lvyanquan for the update, LGTM

PRNDA · 2024-03-04T14:26:10Z

Work like a charm.

lvyanquan · 2024-03-07T06:55:58Z

Rebased to master.

PatrickRen

@lvyanquan Thanks for the pull request! I left a comment.

flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontendOptions.java

…e using `-s` cli args.

PatrickRen

@lvyanquan Thanks for the update! LGTM.

monologuist · 2024-05-30T07:29:54Z

@lvyanquan hi，请问您这个PR，所预期的从savepoint重启Flink CDC Pipeline作业的规范命令是这样吗？

在${FLINK_CDC_HOME}目录下执行：
$ ./bin/flink-cdc.sh mysql-to-doris.yaml -s /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab

因为我没有看到明确的重启命令说明*，所以对此怀有疑惑。并且我认为应当明确指出。

*您引用的链接中，我看到的仍是针对jar作业的重启方式。
$ ./bin/flink run
--detached \
--from-savepoint /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab
./FlinkCDCExample.jar

monologuist · 2024-05-31T01:17:34Z

@lvyanquan hi，请问您这个PR，所预期的从savepoint重启Flink CDC Pipeline作业的规范命令是这样吗？
在${FLINK_CDC_HOME}目录下执行：
$ ./bin/flink-cdc.sh mysql-to-doris.yaml -s /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab
因为我没有看到明确的重启命令说明*，所以对此怀有疑惑。并且我认为应当明确指出。

*您引用的链接中，我看到的仍是针对jar作业的重启方式。 $ ./bin/flink run --detached \ --from-savepoint /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab ./FlinkCDCExample.jar

经多次测试，我确认这个命令可行。

…ache#2959)

github-actions bot added docs Improvements or additions to documentation cli labels Jan 3, 2024

lvyanquan force-pushed the savepoint branch 2 times, most recently from 1ba122d to 8f5ecac Compare January 29, 2024 12:24

leonardBang reviewed Feb 29, 2024

View reviewed changes

leonardBang assigned lvyanquan Feb 29, 2024

leonardBang approved these changes Mar 4, 2024

View reviewed changes

lvyanquan force-pushed the savepoint branch from 1ac386c to 84f13af Compare March 7, 2024 06:55

PatrickRen reviewed Apr 3, 2024

View reviewed changes

flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontendOptions.java Outdated Show resolved Hide resolved

lvyanquan and others added 2 commits April 7, 2024 09:48

[FLINK-34613][cdc][cli] support recover from a specific savepoint fil…

ed380cf

…e using `-s` cli args.

address comment.

4ab2456

lvyanquan force-pushed the savepoint branch from d1d6054 to 4ab2456 Compare April 7, 2024 01:50

PatrickRen approved these changes Apr 9, 2024

View reviewed changes

PatrickRen changed the title ~~[cdc-cli] support recover from a specific savepoint file~~ [FLINK-34613][cdc] Support recover from a specific savepoint file Apr 9, 2024

PatrickRen merged commit b10a666 into apache:master Apr 9, 2024
15 checks passed

wuzhenhua01 pushed a commit to wuzhenhua01/flink-cdc-connectors that referenced this pull request Aug 4, 2024

[FLINK-34613][cdc] Support recover from a specific savepoint file (ap…

641bde4

…ache#2959)

ChaomingZhangCN pushed a commit to ChaomingZhangCN/flink-cdc that referenced this pull request Jan 13, 2025

[FLINK-34613][cdc] Support recover from a specific savepoint file (ap…

02a19fe

…ache#2959)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-34613][cdc] Support recover from a specific savepoint file #2959

[FLINK-34613][cdc] Support recover from a specific savepoint file #2959

lvyanquan commented Jan 3, 2024

lvyanquan commented Jan 3, 2024

Jazzylol commented Jan 30, 2024

lvyanquan commented Jan 31, 2024

leonardBang left a comment

leonardBang Feb 29, 2024

leonardBang Feb 29, 2024

leonardBang Feb 29, 2024

lvyanquan commented Mar 2, 2024

leonardBang left a comment

PRNDA commented Mar 4, 2024

lvyanquan commented Mar 7, 2024

PatrickRen left a comment

PatrickRen left a comment

monologuist commented May 30, 2024

monologuist commented May 31, 2024

[FLINK-34613][cdc] Support recover from a specific savepoint file #2959

[FLINK-34613][cdc] Support recover from a specific savepoint file #2959

Conversation

lvyanquan commented Jan 3, 2024

lvyanquan commented Jan 3, 2024

Jazzylol commented Jan 30, 2024

lvyanquan commented Jan 31, 2024

leonardBang left a comment

Choose a reason for hiding this comment

leonardBang Feb 29, 2024

Choose a reason for hiding this comment

leonardBang Feb 29, 2024

Choose a reason for hiding this comment

leonardBang Feb 29, 2024

Choose a reason for hiding this comment

lvyanquan commented Mar 2, 2024

leonardBang left a comment

Choose a reason for hiding this comment

PRNDA commented Mar 4, 2024

lvyanquan commented Mar 7, 2024

PatrickRen left a comment

Choose a reason for hiding this comment

PatrickRen left a comment

Choose a reason for hiding this comment

monologuist commented May 30, 2024

monologuist commented May 31, 2024