Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Oracle] Fix Oracle CDC fetch two different db name for the same table #2088

Merged
merged 7 commits into from
Feb 28, 2024

Conversation

e-mhui
Copy link
Contributor

@e-mhui e-mhui commented Apr 16, 2023

Fix Oracle CDC fetch two different db name for the same table

Motivation

If the database name configured for Oracle CDC in Flink SQL is lowercase, such as the following example ('database-name' = 'xe'),

CREATE TABLE products (
     ID INT NOT NULL,
     NAME STRING,
     DESCRIPTION STRING,
     WEIGHT DECIMAL(10, 3),
     PRIMARY KEY(id) NOT ENFORCED
     ) WITH (
     'connector' = 'oracle-cdc',
     'hostname' = 'localhost',
     'port' = '1521',
     'username' = 'flinkuser',
     'password' = 'flinkpw',
     'database-name' = 'xe',
     'schema-name' = 'FLINKUSER',
     'table-name' = 'TEST_03');

the following problems will occur:

  1. For the same table (TEST_03),the snapshot phase will use the lowercase database name (xe.FLINKUSER.TEST_03 ), and the incremental phase will use the uppercase database name (XE.FLINKUSER.TEST_03 ).
  2. For the same table (TEST_03 ), after being captured in the snapshot phase, it will be captured again in the incremental phase because their database names are inconsistent.

image

In OracleConnectorConfig, the database name in Flink SQL is converted to uppercase, but the database name used in the snapshot phase is directly taken from Flink SQL. Therefore, when constructing the tableId, the database name in Flink SQL also needs to be converted to uppercase.

企业微信截图_92720bc3-5148-463a-9b35-12c3c8f77570

Modifications

When constructing the tableId, the database name in Flink SQL needs to be converted to uppercase.

@ruanhang1993 ruanhang1993 added this to the V2.5.0 milestone Jul 6, 2023
@ruanhang1993
Copy link
Contributor

@e-mhui Thanks for the contribution.
Please rebase the master branch. And the debezium version has been update to 1.9.7.Final. Could you help to check whether the error still exists?

Besides these, I think maybe we should use database name in upper case at the beginning. For example, in the constructor of the Source.

@e-mhui
Copy link
Contributor Author

e-mhui commented Jul 18, 2023

@e-mhui Thanks for the contribution. Please rebase the master branch. And the debezium version has been update to 1.9.7.Final. Could you help to check whether the error still exists?

Besides these, I think maybe we should use database name in upper case at the beginning. For example, in the constructor of the Source.

@ruanhang1993 Thanks you reply, it has rebased the mater branch. The error still exist in 2.4 and master branch. pls review it again.
In the incremental phase, debezium has already capitalized the database name, so I believe it is necessary to also do so in the snapshot phase.

Copy link
Contributor

@ruanhang1993 ruanhang1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @e-mhui Please rebase master branch and let's pass the CI.

@e-mhui
Copy link
Contributor Author

e-mhui commented Aug 24, 2023

LGTM. @e-mhui Please rebase master branch and let's pass the CI.

Thanks, it has been done.

@e-mhui
Copy link
Contributor Author

e-mhui commented Oct 18, 2023

@ruanhang1993 Hi, pls review it.

@ruanhang1993 ruanhang1993 merged commit 6da9684 into apache:master Feb 28, 2024
17 checks passed
@leonardBang leonardBang modified the milestones: V3.0.0, V3.1.0 Feb 28, 2024
aiwenmo pushed a commit to aiwenmo/flink-cdc-connectors that referenced this pull request Mar 6, 2024
…ot phase to fix fetching two different db name for the same table at the snapshot phase and incremental phase(apache#2088)
joyCurry30 pushed a commit to joyCurry30/flink-cdc-connectors that referenced this pull request Mar 22, 2024
…ot phase to fix fetching two different db name for the same table at the snapshot phase and incremental phase(apache#2088)
wuzhenhua01 pushed a commit to wuzhenhua01/flink-cdc-connectors that referenced this pull request Aug 4, 2024
…ot phase to fix fetching two different db name for the same table at the snapshot phase and incremental phase(apache#2088)
ChaomingZhangCN pushed a commit to ChaomingZhangCN/flink-cdc that referenced this pull request Jan 13, 2025
…ot phase to fix fetching two different db name for the same table at the snapshot phase and incremental phase(apache#2088)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug][Oracle] Oracle CDC fetch two different db name for the same
3 participants