Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tests][oceanbase] Fix oceanbase test failure, possibly caused by some interactions among cases #3712

Merged
merged 2 commits into from
Dec 2, 2024

Conversation

yuxiqian
Copy link
Contributor

@yuxiqian yuxiqian commented Nov 11, 2024

This fixes a mysterious test failure in OceanBaseMySQLModeITCase like this. Recently it is happening quite frequently, possibly brought in by #3439.

This could be stably reproduced locally, by running OceanBaseMySqlModeITCase locally, and make sure testMetadataColumns is executed right after testSnapshotOnly.

This test case will fail, because instead of getting expected

+I(test,inventory,products,101,scooter,Small 2-wheel scooter,3.1400000000), 
+I(test,inventory,products,102,car battery,12V car battery,8.1000000000), 
+I(test,inventory,products,103,12-pack drill bits,12-pack of drill bits with sizes ranging from #40 to #3,0.8000000000), 
+I(test,inventory,products,104,hammer,12oz carpenter's hammer,0.7500000000), 
+I(test,inventory,products,105,hammer,14oz carpenter's hammer,0.8750000000), 
+I(test,inventory,products,106,hammer,16oz carpenter's hammer,1.0000000000), 
+I(test,inventory,products,107,rocks,box of assorted rocks,5.3000000000), 
+I(test,inventory,products,108,jacket,water resistent black wind breaker,0.1000000000), 
+I(test,inventory,products,109,spare tire,24 inch spare tire,22.2000000000), 
+U(test,inventory,products,106,hammer,18oz carpenter hammer,1.0000000000)

values, it is actually receiving:

+I(test,inventory,products,101,scooter,Small 2-wheel scooter,3.1400000000),
+I(test,inventory,products,102,car battery,12V car battery,8.1000000000), 
+I(test,inventory,products,103,12-pack drill bits,12-pack of drill bits with sizes ranging from #40 to #3,0.8000000000), 
+I(test,inventory,products,104,hammer,12oz carpenter's hammer,0.7500000000), 
+I(test,inventory,products,105,hammer,14oz carpenter's hammer,0.8750000000), 
+I(test,inventory,products,106,hammer,16oz carpenter's hammer,1.0000000000), 
+I(test,inventory,products,107,rocks,box of assorted rocks,5.3000000000), 
+I(test,inventory,products,108,jacket,water resistent black wind breaker,0.1000000000), 
+I(test,inventory,products,109,spare tire,24 inch spare tire,22.2000000000), 
+I(test,inventory,products,101,scooter,Small 2-wheel scooter,3.1400000000), 
+I(test,inventory,products,102,car battery,12V car battery,8.1000000000), 
+I(test,inventory,products,103,12-pack drill bits,12-pack of drill bits with sizes ranging from #40 to #3,0.8000000000), 
+I(test,inventory,products,104,hammer,12oz carpenter's hammer,0.7500000000), 
+I(test,inventory,products,105,hammer,14oz carpenter's hammer,0.8750000000), 
+I(test,inventory,products,106,hammer,16oz carpenter's hammer,1.0000000000), 
+I(test,inventory,products,107,rocks,box of assorted rocks,5.3000000000), 
+I(test,inventory,products,108,jacket,water resistent black wind breaker,0.1000000000), 
+I(test,inventory,products,109,spare tire,24 inch spare tire,22.2000000000), 
+I(test,inventory,products,101,scooter,Small 2-wheel scooter,3.1400000000), 
+I(test,inventory,products,102,car battery,12V car battery,8.1000000000)

which makes totally no sense, just like previous data records have not been cleaned up correctly. Running either testSnapshotOnly or testMetadataColumns independently is OK, so this PR creates & destroys OceanBase containers for each test case to avoid breaking CI, as a temporary hackaround.

I'm a little worried that this behavior might not be caused by a bad test case but reflects some connector implementation issue. Needs some advice from @whhe.

@yuxiqian
Copy link
Contributor Author

yuxiqian commented Nov 11, 2024

Another typical failure case is when testAllDataTypes case is running, waitForSinkSize never collects enough rows, and keeps running for an hour, since there's no any timeout mechanisms.

@whhe
Copy link
Member

whhe commented Nov 11, 2024

To avoid reading change events generated by another test case, how about using UniqueDatabase? Starting a new container for each test case may significantly increase the testing time.

There should be a timeout for waitForSinkSize method. Please help to fix it here.

@yuxiqian yuxiqian force-pushed the hotfix/oceanbase-testcases branch from 9b435b0 to 7b731f3 Compare November 11, 2024 09:19
@yuxiqian
Copy link
Contributor Author

Done, please check.

…e interactions among cases

Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>

# Conflicts:
#	flink-cdc-connect/flink-cdc-source-connectors/flink-connector-oceanbase-cdc/src/test/java/org/apache/flink/cdc/connectors/oceanbase/table/OceanBaseMySQLModeITCase.java
Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>
@yuxiqian yuxiqian force-pushed the hotfix/oceanbase-testcases branch from c4783ea to ccce315 Compare November 20, 2024 02:11
@yuxiqian
Copy link
Contributor Author

Rebased with master, could @ruanhang1993 please take a look?

Copy link
Contributor

@ruanhang1993 ruanhang1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ruanhang1993 ruanhang1993 merged commit 26f5880 into apache:master Dec 2, 2024
24 checks passed
ChaomingZhangCN pushed a commit to ChaomingZhangCN/flink-cdc that referenced this pull request Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants