Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stabilize some unreliable tests for smoother CI operation #6151

Merged
merged 1 commit into from
Jul 7, 2023
Merged

Stabilize some unreliable tests for smoother CI operation #6151

merged 1 commit into from
Jul 7, 2023

Conversation

mbien
Copy link
Member

@mbien mbien commented Jul 4, 2023 •
edited
Loading

I checked the last 10 merges to master and all of them failed on first attempt. Most failures were caused by the issue described in #5955

last 10 merge failures including test restarts till green:

BeforeFirstTest: org.netbeans.core.windows.awt.ValidateLayerMenuTest
BeforeFirstTest: org.netbeans.core.windows.awt.ValidateLayerToolbarTest

BeforeFirstTest: org.netbeans.modules.favorites.AddToFavoritesTest
BeforeFirstTest: org.netbeans.modules.favorites.AddToFavoritesTest
BeforeFirstTest: org.netbeans.modules.favorites.HomeFolderTest
BeforeFirstTest: org.netbeans.modules.favorites.RemoveFromFavoritesTest
BeforeFirstTest: org.netbeans.modules.favorites.RemoveFromFavoritesTest
BeforeFirstTest: org.netbeans.modules.favorites.RemoveFromFavoritesTest

BeforeFirstTest: org.netbeans.modules.settings.convertors.MimeTypeTest
BeforeFirstTest: org.netbeans.modules.settings.convertors.MimeTypeTest

testEnumCaseFix_02b: org.netbeans.modules.php.editor.verification.IntroduceSuggestionTest
testEnumCaseFix_02b: org.netbeans.modules.php.editor.verification.IntroduceSuggestionTest
testEnumCaseFix_02b: org.netbeans.modules.php.editor.verification.IntroduceSuggestionTest

testEnumerations_43: org.netbeans.modules.php.editor.csl.GotoDeclarationPHP81Test
testMarkAsRead: org.netbeans.modules.notifications.center.NotificationCenterManagerTest
testPHPTags: org.netbeans.modules.php.editor.csl.FoldingTest

The javafx modules wouldn't load anyway -> lets turn off the javafx cluster for those tests. didn't work

failure:

java.lang.AssertionError: Should have found a nonproblematic provider of needs org.openide.modules.jre.JavaFX among
StandardModule:org.netbeans.libs.javafx.linux jarFile: /home/runner/work/netbeans/netbeans/nbbuild/netbeans/extra/modules/org-netbeans-libs-javafx-linux.jar,  ... long list of jars

fixes #5955

@mbien mbien added CI continuous integration changes tests ci:all-tests [ci] enable all tests Platform [ci] enable platform tests (platform/*) and removed ci:all-tests [ci] enable all tests labels Jul 4, 2023
@mbien
Copy link
Member Author

mbien commented Jul 4, 2023 •
edited
Loading

well, disabling the javafx cluster from specific tests didn't work at all, exception still there:
[junit] java.lang.AssertionError: Should have found a nonproblematic provider of needs org.openide.modules.jre.JavaFX among...

@mbien mbien added the do not merge Don't merge this PR, it is not ready or just demonstration purposes. label Jul 4, 2023
@mbien mbien force-pushed the ci-platform-test-stability branch from 562c3d9 to 88929c5 Compare July 4, 2023 21:32
@mbien
stabilize unreliable tests for smoother CI operation.
3a1176e 
wrap platform/core.windows, favorites and settings into retry script.

mitigates the following exception which causes the tests to be very
unreliable in CI:
java.lang.AssertionError: Should have found a nonproblematic provider
of needs org.openide.modules.jre.JavaFX among [...]

Separate reliable from unreliable PHP tests and wrap unreliable tests
in retry script.
@mbien mbien added the PHP [ci] enable extra PHP tests (php/php.editor) label Jul 6, 2023
@mbien mbien force-pushed the ci-platform-test-stability branch from 88929c5 to a6d3962 Compare July 6, 2023 14:25
@mbien mbien removed the do not merge Don't merge this PR, it is not ready or just demonstration purposes. label Jul 6, 2023
@mbien
Copy link
Member Author

mbien commented Jul 6, 2023

lets try plan b:

  1. separate reliable from unreliable tests
  2. wrap unreliable tests into retry script

@mbien mbien changed the title Stabilize some unreliable tests by not loading javafx cluster Stabilize some unreliable tests for smoother CI operation Jul 6, 2023
@mbien mbien force-pushed the ci-platform-test-stability branch from a6d3962 to 3a1176e Compare July 6, 2023 14:30
@mbien mbien marked this pull request as ready for review July 6, 2023 18:22
@mbien mbien added this to the NB19 milestone Jul 6, 2023
neilcsmith-net
neilcsmith-net approved these changes Jul 7, 2023
View reviewed changes
Copy link
Member

@neilcsmith-net neilcsmith-net left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Good move, for now at least (unreliable tests is something of an oxymoron 😄 )

matthiasblaesing
matthiasblaesing approved these changes Jul 7, 2023
View reviewed changes
Copy link
Contributor

@matthiasblaesing matthiasblaesing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks sane to me. Thank you for looking into this.

@mbien mbien merged commit bbc3145 into apache:master Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@matthiasblaesing matthiasblaesing matthiasblaesing approved these changes

@neilcsmith-net neilcsmith-net neilcsmith-net approved these changes

Assignees
No one assigned
Labels
CI continuous integration changes PHP [ci] enable extra PHP tests (php/php.editor) Platform [ci] enable platform tests (platform/*) tests
Projects
None yet
Milestone
NB19
Development

Successfully merging this pull request may close these issues.

platform/core.windows + platform/favorites tests fail too often
3 participants
@mbien @matthiasblaesing @neilcsmith-net