Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[playwright] Flaky Playwright test in theia-explorer-view.test.ts:56:9 #12063

Closed
planger opened this issue Jan 12, 2023 · 1 comment · Fixed by #12310 or #12373
Closed

[playwright] Flaky Playwright test in theia-explorer-view.test.ts:56:9 #12063

planger opened this issue Jan 12, 2023 · 1 comment · Fixed by #12310 or #12373
Labels
ci issues related to CI / tests playwright issues related to playwright tests

Comments

@planger
Copy link
Contributor

planger commented Jan 12, 2023

Bug Description:

As I'm tracking the executions of the Playwright tests and recording their failures continuously, I observed that there is actually only one test in the past four months which fails from time to time (about 3-4 times a month). I didn't observe any other test failing in this time frame.

Thus, I believe it is worth spending some time investigating whether this flaky test can be stabilized.

See below for the output of the flaky test, when it fails:

[playwright]   1) ../../src/tests/theia-explorer-view.test.ts:56:9 › Theia Explorer View › should show one folder named "sampleFolder" and one file named "sample.txt" 
[playwright] 
[playwright]     Error: expect(received).toBe(expected) // Object.is equality
[playwright] 
[playwright]     Expected: true
[playwright]     Received: false
[playwright] 
[playwright]       56 |     test('should show one folder named "sampleFolder" and one file named "sample.txt"', async () => {
[playwright]       57 |         await explorer.selectTreeNode('sampleFolder');
[playwright]     > 58 |         expect(await explorer.isTreeNodeSelected('sampleFolder')).toBe(true);

See also https://github.com/eclipse-theia/theia/actions/runs/3815148923/jobs/6489956895
@vince-fugnitto vince-fugnitto added ci issues related to CI / tests playwright issues related to playwright tests labels Jan 12, 2023
@ndoschek ndoschek mentioned this issue Feb 2, 2023
Merged
1 task
planger added a commit to eclipsesource/theia that referenced this issue Mar 15, 2023
@planger
playwright: Wait for file stat nodes
ed00d0c 
The `theia-workspace.test.ts` checks for existence of file nodes in the
explorer view to verify whether the workspace has been prepared
correctly. However, occasionally it may happen that the explorer view
doesn't show them yet, as the files are just being loaded.

This resulted in occasionally flaky test executions (eclipse-theia#12063).

To fix this problem, we now use the `existsDirectoryNode` and
`existsFileNode` checks to explicitly wait for them to appear and
only fail if the check times out.

Contributed on behalf of STMicroelectronics.
Fixes eclipse-theia#12063

Change-Id: I4226a7d6f657c6a1363160655da4c94cd0984379
@planger planger mentioned this issue Mar 15, 2023
Merged
1 task
JonasHelming pushed a commit that referenced this issue Mar 17, 2023
@planger
playwright: Wait for file stat nodes (#12310)
ff944de 
The `theia-workspace.test.ts` checks for existence of file nodes in the
explorer view to verify whether the workspace has been prepared
correctly. However, occasionally it may happen that the explorer view
doesn't show them yet, as the files are just being loaded.

This resulted in occasionally flaky test executions (#12063).

To fix this problem, we now use the `existsDirectoryNode` and
`existsFileNode` checks to explicitly wait for them to appear and
only fail if the check times out.

Contributed on behalf of STMicroelectronics.
Fixes #12063

Change-Id: I4226a7d6f657c6a1363160655da4c94cd0984379
@marcdumais-work
Copy link
Contributor

I noticed this issue has been re-happening recently [1]. I tried the Playwright tests locally for the first time and was lucky that I was able to reproduce the failure rather consistently [2]. I will be posting a PR shortly to try to improve the flaky tests I was able to troubleshoot locally.

@planger I would appreciate your review on it. I think you had the correct idea in your related recent PR, but the same issue is happening in other test cases too, so I went with a fix in the test infrastructure. More details in the PR.

[1]: e.g.:
https://github.com/eclipse-theia/theia/actions/runs/4566999149/jobs/8060253354#step:7:422
https://github.com/eclipse-theia/theia/actions/runs/4558355274/jobs/8055182849#step:7:415
https://github.com/eclipse-theia/theia/actions/runs/4564018969/jobs/8053243319#step:7:420

[2]: Way more so that in our CI, which passes most of the time. This suite seems extremely sensitive to timing - t some point I tried adding some entropy by running `stress -c 1 -m 1" during the Playwright tests and the failures went away until I stopped :)

marcdumais-work added a commit that referenced this issue Apr 4, 2023
@marcdumais-work
[ci][playwright] Solidify flaky Playwright tests
96ba4c7 
This is an attempt at solidifying the playwright test suite, that has had testcases
intermitently failing. I have been lucky to be able to observe some race conditions
rather consistently locally, and so be able to debug them and solidify against them.

The race conditions are very slight - the Playwright test suite takes a screenshot
at every failure, and consistently I see the correct state on those, meaning that
the state is what it should be, by the time the screenshot is taken.

In consequence, I wound that, for a given flaky scenario, addressing one race
consition would often mask a second one, present in the same scenario. It's
possible that under a different execution context, more race conditions would
occur. Here are the race coditions I have found:

- When the Theia explorer view is created (at frontend reload or when explicitly
closed and then re-opened in a test case), it can take some time for its file nodes
to appear in the UI. Some tests were assuming that once the view was visible, they
would be too.
- Similarly, it can take some time for the Explorer's view tab to be created, after
the view is. Some test cases were checking immediately for the view tab being
visible, and failing when it was not.
- Also related to the explorer view, method "selectTreeNode" was used to select a file
node and assumed that it would be selected imemdiately after clicking on it.

closes #12063

Signed-off-by: Marc Dumais <marc.dumais@ericsson.com>
marcdumais-work added a commit that referenced this issue Apr 4, 2023
@marcdumais-work
[ci][playwright] Solidify flaky Playwright tests
af7677f 
This is an attempt at solidifying the playwright test suite, that has had testcases
intermitently failing. I have been lucky to be able to observe some race conditions
rather consistently locally, and so be able to debug them and solidify against them.

The race conditions are very slight - the Playwright test suite takes a screenshot
at every failure, and consistently I see the correct state on those, meaning that
the state is what it should be, by the time the screenshot is taken.

In consequence, I wound that, for a given flaky scenario, addressing one race
consition would often mask a second one, present in the same scenario. It's
possible that under a different execution context, more race conditions would
occur. Here are the race coditions I have found:

- When the Theia explorer view is created (at frontend reload or when explicitly
closed and then re-opened in a test case), it can take some time for its file nodes
to appear in the UI. Some tests were assuming that once the view was visible, they
would be too.
- Similarly, it can take some time for the Explorer's view tab to be created, after
the view is. Some test cases were checking immediately for the view tab being
visible, and failing when it was not.
- Also related to the explorer view, in at least one testcase, method "selectTreeNode"
was used to select a file node and assumed that it would be selected immediately after
clicking on it.

closes #12063

Signed-off-by: Marc Dumais <marc.dumais@ericsson.com>
@marcdumais-work marcdumais-work mentioned this issue Apr 4, 2023
Merged
1 task
marcdumais-work added a commit that referenced this issue Apr 4, 2023
@marcdumais-work
[ci][playwright] Solidify flaky Playwright tests
7cedda3 
This is an attempt at solidifying the playwright test suite, that has had testcases
intermitently failing. I have been lucky to be able to observe some race conditions
rather consistently locally, and so be able to debug them and solidify against them.

The race conditions are very slight - the Playwright test suite takes a screenshot
at every failure, and consistently I see the correct state on those, meaning that
the state is what it should be, by the time the screenshot is taken.

In consequence, I wound that, for a given flaky scenario, addressing one race
consition would often mask a second one, present in the same scenario. It's
possible that under a different execution context, more race conditions would
occur. Here are the race coditions I have found:

- When the Theia explorer view is created (at frontend reload or when explicitly
closed and then re-opened in a test case), it can take some time for its file nodes
to appear in the UI. Some tests were assuming that once the view was visible, they
would be too.
- Similarly, it can take some time for the Explorer's view tab to be created, after
the view is. Some test cases were checking immediately for the view tab being
visible, and failing when it was not.
- Also related to the explorer view, in at least one testcase, method "selectTreeNode"
was used to select a file node and assumed that it would be selected immediately after
clicking on it.

closes #12063

Signed-off-by: Marc Dumais <marc.dumais@ericsson.com>
marcdumais-work added a commit that referenced this issue Apr 4, 2023
@marcdumais-work
[ci][playwright] Solidify flaky Playwright tests
cf7d9e8 
This is an attempt at solidifying the playwright test suite, that has had testcases
intermitently failing. I have been lucky to be able to observe some race conditions
rather consistently locally, and so be able to debug them and solidify against them.

The race conditions are very slight - the Playwright test suite takes a screenshot
at every failure, and consistently I see the correct state on those, meaning that
the state is what it should be, by the time the screenshot is taken.

In consequence, I wound that, for a given flaky scenario, addressing one race
consition would often mask a second one, present in the same scenario. It's
possible that under a different execution context, more race conditions would
occur. Here are the race coditions I have found:

- When the Theia explorer view is created (at frontend reload or when explicitly
closed and then re-opened in a test case), it can take some time for its file nodes
to appear in the UI. Some tests were assuming that once the view was visible, they
would be too.
- Similarly, it can take some time for the Explorer's view tab to be created, after
the view is. Some test cases were checking immediately for the view tab being
visible, and failing when it was not.
- Also related to the explorer view, in at least one testcase, method "selectTreeNode"
was used to select a file node and assumed that it would be selected immediately after
clicking on it.

closes #12063

Signed-off-by: Marc Dumais <marc.dumais@ericsson.com>
marcdumais-work added a commit that referenced this issue Apr 4, 2023
@marcdumais-work
[ci][playwright] Solidify flaky Playwright tests
25af1d9 
This is an attempt at solidifying the playwright test suite, that has had testcases
intermitently failing. I have been lucky to be able to observe some race conditions
rather consistently locally, and so be able to debug them and solidify against them.

The race conditions are very slight - the Playwright test suite takes a screenshot
at every failure, and consistently I see the correct state on those, meaning that
the state is what it should be, by the time the screenshot is taken.

In consequence, I wound that, for a given flaky scenario, addressing one race
consition would often mask a second one, present in the same scenario. It's
possible that under a different execution context, more race conditions would
occur. Here are the race coditions I have found:

- When the Theia explorer view is created (at frontend reload or when explicitly
closed and then re-opened in a test case), it can take some time for its file nodes
to appear in the UI. Some tests were assuming that once the view was visible, they
would be too.
- Similarly, it can take some time for the Explorer's view tab to be created, after
the view is. Some test cases were checking immediately for the view tab being
visible, and failing when it was not.
- Also related to the explorer view, in at least one testcase, method "selectTreeNode"
was used to select a file node and assumed that it would be selected immediately after
clicking on it.

closes #12063

Signed-off-by: Marc Dumais <marc.dumais@ericsson.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ci issues related to CI / tests playwright issues related to playwright tests
Projects
None yet
Milestone
No milestone
3 participants
@planger @marcdumais-work @vince-fugnitto