Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: run e2e flaky tests in a separate job #6365

Merged
merged 19 commits into from
Jul 4, 2024
Merged

CI: run e2e flaky tests in a separate job #6365

merged 19 commits into from
Jul 4, 2024

Conversation

potatowagon
Copy link
Contributor

@potatowagon potatowagon commented Jul 2, 2024

Description

  • create a seperate job to run flaky e2e tests. The main e2e job does not run tests tagged flaky. the filtering is via
--grep-invert @flaky

to not run flaky and

--grep @flaky

to run only flaky

  • tag known flaky test in flaky test tracker as flaky
  • had to upgrade playwright to get the filter by tag annotation feature

Test plan

Before

see the current status checks in any of the open PRs

After

monitor status checks for this PR

before accept, all checks should pass. for e2e mac canary run the ci run the command with

npm run prepare-ci && cross-env E2E_PORT=4000 npm run test-e2e-chromium -- --grep-invert @flaky
Screenshot 2024-07-02 at 7 37 17 PM

after accept,

  • all the normal e2e jobs should run with --grep-invert @flaky
  • there will be a new job e2e-tests / flaky in status checks
  • job failures only due to failed test runs; all the tests did trigger

ci pass 100%: https://github.com/facebook/lexical/actions/runs/9775780085/job/26991263423?pr=6365

Copy link

vercel bot commented Jul 2, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
lexical ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 4, 2024 3:15am
lexical-playground ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 4, 2024 3:15am

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 2, 2024
@potatowagon potatowagon marked this pull request as draft July 2, 2024 10:10
Copy link

github-actions bot commented Jul 2, 2024

size-limit report 📦

Path Size
lexical - cjs 28.47 KB (0%)
lexical - esm 28.28 KB (0%)
@lexical/rich-text - cjs 36.86 KB (0%)
@lexical/rich-text - esm 28.08 KB (0%)
@lexical/plain-text - cjs 35.49 KB (0%)
@lexical/plain-text - esm 25.3 KB (0%)
@lexical/react - cjs 38.82 KB (0%)
@lexical/react - esm 29.27 KB (0%)

@potatowagon potatowagon changed the title CI: Seperate flaky tests to be ran in a seperate CI CI: run flaky tests in a separate job Jul 2, 2024
Comment on lines +39 to +41
{
tag: '@flaky',
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only added this, rest are formatting changes

Comment on lines +1436 to +1438
{
tag: '@flaky',
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only added this, rest are formatting changes

@@ -402,7 +402,7 @@ function testSuite(charset) {
await page.focus('div[contenteditable="true"]');

await page.keyboard.type('👨‍👩‍👦‍👦');
if (['chromium', 'webkit'].includes(browserName)) {
if (['chromium', 'webkit', 'firefox'].includes(browserName)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after updating playwright firefox always renders 👨‍👩‍👦‍👦 as 👨‍👩‍👦‍👦

Screenshot 2024-07-03 at 5 10 13 PM

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes the below 2 conditions (charset === 'UTF-16' & charset === 'UTF-8') deadcode, any idea what settings to fall into those conditions? as now all 3 browsers render 👨‍👩‍👦‍👦. if no im thinking of removing it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need playwright update in the same PR ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes because playwright update is needed to have tag annotations

anchorOffset: 2,
anchorPath: [0, 4, 0, 0],
focusOffset: 2,
focusPath: [0, 4, 0, 0],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

firefox behavior changed after updating playwright

Comment on lines -1012 to -1026
if (IS_WINDOWS) {
await assertSelection(page, {
anchorOffset: 2,
anchorPath: [0, 1, 0, 0],
focusOffset: 2,
focusPath: [0, 1, 0, 0],
});
} else {
await assertSelection(page, {
anchorOffset: 2,
anchorPath: [0, 1, 0, 0],
focusOffset: 2,
focusPath: [0, 1, 0, 0],
});
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these 2 are the same so removing the windows condition

Comment on lines -421 to -452
if (charset === 'UTF-16') {
await assertHTML(
page,
html`
<p
class="PlaygroundEditorTheme__paragraph PlaygroundEditorTheme__ltr"
dir="ltr">
<span data-lexical-text="true">👨‍👩</span>
<span
class="PlaygroundEditorTheme__characterLimit PlaygroundEditorTheme__ltr"
dir="ltr">
<span data-lexical-text="true">‍👦‍👦</span>
</span>
</p>
`,
);
} else {
await assertHTML(
page,
html`
<p
class="PlaygroundEditorTheme__paragraph PlaygroundEditorTheme__ltr"
dir="ltr">
<span data-lexical-text="true">👨</span>
<span
class="PlaygroundEditorTheme__characterLimit PlaygroundEditorTheme__ltr"
dir="ltr">
<span data-lexical-text="true">‍👩‍👦‍👦</span>
</span>
</p>
`,
);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confirmed with @zurfyx that these can be removed since now FF behaves the same as chrome and webkit

Sahejkm
Sahejkm previously approved these changes Jul 3, 2024
@potatowagon potatowagon enabled auto-merge July 4, 2024 01:47
@potatowagon potatowagon added this pull request to the merge queue Jul 4, 2024
Merged via the queue into main with commit a7a6a0a Jul 4, 2024
42 checks passed
2wheeh pushed a commit to 2wheeh/lexical that referenced this pull request Jul 17, 2024
@potatowagon potatowagon deleted the flaky-ci branch July 26, 2024 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. extended-tests Run extended e2e tests on a PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants