test(project): add vrt and avt tests through playwright

[joshblack]

Overview

This PR is an exploration into using Playwright for VRT (Visual Regression Testing) and AVT (Accessibility Verification Testing).

Tests are currently located in the e2e directory and were generated from Storybook stories. Each test has roughly the following structure:

• Each component is a test suite containing a number of stories
  • For each story
    • Snapshot the story in each theme
    • Run axe against the story

Currently, the snapshots only run against Chromium with the default browser width. There is the ability to extend snapshots per-browser, if needed, and per-breakpoint.

Developer Experience

A developer can run tests with Playwright using: npx playwright test. They can also filter tests by test name: npx playwright test ActionList or by groups that are listed in the test name: npx playwright test --grep vrt

Reports of a test run are included both in the terminal and in an HTML report that can be opened and interacted with.

Terminal

HTML

A screenshot can be made by using expect(await page.screenshot()).toMatchSnapshot(). A page can be checked for axe violations by writing await expect(page).toHaveNoViolations()

In addition, developers may use the VSCode extension to interact with Playwright: https://playwright.dev/docs/getting-started-vscode

CI

This experiment adds in two separate jobs: vrt and avt. Each job uses Playwright's --grep option to only test for the specific tests that have that tag.

In order to speed up VRT tests, we make use of a vrt-runner job that uses a matrix strategy to create multiple runners that each run a portion of our tests. These runners rollup into the vrt job described above which will report the status of each runner

Drawbacks

• This PR does not include screenshots of interactive states for elements
• Playwright tests can take a while to run locally (around 8min on my machine)
• Playwright tests can be flaky and require re-runs unexpectedly
• Playwright does not seem to remove unused snapshots

Questions

• This draft captures almost everything in storybook, what stories would we prioritize snapshotting?
• Is this preferable to using/paying for Chromatic?
• What kind of interactions would we like to capture in VRT?
• Should tests be auto-generated, manually curated, or some combination of both?

Notes

• Test files were generated using this script: https://gist.github.com/joshblack/416f29592f9ea6c783bb93a232ca7038, we could keep using this script in the project or specify that tests are added to e2e as part of the testing workflow
• Files don't have to live in e2e, instead we could have a convention like ComponentName.visual.test.ts that lives beside the component (and is ignored by Jest)

[changeset-bot]

⚠️ No Changeset found

Latest commit: 01edca3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

size-limit report 📦

Path	Size
dist/browser.esm.js	78.14 KB (0%)
dist/browser.umd.js	78.79 KB (0%)

[jonrohan]

Nice 👍🏻 Maybe we could work together to set this up for view_components next time we pair?

[rezrah]

This PR does not include screenshots of interactive states for elements

Have you considered using Storybook interactions for this? Example of interaction, Example of snaphot.

Playwright tests can take a while to run locally (around 8min on my machine)

What about running these in an Actions runner instead? Generating the snapshots locally-only may limit future contributions and snapshot regeneration to folks in darwin environment (or worse, create duplicates).

Ideally you'd do it in a containerised environment, which you can run locally or on the server. That'll generate filenames with the *-linux.png suffix instead of *-darwin.png. I have this working in Brand so let me know if you want a walkthrough of it.

Should tests be auto-generated, manually curated, or some combination of both?

I recommend self-generating spec files with blanket coverage so you can just run and forget about them. Where you need granular control over some part of the config, like providing timeouts, different viewports, etc.. you can do that in the generation script. Here's a basic example, but can be done in many other ways too.

[langermank]

This draft captures almost everything in storybook, what stories would we prioritize snapshotting?

I would prioritize snapshotting everything within a component's /features directory! And /interactions if it exists for the component. Working on re-formatting all of our stories will make this much easier to accomplish 😄

Is this preferable to using/paying for Chromatic?

I like that everything lives "in house" and can be seen in GitHub (like the actual screenshots). As someone who doesn't contribute to PRC regularly, I don't really know how to access or use Chromatic.

Should tests be auto-generated, manually curated, or some combination of both?

I think auto-generated based on that features directory.. it might be interesting if a story author could opt in a specific story, thinking maybe a more detailed one from the /examples directory.

[joshblack]

Thanks everyone for the feedback! 🙏

I took the suggestions from this PR and created a smaller, more focused one for Button over here: #2520