Try visual regression testing #51287
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kevin940726
added
the
[Type] Automated Testing
|
Size Change: +77 B (0%) Total Size: 1.39 MB
ℹ️ View Unchanged
|
kevin940726
force-pushed
the
try/visual-testing
branch
from
0dc68d1
to
26a07c9
Compare
kevin940726
force-pushed
the
try/visual-testing
branch
from
a10eb96
to
4762d75
Compare
|
Flaky tests detected in 1df3f39. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/5202625090
|
kevin940726
force-pushed
the
try/visual-testing
branch
from
9cfb91c
to
dde0a8b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
A proof of concept of how visual regression testing would work with Playwright.
Prior arts: #46210, #51000.
Why?
While we already have some visual regression testing in place, they all operate only locally and require manual comparison. This PR commits the captured images and rerun them on CI to make this process automatic and cross-platform. The images are now treated as snapshots.
How?
We run the tests locally with docker to get consistent results. That's basically it really.
The idea is that visual testing will be reserved as an advanced feature and only a few experienced testers will be able to generate those images. Hence, the difficulty of using a docker locally is partially justified. This hopefully also limits the amount and size of the generated images because they will have to be cloned for every contributor. Visual regression testing will have to be used sparingly in this approach, but I think it's also a good practice since visual testing can often be too slow and lack focuses.
Testing Instructions
.wp-env.override.jsonin the root directory with the following content if you're now using Linux. (This is a temporary fix until we figure out a way to make accessing localhost in docker seamlessly){ "env": { "tests": { "config": { "WP_SITEURL": "http://host.docker.internal:8889", "WP_HOME": "http://host.docker.internal:8889" } } } }wp-envby runningnpm run env stopthennpm run env start -- --update.npm run test:e2e:dockernpm run test:e2e:docker -- -- -- -uScreenshots or screencast
See the files tab.