Make the Visual regression tests stricter

[obulat]

Problem

The maxDiffPixelRatio set to 0.6 in Playwright test makes us miss actual visual regressions. We should get it back to 0 or at maximum 0.2 to prevent visual regressions from happening, while also keeping the tests not flaky.
I set the priority to high because currently it is very easy to merge in changes that contain visual regressions.

Description

Currently, several tests are failing if you set the maxDiffPixelRatio value to 0. To fix this, we need to update the snapshots, and then set the threshold back to 0. For easier PR review, it would be best to open separate PRs for the following tests in the visual-regression directory:

• components/audio-results-old.spec.ts Add updated audio-results-old snapshots #1981
• components/filters-sidebar.spec.ts Fix filters sidebar regression after merge #1983
• components/header-old.spec.ts Add updated header-old snapshots #1980
• components/header.spec.ts Add updated header snapshots #1986
• pages/homepage.spec.ts Fix homepage search button and update snapshots #1991
• pages/pages.spec.ts (non-existent page) Add updated non-existent page snapshots #1982

After all of these PRs are merged:

• set maxDiffPixelRatio to 0 Make the visual regression tests stricter #1992

Additional context

We could try setting the maxDiffPixelRatio to 0 and see if that makes the tests too flaky. If so, we could increase it to a maximum of 0.2.

The full list of failing tests

visual-regression/components/audio-results-old.spec.ts:27:7 › audio results › screen at breakpoint sm with width 640 › should render small row layout mobile UA with narrow viewport
visual-regression/components/audio-results-old.spec.ts:27:7 › audio results › screen at breakpoint xs with width 340 › should render small row layout mobile UA with narrow viewport
visual-regression/components/audio-results-old.spec.ts:36:5 › audio results › screen at breakpoint 2xl with width 1536 › desktop audio results
visual-regression/components/audio-results-old.spec.ts:36:5 › audio results › screen at breakpoint xl with width 1280 › desktop audio results
visual-regression/components/audio-results-old.spec.ts:36:5 › audio results › screen at breakpoint lg with width 1024 › desktop audio results
visual-regression/components/audio-results-old.spec.ts:36:5 › audio results › screen at breakpoint md with width 768 › desktop audio results
visual-regression/components/filters-sidebar.spec.ts:11:3 › Filters sidebar none selected - ltr
visual-regression/components/filters-sidebar.spec.ts:18:3 › Filters sidebar some filters selected - ltr
visual-regression/components/filters-sidebar.spec.ts:11:3 › Filters sidebar none selected - rtl
visual-regression/components/filters-sidebar.spec.ts:18:3 › Filters sidebar some filters selected - rtl
visual-regression/components/header-old.spec.ts:27:11 › header-old snapshots › ltr › header-old › screen at breakpoint 2xl with width 1536 › filters open
visual-regression/components/header-old.spec.ts:27:11 › header-old snapshots › ltr › header-old › screen at breakpoint xl with width 1280 › filters open
visual-regression/components/header-old.spec.ts:27:11 › header-old snapshots › ltr › header-old › screen at breakpoint lg with width 1024 › filters open
visual-regression/components/header-old.spec.ts:37:11 › header-old snapshots › ltr › header-old › screen at breakpoint 2xl with width 1536 › resting
visual-regression/components/header-old.spec.ts:45:11 › header-old snapshots › ltr › header-old › screen at breakpoint 2xl with width 1536 › scrolled
visual-regression/components/header-old.spec.ts:56:11 › header-old snapshots › ltr › header-old › screen at breakpoint 2xl with width 1536 › searchbar hovered
visual-regression/components/header-old.spec.ts:37:11 › header-old snapshots › ltr › header-old › screen at breakpoint xl with width 1280 › resting
visual-regression/components/header-old.spec.ts:45:11 › header-old snapshots › ltr › header-old › screen at breakpoint xl with width 1280 › scrolled
visual-regression/components/header-old.spec.ts:56:11 › header-old snapshots › ltr › header-old › screen at breakpoint xl with width 1280 › searchbar hovered
visual-regression/components/header-old.spec.ts:37:11 › header-old snapshots › ltr › header-old › screen at breakpoint lg with width 1024 › resting
visual-regression/components/header-old.spec.ts:45:11 › header-old snapshots › ltr › header-old › screen at breakpoint lg with width 1024 › scrolled
visual-regression/components/header-old.spec.ts:56:11 › header-old snapshots › ltr › header-old › screen at breakpoint lg with width 1024 › searchbar hovered
visual-regression/components/header-old.spec.ts:27:11 › header-old snapshots › rtl › header-old › screen at breakpoint 2xl with width 1536 › filters open
visual-regression/components/header-old.spec.ts:27:11 › header-old snapshots › rtl › header-old › screen at breakpoint xl with width 1280 › filters open
visual-regression/components/header-old.spec.ts:27:11 › header-old snapshots › rtl › header-old › screen at breakpoint lg with width 1024 › filters open
visual-regression/components/header-old.spec.ts:27:11 › header-old snapshots › rtl › header-old › screen at breakpoint md with width 768 › filters open
visual-regression/components/header-old.spec.ts:37:11 › header-old snapshots › rtl › header-old › screen at breakpoint 2xl with width 1536 › resting
visual-regression/components/header-old.spec.ts:45:11 › header-old snapshots › rtl › header-old › screen at breakpoint 2xl with width 1536 › scrolled
visual-regression/components/header-old.spec.ts:56:11 › header-old snapshots › rtl › header-old › screen at breakpoint 2xl with width 1536 › searchbar hovered
visual-regression/components/header-old.spec.ts:37:11 › header-old snapshots › rtl › header-old › screen at breakpoint xl with width 1280 › resting
visual-regression/components/header-old.spec.ts:45:11 › header-old snapshots › rtl › header-old › screen at breakpoint xl with width 1280 › scrolled
visual-regression/components/header-old.spec.ts:56:11 › header-old snapshots › rtl › header-old › screen at breakpoint xl with width 1280 › searchbar hovered
visual-regression/components/header-old.spec.ts:37:11 › header-old snapshots › rtl › header-old › screen at breakpoint lg with width 1024 › resting
visual-regression/components/header-old.spec.ts:45:11 › header-old snapshots › rtl › header-old › screen at breakpoint lg with width 1024 › scrolled
visual-regression/components/header-old.spec.ts:56:11 › header-old snapshots › rtl › header-old › screen at breakpoint lg with width 1024 › searchbar hovered
visual-regression/components/header-old.spec.ts:37:11 › header-old snapshots › rtl › header-old › screen at breakpoint md with width 768 › resting
visual-regression/components/header-old.spec.ts:45:11 › header-old snapshots › rtl › header-old › screen at breakpoint md with width 768 › scrolled
visual-regression/components/header-old.spec.ts:56:11 › header-old snapshots › rtl › header-old › screen at breakpoint md with width 768 › searchbar hovered
visual-regression/components/header.spec.ts:29:9 › header › ltr › screen at breakpoint 2xl with width 1536 › filters open
visual-regression/components/header.spec.ts:37:9 › header › ltr › screen at breakpoint 2xl with width 1536 › resting
visual-regression/components/header.spec.ts:47:9 › header › ltr › screen at breakpoint 2xl with width 1536 › scrolled
visual-regression/components/header.spec.ts:57:9 › header › ltr › screen at breakpoint 2xl with width 1536 › searchbar hovered
visual-regression/components/header.spec.ts:69:9 › header › ltr › screen at breakpoint 2xl with width 1536 › searchbar active
visual-regression/components/header.spec.ts:29:9 › header › ltr › screen at breakpoint xl with width 1280 › filters open
visual-regression/components/header.spec.ts:37:9 › header › ltr › screen at breakpoint xl with width 1280 › resting
visual-regression/components/header.spec.ts:47:9 › header › ltr › screen at breakpoint xl with width 1280 › scrolled
visual-regression/components/header.spec.ts:57:9 › header › ltr › screen at breakpoint xl with width 1280 › searchbar hovered
visual-regression/components/header.spec.ts:69:9 › header › ltr › screen at breakpoint xl with width 1280 › searchbar active
visual-regression/components/header.spec.ts:29:9 › header › ltr › screen at breakpoint lg with width 1024 › filters open
visual-regression/components/header.spec.ts:37:9 › header › ltr › screen at breakpoint lg with width 1024 › resting
visual-regression/components/header.spec.ts:47:9 › header › ltr › screen at breakpoint lg with width 1024 › scrolled
visual-regression/components/header.spec.ts:57:9 › header › ltr › screen at breakpoint lg with width 1024 › searchbar hovered
visual-regression/components/header.spec.ts:69:9 › header › ltr › screen at breakpoint lg with width 1024 › searchbar active
visual-regression/components/header.spec.ts:29:9 › header › rtl › screen at breakpoint 2xl with width 1536 › filters open
visual-regression/components/header.spec.ts:37:9 › header › rtl › screen at breakpoint 2xl with width 1536 › resting
visual-regression/components/header.spec.ts:47:9 › header › rtl › screen at breakpoint 2xl with width 1536 › scrolled
visual-regression/components/header.spec.ts:57:9 › header › rtl › screen at breakpoint 2xl with width 1536 › searchbar hovered
visual-regression/components/header.spec.ts:69:9 › header › rtl › screen at breakpoint 2xl with width 1536 › searchbar active
visual-regression/components/header.spec.ts:29:9 › header › rtl › screen at breakpoint xl with width 1280 › filters open
visual-regression/components/header.spec.ts:37:9 › header › rtl › screen at breakpoint xl with width 1280 › resting
visual-regression/components/header.spec.ts:47:9 › header › rtl › screen at breakpoint xl with width 1280 › scrolled
visual-regression/components/header.spec.ts:57:9 › header › rtl › screen at breakpoint xl with width 1280 › searchbar hovered
visual-regression/components/header.spec.ts:69:9 › header › rtl › screen at breakpoint xl with width 1280 › searchbar active
visual-regression/components/header.spec.ts:29:9 › header › rtl › screen at breakpoint lg with width 1024 › filters open
visual-regression/components/header.spec.ts:37:9 › header › rtl › screen at breakpoint lg with width 1024 › resting
visual-regression/components/header.spec.ts:47:9 › header › rtl › screen at breakpoint lg with width 1024 › scrolled
visual-regression/components/header.spec.ts:57:9 › header › rtl › screen at breakpoint lg with width 1024 › searchbar hovered
visual-regression/components/header.spec.ts:69:9 › header › rtl › screen at breakpoint lg with width 1024 › searchbar active
visual-regression/pages/homepage.spec.ts:30:7 › ltr homepage snapshots › screen at breakpoint md with width 768 › ltr full page
visual-regression/pages/homepage.spec.ts:45:9 › ltr homepage snapshots › search input › screen at breakpoint 2xl with width 1536 › focused
visual-regression/pages/homepage.spec.ts:45:9 › ltr homepage snapshots › search input › screen at breakpoint xl with width 1280 › focused
visual-regression/pages/homepage.spec.ts:45:9 › ltr homepage snapshots › search input › screen at breakpoint lg with width 1024 › focused
visual-regression/pages/homepage.spec.ts:38:9 › ltr homepage snapshots › search input › screen at breakpoint md with width 768 › unfocused
visual-regression/pages/homepage.spec.ts:45:9 › ltr homepage snapshots › search input › screen at breakpoint md with width 768 › focused
visual-regression/pages/homepage.spec.ts:45:9 › ltr homepage snapshots › search input › screen at breakpoint sm with width 640 › focused
visual-regression/pages/homepage.spec.ts:45:9 › ltr homepage snapshots › search input › screen at breakpoint xs with width 340 › focused
visual-regression/pages/homepage.spec.ts:30:7 › rtl homepage snapshots › screen at breakpoint md with width 768 › rtl full page
visual-regression/pages/homepage.spec.ts:45:9 › rtl homepage snapshots › search input › screen at breakpoint 2xl with width 1536 › focused
visual-regression/pages/homepage.spec.ts:45:9 › rtl homepage snapshots › search input › screen at breakpoint xl with width 1280 › focused
visual-regression/pages/homepage.spec.ts:45:9 › rtl homepage snapshots › search input › screen at breakpoint lg with width 1024 › focused
visual-regression/pages/homepage.spec.ts:38:9 › rtl homepage snapshots › search input › screen at breakpoint md with width 768 › unfocused
visual-regression/pages/homepage.spec.ts:45:9 › rtl homepage snapshots › search input › screen at breakpoint md with width 768 › focused
visual-regression/pages/pages.spec.ts:31:9 › non-existent ltr page snapshots › screen at breakpoint 2xl with width 1536 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent ltr page snapshots › screen at breakpoint xl with width 1280 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent ltr page snapshots › screen at breakpoint lg with width 1024 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent ltr page snapshots › screen at breakpoint md with width 768 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent ltr page snapshots › screen at breakpoint sm with width 640 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent ltr page snapshots › screen at breakpoint xs with width 340 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent rtl page snapshots › screen at breakpoint 2xl with width 1536 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent rtl page snapshots › screen at breakpoint xl with width 1280 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent rtl page snapshots › screen at breakpoint lg with width 1024 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent rtl page snapshots › screen at breakpoint md with width 768 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent rtl page snapshots › screen at breakpoint sm with width 640 › full page
visual-regression/pages/pages.spec.ts:31:9 › non-existent rtl page snapshots › screen at breakpoint xs with width 340 › full page

Implementation

• 🙋 I would be interested in implementing this feature.

[sarayourfriend]

I worry a little bit about setting the tolerance to 0. If you check out the visual diffs for this PR, for example, the changes amount to some slight shifting in the logo position and maybe a pixel's worth of difference in the leading margin of the page: https://github.com/WordPress/openverse-frontend/pull/1981/files

Are these changes accounted for in changes in the application? If so then my concern is probably safe to avoid.

[obulat]

Are these changes accounted for in changes in the application? If so then my concern is probably safe to avoid.

I think they are, but I'm can't be certain until we merge the other PRs. Considering that previously, when the tolerance was at 0, we did not have problems with the header logo position, I suspect we won't have it now. The biggest flakiness was in the snapshots of elements that are located below a single result image. It might make sense to use the Storybook tests for them. We've already moved the Media reuse tabs test to the Storybook, we probably should also move the External sources popover to the Storybook tests as well.

We might want to set the tolerance to 0, and adjust if we see flakiness.

[sarayourfriend]

Ah! I missed the detail that we'd already been using 0 for the threshold before. Thanks for the helpful explanation 🙏