Automate WCAG testing #1577

yen-tt · 2021-10-19T16:53:31Z

add new Github Action to run WCAG tests on push or pull request to develop, hotfix, and support branches
- will add this to generator-slapshot repo after pr approval
add pageNavigator, similar to theme's pageNavigator, which uses puppeteer API to automate browser navigation.
use axe-core/puppeteer library to generate WCAG report on each page, exit with error if there are violations
- this library have license of type MPL-2.0 (reviewed and approved, for test environment only)

Note:

This pr does not include all possible ui components in the test yet, should make a new item/pr!
This pr does not address all violations listed in the report, should make a new item/pr!

J=SLAP-1634
TEST=manual/auto

run npm run wcag locally, and see that a wcag report is properly generated for each url, and listed out the violations to be addressed.
See that github action ran on this pr

- add new Github Action to run WCAG tests on push or pull request to develop, hotfix, and support branches - add pageNavigator, similar to theme's pageNavigator, which uses puppeteer API to automate browser nagivation. - use axe-core/puppeteer library to generate WCAG report on each page J=SLAP-1634 TEST=manual/auto - run 'npm run wcag' locally, and see that a wcag report is properly generated for each url, and listed out the violations to be addressed (should have another item(s) to do so) - testing github action on this pr..

…y notices check

coveralls · 2021-10-19T16:57:14Z

Coverage increased (+0.01%) to 61.223% when pulling 26559e9 on dev/automate-wcag-test into 383fb3d on develop.

.github/workflows/wcag_test.yml

tests/acceptance/wcag/index.js

tests/acceptance/wcag/pagenavigator.js

tmeyer2115 · 2021-10-19T17:15:24Z

tests/acceptance/wcag/utils.js

@@ -0,0 +1,44 @@
+
+/**
+ * Waits until the content of the page stops changing.


For my own edification, the goto method does not wait until the DOM of the page is fully loaded? Instead of rolling our own wait here, could we use page.waitForNavigation()? It's referenced a couple places:

https://stackoverflow.com/questions/52497252/puppeteer-wait-until-page-is-completely-loaded

https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#pagewaitfornavigationoptions

No, goto doesn't wait for content to be fully loaded. I think we tried that and with waitForNavigation/other waitFor approach before in theme and it didn't work well so we resolve it with our custom wait. I tested waitForNavigation out locally just now with different network and DOM content loaded options, it's always stuck on the first page.

…ch-ui into dev/automate-wcag-test

…y notices check

…ch-ui into dev/automate-wcag-test

tests/acceptance/pagenavigator.js

cea2aj · 2021-10-21T13:20:09Z

tests/acceptance/wcag/index.js

+  reporter: 'no-passes',
+  runOnly: {
+    type: 'tag',
+    values: ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag2aaa']


I would check with Rose, but I think currently we are only trying to meet WCAG AA compliance. (We can leave AAA in here for now, but when we go to address these concerns, we may want to bump down to just AA)

Rose said Level AA for WCAG 2.0 and 2.1 should be good!

cea2aj · 2021-10-21T13:29:19Z

Maybe we should merge this PR into a feature branch rather than develop until we address the failures? If we merge into develop, every PR will fail to pass the tests

tmeyer2115 · 2021-10-21T13:40:54Z

Maybe we should merge this PR into a feature branch rather than develop until we address the failures? If we merge into develop, every PR will fail to pass the tests

IMO, I would rather we merge this in now. Long-lived feature branches can become cumbersome. Merge conflicts can arise, etc. You have to constantly rebase. Especially here. If we don't rebase constantly, then the WCAG checks wouldn't catch any new failures added in develop. If possible, I think we should ensure that WCAG failures do not block the merging of a PR. They only register as warnings (if that's a thing). Once we are fully WCAG compliant, we can change it so that failure would block a PR.

cea2aj · 2021-10-21T13:51:31Z

Maybe we should merge this PR into a feature branch rather than develop until we address the failures? If we merge into develop, every PR will fail to pass the tests

IMO, I would rather we merge this in now. Long-lived feature branches can become cumbersome. Merge conflicts can arise, etc. You have to constantly rebase. Especially here. If we don't rebase constantly, then the WCAG checks wouldn't catch any new failures added in develop. If possible, I think we should ensure that WCAG failures do not block the merging of a PR. They only register as warnings (if that's a thing). Once we are fully WCAG compliant, we can change it so that failure would block a PR.

Makes sense. We'll just want to watch out for the boy who cried wolf so that we don't start ignoring tests because they are always failing. It doesn't look like warnings are a thing except for code annotations which probably isn't what we are looking for.

## Version 1.12.0 ### Features - Allow search rate tracking (#1558) - Add support for setting and changing the visitor and passing it to answers-core (#1564) - Add support for the auth token that is passed in from the config (#1566) - Add an `environment` field to support consumer auth in Sandbox (#1597) - Allow components to override the beforeMount function (#1547) - Add distance to the card data and a function to format it (#1550) - WCAG updates (allow pagination with Enter (#1575), identify current page in navigation tab (#1576), update autocomplete screen reader support (#1578, #1579)) ### Changes - Update directAnswers component data to include the searcher (#1596) - Use custom alerts instead of window.alert (#1549) - Update Mapbox version to match the Theme (#1551) - Internal repo changes (#1562, #1577) ### Bugfixes - Fix console error which would appear on google maps (#1548) - Fix FAQ expansion when default is expanded (#1553) - Fix error for searches on page load with no businessId (#1561)

Yen Truong and others added 2 commits October 19, 2021 12:49

Automated update to THIRD-PARTY-NOTICES from github action's 3rd part…

cf7b570

…y notices check

tmeyer2115 reviewed Oct 19, 2021

View reviewed changes

Yen Truong added 3 commits October 19, 2021 13:30

guard error

74f11dc

Merge branch 'dev/automate-wcag-test' of github.com:yext/answers-sear…

5a6cd31

…ch-ui into dev/automate-wcag-test

test remove click

1a71186

yen-tt added the WIP Work in progress label Oct 19, 2021

Yen Truong and others added 12 commits October 19, 2021 14:04

feedback

7fb74a0

documentation

c05b89a

waitTillNavigation test

2c40747

test click null

51f8617

use $eval

2f5d068

different selector

34b0396

try classname css selector

9d5f564

Automated update to THIRD-PARTY-NOTICES from github action's 3rd part…

e61fb2c

…y notices check

different input selector

c8d7d6c

Merge branch 'dev/automate-wcag-test' of github.com:yext/answers-sear…

27e000e

…ch-ui into dev/automate-wcag-test

revert changes

faa4e06

revert change

89ff890

yen-tt force-pushed the dev/automate-wcag-test branch from 1466842 to 89ff890 Compare October 19, 2021 20:24

Yen Truong added 2 commits October 19, 2021 16:28

percy update

6f4938e

rename

001833e

yen-tt removed the WIP Work in progress label Oct 19, 2021

yen-tt requested review from nmanu1, cea2aj and tmeyer2115 October 19, 2021 20:42

tmeyer2115 reviewed Oct 20, 2021

View reviewed changes

tests/acceptance/pagenavigator.js Show resolved Hide resolved

vilation reults length check

26559e9

yen-tt requested a review from tmeyer2115 October 20, 2021 20:10

cea2aj reviewed Oct 21, 2021

View reviewed changes

tmeyer2115 approved these changes Oct 21, 2021

View reviewed changes

yen-tt merged commit c3ae4b7 into develop Oct 21, 2021

yen-tt deleted the dev/automate-wcag-test branch October 21, 2021 15:03

nmanu1 mentioned this pull request Nov 16, 2021

Version 1.12 #1604

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automate WCAG testing #1577

Automate WCAG testing #1577

yen-tt commented Oct 19, 2021 •

edited

Loading

coveralls commented Oct 19, 2021 •

edited

Loading

tmeyer2115 Oct 19, 2021

yen-tt Oct 19, 2021 •

edited

Loading

cea2aj Oct 21, 2021 •

edited

Loading

yen-tt Oct 21, 2021

yen-tt Oct 21, 2021

cea2aj commented Oct 21, 2021

tmeyer2115 commented Oct 21, 2021

cea2aj commented Oct 21, 2021 •

edited

Loading

		@@ -0,0 +1,44 @@

		/**
		* Waits until the content of the page stops changing.

Automate WCAG testing #1577

Automate WCAG testing #1577

Conversation

yen-tt commented Oct 19, 2021 • edited Loading

coveralls commented Oct 19, 2021 • edited Loading

tmeyer2115 Oct 19, 2021

Choose a reason for hiding this comment

yen-tt Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

cea2aj Oct 21, 2021 • edited Loading

Choose a reason for hiding this comment

yen-tt Oct 21, 2021

Choose a reason for hiding this comment

yen-tt Oct 21, 2021

Choose a reason for hiding this comment

cea2aj commented Oct 21, 2021

tmeyer2115 commented Oct 21, 2021

cea2aj commented Oct 21, 2021 • edited Loading

yen-tt commented Oct 19, 2021 •

edited

Loading

coveralls commented Oct 19, 2021 •

edited

Loading

yen-tt Oct 19, 2021 •

edited

Loading

cea2aj Oct 21, 2021 •

edited

Loading

cea2aj commented Oct 21, 2021 •

edited

Loading