Skip to content

Commit

Permalink
refactor: exclude descendancy based rules from Jest API and non-a11y …
Browse files Browse the repository at this point in the history 
…failures from custom reporter (#78)
  • Loading branch information
Mohan Raj Rajamanickam authored Oct 26, 2021
1 parent 8432157 commit 93de21f
Show file tree
Hide file tree
Showing 26 changed files with 1,019 additions and 3,194 deletions.
1 change: 0 additions & 1 deletion .github/CODEOWNERS
View file
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
* @salesforce/sa11y-reviewers
/packages/preset-rules @cordeliadillon @salesforce/sa11y-reviewers
120 changes: 65 additions & 55 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,64 +3,74 @@
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

- [3.0.0 (2021-09-16)](#300-2021-09-16)
- [0.11.2-beta (2021-09-13)](#0112-beta-2021-09-13)
- [WDIO](#wdio)
- [0.11.1-beta (2021-09-10)](#0111-beta-2021-09-10)
- [Automatic checks](#automatic-checks)
- [Results processor](#results-processor)
- [WDIO](#wdio-1)
- [0.11.0-beta (2021-08-26)](#0110-beta-2021-08-26)
- [BREAKING CHANGES ⚠️](#breaking-changes-)
- [Features](#features)
- [0.10.0-beta (2021-08-04)](#0100-beta-2021-08-04)
- [Features](#features-1)
- [0.9.1-beta (2021-07-05)](#091-beta-2021-07-05)
- [Fix](#fix)
- [0.9.0-beta (2021-06-15)](#090-beta-2021-06-15)
- [Features](#features-2)
- [0.8.0-beta (2021-03-19)](#080-beta-2021-03-19)
- [Features](#features-3)
- [BREAKING CHANGES](#breaking-changes)
- [Refactor](#refactor)
- [0.7.0-beta (2021-02-24)](#070-beta-2021-02-24)
- [Features](#features-4)
- [0.6.0-beta (2021-01-22)](#060-beta-2021-01-22)
- [Features](#features-5)
- [0.5.0-beta (2020-12-01)](#050-beta-2020-12-01)
- [Features](#features-6)
- [0.4.1-beta (2020-11-24)](#041-beta-2020-11-24)
- [Bug Fixes](#bug-fixes)
- [0.4.0-beta (2020-10-31)](#040-beta-2020-10-31)
- [Features](#features-7)
- [Refactor](#refactor-1)
- [0.3.2-beta (2020-09-22)](#032-beta-2020-09-22)
- [Bug Fixes](#bug-fixes-1)
- [Refactor](#refactor-2)
- [0.3.1-beta (2020-08-20)](#031-beta-2020-08-20)
- [Refactor](#refactor-3)
- [0.3.0-beta (2020-08-19)](#030-beta-2020-08-19)
- [Features](#features-8)
- [0.2.0-beta (2020-06-25)](#020-beta-2020-06-25)
- [Features](#features-9)
- [Bug Fixes](#bug-fixes-2)
- [BREAKING CHANGES](#breaking-changes-1)
- [v0.1.0-alpha (2020-06-02)](#v010-alpha-2020-06-02)
- [Features](#features-10)
- [(2020-05-29)](#2020-05-29)
- [Features](#features-11)
- [(2020-05-07)](#2020-05-07)
- [Bug Fixes](#bug-fixes-3)
- [Features](#features-12)
- [(2020-04-07)](#2020-04-07)
- [Bug Fixes](#bug-fixes-4)
- [Features](#features-13)
- [(2020-03-20)](#2020-03-20)
- [Bug Fixes](#bug-fixes-5)
- [Features](#features-14)
- [3.0.1 (2021-10-21)](#301-2021-10-21)
- [Refactor](#refactor)
- [3.0.0 (2021-09-16)](#300-2021-09-16)
- [0.11.2-beta (2021-09-13)](#0112-beta-2021-09-13)
- [WDIO](#wdio)
- [0.11.1-beta (2021-09-10)](#0111-beta-2021-09-10)
- [Automatic checks](#automatic-checks)
- [Results processor](#results-processor)
- [WDIO](#wdio-1)
- [0.11.0-beta (2021-08-26)](#0110-beta-2021-08-26)
- [BREAKING CHANGES ⚠️](#breaking-changes-)
- [Features](#features)
- [0.10.0-beta (2021-08-04)](#0100-beta-2021-08-04)
- [Features](#features-1)
- [0.9.1-beta (2021-07-05)](#091-beta-2021-07-05)
- [Fix](#fix)
- [0.9.0-beta (2021-06-15)](#090-beta-2021-06-15)
- [Features](#features-2)
- [0.8.0-beta (2021-03-19)](#080-beta-2021-03-19)
- [Features](#features-3)
- [BREAKING CHANGES](#breaking-changes)
- [Refactor](#refactor-1)
- [0.7.0-beta (2021-02-24)](#070-beta-2021-02-24)
- [Features](#features-4)
- [0.6.0-beta (2021-01-22)](#060-beta-2021-01-22)
- [Features](#features-5)
- [0.5.0-beta (2020-12-01)](#050-beta-2020-12-01)
- [Features](#features-6)
- [0.4.1-beta (2020-11-24)](#041-beta-2020-11-24)
- [Bug Fixes](#bug-fixes)
- [0.4.0-beta (2020-10-31)](#040-beta-2020-10-31)
- [Features](#features-7)
- [Refactor](#refactor-2)
- [0.3.2-beta (2020-09-22)](#032-beta-2020-09-22)
- [Bug Fixes](#bug-fixes-1)
- [Refactor](#refactor-3)
- [0.3.1-beta (2020-08-20)](#031-beta-2020-08-20)
- [Refactor](#refactor-4)
- [0.3.0-beta (2020-08-19)](#030-beta-2020-08-19)
- [Features](#features-8)
- [0.2.0-beta (2020-06-25)](#020-beta-2020-06-25)
- [Features](#features-9)
- [Bug Fixes](#bug-fixes-2)
- [BREAKING CHANGES](#breaking-changes-1)
- [v0.1.0-alpha (2020-06-02)](#v010-alpha-2020-06-02)
- [Features](#features-10)
- [(2020-05-29)](#2020-05-29)
- [Features](#features-11)
- [(2020-05-07)](#2020-05-07)
- [Bug Fixes](#bug-fixes-3)
- [Features](#features-12)
- [(2020-04-07)](#2020-04-07)
- [Bug Fixes](#bug-fixes-4)
- [Features](#features-13)
- [(2020-03-20)](#2020-03-20)
- [Bug Fixes](#bug-fixes-5)
- [Features](#features-14)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

# 3.0.1 (2021-10-21)

### Refactor

- exclude descendancy rules from Jest API
- these are rules that might fail at unit/component level but might pass at page level
- change sa11y results processor to output only a11y failures

# [3.0.0](https://github.com/salesforce/sa11y/releases/tag/v3.0.0) (2021-09-16)

- Moving to unified versioning of @sa11y packages instead of independent versioning for reducing complexity of managing independent versions
Expand Down
1 change: 1 addition & 0 deletions cSpell.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"ignoreWords": [
"circleci",
"CNCF",
"descendancy",
"dlitem",
"doctoc",
"goog",
Expand Down
2 changes: 1 addition & 1 deletion lerna.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"packages": ["packages/*"],
"version": "3.0.0",
"version": "3.0.1",
"exact": true,
"npmClient": "yarn",
"useWorkspaces": true,
Expand Down
26 changes: 13 additions & 13 deletions package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@
}
},
"devDependencies": {
"@babel/core": "7.15.5",
"@babel/preset-env": "7.15.6",
"@babel/core": "7.15.8",
"@babel/preset-env": "7.15.8",
"@babel/preset-typescript": "7.15.0",
"@commitlint/cli": "13.1.0",
"@commitlint/config-conventional": "13.1.0",
"@commitlint/cli": "13.2.1",
"@commitlint/config-conventional": "13.2.0",
"@types/jest": "26.0.24",
"@types/node": "15.14.1",
"@typescript-eslint/eslint-plugin": "4.31.1",
Expand All @@ -68,19 +68,19 @@
"@wdio/mocha-framework": "6.10.4",
"@wdio/spec-reporter": "6.11.0",
"@wdio/sync": "6.10.4",
"babel-jest": "27.2.0",
"chromedriver": "93.0.1",
"babel-jest": "27.3.1",
"chromedriver": "94.0.0",
"codecov": "3.8.3",
"commitizen": "4.2.4",
"conventional-changelog-cli": "2.1.1",
"cspell": "5.9.1",
"cspell": "5.12.3",
"depcheck": "1.4.2",
"doctoc": "2.0.1",
"doctoc": "2.1.0",
"eslint": "7.32.0",
"eslint-config-prettier": "8.3.0",
"eslint-import-resolver-typescript": "2.5.0",
"eslint-plugin-eslint-comments": "3.2.0",
"eslint-plugin-import": "2.24.2",
"eslint-plugin-import": "2.25.2",
"eslint-plugin-jest": "24.4.0",
"eslint-plugin-markdown": "2.2.1",
"eslint-plugin-notice": "0.9.10",
Expand All @@ -90,14 +90,14 @@
"husky": "7.0.2",
"jest": "26.6.3",
"lerna": "4.0.0",
"lint-staged": "11.1.2",
"lint-staged": "11.2.3",
"lockfile-lint": "4.6.2",
"markdown-link-check": "3.8.7",
"package-dependency-graph": "1.11.2",
"prettier": "2.4.0",
"prettier": "2.4.1",
"rimraf": "3.0.2",
"ts-node": "10.1.0",
"typescript": "4.4.3",
"ts-node": "10.3.0",
"typescript": "4.4.4",
"vertioner": "1.0.6",
"wdio-chromedriver-service": "6.0.4",
"webdriverio": "6.12.1"
Expand Down
10 changes: 5 additions & 5 deletions packages/assert/package.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@sa11y/assert",
"version": "3.0.0",
"version": "3.0.1",
"description": "Provides assertAccessible API to check DOM for accessibility issues",
"license": "BSD-3-Clause",
"homepage": "https://github.com/salesforce/sa11y/tree/master/packages/assert#readme",
Expand All @@ -21,13 +21,13 @@
"dist/**/*.d.ts*"
],
"dependencies": {
"@sa11y/common": "3.0.0",
"@sa11y/format": "3.0.0",
"@sa11y/preset-rules": "3.0.0",
"@sa11y/common": "3.0.1",
"@sa11y/format": "3.0.1",
"@sa11y/preset-rules": "3.0.1",
"axe-core": "4.3.3"
},
"devDependencies": {
"@sa11y/test-utils": "3.0.0"
"@sa11y/test-utils": "3.0.1"
},
"publishConfig": {
"access": "public"
Expand Down
16 changes: 8 additions & 8 deletions packages/browser-lib/package.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@sa11y/browser-lib",
"version": "3.0.0",
"version": "3.0.1",
"description": "Provides a minified version of selected `@sa11y` libraries to be injected into a browser (using webdriver) and executed from integration testing workflows.",
"license": "BSD-3-Clause",
"homepage": "https://github.com/salesforce/sa11y/tree/master/packages/browser-lib#readme",
Expand Down Expand Up @@ -34,17 +34,17 @@
"url": "https://github.com/salesforce/sa11y/issues"
},
"dependencies": {
"@sa11y/format": "3.0.0",
"@sa11y/preset-rules": "3.0.0",
"@sa11y/format": "3.0.1",
"@sa11y/preset-rules": "3.0.1",
"axe-core": "4.3.3"
},
"devDependencies": {
"@rollup/plugin-commonjs": "20.0.0",
"@rollup/plugin-node-resolve": "13.0.4",
"@rollup/plugin-commonjs": "21.0.1",
"@rollup/plugin-node-resolve": "13.0.6",
"@rollup/plugin-replace": "3.0.0",
"@sa11y/common": "3.0.0",
"@sa11y/test-utils": "3.0.0",
"rollup": "2.56.3",
"@sa11y/common": "3.0.1",
"@sa11y/test-utils": "3.0.1",
"rollup": "2.58.0",
"rollup-plugin-polyfill-node": "0.7.0",
"rollup-plugin-progress": "1.1.2",
"rollup-plugin-sizes": "1.0.4",
Expand Down
2 changes: 1 addition & 1 deletion packages/common/package.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@sa11y/common",
"version": "3.0.0",
"version": "3.0.1",
"description": "Common utilities, constants, error messages for @sa11y",
"license": "BSD-3-Clause",
"homepage": "https://github.com/salesforce/sa11y/tree/master/packages/common#readme",
Expand Down
8 changes: 4 additions & 4 deletions packages/format/package.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@sa11y/format",
"version": "3.0.0",
"version": "3.0.1",
"description": "Accessibility results re-formatter",
"license": "BSD-3-Clause",
"homepage": "https://github.com/salesforce/sa11y/tree/master/packages/format#readme",
Expand All @@ -24,12 +24,12 @@
"dist/**/*.d.ts*"
],
"dependencies": {
"@sa11y/preset-rules": "3.0.0",
"@sa11y/preset-rules": "3.0.1",
"axe-core": "4.3.3"
},
"devDependencies": {
"@sa11y/common": "3.0.0",
"@sa11y/test-utils": "3.0.0"
"@sa11y/common": "3.0.1",
"@sa11y/test-utils": "3.0.1"
},
"publishConfig": {
"access": "public"
Expand Down
45 changes: 34 additions & 11 deletions packages/jest/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Accessibility matcher for [Jest](https://jestjs.io)
- [Test module level](#test-module-level)
- [Usage](#usage)
- [Caveats](#caveats)
- [Disabled checks](#disabled-checks)
- [Automatic checks](#automatic-checks)
- [Using environment variables](#using-environment-variables)
- [Sa11y results processor](#sa11y-results-processor)
Expand Down Expand Up @@ -126,16 +127,38 @@ it('should be accessible', async () => {
- **async**: `toBeAccessible` **must** be invoked with `async/wait` or `Promise` or the equivalent supported asynchronous method in your environment
- Not invoking it async would result in incorrect results e.g. no issues reported even when the page is not accessible
- `Promise` should not be mixed together with `async/wait`. Doing so could result in Jest timeout and other errors.
- In spite of using async/await correctly if you run into the error `Axe is already running. Use await axe.run() to wait for the previous run to finish before starting a new run.` try running tests serially by using Jest's [run in band option](https://jestjs.io/docs/cli#--runinband).
- **useRealTimers**: ⏲ When Timer is mocked (e.g. `jest.useFakeTimers()`) accessibility API can timeout. Before invoking the accessibility API switch to the real timer (e.g. `jest.useRealTimers()`).
- **DOM**: 💡 The accessibility checks _cannot_ be run on static HTML markup. They can only be run against a rendered DOM.
- **color-contrast**: 🍭 Color-contrast check is disabled for Jest tests as it [does not work in JSDOM](https://github.com/dequelabs/axe-core/issues/595)
- **audio, video**: 📹 Accessibility of `audio`, `video` elements cannot be checked with Jest as they are [stubbed out in JSDOM](https://github.com/jsdom/jsdom/issues/2155)
- **template**: [`<template>` elements](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/template) are not rendered in DOM and hence cannot be checked directly without rendering. They have to be rendered before they can be checked.

#### Disabled checks

Following rules are disabled for the Sa11y Jest API

- **descendancy checks**:
- Following rules are disabled as they might fail at unit/component level but might pass at page level
- aria-required-children
- aria-required-parent
- dlitem
- definition-list
- list
- listitem
- landmark-one-main
- Following rules are not disabled, but they might pass at the unit/component level but might fail at integration/page level.
- landmark-banner-is-top-level
- landmark-complementary-is-top-level
- landmark-contentinfo-is-top-level
- landmark-main-is-top-level
- landmark-no-duplicate-banner
- landmark-no-duplicate-contentinfo
- **color-contrast**: 🍭 Color-contrast check is disabled as it [does not work in JSDOM](https://github.com/dequelabs/axe-core/issues/595)
- **audio, video**: 📹 Accessibility of `audio`, `video` elements cannot be checked as they are [stubbed out in JSDOM](https://github.com/jsdom/jsdom/issues/2155)
- **real browser**: If you need to check for color-contrast, audio/video elements or any other checks which need the element to be rendered visually please use a real browser to test e.g. using [`@sa11y/wdio`](https://github.com/salesforce/sa11y/tree/master/packages/wdio#readme)

## Automatic checks

The sa11y API can be setup to be automatically invoked at the end of each test as an alternative to adding the `toBeAccessible` API at the end of each test.
The Sa11y Jest API can be setup to be automatically invoked at the end of each test as an alternative to adding the `toBeAccessible` API at the end of each test.

- When automatic checks are enabled each child element in the DOM body will be checked for a11y and failures reported as part of the test.

Expand All @@ -154,14 +177,14 @@ Automatic checks can also be enabled using environment variables
SA11Y_AUTO=1 SA11Y_CLEANUP=1 jest
```

- Invoking `jest` with environment variables as above will enable automatic checks with no changes required to `setup()`
- The environment variables can be used to set up parallel builds e.g., in a CI environment without code changes to `setup()` to opt-in to automatic checks
- Setting `SA11Y_DEBUG=1` will output verbose logging
- `SA11Y_AUTO_FILTER` can be used to specify a comma seperated list of test file paths to filter for automatic checks
- When specified, automatic checks will be run only on given files
- A file can be excluded from automatic checks by prefixing the filename with `!`
- e.g. `SA11Y_AUTO_FILTER='!exclude.test.js'`
- The file paths can be also be expressed as [regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Cheatsheet) if necessary
- Invoking `jest` with environment variables as above will enable automatic checks with no changes required to `setup()`
- The environment variables can be used to set up parallel builds e.g., in a CI environment without code changes to `setup()` to opt-in to automatic checks
- Setting `SA11Y_DEBUG=1` will output verbose logging
- `SA11Y_AUTO_FILTER` can be used to specify a comma seperated list of test file paths to filter for automatic checks
- When specified, automatic checks will be run only on given files
- A file can be excluded from automatic checks by prefixing the filename with `!`
- e.g. `SA11Y_AUTO_FILTER='!exclude.test.js'`
- The file paths can be also be expressed as [regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Cheatsheet) if necessary

### Sa11y results processor

Expand Down
Loading

0 comments on commit 93de21f

Please sign in to comment.