Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate WCAG testing #987

Merged
merged 15 commits into from
Oct 21, 2021
Merged

Automate WCAG testing #987

merged 15 commits into from
Oct 21, 2021

Conversation

yen-tt
Copy link
Contributor

@yen-tt yen-tt commented Oct 20, 2021

  • add github action to automate wcag test on pull requests to release, hotfix, support, and develop branches
  • add axe-core/puppeteer to repo, and run analysis on all pages used in percy snapshot tests

J=1644
TEST=manual/auto

  • ran npm run wcag with puppeteer's browser headless mode disabled, see that all pages are loaded and analyzed by axe-core, see that the violation results are console log out.
  • see that github action wcag test ran in this pr

- add github action to automate wcag test on release, hotfix, support, and develop branches
- add axe-core/puppeteer to repo, and run analysis on all pages used in percy snapshot tests

J=1644
TEST=manual/auto

- ran `npm run wcag` with puppeteer's browser headless mode disabled, see that all pages are loaded and analyzed by axe-core, see that the violation results are console log out.
@coveralls
Copy link

coveralls commented Oct 20, 2021

Coverage Status

Coverage decreased (-0.01%) to 9.18% when pulling fc139fa on dev/automate-wcag into 9b70dfe on develop.

@yen-tt yen-tt added the WIP label Oct 20, 2021
@yen-tt yen-tt removed the WIP label Oct 20, 2021
.github/workflows/wcag_test.yml Show resolved Hide resolved
}


async _analyzeUniversalSearch() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to mimic the exact flow of the Photographer class. I think we should do some re-factoring here. For both classes, we go to the same pages, run the same queries, etc. If we add a new page or want to run a new query, we will have to update both files.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure if we would always test the same page flow between percy and wcag. But I guess anything UI related snapshots would be best to run by wcag analysis as well. I will do some refactoring

Copy link
Member

@cea2aj cea2aj Oct 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we move towards a CAC approach? Something like:

interface TestCommand {
  type: 'click',
  params?: any[]
}

interface TestLocation {
  pageType: 'vertical' | 'universal',
  page?: string
  name: string
  viewPort?: {
    height?: number,
    width?: number
  }
  commands?: TestCommand[]
}

const testingLocations : TestLocation[] = [
  {
    name: 'universal-search',
    pageType: 'universal'
  },
  {
    name: 'vertical-grid-search',
    pageType: 'vertical',
    page: 'people'
  },
  {
    name: 'universal-search-faq-accordion',
    pageType: 'universal',
    commands: [
      { type: 'click', params: ['.HitchhikerFaqAccordion-toggle']}
    ]
  }
]

Rather than an array of TestLocations, we could remove the name from TestLocation and have a Record<name,TestLocation>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, that makes sense. Although we wouldn't have nice interface/typing since theme is only in js right now. Yeah I think having a PageOperator that runs through these test configs would be good.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would this TestLocation[] fit into what we have currently? Would the Photographer and WCAGReporter classes take them in? Then, for each entry in the array, the respective class would know what action to take? Another approach is to simply have a class that does all the navigating, clicking, etc. and takes the snapshot and does the WCAG analysis. For example:

    await this._pageNavigator.gotoUniversalPage({ query: 'what if i forget my password?'});
    await this._pageNavigator.click('.HitchhikerFaqAccordion-toggle')
    await this._camera.snapshot('universal-search--faq-accordion');
    this.results.push(await this._analyzer.analyze());

This approach is fairly straightforward and we don't need to create a CaC framework.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd need to create some new code such as the PageOperator that Yen mentioned which would parse the file and run the commands. I like the idea of the CaC approach, but if we'd rather close this out sooner, then it probably makes more sense to do as you suggested

Copy link
Contributor Author

@yen-tt yen-tt Oct 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the CaC approach is pretty neat, and it might also help us out with organizing the multilangphotographer's tests as well. I imagine it's something like this:

while(pageOperator.hasNextTestLocation()) {
   const testContext = await pageOperator.nextTestLocation();
   this._camera.snapshot(testContext); OR this.results.push(await this._analyzer.analyze());
}

I think I have most of the pageOperator code that would parse the file and run the commands, but yes there would need to be more changes in other files and writing up the testLocations itself.
Perhaps, we can keep this PR as is for the format, and make a new item to address the CaC or page navigation refactoring in general?
@tmeyer2115 @cea2aj

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that works, I'll add the CaC idea to the 'ideas for product doc' and we can keep this PR on the current approach

tests/wcag/index.js Show resolved Hide resolved
@yen-tt yen-tt merged commit 6327a2d into develop Oct 21, 2021
@yen-tt yen-tt deleted the dev/automate-wcag branch October 21, 2021 18:11
@nmanu1 nmanu1 mentioned this pull request Nov 16, 2021
nmanu1 added a commit that referenced this pull request Nov 16, 2021
## Version 1.26.0
### Features
- Add support for the auth token, with a new global_config `token` field  (#976)
- Add support for the visitor, which can be set or changed at runtime (#970)
- Add support for generic thumbs up/down feedback buttons to all cards (#973, #1005)
- Handle page and template names with spaces (#988)

### Changes
- Update styling of navigation ‘More’ menu (#989)
- Update hover color of search bar buttons (#1000)
- Internal repo changes (automate WCAG testing (#987))

### Bugfixes
- Fix console error which would appear on google maps (#955)
- Fix extra query when loading in iframe integrations (#986)
- Fix bug that displayed links that didn’t exist (#990)
- Allow microphone access for voice search in iframe (#991)
- Fix issue with custom search icon size (#992)
- Fix bug with new prop comments not being added (#1004)
- Fix distance alignment on location cards (#1006)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants