Failing test: X-Pack Accessibility Tests.x-pack/test/accessibility/apps/lens·ts - Lens "after all" hook for "saves lens chart"

[kibanamachine]

A test failed on a tracked branch

Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="deleteSelectedItems"])
Wait timed out after 10057ms
    at /dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at onFailure (/dev/shm/workspace/parallel/4/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/4/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at Retry.try (/dev/shm/workspace/parallel/4/kibana/test/common/services/retry/retry.ts:32:14)
    at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/4/kibana/test/functional/services/common/find.ts:419:7)
    at TestSubjects.click (/dev/shm/workspace/parallel/4/kibana/test/functional/services/common/test_subjects.ts:105:7)
    at ListingTable.clickDeleteSelected (/dev/shm/workspace/parallel/4/kibana/test/functional/services/listing_table.ts:141:7)
    at Context.<anonymous> (test/accessibility/apps/lens.ts:28:7)
    at Object.apply (/dev/shm/workspace/parallel/4/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-accessibility (Project:Accessibility)

[kibanamachine]

New failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[spalger]

10 failures in master, 7.x, and PRs in this suite since #88623 was merged 12 hours ago. CC @markov00

Skipped
master: 97c7f5c
7.x/7.12: 2e1d055

[kibanamachine]

New failure: Jenkins Build

[flash1293]

It looks like this failed because the sample data wasn't installed correctly - after navigating to visualize it was redirected to index pattern management. Maybe some wait time before navigating to visualize would do the trick?

[flash1293]

I think this is an instability of either sample data or the way Elasticsearch is used in the functional tests. In some cases the sample data is available, but the saved objects are missing, in some cases not even those are loaded:

I checked the code of sample data and they create the saved objects as last step with the default refresh parameter which is "wait_for", so the UI shouldn't refresh before the saved objects are available in Elasticsearch.

@spalger This looks somewhat similar to another failure we were looking into: #77567 It looks like in some situations saved objects or other Elasticsearch data is not immediately available.

Could you take a look from this new angle? Based on the code it seems very unlikely this is related to migration somehow.

[flash1293]

The failures are pretty rare, If we had some CI instability in the last few days, that might explain it as well (that's not scientific but personally I saw quite a few PRs failing in weird and flaky ways lately).

[flash1293]

I hope #91000 fill fix this one as well

[kibanamachine]

New failure: CI Build - master

[kibanamachine]

New failure: CI Build - master

[kibanamachine]

New failure: CI Build - master

[flash1293]

Seems like a real failure, not sure what started to trigger it:

VIOLATION
--
  | │   [aria-input-field-name]: Ensures every ARIA input field has an accessible name
  | │     Help: https://dequeuniversity.com/rules/axe/4.0/aria-input-field-name?application=axeAPI
  | │     Elements:
  | │       - .euiSuperSelect__listbox
  | │       at AccessibilityService.assertValidAxeReport (/opt/local-ssd/buildkite/builds/kb-cigroup-6-5ddd82f32068eec7/elastic/kibana-hourly/kibana/test/accessibility/services/a11y/a11y.ts:76:13)
  | │       at AccessibilityService.testAppSnapshot (/opt/local-ssd/buildkite/builds/kb-cigroup-6-5ddd82f32068eec7/elastic/kibana-hourly/kibana/test/accessibility/services/a11y/a11y.ts:49:10)
  | │       at runMicrotasks (<anonymous>)
  | │       at processTicksAndRejections (node:internal/process/task_queues:96:5)
  | │       at Context.<anonymous> (test/accessibility/apps/lens.ts:90:7)
  | │       at Object.apply (/opt/local-ssd/buildkite/builds/kb-cigroup-6-5ddd82f32068eec7/elastic/kibana-hourly/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function

Screenshot doesn't show anything weird

Maybe this is an EUI problem? @chandlerprall is there anything known about this problem already by any chance? euiSuperSelect__listbox seems like to be an internal class name so I guess it's not limited to this single place.

[chandlerprall]

The euiSuperSelect__listbox element hasn't be changed in quite a while, but perhaps the AXE rule was changed or expanded and now errors?

https://github.com/elastic/eui/blob/master/src/components/form/super_select/super_select.tsx#L361-L366

        <div
          className="euiSuperSelect__listbox"
          role="listbox"
          aria-activedescendant={valueOfSelected}
          tabIndex={0}
        >

Seems valid that it should be given an accessible name as it has role + tabIndex defined. @1Copenut any thoughts here?

[1Copenut]

UPDATE:
Logged into EUI as elastic/eui#5292

Looking at the issue locally on EUI, I have been able to re-create it. I'm guessing the functional test may have run with the EuiSuperSelect still expanded, and caught the listbox without an accessible label. This issue should be addressed in EUI. I'll add a ticket now and update my comment with a number in a few.

[flash1293]

Awesome, thanks @1Copenut - I will mark this as blocked until the EUI fix lands in Kibana

[flash1293]

Same underlying problem as #115614

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[flash1293]

I'm not sure where this failure is coming from. It says

[00:11:29]       │   [aria-valid-attr-value]: Ensures all ARIA attributes have valid values
[00:11:29]       │     Impact: critical
[00:11:29]       │     Help: https://dequeuniversity.com/rules/axe/4.0/aria-valid-attr-value?application=axeAPI
[00:11:29]       │     Elements:
[00:11:29]       │       - <input type="search" placeholder="Filter options" class="euiFieldSearch euiFormControlLayout--1icons euiFieldSearch--fullWidth euiFieldSearch-isClearable euiSelectableSearch lnsChartSwitch__search" autocomplete="off" aria-haspopup="listbox" aria-activedescendant="iec0203c0-c585-11ec-b7e1-a710218dd424_listbox_option--1" aria-label="Filter options" data-test-subj="lnsChartSwitchSearch" value="ine" role="combobox" aria-autocomplete="list" aria-expanded="true" aria-controls="iec0203c0-c585-11ec-b7e1-a710218dd424_listbox" aria-owns="iec0203c0-c585-11ec-b7e1-a710218dd424_listbox">

But the dom doesn't even contain this element anymore in the failure snapshot. Also the aria attributes seem valid to me provided the referenced controlled listbox component actually exists.

I'm fixing this by making sure the test is executed as long as the chart switcher is fully rendered.

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[flash1293]

The failure from above didn't have the fix yet