Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

object-is-loaded-matches test flakiness in nightly CI builds #4529

Closed
1 task done
dbjorge opened this issue Jul 8, 2024 · 1 comment · Fixed by #4530
Closed
1 task done

object-is-loaded-matches test flakiness in nightly CI builds #4529

dbjorge opened this issue Jul 8, 2024 · 1 comment · Fixed by #4530
Labels
ungroomed Ticket needs a maintainer to prioritize and label

Comments

@dbjorge
Copy link
Contributor

dbjorge commented Jul 8, 2024

Product

axe-core

Product Version

4.9.1

Latest Version

  • I have tested the issue with the latest version of the product

Issue Description

Expectation

Test should work consistently

Actual

Test regularly fails (example) like this:

  object-is-loaded-matches
    ✗ returns false if the object shows any content
        expected true to be false
        AssertionError@node_modules/chai/chai.js:9227:13
        [3]</module.exports/Assertion.prototype.assert@node_modules/chai/chai.js:253:13
        [5]</module.exports/<@node_modules/chai/chai.js:1101:10
        propertyGetter@node_modules/chai/chai.js:7985:29
        proxyGetter@node_modules/chai/chai.js:9025:22
        [6]</module.exports/assert.isFalse@node_modules/chai/chai.js:4638:5
        @test/rule-matches/object-is-loaded-matches.js:37:12

Often passes on rerun

How to Reproduce

Wait for CI to fail

Additional context

n/a

@dbjorge dbjorge added the ungroomed Ticket needs a maintainer to prioritize and label label Jul 8, 2024
dbjorge added a commit that referenced this issue Jul 8, 2024
…re time to load <object> data (#4530)

These test failures are happening because the test isn't allowing
sufficient time for slower test scenarios (headed browsers in CI
machines) to finish setting up test fixtures before testing against
them.

I experimented with using `load`/`error` events on the fixtures instead
of a flat time, but this didn't work reliably; in the case where the
browser renders fallback content, it fires the `error` event *before*
the fallback content is ready, at least in Chromium stable. I wanted to
stick to a timebox for this so I settled for just increasing the time
allotment instead.

Closes: #4529
@dbjorge
Copy link
Contributor Author

dbjorge commented Jul 8, 2024

change only impacts unit tests, no QA required

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ungroomed Ticket needs a maintainer to prioritize and label
Projects
None yet
1 participant