-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[testharness.js] Ignore errors after completion #20239
base: master
Are you sure you want to change the base?
[testharness.js] Ignore errors after completion #20239
Conversation
Although uncaught exceptions are often an indication of errors in test design, the harness status cannot be safely modified following completion, so these exceptions must be ignored.
@gsnedders @jgraham Mind weighing in on this? |
infrastructure/expected-fail/unhandled-rejection-following-subtest.html
Outdated
Show resolved
Hide resolved
I assumed otherwise, but it looks like I was wrong. When cleaning up after running a script, a browser is supposed to perform a microtask checkpoint, and that involves notifying about rejected promises. Since the This event ordering is contentious, so I've submitted a test for it: #20329 |
No description provided. |
1 similar comment
No description provided. |
I think I'd quite like to do both this and freeze as much as possible when we're done? |
That's actually one alternative I'm not comfortable with. If we use both solutions, then one will be untested, and the redundancy may confuse future contributors. |
Although uncaught exceptions are often an indication of errors in test design, the harness status cannot be safely modified following completion, so these exceptions must be ignored.
Here are some comparisons between test results collected from
master
and test results collected with this patch applied:As usual, flaky tests make interpreting these comparisons a little tricky. Firefox's results don't seem to be influenced by this change, but Chrome's results have some differences.
I found 5 tests which report
ERROR
consistently inmaster
andOK
consistently with the patch applied.content-security-policy/script-src/script-src-strict_dynamic_hashes.html
- this test has a bug which I've reported with this issuehtml/syntax/parsing/html5lib_innerHTML_adoption01.html
- this test has a bug which I've tried to fix with this patchinfrastructure/expected-fail/unhandled-rejection-following-subtest.html
- this is a test for the behavior under consideration. I've added a "fixup!" commit to this branch to remove itmediacapture-image/MediaStreamTrack-getConstraints-fast.html
- this test has a bug which I've tried to fix with this patchtrusted-types/block-string-assignment-to-DOMWindowTimers-setTimeout-setInterval.tentative.html
- this test intentionally triggers an uncaught exception, so it seems to expect the behavior proposed here. As a "tentative" test, it may be that the authors aren't yet scrutinizing the results on wpt.fyiI found 5 tests which are flaky in
master
and consistent with the patch applied:content-security-policy/script-src/script-src-strict_dynamic_in_img-src.html
html/webappapis/microtask-queuing/queue-microtask-exceptions.any.worker.html
offscreen-canvas/*
(coincidentally, we just recently recognized that these tests are invalid)workers/baseurl/alpha/sharedworker-in-worker.html
workers/interfaces/WorkerGlobalScope/onerror/exception-in-onerror.html
If the effect of this change seems agreeable, I'd be happy to discuss alternative implementations. For instance, we could freeze the status object at the moment of completion, or we could call
structured_clone
(which would have a similar effect due to memoization).