Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[testharness.js] Ignore errors after completion #20239

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

jugglinmike
Copy link
Contributor

Although uncaught exceptions are often an indication of errors in test design, the harness status cannot be safely modified following completion, so these exceptions must be ignored.


Here are some comparisons between test results collected from master and test results collected with this patch applied:

As usual, flaky tests make interpreting these comparisons a little tricky. Firefox's results don't seem to be influenced by this change, but Chrome's results have some differences.

I found 5 tests which report ERROR consistently in master and OK consistently with the patch applied.

I found 5 tests which are flaky in master and consistent with the patch applied:

If the effect of this change seems agreeable, I'd be happy to discuss alternative implementations. For instance, we could freeze the status object at the moment of completion, or we could call structured_clone (which would have a similar effect due to memoization).

Although uncaught exceptions are often an indication of errors in test
design, the harness status cannot be safely modified following
completion, so these exceptions must be ignored.
@jugglinmike
Copy link
Contributor Author

@gsnedders @jgraham Mind weighing in on this?

@jugglinmike
Copy link
Contributor Author

This is hit during test collection, right? So this should still ERROR?

I assumed otherwise, but it looks like I was wrong.

When cleaning up after running a script, a browser is supposed to perform a microtask checkpoint, and that involves notifying about rejected promises. Since the load event doesn't fire until a couple tasks later (and since all of the relevant operations use the DOM Manipulation task source), it seems like we can expect the unhandledrejection event to occur before the harness has transitioned to the COMPLETE phase.

This event ordering is contentious, so I've submitted a test for it: #20329

@jugglinmike
Copy link
Contributor Author

No description provided.

1 similar comment
@jugglinmike
Copy link
Contributor Author

No description provided.

@wpt-pr-bot wpt-pr-bot requested a deployment to wpt-preview-20239 November 20, 2019 20:23 Abandoned
@gsnedders
Copy link
Member

If the effect of this change seems agreeable, I'd be happy to discuss alternative implementations. For instance, we could freeze the status object at the moment of completion, or we could call structured_clone (which would have a similar effect due to memoization).

I think I'd quite like to do both this and freeze as much as possible when we're done?

@jugglinmike
Copy link
Contributor Author

That's actually one alternative I'm not comfortable with. If we use both solutions, then one will be untested, and the redundancy may confuse future contributors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants