Backporting test fixes to maintenance branches

[gibfahn]

See @MylesBorins comment in: nodejs/node#12567 (comment)

@gibfahn by the maintenance contract we are no longer supposed to be backporting fixes to tests. It didn't land because we have not been super adamant about making sure tests that don't land cleanly backport.

Personally I'm -1 on making exceptions for tests, but willing to reconsider if @nodejs/lts thinks this should land

We should have the discussion and clarify whether we think any test fixes should be backported, and if so which ones.

This was originally in #188, but I thought we should record our decision separately.

[gibfahn]

My initial feeling on this is that we should only backport tests if:

• The test is failing (or failing intermittently) on that branch, and someone is actually requesting the backport, and
• We're already doing a (non-security) release

So it would only apply to actual test-case issues, and it would only be included in a (vanishingly rare) non-security maintenance release. PRs could land on v4.x-staging and sit there until we did a release, which might be never.

[sam-github]

Seems reasonable to me. Its worth having the test suite in good order for when a release has to happen. Not worth landing the flood of test PRs.

[MylesBorins]

I'd be open to landing test fixes for things that are broken... in fact perhaps this is a bigger question about what we are willing to land on maintenance if people are requesting it.

Looking forward to discussing this afternoon

[jasnell]

The maintenance phase is specifically limited to critical fixes. Updates to test do not count as critical. We did not update the tests in 0.12 or 0.10 unless those were tied to specific other changes that were being made. I'm -1 on backporting tests unless those changes fall into the "critical" category.

[mhdawson]

I'm +1 to fixing things that are broken and causing people problems. The key being that people are asking for the fix.

[refack]

+1 for no semver-minor (leave a window for an LTS WG consensus override vote)

[gibfahn]

Closing as agreed (no backporting by default, we can change our mind going forward if we want).