Make 'warnings' and 'deselected' in assert_outcomes optional

[nicoddemus]

Fix #9471

I don't think we need a CHANGELOG here, as this has not really been officially released yet.

[The-Compiler]

I'm not sure if I like the approach to be honest... while it is indeed convenient, I fear the asymmetry could lead to a silent mode of failure, where people expect e.g. warnings to be checked, but that does not happen.

[nicoddemus]

I was trying to fix the regression... should we keep the regression then? Fix this some other way?

[The-Compiler]

Agreed that this would fix the regression - but I'm not sure if it's the right way forward. I think I personally would prefer documenting this as a breaking change instead.

However it sounded like @RonnyPfannschmidt was in favor of this approach in #9471, so I might be overridden on that. 😉

Maybe @okken has an opinion on this?

[RonnyPfannschmidt]

I totally agree with the sentient of @TheCompiler

My personal preference is to fix the regression first, then think about warning and /or plannable breaking changes

The problem is that the api initially was incomplete, and likely still is incomplete, so we ought to consider the states explicitly and implicitly checked for

[nicoddemus]

OK, closing this then. 👍

[RonnyPfannschmidt]

reopening as its preferable not to break users while we figure the correct long term solution

[The-Compiler]

Let me note that we already had breaking changes to this API (without any kind of deprecation period) in pytest 6, via #6505. I've seen some version checks to account for that in the wild, but don't remember where.

So I guess the question is whether we'd rather:

1. Avoid additional breakage, but in return have an API which is inconsistent and could silently hide problems, especially with warnings (yes, they have been hidden before expose warnings= to pytester assert_outcomes() #8952 I think - but it seems like a good thing to make pytester tests fail if the subprocess has warnings by default, no)?
2. Have another breaking change there, accept the breakage, but of course document it.

I'd prefer 2) over an inconsistent API - but then again 1) would probably mean that if we clean this up in form of some kind of different API, plugins then need to account for four different ways of doing the same thing, which isn't ideal either...

Thoughts? I'm fine with merging this and defer a better API to another release, but I feel like there is a trade-off...

also cc-ing @symonk

[The-Compiler]

Bump? I feel like we need a decision here to avoid blocking the 7.0.0 final. As said, I'm fine with being overridden on this, I just hope it won't paint us into a corner.

[nicoddemus]

The safest one, feature wise, is 2: close this PR and accept the breakage. But I don't have strong feelings either, so what you folks decide it is fine.

[RonnyPfannschmidt]

the breakage will be a utter pain for all plugin testsuites supporting multiple pytest versions,

[The-Compiler]

ok, let's go for this then to get things unstuck, and hope we can come up with a better API altogether in the future...