Make PipTestEnvironment stderr checks more assertive

[chrahunt]

We can tighten the tests in the following way. Rename the PipTestEnvironment.run() arguments of allow_stderr_warning and allow_stderr_error arguments to expect_stderr_warning and expect_stderr_error:

pip/tests/lib/__init__.py

Lines 501 to 507 in 2bfafc9

	def run(self, *args, **kw):
	"""
	:param allow_stderr_warning: whether a logged warning (or
	deprecation message) is allowed in stderr.
	:param allow_stderr_error: whether a logged error is allowed in
	stderr. Passing True for this argument implies
	`allow_stderr_warning` since warnings are weaker than errors.

and make the test fail if the argument is passed and a warning or error isn't present, respectively. The idea is that the argument should only be passed if it's actually needed, unlike the current case where the test is passing if the argument is provided unnecessarily.

Also, to do this it might be necessary to do a check of some kind to know if the argument should be passed. For example, whether a deprecation warning is emitted can depend on what version is running.

Posted in its original form by @cjerdonek in #161 (comment)

[cjerdonek]

A related task is to remove the expect_stderr argument since it's redundant with allow_stderr_warning.

[atugushev]

Hello @cjerdonek,

Could you help me to clarify a few cases?

Cases for expect_stderr_warning=True

run(command, expect_stderr_warning=True)

1. if a command produces no stderr
   ... then raises a RuntimeError("expect a warning")
2. if a command produces stderr: "WARNING: Hello, world!"
   ... then nothing raises
3. if a command produces stderr: "ERROR: Hello, world!"
   ... then raises a RuntimeError("expect a warning")

Cases for expect_stderr_error=True

run(command, expect_stderr_error=True)

1. if a command produces no stderr
   ... then raises a RuntimeError("expect an error")
2. if a command produces stderr: "WARNING: Hello, world!"
   ... then raises a RuntimeError("expect an error")
3. if a command produces stderr: "ERROR: Hello, world!"
   ... then nothing raises

Questions:

• Are these statements true?
• Should expect_stderr_error=True imply expect_stderr_warning=True?

[pradyunsg]

My answers:

• Yes.
• No, it should require that.

[ssurbhi560]

Hey @atugushev , can I please take your work further and work on this issue, if you are not onto this anymore? :-)

[atugushev]

Hey @ssurbhi560. Thanks for pinging me! Can't really find a time to get this done. Would you like to pick it up? Please, feel free to 🙏

[ssurbhi560]

@atugushev , I would be glad to pick it up and take this further.
Thank you ! :)

[gutsytechster]

Hey @ssurbhi560 are you working over it? If not, then I'd like to take up this issue. :)

[ssurbhi560]

Hey @gutsytechster! I won't be able to work on this right now, feel free to work on it. :)

[gutsytechster]

Here, does it mean that for an error to be expected to raise, the variables expect_stderr_error and expect_stderr_warning would be both True? However, in case of an expected warning, only expect_stderr_warning would be True but not expect_stderr_error.