Get traceback-trimming working under Python 3.11

[Zac-HD]

Follow-up to #3294 - if we remove the workarounds in conftest.py, two tests fail:

=================================== FAILURES ===================================
_____________________ test_healthcheck_traceback_is_hidden _____________________
[gw0] linux -- Python 3.11.0 .tox/py311-full/bin/python
Traceback (most recent call last):
  File "tests/pytest/test_capture.py", line 106, in test_healthcheck_traceback_is_hidden
    assert timeout_line - def_line == expected
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: assert (122 - 11) == 7

_________________ test_tracebacks_omit_hypothesis_internals[1] _________________
[gw0] linux -- Python 3.11.0 .tox/py311-full/bin/python
Traceback (most recent call last):
  File "tests/cover/test_traceback_elision.py", line 26, in test_tracebacks_omit_hypothesis_internals
    simplest_failure()
    ^^^^^^^^^^^^^^^^^^
  File "tests/cover/test_traceback_elision.py", line 21, in simplest_failure
    @given(st.just(False))
               ^^^
  File "contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "hypothesis/utils/dynamicvariables.py", line 33, in with_value
    yield
    ^^^^^
  File "hypothesis/_settings.py", line 340, in local_settings
    yield s
    ^^^^^^^
  File "hypothesis/core.py", line 1220, in wrapped_test
    raise the_error_hypothesis_found
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "hypothesis/utils/dynamicvariables.py", line 33, in with_value
    yield
    ^^^^^
  File "hypothesis/_settings.py", line 340, in local_settings
    yield s
    ^^^^^^^
  File "contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "hypothesis/internal/entropy.py", line 112, in deterministic_PRNG
    yield
    ^^^^^
  File "contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "hypothesis/utils/dynamicvariables.py", line 33, in with_value
    yield
    ^^^^^
  File "hypothesis/core.py", line 656, in run
    return test(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "tests/cover/test_traceback_elision.py", line 21, in simplest_failure
    @given(st.just(False))
               ^^^^
  File "hypothesis/core.py", line 597, in test
    result = self.test(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tests/cover/test_traceback_elision.py", line 23, in simplest_failure
    raise ValueError()
    ^^^^^^^^^^^^^^^^^^
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tests/cover/test_traceback_elision.py", line 32, in test_tracebacks_omit_hypothesis_internals
    assert len(tb) == 4
    ^^^^^^^^^^^^^^^^^^^
AssertionError: assert 19 == 4

=========================== short test summary info ============================
FAILED tests/cover/test_traceback_elision.py::test_tracebacks_omit_hypothesis_internals[1]
FAILED tests/pytest/test_capture.py::test_healthcheck_traceback_is_hidden
= 2 failed, 2[723](https://github.com/HypothesisWorks/hypothesis/runs/6050406931?check_suite_focus=true#step:6:723) passed, 3 skipped, 1 xfailed, 10 warnings in 330.33s (0:05:30) =

After brief investigation, it looks like this is a a real change that we'll need to adapt our code to; and since traceback-trimming is pretty important to our user's experience I'd like to ship that well before 3.11.0 final.

[Cheukting]

I will give this a go :-)

[Cheukting]

@Zac-HD I think this is cause by the difference in tracestack in 3.10 and 3.11:

3.10:

========================================= FAILURES ==========================================
___________________________ test_healthcheck_traceback_is_hidden ____________________________

    @given(integers().map(lambda x: time.sleep(0.2)))
>   def test_healthcheck_traceback_is_hidden(x):

test.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py:470: in run
    self._run()
../hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py:876: in _run
    self.generate_new_examples()
../hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py:680: in generate_new_examples
    minimal_example = self.cached_test_function(
../hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py:1055: in cached_test_function
    self.test_function(data)
  ...

3.11:

======================================= FAILURES ========================================
_________________________ test_healthcheck_traceback_is_hidden __________________________

  @given(integers().map(lambda x: time.sleep(0.2)))
>   def test_healthcheck_traceback_is_hidden(x):

test.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.pyenv/versions/3.11.0a6/lib/python3.11/contextlib.py:155: in __exit__
  self.gen.throw(typ, value, traceback)
../../.pyenv/versions/3.11.0a6/lib/python3.11/contextlib.py:155: in __exit__
  self.gen.throw(typ, value, traceback)
../hypothesis/hypothesis-python/src/hypothesis/utils/dynamicvariables.py:33: in with_value
  yield
../hypothesis/hypothesis-python/src/hypothesis/_settings.py:340: in local_settings
  yield s
../../.pyenv/versions/3.11.0a6/lib/python3.11/contextlib.py:155: in __exit__
  self.gen.throw(typ, value, traceback)
../../.pyenv/versions/3.11.0a6/lib/python3.11/contextlib.py:155: in __exit__
  self.gen.throw(typ, value, traceback)
...

as you see in 3.11 the stack quickly go to exit in python 3.11 contextlib.py itself which causes the check in the while here to fail as it is not a hypothesis file.

So what do you think? shall we trim the exit in 3.11 as well (which we just need to add extras in the check) or is there any nicer way to treat it?

[Zac-HD]

Yeah, let's do it! Specifically, trim frames from contextlib in addition to those from Hypothesis on Python 3.11+

I also considered replacing the @contextmanager decorator with explicit uses of __enter__ and __exit__, but this would be pretty annoying to maintain since we'd have to be super-strict or worry about reintroducing the problem on some less-common code path.

[Cheukting]

Oh also one thing that bugs me a lot, why we have a is True there?

[Cheukting]

By the way, I wonder if the contextlib exit stuff will still exist after 3.11 release? (currently at alpha) cause it seems it can still vary a lot (e.g. adding the tomllib) when realse. See psf/black#2983

It seems to me that it is weird that the stacktrace of contextlib is there and I am thinking that it may got removed when 3.11 is released. If so, this will be automatically fix isn't it?

Yeah, let's do it! Specifically, trim frames from contextlib in addition to those from Hypothesis on Python 3.11+

I also considered replacing the @contextmanager decorator with explicit uses of __enter__ and __exit__, but this would be pretty annoying to maintain since we'd have to be super-strict or worry about reintroducing the problem on some less-common code path.

[Cheukting]

I have created the PR anyway, if the problem still exist next week when it got release we are ready.

[Zac-HD]

We can also just ask some core devs at the sprints if this is intentional - and either handle it in Hypothesis if it is, or try to get it fixed+tested in CPython if it's not 🙂

[Cheukting]

Yes I love this idea, let's see tomorrow (or today if they are around)

[Cheukting]

Or we can just open an issue there, someone will have an answer.

[Zac-HD]

Go for it, and tag me + reference this issue?

[Zac-HD]

Closing this in favor of python/cpython#92202; I'll remove the test-skips when we start running beta1 in CI.