Test inside pytest_runtest_call hook

[ConorMacBride]

This changes the plugin's pytest integration to be the same as pytest-mpl, which is able to handle intercepting returned values for further testing.

Fix #35

[ConorMacBride]

@pllim I think the CI workflow is disabled

[pllim]

Let me close/reopen.

[pllim]

Might have to drop Python 3.7 ?

[ConorMacBride]

Could just ignore the warning? Which would you prefer?

[pllim]

Ah, true, you could.

[pllim]

Thanks. I don't grok the internals here, so I pinged the people who has release rights for review (https://pypi.org/project/pytest-arraydiff).

[ConorMacBride]

#35 should only be an issue when pytest-arraydiff is either not installed or not enabled. I think it's also worth updating for the enabled case here because we found that the existing wrapper method didn't test class based tests.

[pllim]

I think it's also worth updating for the enabled case here because we found that the existing wrapper method didn't test class based tests.

Do you plan to do that in this PR? If not, please open a new issue for it. Thanks!

[ConorMacBride]

I already fixed that in the first commit. I was just justifying the change which I later realised wasn't necessary to fix the actual issue referenced. 😆

I've now confirmed that pytest-arraydiff doesn't work with class-based tests on recent pytest versions. You can see in the logs that it only shows the warning for the class test (because it wasn't able to wrap the test method for array testing): https://github.com/astropy/pytest-arraydiff/actions/runs/2914788915/jobs/4644034561#step:5:148

I tried modifying the return value of tests/test_pytest_arraydiff.py::TestClass::test_succeeds_class, however, it always passed on the main branch. Although, with this PR it failed as expected when modified.

[pllim]

If you don't hear back from the actual package maintainers, try poke them on Astropy Slack. Thanks!

[bsipocz]

minor comment only, you need Tom to sign off on the actual code changes

[astrofrog]

I will try and review this shortly.

I do wonder if there would be scope to develop a pytest plugin that deals with getting return values that both pytest-mpl and this plugin could depend on to avoid duplicating infrastructure?

[astrofrog]

Looks good to me, thanks!