Using pytest for integration tests.

[prady0t]

Description

Running integration tests with pytest. Almost 50% of the integration tests were failing due to multiple tests accessing test_model.json file leading to race conditions and file corruption. We have used uuid to generate unique filename for every test. With this approach we can run all integration tests in parallel. Overall time to run integration tests is approx. 7-8 min. now

Addresses #3617

Type of change

Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist:

• No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
• All tests pass: $ python run-tests.py --all (or $ nox -s tests)
• The documentation builds: $ python run-tests.py --doctest (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ python run-tests.py --quick (or $ nox -s quick).

Further checks:

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[agriyakhetarpal]

I merged develop (it looked like your branch is out of date) to trigger a CI run with caching and to try out with the tip of the trunk. The macOS ARM builds are generally faster, I hope we'll be able to see under 10-minute runs pretty soon

[agriyakhetarpal]

This is looking very good for local development purposes, and would have been even better if we had larger GitHub runners. Here are the results from a few local runs from my macOS machine with 11 cores:

Running the integration tests via serial execution

$ python -m pytest tests/integration -n1

============================== 344 passed, 1 skipped in 563.97s (0:09:23) ==============================

Running the integration tests via parallel execution

$ python run-tests.py --integration

============================== 344 passed, 1 skipped in 100.53s (0:01:40) ==============================

From 0:09:23 to 0:01:40 – a massive boost 🎉 an ~82% reduction in test time, and a ~4.6x faster test suite.

[agriyakhetarpal]

This looks great, thanks, @prady0t! As discussed, cleaning up the run-tests.py file will be great to look at in the coming weeks, there are a lot of if-else conditions in there. I did test out run-tests.py --unit and run-tests.py --integration along with their nox equivalents, and all of them are collecting the expected tests.

[prady0t]

This looks great, thanks, @prady0t! As discussed, cleaning up the run-tests.py file will be great to look at in the coming weeks, there are a lot of if-else conditions in there. I did test out run-tests.py --unit and run-tests.py --integration along with their nox equivalents, and all of them are collecting the expected tests.

Are we planning to eventually completely remove run_test.py file?

[agriyakhetarpal]

Are we planning to eventually completely remove run_test.py file?

Yes, it looks like there is a bunch of legacy, verbose code that is better replaced by keeping all of the testing commands/infra inside noxfile.py. We can use advanced features from Nox and improve the sessions to provide a similar experience, plus document the options better. Let's keep this plus testing the built wheels in mind for this week's discussions.