Tests: Speed up test suite by invoking tests in parallel, using `pytest-xdist` #690

amotl · 2022-10-10T22:58:15Z

By using the excellent pytest-xdist package, I was able to reduce the runtime of the test suite from 45 to 15 (no coverage) / 25 (with coverage) seconds, tested on machines with 8 and 16 CPUs.

# Tests only.
pytest --numprocesses=auto

# Tests, with coverage reporting.
pytest --numprocesses=auto --cov

It apparently also improves runtime on Travis CI, so it would be a big resource saver.

Before: Ran for 6 min 58 sec; Total time 42 min 45 sec
After: Ran for 4 min 47 sec; Total time 30 min 59 sec

Running the tests in parallel also ensures there are no spots where shared resources are not properly accessed in concurrent situations. It will reveal such shortcomings very quickly.

codecov-commenter · 2022-10-10T23:20:43Z

Codecov Report

Base: 100.00% // Head: 100.00% // No change to project coverage 👍

Coverage data is based on head (6b02e73) compared to base (d895471).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #690   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          113       113           
  Lines        14533     14533           
  Branches      2787      2787           
=========================================
  Hits         14533     14533

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

amotl · 2022-10-11T12:01:58Z

Unfortunately, I have not been able to make code coverage tracking work on Travis CI, so I had to remove 1e2d155 and 8bcc6e1 again. Maybe it can be brought back later.

caronc · 2022-10-11T13:41:12Z

So you dropped code coverage completely with this?

amotl · 2022-10-11T17:24:25Z

test_plugin_twitter_tweet_attachments failed on PyPy 3.7 at ¹. It may be a fluke. Because this patch should not change anything significant about the current runtime behavior of the test suite on CI, I believe it has been there before?

I've just started a new build in order to keep the reference to the failure with build #585326200 ¹. It succeeded.

https://app.travis-ci.com/github/caronc/apprise/jobs/585326200#L738 ↩ ↩²

caronc · 2022-10-11T19:15:34Z

test_plugin_twitter_tweet_attachments failed on PyPy 3.7 at 1. It may be a fluke. Because this patch should not change anything significant about the current runtime behavior of the test suite on CI, I believe it has been there before?

This happens ALL the time; every now and then a random test that never failed (ever) in the past just fails on TravisCI. It's usually always related to the Attachments. But giving the CI a kick again always has it work the second time around. So no worries at all here; at this point it's more of nuisance then anything; it just doesn't happen enough for me to investigate it further at this time.

I can't tell if it's a Travis-CI issue, or the way my tests are written, or if if it's some mulch-threading issue with Apprise and the timing of some of the attachment tests (causing it to pass 99.9% of the time).

If it happens again, even if I'm not in front of my PC, i can give easily give Travis CI runner a kick. Feel free to just ask me to do so... (or do a force-push on the branch again; that'll trigger it to go again too). 🤷‍♂️

caronc · 2022-10-11T22:15:06Z

I really appreciate your efforts here, but i don't want to drop code coverage. I personally don't think the 2 min of resource time gained is worth not knowing the coverage on every commit pushed upstream.

amotl

Hi again,

So you dropped code coverage completely with this?
I really appreciate your efforts here, but i don't want to drop code coverage.

I totally agree with you, code coverage tracking and reporting is very important, it must not go away. Thus, I would never dare to remove it ;]. I think there only was a small hiccup at the time when you looked at the outcome here, which has now been fixed.

Other than diligently going through each line of the configuration changes within the review comments, let me explain the situation.

With 1e2d155 and 8bcc6e1, I tried to bring improved efficiency to both the developer sandbox and CI. However, the changes had to be reverted, because I haven't been able to make it work on CI together with coverage tracking.

I think the reason are some shenanigans related to tox, where I didn't have the energy to find out about the root cause.

So, this patch just adds the foundation, and might set the stage for bringing it to CI later. However, it already brings improved efficiency to the developer's sandbox, as you will be able to run the test suite in half of the time as before.

Both of those commands will work as intended, as announced in the original post.

To use it, invoke:

pytest -vvv --numprocesses=auto

To use it with coverage tracking and reporting, invoke:

pytest -vvv --numprocesses=auto --cov

Please note it is a must to use pytest-cov, running the process through the coverage program does not work here.

With kind regards,
Andreas.

amotl · 2022-10-11T22:24:21Z

dev-requirements.txt

@@ -2,5 +2,6 @@ coverage
 flake8
 pytest
 pytest-cov
+pytest-xdist


This line only adds the pytest-xdist package to the list of dependencies.

amotl · 2022-10-11T22:27:06Z

.coveragerc

 branch = True
+parallel = True


Using this option is obligatory for running tests in parallel. It will create many files like .coverage.{hostname}.23098.826306, which will be combined by coverage combine later. Following up on this, I think using the --parallel option within tox.ini can be removed.

I think using the --parallel option within tox.ini can be removed.

Implemented with 290ffc1.

amotl · 2022-10-11T22:29:20Z

.coveragerc

@@ -1,6 +1,7 @@
 [run]
-disable_warnings = no-data-collected
+data_file = .coverage-reports/.coverage


Because many files like .coverage.{hostname}.23098.826306 will be created, I don't want to make them appear in the root directory. It is better to materialize them inside a dedicated subdirectory, which can easily be made excluded from being indexed by your favorite IDE.

With this setting, the files will be materialized like .coverage-reports/.coverage.{hostname}.23098.826306.

amotl · 2022-10-11T22:31:26Z

tox.ini

@@ -131,4 +131,5 @@ deps = coverage
 skip_install = true
 commands=
   coverage combine
+   coverage xml


This is needed to write out a coverage.xml file to the root directory. The reason is that the combined .coverage-reports/.coverage file can't be picked up by the codecov program because of codecov/codecov-python#198. However, it fortunately does pickup coverage.xml instead.

amotl · 2022-10-11T22:33:04Z

.coveragerc

@@ -1,6 +1,7 @@
 [run]
-disable_warnings = no-data-collected


Last but not least, the commit f475704 removes this line. A rationale is provided in its commit message.

It was important to discover that running the tests with pytest-xdist on Travis CI did not produce any coverage reports at all. This setting disabled that warning, so we didn't see it, which was unfortunate.

caronc · 2022-10-11T23:04:11Z

This is amazing; i didn't realize the small changes needed at the end. Do we need anything in the .giignore for .coverage-reports/ (for local runs)? Otherwise this is fantastic and now and easy merge!

You are awesome Andreas! 🚀 ❤️

amotl · 2022-10-11T23:07:45Z

Do we need anything in the .gitignore for .coverage-reports/ (for local runs)?

Your attention to detail is amazing. I also expected that we need another item, but apparently, the item .coverage.* covers it. I don't know why, maybe the globbing syntax is regex-like?

amotl · 2022-10-11T23:35:48Z

Re. #690 (comment):

test_plugin_twitter_tweet_attachments failed on PyPy 3.7. It may be a fluke.

This happens ALL the time; every now and then a random test that never failed (ever) in the past just fails on TravisCI.

Thanks a stack for sharing your observations. Let us work diligently here, so I created #692.

It uses the excellent `pytest-xdist` package. To use it, invoke: pytest -vvv --numprocesses=auto To use it with coverage tracking and reporting, invoke: pytest -vvv --numprocesses=auto --cov --cov-report=term-missing

It was important to discover that running the tests with `pytest-xdist` on Travis CI did not produce any coverage reports at all. This setting disabled that warning, which was unfortunate.

It is now already defined within `.coveragerc`.

…t` (caronc#690)

amotl force-pushed the amo/parallel-testing branch from 5a2188e to 3adcb41 Compare October 10, 2022 23:13

amotl force-pushed the amo/parallel-testing branch 2 times, most recently from d3ec373 to 8bcc6e1 Compare October 11, 2022 00:35

amotl changed the title ~~Tests: Speed up test suite by invoking tests in parallel~~ [WIP] Tests: Speed up test suite by invoking tests in parallel Oct 11, 2022

amotl force-pushed the amo/parallel-testing branch from 8bcc6e1 to 7ca2b42 Compare October 11, 2022 12:02

amotl changed the title ~~[WIP] Tests: Speed up test suite by invoking tests in parallel~~ Tests: Speed up test suite by invoking tests in parallel Oct 11, 2022

amotl marked this pull request as ready for review October 11, 2022 12:11

amotl force-pushed the amo/parallel-testing branch from 7ca2b42 to 337db20 Compare October 11, 2022 16:28

amotl mentioned this pull request Oct 11, 2022

Python .coverage files in nested directories ignored due to ignored_report codecov/codecov-python#198

Closed

amotl force-pushed the amo/parallel-testing branch from 337db20 to 94cad72 Compare October 11, 2022 17:00

amotl force-pushed the amo/parallel-testing branch from 94cad72 to f475704 Compare October 11, 2022 17:24

amotl commented Oct 11, 2022

View reviewed changes

amotl mentioned this pull request Oct 11, 2022

Tests: Collection of flukes on CI #692

Open

amotl changed the title ~~Tests: Speed up test suite by invoking tests in parallel~~ Tests: Speed up test suite by invoking tests in parallel, using pytest-xdist Oct 12, 2022

amotl added 3 commits October 14, 2022 04:26

Tests: Speed up test suite by invoking test cases in parallel

53107a5

It uses the excellent `pytest-xdist` package. To use it, invoke: pytest -vvv --numprocesses=auto To use it with coverage tracking and reporting, invoke: pytest -vvv --numprocesses=auto --cov --cov-report=term-missing

Tests: Enable no-data-collected warning again

068af4f

It was important to discover that running the tests with `pytest-xdist` on Travis CI did not produce any coverage reports at all. This setting disabled that warning, which was unfortunate.

Chore: Remove --parallel option to coverage from tox.ini

0b5b99a

It is now already defined within `.coveragerc`.

amotl force-pushed the amo/parallel-testing branch from 290ffc1 to 0b5b99a Compare October 14, 2022 02:26

Chore: Use --verbosity=3 option to pytest

6b02e73

amotl force-pushed the amo/parallel-testing branch from 8b8587e to 6b02e73 Compare October 14, 2022 04:29

amotl mentioned this pull request Oct 14, 2022

Resolve ambiguity with apprise.plugins module namespace #687

Merged

caronc merged commit 00f6626 into caronc:master Oct 14, 2022

caronc deleted the amo/parallel-testing branch October 14, 2022 18:58

amotl added a commit to panodata/apprise that referenced this pull request Oct 18, 2022

Speed up test suite by invoking tests in parallel, using `pytest-xdis…

a07a217

…t` (caronc#690)

amotl added a commit to panodata/apprise that referenced this pull request Oct 18, 2022

Speed up test suite by invoking tests in parallel, using `pytest-xdis…

bcd5927

…t` (caronc#690)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests: Speed up test suite by invoking tests in parallel, using `pytest-xdist` #690

Tests: Speed up test suite by invoking tests in parallel, using `pytest-xdist` #690

amotl commented Oct 10, 2022 •

edited

Loading

codecov-commenter commented Oct 10, 2022 •

edited

Loading

amotl commented Oct 11, 2022

caronc commented Oct 11, 2022

amotl commented Oct 11, 2022 •

edited

Loading

caronc commented Oct 11, 2022

caronc commented Oct 11, 2022

amotl left a comment •

edited

Loading

amotl Oct 11, 2022

amotl Oct 11, 2022 •

edited

Loading

amotl Oct 12, 2022

amotl Oct 11, 2022

amotl Oct 11, 2022 •

edited

Loading

amotl Oct 11, 2022 •

edited

Loading

caronc commented Oct 11, 2022

amotl commented Oct 11, 2022 •

edited

Loading

amotl commented Oct 11, 2022

Tests: Speed up test suite by invoking tests in parallel, using pytest-xdist #690

Tests: Speed up test suite by invoking tests in parallel, using pytest-xdist #690

Conversation

amotl commented Oct 10, 2022 • edited Loading

codecov-commenter commented Oct 10, 2022 • edited Loading

Codecov Report

amotl commented Oct 11, 2022

caronc commented Oct 11, 2022

amotl commented Oct 11, 2022 • edited Loading

Footnotes

caronc commented Oct 11, 2022

caronc commented Oct 11, 2022

amotl left a comment • edited Loading

Choose a reason for hiding this comment

amotl Oct 11, 2022

Choose a reason for hiding this comment

amotl Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

amotl Oct 12, 2022

Choose a reason for hiding this comment

amotl Oct 11, 2022

Choose a reason for hiding this comment

amotl Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

amotl Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

caronc commented Oct 11, 2022

amotl commented Oct 11, 2022 • edited Loading

amotl commented Oct 11, 2022

Tests: Speed up test suite by invoking tests in parallel, using `pytest-xdist` #690

Tests: Speed up test suite by invoking tests in parallel, using `pytest-xdist` #690

amotl commented Oct 10, 2022 •

edited

Loading

codecov-commenter commented Oct 10, 2022 •

edited

Loading

amotl commented Oct 11, 2022 •

edited

Loading

amotl left a comment •

edited

Loading

amotl Oct 11, 2022 •

edited

Loading

amotl Oct 11, 2022 •

edited

Loading

amotl Oct 11, 2022 •

edited

Loading

amotl commented Oct 11, 2022 •

edited

Loading