Tests: Collection of flukes on CI

[amotl]

Hi again,

while submitting some patches recently, we observed occasional flaky behavior of the test suite on CI. ¹². This issue intends to collect references to them, in order to find out about the root cause.

At ², we discovered test_plugin_twitter_tweet_attachments failed on PyPy 3.7 ³ with #690. @caronc responded to this with:

This happens ALL the time; every now and then a random test that never failed (ever) in the past just fails on TravisCI. It's usually always related to the Attachments. [...] I can't tell if it's a Travis-CI issue, or the way my tests are written, or if if it's some mulch-threading issue with Apprise and the timing of some of the attachment tests (causing it to pass 99.9% of the time).

Thank you for sharing your knowledge here, I will have another look at the test cases related to attachments when I can find some time.

If it happens again, even if I'm not in front of my PC, i can give easily give Travis CI runner a kick. Feel free to just ask me to do so... (or do a force-push on the branch again; that'll trigger it to go again too). 🤷‍♂️

I think it will be fortunate to use the "force-push" variant, in order to keep references to the failed builds on Travis CI, in order to build up more knowledge where and how often this error happens. If you like that approach, feel free to add corresponding references to this issue while we go.

With kind regards,
Andreas.

Footnotes

1. https://github.com/caronc/apprise/pull/681#issuecomment-1272408168 ↩

2. https://github.com/caronc/apprise/pull/690#issuecomment-1275036257 ↩ ↩²

3. https://app.travis-ci.com/github/caronc/apprise/jobs/585326200#L738 ↩

[amotl]

It looks like there is a mixup of NotifyMSTeams vs. NotifyMatrix here. It happened on PyPy 3.9 with #694.

________________________ test_plugin_msteams_templating ________________________
[...]

>       assert mock_post.call_args_list[0][0][0].startswith(
            'https://outlook.office.com/webhook/')
E       AssertionError: assert False
E        +  where False = <bound method str.startswith of 'http://host:/_matrix/client/r0/logout'>('https://outlook.office.com/webhook/')
E        +    where <bound method str.startswith of 'http://host:/_matrix/client/r0/logout'> = 'http://host:/_matrix/client/r0/logout'.startswith
test/test_plugin_msteams.py:371: AssertionError

There is also a warning.

test/test_plugin_msteams.py::test_plugin_msteams_templating
  /home/travis/build/caronc/apprise/.tox/pypy39/lib/pypy3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: : <function NotifyMatrix.__del__ at 0x000000000a4848e0>
  Traceback (most recent call last):
    File "/home/travis/build/caronc/apprise/apprise/plugins/NotifyMatrix.py", line 1020, in _fetch
      response = loads(r.content)
  AttributeError: 'Request' object has no attribute 'content'

-- https://app.travis-ci.com/github/caronc/apprise/jobs/585483224#L915

[caronc]

This might be tied to the reloading of modules that already executed code (and were populated in that matrix found in common.py (still pointing to the old module from a previous load) during testing. I have had lots of issues with this. The windows plugin is about the only one that you can't escape this kind of hack. This comment could be a red herring, but if the test passes fine with exclusively called, then this could be the case. What is the status if you run:

./bin/test.sh msteams

I'll bet it might work fine

[amotl]

Hi Chris,

thanks for your response. The error reported above was only a fluke on CI, it is not deterministically reproducable right now, neither individually nor in combination. I also never observed it on my workstation, even when running the tests in parallel using pytest-xdist.

I will definitively take all your valuable information into account (very much appreciated, please keep it coming) when eventually giving dedicated love to this part of the test harness within any future iterations.

I think it is acceptable to have a small amount of flukes in the CI setup of a project as heavy as Apprise. But this doesn't mean we can't do better, so please consider my efforts here only as a long-term endeavor.

With kind regards,
Andreas.

[amotl]

Another fluke on PyPy 3.9 with #690.

________________________ test_plugin_fcm_general_oauth _________________________

        # Test our call count
>       assert mock_post.call_count == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <MagicMock name='post' id='301617512'>.call_count
test/test_plugin_fcm.py:377: AssertionError

FAILED test/test_plugin_fcm.py::test_plugin_fcm_general_oauth - AssertionErro...

-- https://app.travis-ci.com/github/caronc/apprise/jobs/585628603#L763

[caronc]

I applaud your efforts to improve the CI and testing. I feel you're going through the same circles i did and eventually gave up as it's only reproducible randomly on Travis-CI. You seemed like a fresh pair of eyes and you were so engaged in the matter, so i didn't want to stop you.

As discussed earlier, I personally believe the random failing of these modules is related to the (test case) hacking of the module loading just for test coverage. I think dropping all of this (except windows one) and leveraging the the bare container is the way out. With the bare container, perhaps there is a way to drop the glib module prior to the test suite being ran so the edge case to auto disable them is met?

Alternatively, place the pragma: no branch on the import error catching aa we know works. This would allow us not to not reload any modules at all (this would reduce even the logic/fixtures for the reloading). The windows one is still around only because we need to fake that the modules exist so it's testable in Linux. Perhaps there are windows runners we can leverage with Travis-CI; that would simply life even more?

[amotl]

Dear Chris,

thanks for your response.

I applaud your efforts to improve the CI and testing.

Thank you, I appreciate it.

As discussed earlier, I personally believe the random failing of these modules is related to the (test case) hacking of the module loading just for test coverage.

I tried my best to reflect the state of the onion within #697, did you read my ramblings over there?

I think dropping all of this [...] is the way out.

I agree. Just to make sure our ideas match up reasonably: Is your proposal going into the same direction as I proposed in #697 in any way, or would it be an alternative path out of the module reloading hell you would be in favor to consider?

I think [...] leveraging the bare container is the way out.

I hear you. I don't want to argue too much against having such a bare container. It is only that I found it to be a little anomaly in the whole process, and that it pays a small toll on maintaining a mind map of how things work together.

Perhaps there are windows runners we can leverage with Travis-CI; that would simply life even more?

If I still can find more time to dedicate to the code base, I will surely use a Windows runner, probably on GHA, to improve the test cases on this matter. The build at [1] shows the current state of affairs on this matter.

With kind regards,
Andreas.

[1] https://github.com/panodata/apprise/actions/runs/3238651908

[amotl]

#697 is out of the loop, sorry for the noise. So, let's follow your proposal and keep the bare runner for that purpose?

On the other details, I am not fully convinced yet that the flukes are caused by module reloading woes. Let's see if it will get better with #687 down the line.

If you think it is more appropriate, feel free to move this issue into a discussion. My intention is to mereley use it to track all flukes I am observing, mostly on CI. One first outcome from tracking the three items here is, that all of them only happened on PyPy builds, CPython seems to be fine, until further notice.

[amotl]

PyPy 3.6 strikes again with #698. Another apparent mixup between NotifyMSTeams vs. NotifyMatrix here, no?

________________________ test_plugin_msteams_templating ________________________
[...]

        assert mock_post.called is True
>       assert mock_post.call_args_list[0][0][0].startswith(
            'https://outlook.office.com/webhook/')
E       AssertionError: assert False
E        +  where False = <bound method str.startswith of 'http://host:/_matrix/client/r0/logout'>('https://outlook.office.com/webhook/')
E        +    where <bound method str.startswith of 'http://host:/_matrix/client/r0/logout'> = 'http://host:/_matrix/client/r0/logout'.startswith
test/test_plugin_msteams.py:225: AssertionError

----------------------------- Captured stderr call -----------------------------
Exception ignored in: <bound method NotifyMatrix.__del__ of <apprise.plugins.NotifyMatrix.NotifyMatrix object at 0x000000000a7ca448>>
Traceback (most recent call last):
  File "/home/travis/build/caronc/apprise/apprise/plugins/NotifyMatrix.py", line 1020, in _fetch
    response = loads(r.content)
AttributeError: 'Request' object has no attribute 'content'

-- https://app.travis-ci.com/github/caronc/apprise/jobs/585672683#L826

[caronc]

That's a bit of a new one. Your amazing and growing merge request has gotten so big (with so many fine little fixes in it), it's hard to see where/what is triggering some of these now 😉 .

the Request() object is mocked in a lot of these cases. I wonder if .content isn't being defined. But then i guess if that were the case all other test cases (on all other Py branches would have failed). On that same link you shared on lines 789 and 790. I wonder if this is just a simple case of setting something like mock_post.return_value.content = '{}' for bullet proofing. Still really strange all the errors you're getting. 😕

[caronc]

You made another comment about the CI Credits; don't' worry about them. Travis CI is fantastic and while i only get credits now in small bursts (that have run out fast only because of the amazing extensive testing you've done), they're all free.

So don't worry about exhausting credits at all. I know it's a bit of delay/thump in your productivity, but dont' let it hold you back or feel you need to work around limiting the occurrences of runners.

[amotl]

Your amazing and growing merge request has gotten so big (with so many fine little fixes in it), it's hard to see where/what is triggering some of these now 😉.

Are you referring to #687? Well, all the flukes have not been on this one, but on the rather small changes with #690, #694, and #698 instead, which do not leverage the improvements from #687 yet - I've just retroactively edited the reports above to include those references. Until further contrary insights, I am slightly confident that #687 will improve the situation on this matter.

the Request() object is mocked in a lot of these cases. I wonder if .content isn't being defined. But then i guess if that were the case all other test cases (on all other Py branches would have failed) [...] Still really strange all the errors you're getting.

Yeah, it's weird. Thanks so far for sharing your analysis on all the occurrances right away, I will try to take them into consideration when eventually dedicating some time to this specific topic.

I will keep on posting all my observations here, maybe we will see the light. So far, I can only say the spots have only happened on PyPy builds, and never on #687 yet.

[amotl]

Another fluke on PyPy 3.9, on behalf of #700.

________________________ test_plugin_fcm_general_oauth _________________________

        # Now we test using a valid Project ID and data parameters
        obj = Apprise.instantiate(
            'fcm://mock-project-id/device/#topic/?keyfile={}'
            '&+key=value&+key2=value2'
            '&image_url=https://example.com/interesting.png'.format(str(path)))
>       assert mock_post.call_count == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <MagicMock name='post' id='221416880'>.call_count
test/test_plugin_fcm.py:396: AssertionError

-- https://app.travis-ci.com/github/caronc/apprise/jobs/585748460#L803

[amotl]

Some mocking problems on PyPy 3.6 with test_plugin_twitter_dm_attachments (test/test_plugin_twitter.py:584), on behalf of #702.

https://github.com/caronc/apprise/actions/runs/3271649948/jobs/5381689500#step:10:535

[amotl]

On behalf of #707, this guy failed on CPython 3.9 on Windows.

______________________________ test_notify_base _______________________________
>       assert elapsed > 0.5 and elapsed < 1.5
E       assert (0.4999514000000005 > 0.5)

test\test_notify_base.py:147: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3273052301/jobs/5384798729#step:10:469

[caronc]

The previous runner was so slow, this test passed. 0.4995 is pretty close. I think the test just needs to be tweaked to allow for this. No worries here at all

[amotl]

PyPy 3.8 on behalf of #707.

________________________ test_plugin_fcm_general_oauth _________________________

         mock_post.reset_mock()
        obj = Apprise.instantiate(
            'fcm://mock-project-id/device/?keyfile={}'
            '&color=#12AAbb'.format(str(path)))
>       assert mock_post.call_count == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <MagicMock name='post' id='351584512'>.call_count

test/test_plugin_fcm.py:508: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3277555883/jobs/5394983838#step:10:537

[amotl]

PyPy 3.9 on behalf of #707. Two in one.

FAILED test/test_plugin_msteams.py::test_plugin_msteams_templating - Assertio...
FAILED test/test_plugin_twitter.py::test_plugin_twitter_tweet_attachments - S...

test_plugin_msteams_templating

________________________ test_plugin_msteams_templating ________________________

>       assert mock_post.call_args_list[0][0][0].startswith(
            'https://outlook.office.com/webhook/')
E       AssertionError: assert False
E        +  where False = <bound method str.startswith of 'http://host:/_matrix/client/r0/logout'>('https://outlook.office.com/webhook/')
E        +    where <bound method str.startswith of 'http://host:/_matrix/client/r0/logout'> = 'http://host:/_matrix/client/r0/logout'.startswith

test/test_plugin_msteams.py:372: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3277609113/jobs/5395106847#step:10:556

test_plugin_twitter_tweet_attachments

____________________ test_plugin_twitter_tweet_attachments _____________________

 >       assert obj.notify(
            body='body', title='title', notify_type=NotifyType.INFO,
            attach=attach) is True

test/test_plugin_twitter.py:909:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apprise/plugins/NotifyBase.py:302: in notify
    if not self.send(body=chunk['body'], title=chunk['title'],
apprise/plugins/NotifyTwitter.py:345: in send
    return getattr(self, '_send_{}'.format(self.mode))(
apprise/plugins/NotifyTwitter.py:412: in _send_tweet
    postokay, response = self._fetch(
apprise/plugins/NotifyTwitter.py:708: in _fetch
    r = fn(
/opt/hostedtoolcache/PyPy/3.9.12/x64/lib/pypy3.9/unittest/mock.py:1092: in __call__
    return self._mock_call(*args, **kwargs)
/opt/hostedtoolcache/PyPy/3.9.12/x64/lib/pypy3.9/unittest/mock.py:1096: in _mock_call
    return self._execute_mock_call(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>               result = next(effect)
E               StopIteration

/opt/hostedtoolcache/PyPy/3.9.12/x64/lib/pypy3.9/unittest/mock.py:1153: StopIteration
=============================== warnings summary ===============================
test/test_plugin_msteams.py::test_plugin_msteams_templating
  /opt/hostedtoolcache/PyPy/3.9.12/x64/lib/pypy3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: : <function NotifyMatrix.__del__ at 0x00000000084e1d80>
  
  Traceback (most recent call last):
    File "/home/runner/work/apprise/apprise/apprise/plugins/NotifyMatrix.py", line 1020, in _fetch
      response = loads(r.content)
  AttributeError: 'Request' object has no attribute 'content'
  
  During handling of the above exception, another exception occurred:

-- https://github.com/caronc/apprise/actions/runs/3277609113/jobs/5395106847#step:10:836

[amotl]

Another fluke on PyPy 3.9 on behalf of #707. It already happened at #692 (comment) and a different one at #692 (comment).

________________________ test_plugin_fcm_general_oauth _________________________

        # Test our call count
>       assert mock_post.call_count == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <MagicMock name='post' id='91076936'>.call_count
test/test_plugin_fcm.py:361: AssertionError

FAILED test/test_plugin_fcm.py::test_plugin_fcm_general_oauth - AssertionErro...

-- https://github.com/caronc/apprise/actions/runs/3277845543/jobs/5395580516#step:10:396

[caronc]

Just adding to fluke collection; just a change to the README.md file alone had to be built 3 times (3 times a charm). All failures were with PyPy3.7 Runner.

• Failure 1
• Failure 2

[amotl]

Thanks for reporting. I see that both test_plugin_pushover_attachments and test_plugin_fcm_general_oauth strike again, and again on PyPy.

To be more specific, we did not see a fluke on test_plugin_pushover_attachments yet, but on others with _attachments, namely test_plugin_twitter_tweet_attachments and test_plugin_twitter_dm_attachments.

---

 _______________________ test_plugin_pushover_attachments _______________________

>       assert mock_post.call_count == 2
E       AssertionError: assert 23 == 2
E        +  where 23 = <MagicMock name='post' id='188343032'>.call_count

test/test_plugin_pushover.py:244: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3307979727/jobs/5460021990#step:10:423

________________________ test_plugin_fcm_general_oauth _________________________

        # Test our call count
>       assert mock_post.call_count == 2
E       AssertionError: assert 3 == 2
E        +  where 3 = <MagicMock name='post' id='388993832'>.call_count

test/test_plugin_fcm.py:514: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3307979727/jobs/5460189083#step:10:553

[amotl]

Another fluke on PyPy 3.9 on behalf of #727. It is test_plugin_fcm_general_oauth again.

________________________ test_plugin_fcm_general_oauth _________________________

        # Test our call count
>       assert mock_post.call_count == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <MagicMock name='post' id='58434072'>.call_count

test/test_plugin_fcm.py:372: AssertionError

[amotl]

PyPy 3.6 failed on test_plugin_twitter_tweet_attachments again, on behalf of #727.

         # Send our notification (again); this time there willb e more tweet logging
>       assert obj.notify(
            body='body', title='title', notify_type=NotifyType.INFO,
            attach=attach) is True

test/test_plugin_twitter.py:744: 

-- https://github.com/caronc/apprise/actions/runs/3358072072/jobs/5564532037#step:10:472

[amotl]

test_plugin_twitter_dm_attachments and test_plugin_twitter_tweet_attachments are the most prominent causes for failed test runs on CI, having been observed at [1]. I started to refactor the test cases with #728, in order to improve the situation.

[1] https://github.com/caronc/apprise/actions/runs/3358072072

[amotl]

On behalf of #728, when refactoring the test cases, we observed this one on PyPy3.8 at [1]:

test/test_plugin_twitter.py::test_plugin_twitter_tweet_attachments_bad_media_response_unparseable
  /opt/hostedtoolcache/PyPy/3.8.13/x64/lib/pypy3.8/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: : <function NotifyTwist.__del__ at 0x0000000006a74d40>
  
  Traceback (most recent call last):
    File "/home/runner/work/apprise/apprise/apprise/plugins/NotifyTwist.py", line 791, in __del__
      self.logout()
    File "/home/runner/work/apprise/apprise/apprise/plugins/NotifyTwist.py", line 353, in logout
      postokay, response = self._fetch('users/logout')
    File "/home/runner/work/apprise/apprise/apprise/plugins/NotifyTwist.py", line 641, in _fetch
      timeout=self.request_timeout,
    File "/opt/hostedtoolcache/PyPy/3.8.13/x64/lib/pypy3.8/unittest/mock.py", line 1081, in __call__
      return self._mock_call(*args, **kwargs)
    File "/opt/hostedtoolcache/PyPy/3.8.13/x64/lib/pypy3.8/unittest/mock.py", line 1085, in _mock_call
      return self._execute_mock_call(*args, **kwargs)
    File "/opt/hostedtoolcache/PyPy/3.8.13/x64/lib/pypy3.8/unittest/mock.py", line 1142, in _execute_mock_call
      result = next(effect)
  StopIteration

Isn't it weird that something related to NotifyTwist pops up here? The first kind of "mixup" has been reported at #692 (comment), maybe both are related?

[1] https://github.com/caronc/apprise/actions/runs/3371826071/jobs/5594557259#step:10:456

[amotl]

On behalf of #728, PyPy3.8 strikes again.

________________________ test_plugin_fcm_general_oauth _________________________
>       assert mock_post.call_count == 2
E       AssertionError: assert 3 == 2
E        +  where 3 = <MagicMock name='post' id='323378824'>.call_count

test/test_plugin_fcm.py:514: AssertionError

______________________ test_plugin_pushbullet_attachments ______________________
 >       assert mock_post.call_count == 4
E       AssertionError: assert 25 == 4
E        +  where 25 = <MagicMock name='post' id='288615968'>.call_count

test/test_plugin_pushbullet.py:226: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3371988866/jobs/5594890639

[amotl]

On PyPy3.8 again, with the refactorings from #728.

________________ test_plugin_fcm_keyfile_parse_keyfile_failures ________________

>       assert mock_post.call_count == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <MagicMock name='post' id='359348776'>.call_count

test/test_plugin_fcm.py:713: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3376304064/jobs/5603868430#step:10:446

[caronc]

It's definitely weird; it's a always a mock result that fails with completely erroneous information. It's pure randomness makes it impossible to determine its cause.

[amotl]

PyPy3.9 and PyPy3.8 again, also with the refactorings from #728.

____________ test_plugin_twitter_dm_attachments_invalid_attachment _____________

        # No post request as attachment is not good.
>       assert mock_post.call_count == 0
E       AssertionError: assert 4 == 0
E        +  where 4 = <MagicMock name='post' id='221399936'>.call_count

test/test_plugin_twitter.py:682: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3383323341/jobs/5619109413#step:10:394

___________________ test_plugin_fcm_keyfile_parse_user_agent ___________________

>       assert mock_post.call_count == 1
E       AssertionError: assert 2 == 1
E        +  where 2 = <MagicMock name='post' id='338869000'>.call_count

test/test_plugin_fcm.py:657: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3383386910/jobs/5619232781#step:10:394

[amotl]

I've refactored the test modules test_plugin_twitter, test_plugin_fcm, test_plugin_msteams a bit. Let's see how this story continues after integrating #728.

[amotl]

PyPy3.9.

__________________ ERROR collecting test/test_config_base.py ___________________
/opt/hostedtoolcache/PyPy/3.9.12/x64/lib/pypy3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1030: in _gcd_import
    ???
<frozen importlib._bootstrap>:1007: in _find_and_load
    ???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
/opt/hostedtoolcache/PyPy/3.9.12/x64/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py:159: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
/opt/hostedtoolcache/PyPy/3.9.12/x64/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py:339: in _rewrite_test
    co = compile(tree, strfn, "exec", dont_inherit=True)
E   TypeError: expected some sort of stmt, got 'visit_Assert'

-- https://github.com/caronc/apprise/actions/runs/3465267998/jobs/5787797672#step:10:37

[amotl]

PyPy3.9 again.

_______________________ test_plugin_fcm_oauth_no_colors ________________________

mock_post = <MagicMock name='post' id='66486064'>

    @pytest.mark.skipif(
        'cryptography' not in sys.modules, reason="Requires cryptography")
    def test_plugin_fcm_oauth_no_colors(mock_post):
        """
        Verify `color=no` work as intended.
        """
    
        obj = Apprise.instantiate(
            f'fcm://mock-project-id/device/?keyfile={FCM_KEYFILE}'
            '&color=no')
>       assert mock_post.call_count == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <MagicMock name='post' id='66486064'>.call_count

test/test_plugin_fcm.py:543: AssertionError

-- https://github.com/caronc/apprise/actions/runs/3665208554/jobs/6196128736#step:10:125

[amotl]

Dear Chris,

I hope you are doing well. I wish you and your family and happy new year.

On this matter, may I ask how it was going lately? Did you observe any of the listed, or otherwise similar hiccups on CI?

With kind regards,
Andreas.

[caronc]

Sorry for the late response. It still fails on the odd deployment yeah. Like in the past, it's nothing a kick doesn't fix though

[caronc]

Not a fluke, but GitHub Python v3.6 seems totally broken today:
source.

I've tried 7 times now over the past 12 hrs without success

@amotl any advice on this one? Still failing. Not sure if there is GitHub support that can action this. It's strange because it built fine in the rebased pull request being merged.

[amotl]

Hi Chris,

the references to the CI runs you shared show that the builds on PyPy are failing, while the build on Python 3.6.8 succeeds. Can you clarify? Does it still happen?

With kind regards,
Andreas.

[caronc]

Yeah, yeah, tests consistently fail now but for a different reason. It seems pytest just seg faults on startup. It works fine for me, so I'm "guessing" it might be changes on GitHubs side of things. Light they reduced the memory per container? Or perhaps I'm testing with a different version of pytest then the one pulled by GitHub. I haven't had time to investigate

[amotl]

tests consistently fail now but for a different reason

I just wanted to go ahead and remove testing on GHA for PyPy, if that would fail, but then I inspected GH-854 as a sample, and it looks like every matrix slot actually succeeded?

[caronc]

I'm seeing core dumps now that don't even begin running at all, eg: https://github.com/caronc/apprise/actions/runs/4754818509/jobs/8448127232

Edit: For your reference to the ticket, the test failing began after that pr. It'll likely pass testing on your personal pc, but die if rebuilt by GitHub now.

[amotl]

I've submitted GH-868 in order to disable testing on PyPy 3.6 and 3.7 for now.