Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for recent CPython and PyPy versions, drop support for Python 2 #680

Merged
merged 31 commits into from
Oct 8, 2022
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
6aa4f09
CI: Update from Ubuntu 16 (Xenial Xerus) to Ubuntu 20 (Focal Fossa)
amotl Oct 6, 2022
b1ad368
CI: Trim Python 3.7 version
amotl Oct 6, 2022
b7ff3e5
CI: Pin to `cryptography<3.4`, 3.3 is the last one not needing Rust
amotl Oct 6, 2022
e67c899
CI: Modernize PyPy versions
amotl Oct 6, 2022
ccb82e4
Tests: Skip AWS SES tests on PyPy, they are stalling on Travis CI
amotl Oct 6, 2022
d71de17
CI: Don't run flake8 on PyPy 3.7, it croaks
amotl Oct 6, 2022
4f9ef57
CI: Add support for CPython 3.10
amotl Oct 6, 2022
80680d5
Package metadata: Explicitly designate supported Python versions
amotl Oct 6, 2022
ce20676
Tests: Skip `test_plugin_fcm_general_legacy` on PyPy, it is flaky
amotl Oct 6, 2022
cbd1dc7
Tests: Skip `test_msteams_yaml_config` on PyPy, it is flaky
amotl Oct 7, 2022
0a6558f
Remove most Python 2 specific code
amotl Oct 7, 2022
9e20760
dropped six and updated rpm packaging
caronc Oct 7, 2022
1958018
more python v2.x legacy code removal handling
caronc Oct 7, 2022
2ec6f5f
io.open switch to builtins.open
caronc Oct 7, 2022
4f6e18a
Chore: Mitigate `PytestConfigWarning: Unknown config option: strict`
amotl Oct 7, 2022
919de0e
Chore: Mitigate `DeprecationWarning: invalid escape sequence '\.'`
amotl Oct 7, 2022
3a92897
Add runtime warning about `async def async_notify`
amotl Oct 7, 2022
369ad9b
Chore: Fix a few `noqa` tags
amotl Oct 7, 2022
e77d155
Chore: Add a few inline comments about reviewing Python2-compatible code
amotl Oct 7, 2022
19043f2
Chore: Fix typos and improve wording
amotl Oct 7, 2022
4e00bf5
CI: Pre-install `wheel` package
amotl Oct 7, 2022
30b7cf1
CI: Do not install `flake8` on PyPy 3.7
amotl Oct 7, 2022
470760a
CI: Pin to `cryptography<3.4` with PyPy only
amotl Oct 7, 2022
a1bcbfc
Chore: Mitigate `DeprecationWarning: invalid escape sequence '\g'`
amotl Oct 7, 2022
2e2ee1b
Repository: Fix wrong .gitignore for `.spec` files
amotl Oct 7, 2022
3436daa
Packaging: Remove `six` runtime dependency
amotl Oct 7, 2022
0a8c012
Packaging: Remove `python-mock` test-time dependency
amotl Oct 7, 2022
d284300
CI: Install `cryptography<3.4` on PyPy before everything else
amotl Oct 7, 2022
4576702
CI: Also install `cryptography<3.4` within tox environment
amotl Oct 7, 2022
70af51b
specfile update
caronc Oct 8, 2022
7e47cfd
just rolling back the warning as it wont' be an issue
caronc Oct 8, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,6 @@ sdist/
.installed.cfg
*.egg

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Allow RPM SPEC files despite pyInstaller ignore
!packaging/redhat/*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt
Expand Down
41 changes: 26 additions & 15 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
language: python

dist: xenial
dist: focal

addons:
apt:
Expand All @@ -11,40 +11,51 @@ matrix:
include:
- python: "3.6"
env: TOXENV=py36
- python: "3.7.7"
- python: "3.7"
env: TOXENV=py37
- python: "3.8"
env: TOXENV=py38
- python: "3.9"
env: TOXENV=py39
- python: "3.9-dev"
env: TOXENV=py39-dev
- python: "3.10"
env: TOXENV=py310
# PyPy Environments
- python: "pypy2.7-6.0"
env: TOXENV=pypy
- python: "pypy3.5-7.0"
env: TOXENV=pypy3
- python: "pypy3.6-7.3.3"
env: TOXENV=pypy36
- python: "pypy3.7-7.3.9"
env: TOXENV=pypy37
- python: "pypy3.8-7.3.9"
env: TOXENV=pypy38
- python: "pypy3.9-7.3.9"
env: TOXENV=pypy39
# An extra environment where additional packages are not installed
- python: "3.9"
env:
- TOXENV=bare

install:
- pip install babel
# upgrade tox, pip, and virtualenv so Python 3.6 will build crytography:
# https://travis-ci.community/t/pip-install-cryptography-fails-on-py36/11233
- pip install -U tox pip virtualenv
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any value in bringing this back? This 1 liner allowed all pip calls to install cryptography (regardless of what version)

Copy link
Collaborator Author

@amotl amotl Oct 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's right there, just tweaked it a bit and accidentally removed your comment.

# Use up-to-date versions of tox, pip, virtualenv, and wheel.
pip install --upgrade tox pip virtualenv wheel


# Use up-to-date versions of tox, pip, virtualenv, and wheel.
- pip install --upgrade tox pip virtualenv wheel

# cryptography 3.3 is the last one not needing a Rust toolchain. Let's use it for PyPy.
- if [[ $TOXENV == 'pypy'* ]]; then pip install "cryptography<3.4"; fi

# Install project dependencies.
- pip install codecov
- pip install -r dev-requirements.txt
- pip install -r requirements.txt

# bare installs do not include extra package dependencies
- if [[ $TOXENV != 'bare' ]]; then pip install -r all-plugin-requirements.txt; fi
# pypy and bare installs do not include dbus-python
- if [[ $TOXENV != 'bare' ]] && [[ $TRAVIS_PYTHON_VERSION != 'pypy'* ]]; then travis_retry pip install dbus-python; fi
# Python 3.7 importlib-metadata becomes incompatible with flake8 unless we use
# a version that still supports EntryPoints.get(); tox.ini updated to not call flake; this was
# the only way around the Travis CI Builder issues
- if [[ $TOXENV == 'py37' ]]; then pip uninstall --yes flake8; fi

# Fix/workaround: Python 3.7 importlib-metadata becomes incompatible with flake8,
# unless we use a version that still supports EntryPoints.get().
# `tox.ini` has been updated to not call flake8 on Python 3.7.
- if [[ $TOXENV == 'py37' || $TOXENV == 'pypy37' ]]; then pip uninstall --yes flake8; fi

# run tests
script:
Expand Down
14 changes: 0 additions & 14 deletions Dockerfile.py27

This file was deleted.

148 changes: 36 additions & 112 deletions apprise/Apprise.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
# THE SOFTWARE.

import os
import six
import warnings
from itertools import chain
from . import common
from .conversion import convert_between
Expand All @@ -44,13 +44,13 @@
from . import plugins
from . import __version__

# Python v3+ support code made importable so it can remain backwards
# Python v3+ support code made importable, so it can remain backwards
# compatible with Python v2
# TODO: Review after dropping support for Python 2.
from . import py3compat
Copy link
Collaborator Author

@amotl amotl Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the py3compat module may be revisited in a subsequent iteration? I will add a corresponding TODO marker here, if you don't have any objections.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree; this definitely needs to be revisited... a lot of @YoRyan's work with creating the .pyi files can probably be merged now into the .py files themselves. I don't think it's a bit deal if it doesn't happen in this PR though, it's getting quite big now.

I think it's great the extra CI tasks you added

ASYNCIO_SUPPORT = not six.PY2


class Apprise(object):
class Apprise:
"""
Our Notification Manager

Expand Down Expand Up @@ -124,7 +124,7 @@ def instantiate(url, asset=None, tag=None, suppress_exceptions=True):
# Prepare our Asset Object
asset = asset if isinstance(asset, AppriseAsset) else AppriseAsset()

if isinstance(url, six.string_types):
if isinstance(url, str):
caronc marked this conversation as resolved.
Show resolved Hide resolved
# Acquire our url tokens
results = plugins.url_to_dict(
url, secure_logging=asset.secure_logging)
Expand Down Expand Up @@ -247,7 +247,7 @@ def add(self, servers, asset=None, tag=None):
# prepare default asset
asset = self.asset

if isinstance(servers, six.string_types):
if isinstance(servers, str):
# build our server list
servers = parse_urls(servers)
if len(servers) == 0:
Expand Down Expand Up @@ -275,7 +275,7 @@ def add(self, servers, asset=None, tag=None):
self.servers.append(_server)
continue

elif not isinstance(_server, (six.string_types, dict)):
elif not isinstance(_server, (str, dict)):
logger.error(
"An invalid notification (type={}) was specified.".format(
type(_server)))
Expand Down Expand Up @@ -306,7 +306,7 @@ def clear(self):

def find(self, tag=common.MATCH_ALL_TAG, match_always=True):
"""
Returns an list of all servers matching against the tag specified.
Returns a list of all servers matching against the tag specified.

"""

Expand Down Expand Up @@ -347,14 +347,14 @@ def notify(self, body, title='', notify_type=common.NotifyType.INFO,
body_format=None, tag=common.MATCH_ALL_TAG, match_always=True,
attach=None, interpret_escapes=None):
"""
Send a notification to all of the plugins previously loaded.
Send a notification to all the plugins previously loaded.

If the body_format specified is NotifyFormat.MARKDOWN, it will
be converted to HTML if the Notification type expects this.

if the tag is specified (either a string or a set/list/tuple
of strings), then only the notifications flagged with that
tagged value are notified. By default all added services
tagged value are notified. By default, all added services
are notified (tag=MATCH_ALL_TAG)

This function returns True if all notifications were successfully
Expand All @@ -363,59 +363,37 @@ def notify(self, body, title='', notify_type=common.NotifyType.INFO,
simply having empty configuration files that were read.

Attach can contain a list of attachment URLs. attach can also be
represented by a an AttachBase() (or list of) object(s). This
represented by an AttachBase() (or list of) object(s). This
identifies the products you wish to notify

Set interpret_escapes to True if you want to pre-escape a string
such as turning a \n into an actual new line, etc.
"""

if ASYNCIO_SUPPORT:
return py3compat.asyncio.tosync(
self.async_notify(
body, title,
notify_type=notify_type, body_format=body_format,
tag=tag, match_always=match_always, attach=attach,
interpret_escapes=interpret_escapes,
),
debug=self.debug
)

else:
try:
results = list(
self._notifyall(
Apprise._notifyhandler,
body, title,
notify_type=notify_type, body_format=body_format,
tag=tag, attach=attach,
interpret_escapes=interpret_escapes,
)
)

except TypeError:
# No notifications sent, and there was an internal error.
return False

else:
if len(results) > 0:
# All notifications sent, return False if any failed.
return all(results)

else:
# No notifications sent.
return None
return py3compat.asyncio.tosync(
self.async_notify(
body, title,
notify_type=notify_type, body_format=body_format,
tag=tag, match_always=match_always, attach=attach,
interpret_escapes=interpret_escapes,
),
debug=self.debug
)

def async_notify(self, *args, **kwargs):
Copy link
Collaborator Author

@amotl amotl Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring of the async_notify method says:

This method is an async method that should be awaited on, even if it is missing the async keyword in its signature. This is omitted to preserve syntax compatibility with Python 2.

After dropping support for Python 2, the async keyword may be added back, if that makes sense?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you can add async; I wrote the async_notify() method with just this scenario in mind. 🙂

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not get hasty on this one... a lot of people have adopted notify into their code. they'll all break and have to add await in front of it and start tracking loops. Leave the async_notify() as is for now.

Copy link
Contributor

@YoRyan YoRyan Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to be worried; there is no backwards compatibility issue. notify() will continue to exist, and Python 3 async users already had to write await async_notify(). Switching to

async def async_notify(self, *args, **kwargs)

just makes it a little more obvious that this method must be called with await. 😉

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, if you look at the corresponding .pyi file, you'll see that the type annotation already has the async keyword included.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also pleased to meet you, Andreas!

Even if that function has been declared async, it has not been defined as such. So, chances are, that people are using it within synchronous code. Even if it would have been wrong not to await the future, their code would still have mostly worked.

The function already returns a future, hence every return value being wrapped with py3compat.asyncio.toasyncwrapvalue. I do believe it's indistinguishable from an async function with the wrapper removed. Because if you don't use await, you have to unwrap the future.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After touching some stuff here and blowing it all up, i think I'll leave it the way it is for this commit since everything else works so well. @YoRyan is an async master... he breaths in air and outputs asynchronous threaded Python 😉. i think i'll wait for him to review the async part when this request is all pushed upstream.

Aww, thanks. 😁 Yes, we absolutely can leave this for later, and I would be happy to make that PR.

For the record, if you add the async keyword, it's also necessary to remove the py3compat.asyncio.toasyncwrapvalue calls. Basically, the following two functions are equivalent:

def myAsyncFunction():
    return py3compat.asyncio.toasyncwrapvalue("Hello, World!")
async def myAsyncFunction():
    return "Hello, World!"

Copy link
Collaborator Author

@amotl amotl Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for providing an example. Let me demonstrate, while the returning behavior is being made equivalent, the invocation behavior, at runtime, is not.

The following code, where the async_foo function is called synchronously, will run all its main body, before issuing a RuntimeWarning: coroutine 'toasyncwrapvalue' was never awaited when returning.

On the other hand, when you will prefix that routine with the async keyword, the function will not be invoked at all. Instead, Python will issue a RuntimeWarning: coroutine 'foo_async' was never awaited.

On that matter, I think Chris is absolutely right:

A lot of people have adopted notify into their code. They'll all break and have to add await in front of it and start tracking loops. Leave the async_notify() as is for now.

However, it will not break because of not waiting for the completion of the function's future, that would only be about getting a return value anyway, and making sure it completed. It will break for all people who either unaware or errorneously are invoking the function synchronously, and then suddenly finding it would stop being executed, which might be many.

All users who are doing it this way, must change their code to await the function, and make sure they will invoke their code asynchronously with asyncio.run(main()). Many may not have heard about all of that at all.

I think this would be a change which should only be done on a major release bump, designated as "breaking change".

import apprise


def async_foo():
    raise ValueError("foo")
    return apprise.py3compat.asyncio.toasyncwrapvalue("Hello, World!")


def main():
    async_foo()


if __name__ == "__main__":
    main()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I understand now. I had not considered synchronous callers who would not be interested in the return value and who might inadvertently discard the future.

I absolutely intended for async_notify() to behave just like an async def function, and it's unfortunate - given the constraint of having to be syntactically compatible with Python 2 - that I was unable to achieve a perfect replacement.

But in contrast to "a lot of people," I doubt this breakage would affect that many at all. The new async method was only introduced fairly recently, and the "async_" prefix combined with the type annotation should have served as reminders that the returned future needs to be await'ed. Nevertheless, yes, it is a breakage, and I agree it should be documented for a future PR and release.

Copy link
Collaborator Author

@amotl amotl Oct 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi again,

The new async method was only introduced recently.

It is still 1.5 years, though ;].

But in contrast to "a lot of people," I doubt this breakage would affect that many at all.

Sure, I think I might have exeggarated this a bit. I only wanted to outline the potential breaking change with insistence, but not overshoot the discussion. It may have happened - apologies for that.

Nevertheless, yes, it is a breakage, and I agree it should be documented for a future PR and release.

I think Chris will be happy if you continue throwing in your expertise on this matter. As we observed yesterday with a testing build on Copr, just slapping an async keyword will not immediately work, but instead break some of Apprise's own tests.

Whether the improvement may be released within the 1.x series, with a corresponding admonition in the changelog, or just right away with a new 2.x release, is of course up to you. My verdict is: Why not just make a 2.x when the switchover is ready - I think it will not do any harm whatsoever.

Keep up the spirit, thanks for all the excellent discussions, and with kind regards,
Andreas.

"""
Send a notification to all of the plugins previously loaded, for
Send a notification to all the plugins previously loaded, for
asynchronous callers. This method is an async method that should be
awaited on, even if it is missing the async keyword in its signature.
(This is omitted to preserve syntax compatibility with Python 2.)

The arguments are identical to those of Apprise.notify(). This method
is not available in Python 2.
The arguments are identical to those of Apprise.notify().

# TODO: Add `async` keyword.
"""
warnings.warn("A future version of Apprise will add the `async` "
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The #TODO ref is good, but the async is already in place today

We don't need this warning at all. We just didn't have the async keyword there (and had some magic in front of it, so the code could co-exist in a Python v2.7 environment. we're good here for sure.

Copy link
Collaborator Author

@amotl amotl Oct 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I am not right on track yet, but didn't we just observe that adding async breaks the test suite completely, as it will likewise do for others?

Maybe it is too late at night already for me to correctly understand why adding async is not problematic ;]. Feel free to adjust as necessary. Thanks for your patience!

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might be right here; but to my knowledge we were async code that under the hood created an async object. The chagne Ryan would make would be more then the bug i introduced by only updating one function. The wrappers under the hood will disappear . But i could also be misinterpreting what is wrong here.

"keyword to `async_notify`. Please make sure your code "
"will correctly `await` the returned future.")

try:
coroutines = list(
Expand Down Expand Up @@ -477,7 +455,7 @@ def _notifyall(self, handler, body, title='',
tag=common.MATCH_ALL_TAG, match_always=True, attach=None,
interpret_escapes=None):
"""
Creates notifications for all of the plugins loaded.
Creates notifications for all the plugins loaded.

Returns a generator that calls handler for each notification. The first
and only argument supplied to handler is the server, and the keyword
Expand All @@ -496,23 +474,11 @@ def _notifyall(self, handler, body, title='',
raise TypeError(msg)

try:
if six.PY2:
# Python 2.7 encoding support isn't the greatest, so we try
# to ensure that we're ALWAYS dealing with unicode characters
# prior to entrying the next part. This is especially required
# for Markdown support
if title and isinstance(title, str): # noqa: F821
title = title.decode(self.asset.encoding)

if body and isinstance(body, str): # noqa: F821
body = body.decode(self.asset.encoding)

else: # Python 3+
if title and isinstance(title, bytes): # noqa: F821
title = title.decode(self.asset.encoding)
if title and isinstance(title, bytes):
title = title.decode(self.asset.encoding)

if body and isinstance(body, bytes): # noqa: F821
body = body.decode(self.asset.encoding)
if body and isinstance(body, bytes):
body = body.decode(self.asset.encoding)

except UnicodeDecodeError:
msg = 'The content passed into Apprise was not of encoding ' \
Expand Down Expand Up @@ -580,43 +546,12 @@ def _notifyall(self, handler, body, title='',
.encode('ascii', 'backslashreplace')\
.decode('unicode-escape')

except UnicodeDecodeError: # pragma: no cover
# This occurs using a very old verion of Python 2.7
# such as the one that ships with CentOS/RedHat 7.x
# (v2.7.5).
conversion_body_map[server.notify_format] = \
conversion_body_map[server.notify_format] \
.decode('string_escape')

conversion_title_map[server.notify_format] = \
conversion_title_map[server.notify_format] \
.decode('string_escape')

amotl marked this conversation as resolved.
Show resolved Hide resolved
except AttributeError:
# Must be of string type
msg = 'Failed to escape message body'
logger.error(msg)
raise TypeError(msg)

if six.PY2:
# Python 2.7 strings must be encoded as utf-8 for
# consistency across all platforms
if conversion_body_map[server.notify_format] and \
isinstance(
conversion_body_map[server.notify_format],
unicode): # noqa: F821
conversion_body_map[server.notify_format] = \
conversion_body_map[server.notify_format]\
.encode('utf-8')

if conversion_title_map[server.notify_format] and \
isinstance(
conversion_title_map[server.notify_format],
unicode): # noqa: F821
conversion_title_map[server.notify_format] = \
conversion_title_map[server.notify_format]\
.encode('utf-8')

yield handler(
server,
body=conversion_body_map[server.notify_format],
Expand Down Expand Up @@ -669,12 +604,12 @@ def details(self, lang=None, show_requirements=False, show_disabled=False):

# Standard protocol(s) should be None or a tuple
protocols = getattr(plugin, 'protocol', None)
if isinstance(protocols, six.string_types):
if isinstance(protocols, str):
protocols = (protocols, )

# Secure protocol(s) should be None or a tuple
secure_protocols = getattr(plugin, 'secure_protocol', None)
if isinstance(secure_protocols, six.string_types):
if isinstance(secure_protocols, str):
secure_protocols = (secure_protocols, )

# Add our protocol details to our content
Expand Down Expand Up @@ -779,15 +714,8 @@ def __getitem__(self, index):

def __bool__(self):
"""
Allows the Apprise object to be wrapped in an Python 3.x based 'if
statement'. True is returned if at least one service has been loaded.
"""
return len(self) > 0

def __nonzero__(self):
"""
Allows the Apprise object to be wrapped in an Python 2.x based 'if
statement'. True is returned if at least one service has been loaded.
Allows the Apprise object to be wrapped in an 'if statement'.
True is returned if at least one service has been loaded.
"""
return len(self) > 0

Expand All @@ -807,7 +735,3 @@ def __len__(self):
"""
return sum([1 if not isinstance(s, (ConfigBase, AppriseConfig))
else len(s.servers()) for s in self.servers])


if six.PY2:
del Apprise.async_notify
1 change: 0 additions & 1 deletion apprise/Apprise.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,5 @@ class Apprise:
def pop(self, index: int) -> ConfigBase: ...
def __getitem__(self, index: int) -> ConfigBase: ...
def __bool__(self) -> bool: ...
def __nonzero__(self) -> bool: ...
def __iter__(self) -> Iterator[ConfigBase]: ...
def __len__(self) -> int: ...
Loading