make celery a required dependency and the celery extra's optional dependencies

[woutdenolf]

The modifications in #172 can cause issues like #205.

When installing pytest-redis without celery[redis,pymemcache], pytest becomes unusable because the pytest plugin system causes pytest-redis to be imported, which in turn imports celery, redis and memcache.

For example

pip install pytest-redis

pytest -h

Traceback (most recent call last):
  ...
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/users/denolf/virtualenvs/pybox_VH60vO/ubuntu_20_04/py38/lib/python3.8/site-packages/pluggy/_manager.py", line 414, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/users/denolf/.pyenv/ubuntu_20_04/versions/3.8.5/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  ...
  File "/users/denolf/virtualenvs/pybox_VH60vO/ubuntu_20_04/py38/lib/python3.8/site-packages/pytest_celery/api/backend.py", line 9, in <module>
    from pytest_celery.api.base import CeleryTestCluster
  File "/users/denolf/virtualenvs/pybox_VH60vO/ubuntu_20_04/py38/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 175, in exec_module
    exec(co, module.__dict__)
  File "/users/denolf/virtualenvs/pybox_VH60vO/ubuntu_20_04/py38/lib/python3.8/site-packages/pytest_celery/api/base.py", line 15, in <module>
    from celery import Celery
ModuleNotFoundError: No module named 'celery'

We could make celery an optional dependency of pytest-redis but I doubt this is meaningful. What is meaningful is to make the celery extra's optional: celery[redis] and celery[pymemcache].

That's what I tried to do in this MR by making the memcached and redis vendor modules optional. I would like to do this

[options]
install_requires = 
    celery<6.0.0

[options.extras_require]
full =
    celery[redis,pymemcached]

But I cannot figure out how to do this in poetry (I believe python-poetry/poetry-core#613 is needed for that). For now I did this

[tool.poetry.dependencies]
celery = { version = "<6.0.0" }
redis = { version = ">=4.5.2,<6.0.0,!=4.5.5", optional = true }
python-memcached = { version = ">=1.61", optional = true }

[tool.poetry.extras]
full = ["redis", "python-memcached"]

but this means we are managing the redis and python-memcached version limits ourselves which is not what we want.

Edit: python-poetry/poetry-core#613 (comment) shows that there is a way to do it in poetry

[tool.poetry.dependencies]
celery = [
    { version = "<6.0.0" },
    { version = "<6.0.0", extras = ["redis", "pymemcache"], markers = "extra == 'full'" }
]

[tool.poetry.extras]
full = []

[woutdenolf]

@Nusnus Let me know if celery itself should stay optional as well.

[Nusnus]

@woutdenolf

@Nusnus Let me know if celery itself should stay optional as well.

It appears you’ve edited this PR while I was answering the other issue :)
Let’s keep the conversation in one place for our own sake -> over the issue.

When there’s a final conclusion, we can discuss what is the preferred change.

[woutdenolf]

@Nusnus I changed this PR to include only the changes that allow pytest-celery to run when the celery extra's redis and memached are not installed.

[woutdenolf]

Note that after this PR it is enough to do

pip install celery  # or whatever the needs are
pip install pytest-celery

When you try to use the redis or memcached backends while the celery extra's are not installed you get

from pytest_celery import MemcachedContainer

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'MemcachedContainer' from 'pytest_celery'

I guess it would be more informative if this traceback included the MissingCeleryDependency trace as well

Traceback (most recent call last):
  File "/home/denolf/projects/pytest-celery/src/pytest_celery/vendors/memcached/__init__.py", line 10, in <module>
    import memcache  # noqa F401
ModuleNotFoundError: No module named 'memcache'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/denolf/projects/pytest-celery/src/pytest_celery/__init__.py", line 33, in <module>
    from pytest_celery.vendors.memcached.api import *
  File "/home/denolf/projects/pytest-celery/src/pytest_celery/vendors/memcached/__init__.py", line 12, in <module>
    raise MissingCeleryDependency("celery extra dependency missing: celery[pymemcache]")
pytest_celery.vendors.MissingCeleryDependency: celery extra dependency missing: celery[pymemcache]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'MemcachedContainer' from 'pytest_celery'

Not sure how to achieve this though.

[woutdenolf]

I tried to find a better name for MissingCeleryDependency. We cannot say MissingBackend or MissingBroker because redis can be both. Perhaps MissingVendor? Still a cryptic name but this exception never bubbles up to the user so perhaps not a problem.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 47 lines in your changes are missing coverage. Please review.

Project coverage is 25.75%. Comparing base (40ee732) to head (8afec66).
Report is 7 commits behind head on main.

Files	Patch %	Lines
src/pytest_celery/defaults.py	0.00%	20 Missing ⚠️
src/pytest_celery/__init__.py	0.00%	15 Missing ⚠️
src/pytest_celery/vendors/memcached/__init__.py	0.00%	5 Missing ⚠️
src/pytest_celery/vendors/redis/__init__.py	0.00%	5 Missing ⚠️
src/pytest_celery/vendors/__init__.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #206      +/-   ##
==========================================
- Coverage   28.01%   25.75%   -2.26%     
==========================================
  Files          36       39       +3     
  Lines        1053     1087      +34     
  Branches      213      215       +2     
==========================================
- Hits          295      280      -15     
- Misses        730      781      +51     
+ Partials       28       26       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.