SystemError when switching projects

[mrclary]

Description

What steps will reproduce the problem?

• Open Spyder project
• Switch to another project (error may not manifest on first change)
• Switch back to first project (Issue reporter window appears)

Traceback

  File "watchdog/observers/fsevents.pyc", line 283, in run
SystemError: <built-in function add_watch> returned NULL without setting an error

Versions

• Spyder version: 4.3.0.dev0
• Python version: 3.9.1
• Qt version: 5.12.10
• PyQt5 version: 5.12.3
• Operating System: Darwin 19.6.0

Dependencies

# Mandatory:
applaunchservices >=0.1.7       :  0.2.1 (OK)
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree >=3.0.2            :  3.1.0 (OK)
IPython >=7.6.0                 :  7.20.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  22.0.1 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.8.0 (OK)
pygments >=2.0                  :  2.8.0 (OK)
pylint >=1.0                    :  2.6.2 (OK)
pyls >=0.36.2;<1.0.0            :  0+unknown (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.0             :  0.3.2 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.2 (OK)
qtconsole >=5.0.1               :  5.0.2 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
setuptools >=39.0.0             :  52.0.0 (OK)
sphinx >=0.6.6                  :  3.5.0 (OK)
spyder_kernels >=1.10.1;<1.11.0 :  1.11.0.dev0 (OK)
textdistance >=4.2.0            :  4.2.1 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog >=0.10.3               :  2.0.0 (OK)
zmq >=17                        :  22.0.3 (OK)

# Optional:
cython >=0.21                   :  0.29.21 (OK)
matplotlib >=2.0.0              :  3.3.4 (OK)
numpy >=1.7                     :  1.20.1 (OK)
pandas >=1.1.1                  :  1.2.2 (OK)
scipy >=0.17.0                  :  1.6.0 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

[ccordoba12]

It seems this was fixed in the latest version of watchdog (2.0.1):

gorakhargosh/watchdog#763

Could you update and try again?

[mrclary]

watchdog=2.0.1 does not seem to resolve the issue. The above traceback and package list was from using the macOS application.

I can also confirm the issue manifests with bootstrapping from a conda environment.

Dependencies

# Mandatory:
applaunchservices >=0.1.7       :  0.2.1 (OK)
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree >=3.0.2            :  3.1.0 (OK)
IPython >=7.6.0                 :  7.20.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  22.0.1 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.8.0 (OK)
pygments >=2.0                  :  2.7.4 (OK)
pylint >=1.0                    :  2.6.0 (OK)
pyls >=0.36.2;<1.0.0            :  v4.2.1+130.g1c4c84e72 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.0             :  0.3.0 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.1 (OK)
qtconsole >=5.0.1               :  5.0.2 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
rtree >=0.8.3                   :  0.9.4 (OK)
setuptools >=39.0.0             :  52.0.0.post20210125 (OK)
sphinx >=0.6.6                  :  3.4.3 (OK)
spyder_kernels >=1.10.1;<1.11.0 :  1.11.0.dev0 (OK)
textdistance >=4.2.0            :  4.2.0 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog >=0.10.3               :  2.0.1 (OK)
zmq >=17                        :  20.0.0 (OK)

# Optional:
cython >=0.21                   :  0.29.21 (OK)
matplotlib >=2.0.0              :  3.3.4 (OK)
numpy >=1.7                     :  1.19.2 (OK)
pandas >=1.1.1                  :  1.2.1 (OK)
scipy >=0.17.0                  :  1.6.0 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

resulting in a similar traceback

  File "/Users/rclary/opt/miniconda3/envs/spy-dev/lib/python3.9/site-packages/watchdog/observers/fsevents.py", line 283, in run
    _fsevents.add_watch(self, self.watch, callback, self.pathnames)
SystemError: <built-in function add_watch> returned NULL without setting an error

Note that the most recent version of watchdog that works is 1.0.2, which is the latest available on conda's main channel). Versions 2.0.0 and 2.0.1 manifest this issue and are available via pip (so are installed in macOS application) but are not yet available on conda's main channel.

Not sure if gorakhargosh/watchdog#763 simply failed to completely resolve gorakhargosh/watchdog#762 or whether this is a different issue.

Should we change constraint for watchdog to >=0.10.3, <2.0.0?

[BoboTiG]

It seems something else than the reported issue on watchdog. What paths are being handled?

Can you provide a minimal code to reproduce?

[ccordoba12]

@BoboTiG, thanks for chiming in! It seems you need to open and close one of our projects several times for this to happen.

[mrclary]

@ccordoba12, do we want to defer this issue until further updates to watchdog. We'll need to retest and unconstrain the version when possible.

[BoboTiG]

We will investigate on our side and come with a fix ASAP.

[ccordoba12]

do we want to defer this issue until further updates to watchdog

Sorry but I had to fix it now because we're planning to release 4.2.2 tomorrow.

We'll need to retest and unconstrain the version when possible.

Sure, once this is fixed on the Watchdog side, we'll remove the constrain I added on PR #14791.

[ccordoba12]

We will investigate on our side and come with a fix ASAP.

Thanks for your help @BoboTiG!

[BoboTiG]

@ccordoba12 release v2.0.2 with the fix is out.

The bug was present since a while (inclusing v1.0.2) but it was just ignored.

[ccordoba12]

Thanks a lot for your help @BoboTiG! Since we're going to release our version 4.2.2 today, we'll leave the current constraint in place (just because we don't have enough time to test it on our side). But we'll remove it in our next version, to be released in a month or so.

[mrclary]

With artifact from #14797 I still get the following error.

  File "watchdog/observers/fsevents.pyc", line 283, in run
RuntimeError: Cannot add watch <ObservedWatch: path=/Users/rclary/Documents/Python/spyder-kernels, is_recursive=True> - it is already scheduled

watchdog is not updated on conda main or conda-forge channels yet, so can't test bootstrap.

[BoboTiG]

Yes, the old behavior was bad as it was hiding the error (a bug in our implementation). You will need to catch the error or not add watch for already watched folders.

[ccordoba12]

With artifact from #14797 I still get the following error.

@mrclary, could you open a new issue for this? That way we won't forget to fix it in a future release.