macOS: TypeError: PyCObject_AsVoidPtr called with null pointer

[exarkun]

On macOS this program fails:

import time

from os import mkdir, unlink, rmdir
from os.path import join, exists

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

def main(local_path):
    if not exists(local_path):
        mkdir(local_path)
    a = join(local_path, "a")
    mkdir(a)

    o = Observer()
    o.start()
    w = o.schedule(FileSystemEventHandler(), a, recursive=False)

    rmdir(a)
    time.sleep(0.1)

    o.unschedule(w)

main("PyCObject_AsVoidPtr_called_with_null_pointer")

with this traceback:

$ python err6.py
/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:1151: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
  return _active[_get_ident()]
Traceback (most recent call last):
  File "err6.py", line 24, in <module>
    main("PyCObject_AsVoidPtr_called_with_null_pointer")
  File "err6.py", line 22, in main
    o.unschedule(w)
  File "/Users/administrator/Environments/watchdog-it-all/lib/python2.7/site-packages/watchdog/observers/api.py", line 345, in unschedule
    self._remove_emitter(emitter)
  File "/Users/administrator/Environments/watchdog-it-all/lib/python2.7/site-packages/watchdog/observers/api.py", line 225, in _remove_emitter
    emitter.join()
  File "/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 930, in join
    if self is current_thread():
  File "/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 1151, in currentThread
    return _active[_get_ident()]
TypeError: PyCObject_AsVoidPtr called with null pointer

[exarkun]

(edited example to shorten)

[exarkun]

The traceback doesn't point at the implementation of watchdog_stop in watchdog_fsevents.c but that seems to be where the problem arises. It's not in the traceback because there's no check for an error result from PyCObject_AsVoidPtr so the exception sticks around until some other random code happens to notice and raise it.

[exarkun]

When the watched directory is deleted the emitter thread for it exits (read_events raises OSError(ENOENT)). Then later when the watch is unscheduled there is no event loop running for it anymore, thus the PyCObject_AsVoidPtr error.

If I change BaseObserver._remove_emitter to:

    def _remove_emitter(self, emitter):
        del self._emitter_for_watch[emitter.watch]
        self._emitters.remove(emitter)
        if emitter.is_alive():
            emitter.stop()
            try:
                emitter.join()
            except RuntimeError:
                pass

to avoid the stop/join if it is already stopped then the exception goes away.

[BoboTiG]

Hello @exarkun,

Thanks for the analysis 👍
Go ahead and open a PR if you would like to :)

[exarkun]

I looked around for tests for Observer behavior but I didn't find a module that looked suitable. Do you have any suggestions about where a test for this should go?

[BoboTiG]

I will have a look this weekend ;)

[exarkun]

ping

[BoboTiG]

A test should go into test_fsevents.py. Do you want me to try?

[exarkun]

Sure, that'd be great.

[BoboTiG]

OK it should be just fine like that:

From 7f5b5af0b3de37445f95be293967427d3b1a66e6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micka=C3=ABl=20Schoentgen?= <contact@tiger-222.fr>
Date: Tue, 19 Mar 2019 15:25:38 +0100
Subject: [PATCH] Add test case

---
 tests/test_fsevents.py | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/tests/test_fsevents.py b/tests/test_fsevents.py
index f0d80066..c37797c6 100644
--- a/tests/test_fsevents.py
+++ b/tests/test_fsevents.py
@@ -8,8 +8,11 @@
 
 import logging
 import os
+import time
 from functools import partial
+from os import mkdir, rmdir
 
+from watchdog.observers import Observer
 from watchdog.observers.api import ObservedWatch
 from watchdog.observers.fsevents import FSEventsEmitter
 
@@ -41,6 +44,17 @@ def start_watching(path=None, use_full_emitter=False):
     emitter.start()
 
 
+@pytest.fixture
+def observer():
+    obs = Observer()
+    yield obs
+    obs.stop()
+    try:
+        obs.join()
+    except RuntimeError:
+        pass
+
+
 def test_remove_watch_twice():
     """
 ValueError: PyCapsule_GetPointer called with invalid PyCapsule object
@@ -63,3 +77,25 @@ def on_thread_stop(self):
     emitter.stop()
     # This is allowed to call several times .stop()
     emitter.stop()
+
+
+def test_unschedule_removed_folder(observer):
+    """
+TypeError: PyCObject_AsVoidPtr called with null pointer
+The above exception was the direct cause of the following exception:
+
+def on_thread_stop(self):
+    if self.watch:
+        _fsevents.remove_watch(self.watch)
+E       SystemError: <built-in function stop> returned a result with an error set
+
+(FSEvents.framework) FSEventStreamStop(): failed assertion 'streamRef != NULL'
+(FSEvents.framework) FSEventStreamInvalidate(): failed assertion 'streamRef != NULL'
+(FSEvents.framework) FSEventStreamRelease(): failed assertion 'streamRef != NULL'
+    """
+    a = p("a")
+    mkdir(a)
+    w = observer.schedule(event_queue, a, recursive=False)
+    rmdir(a)
+    time.sleep(0.1)
+    observer.unschedule(w)