Skip to content
This repository has been archived by the owner on Oct 31, 2023. It is now read-only.

Commit

Permalink
Register and unload in a python atexit handler. (#53)
Browse files Browse the repository at this point in the history
This works around some nasty thread exit behavior from python, described in more detail at python/cpython#87135 .
This behavior was the source of moolib crashes and deadlocks on exit.
Luckily, we can essentially unload moolib in an atexit handler - this waits for any callbacks etc to finish, and prevents any more callbacks from being issued. Any python Rpc objects will still be valid, but closed (any operations on them will silently fail).
CPython will also then not have an opportunity to exit our async threads, which would result in deadlocks.
  • Loading branch information
tscmoo authored Sep 18, 2022
1 parent 37173c5 commit 788450c
Showing 1 changed file with 10 additions and 3 deletions.
13 changes: 10 additions & 3 deletions src/moolib.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1503,13 +1503,20 @@ void set_log_level(py::object level) {
}
}

static bool moduleUnloaded = false;

PYBIND11_MODULE(_C, m) {
moduleIncRef();
rpc::moolibModule = m.ptr();
m.add_object("_cleanup", py::capsule([] {
moduleUnload();
rpc::moolibModule = nullptr;
if (!moduleUnloaded) {
moduleUnload();
}
}));
py::module::import("atexit").attr("register")(py::cpp_function([] {
if (!moduleUnloaded) {
moduleUnload();
}
}));

// numpy forks during import, and pybind can trigger a numpy import
// during serialization (just for checking if the input is a numpy array).
Expand Down

0 comments on commit 788450c

Please sign in to comment.