-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stale reference needs a gc.collect to be collected in cloudpickle_fast #327
Comments
I found out the issue. To understand its origin, first note that this: class MyObject:
def my_method(self):
pass
my_object = MyObject()
my_object.looped_method = my_object.my_method Generates a cycle. So Now, for performance reasons, this exact pattern shows up in the cc @ogrisel @tomMoral (because it originally happened in |
As mentioned by @ogrisel, we can try to break the reference cycle by attempting to remove the I did not find any clean way do update def _cleanup(self):
# break the reference cycle involving the current pickler because of it
# directly holding a reference to a bound reducer_override method at
# the C-level. This is done by re-setting reducer_override to a faulty
# function not referencing the current pickler, and triggering the
# C-level update using a smoke dump call, made to error out without
# writing anything into the previous file object.
def _reducer_override_placeholder(obj):
raise RuntimeError
def _f():
pass
self.reducer_override = _reducer_override_placeholder
try:
Pickler.dump(self, _f)
except RuntimeError:
pass which should be invoked after a However, the fix for this problem in |
I created an issue in bugs.python.org and submitted a patch (python/cpython#18266) EDIT: This issue is now fixed on Python upstream, and will be present in the next Python 3.8.x. |
We noticed test failures in joblib/loky#229 using
Python3.8
. After some investigations, this bug can be reproduced withcloudpickle
only: apparently, usingPython3.8
and the newreducer_override
mechanism, something holds a stale reference until agc.collect
call is performed:The text was updated successfully, but these errors were encountered: