-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cloudpickle_fast causes pickled object to be uncollectable #343
Comments
@pierreglaser could this be a duplicate of #327 fixed upstream in https://bugs.python.org/issue39492? |
Yes. @edoakes this bug will be fixed starting Python 3.8.2. |
I enabled test on Python nightly by default in #344 and I confirm that the new test introduced in: https://github.com/cloudpipe/cloudpickle/pull/334/files is not skipped in Python 3.9-dev: https://github.com/cloudpipe/cloudpickle/pull/344/checks?check_run_id=446363404 while it is skipped on Python 3.8.1: https://github.com/cloudpipe/cloudpickle/pull/344/checks?check_run_id=446363575 |
Closing this since it got resolved. |
I work on Ray, where we're now using
cloudpickle_fast
for object serialization (thanks for the great work!). We have recently started working on distributed reference counting in Ray, which depends on local Python object reference counting.When running some tests for our reference counting implementation, we ran into an issue where some serialized objects were uncollectable (due to a circular reference), causing them not to be deleted until a call to the garbage collector. After some investigation, it seems the problem is in
cloudpickle_fast
as shown by the following example:We expect that the
__del__
method of theShortLivedObject
will be called afterpickle.dumps
returns, as the only references to the object are removed. When running this example on Python 3.7 or earlier (not usingcloudpickle_fast
), this is indeed the case. However, when running on Python 3.8 (usingcloudpickle_fast
),ShortLivedObject.__del__
is not called until the call togc.collect()
.Any help would be much appreciated!
FYI: setting
gc.DEBUG_SAVEALL
and printinggc.garbage
after the collect call shows the following:The text was updated successfully, but these errors were encountered: