Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can't pickle _abc_data objects defined in __main__ #595

Closed
NiklasRosenstein opened this issue May 12, 2023 · 18 comments
Closed

can't pickle _abc_data objects defined in __main__ #595

NiklasRosenstein opened this issue May 12, 2023 · 18 comments

Comments

@NiklasRosenstein
Copy link

NiklasRosenstein commented May 12, 2023

This is a follow up to #332

It appears it doesn't work when the base class is defined in the __main__ module:

import abc, dill

class Base(abc.ABC):
    pass

dill.dumps(Base())  # TypeError: cannot pickle '_abc._abc_data' object

vs.

# tmp.py
import abc, dill

class Base(abc.ABC):
    pass

if __name__ == '__main__':a
    from tmp import Base

dill.dumps(Base())  # Ok!

Tested in CPython 3.10 and 3.11

I suppose it is okay if that is simply not supported, but given that in the general case pickling ABCs is supposed to work, the error message is not very helpful. It would be good to include a reason for why the pickling failed.

I'm currently trying to debug an issue where this same error is raised but I don't yet see yet how this particular reproduction of the issue maps to my case. I was stuck at this reproduction for a while, not understanding why people claim it works in #332.

What's even more interesting is that in the thing I'm trying to debug, Dill 0.3.5.1 works without issues but Dill 3.6 raises the above error.

@NiklasRosenstein
Copy link
Author

NiklasRosenstein commented May 12, 2023

FWIW it's working well with the latest version of Dill (both, the example above without the from tmp import Base as well as the issue I was trying to debug). After running a bisect, I can confirm that #580 fixed the issue. The MR has been merged ~ 1.5 months ago. Any chance we can get a release for the latest Dill version soon?

@NiklasRosenstein
Copy link
Author

It also appears to fix another issue I have encountered with Dill when it needs to serialize an object from a PEP 420 namespace package (kraken-build/kraken#1). Unsure how related this is to the ABC serialization fix, though.

@anivegesana
Copy link
Contributor

There are no releases that have the ABC functionality yet. If you try the example with the master branch, it works:

pip install git+https://github.com/uqfoundation/dill

@NiklasRosenstein
Copy link
Author

NiklasRosenstein commented May 12, 2023

Thanks @anivegesana! I figured that bit out already ;-) Hence my question about the plans for a new release with the ABC functionality.

@mmckerns
Copy link
Member

A release is imminent.

@NiklasRosenstein
Copy link
Author

Hey @mmckerns, do you have an update on the next release?

@chamini2
Copy link

Hey, @mmckerns, we just hit this and we really need a new dill release. Is there a timeline anywhere?

@mmckerns
Copy link
Member

It's on my schedule for tomorrow. It was today, but it got bumped. It's been bumped a lot... but it should happen in the next day or so. I've been dealing with some health issues for the past year and half that have thrown my schedule way off. Anyway, I expect to cut a release tomorrow.

@chamini2
Copy link

@mmckerns , thanks for taking the time to reply. Hope you get better soon. Let me know if you need any help with the project to keep going forward, our company would love to support development of such a great project.

@mmckerns
Copy link
Member

Thanks for the kind words and the offer. I think I'm finally headed toward a recovery. I'm still a few hours away from a release, but t'll get it put to bed shortly.

@mmckerns mmckerns added this to the dill-0.3.8 milestone Aug 19, 2023
@mmckerns
Copy link
Member

Python 3.8.17 (default, Jun 11 2023, 01:54:00) 
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import abc, dill
>>> 
>>> class Base(abc.ABC):
...     pass
... 
>>> dill.dumps(Base())
b'\x80\x04\x95\xcc\x00\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x0c_create_type\x94\x93\x94(\x8c\x03abc\x94\x8c\x07ABCMeta\x94\x93\x94\x8c\x04Base\x94h\x03\x8c\x03ABC\x94\x93\x94\x85\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x07__doc__\x94N\x8c\x13__abstractmethods__\x94(\x91\x94\x8c\r__slotnames__\x94]\x94ut\x94R\x94\x8c\x08builtins\x94\x8c\x07setattr\x94\x93\x94h\x13\x8c\x0c__qualname__\x94h\x06\x87\x94R0)\x81\x94.'
>>> dill.__version__
'0.3.8.dev0'

this works, so I'm closing it.

@NiklasRosenstein
Copy link
Author

Is there any chance we could get a patch release earlier than before the entire 0.3.8 milestone is completed? There are 8 issues still open in that milestone. It would be good to release non-breaking changes that fix issues consumers are facing sooner than later?

Thanks,

@mmckerns
Copy link
Member

mmckerns commented Sep 6, 2023

The plan is to cut a release sometime around the python 3.12.0 final: Monday, 2023-10-02 with formal python 3.12 support.

@NiklasRosenstein
Copy link
Author

NiklasRosenstein commented Oct 23, 2023

Hi @mmckerns 👋 Python 3.12 is available for a few weeks now, is dill ready to cut a 0.3.8 release? :)

@mmckerns
Copy link
Member

yes. 3.12 support is ready to go. 3.13.0a1 support is nearly complete. I've been a bit swamped with end of FY closeout activities, but should be out asap.

NiklasRosenstein added a commit to kraken-build/kraken that referenced this issue Nov 8, 2023
…kraken-build (#125)

* Merge kraken-common, kraken-core, kraken-std and kraken-wrapper into kraken-build

* kraken-build/: improvement: Merge `kraken-common`, `kraken-core`, `kraken-std` and `kraken-wrapper` packages into a single `kraken-build` package.

* Updated PR references in 1 changelogs.

skip-checks: true

* update CI, disable Selftest and Kraken-std integration tests, fix kraken-build/pyproject.toml

* format imports with isort

* upgrade Mypy python version

* one more isort

* adjustments

* install poetry/pdm into CI environment

* Move Python projects for integration tests into examples/ folder

* also move Cargo examples for integration tests into examples/ folder

* Improve Python integration tests by unsetting the locally activate virtual environment

* fix install pdm in ci

* comment out 3.11 and 3.12 for now because of Dill

* update Python constraint due to Dill

* add mention to readme that we cant use 3.11+ until Dill 0.3.8 is released (uqfoundation/dill#595)

* remove unused import

* run rustup update to make sure cargo is available

* use example_dir

* fix

* disable pyupgrade for now, we will do that in another MR for kraken-common and kraken-wrapper which is now elevated to 3.10+

* re-enable selftest

* need poetry/rustup in selftest as well

* improve integration test for Python when running locally where Poetry can cache a copy of the same wheel with a different hash from a previous test run

* also fix PDM for repeated tests locally

* mark deactivate_venv fixture as autouse

* doctest fix

---------

Co-authored-by: GitHub Action <github-action@users.noreply.github.com>
@NiklasRosenstein
Copy link
Author

NiklasRosenstein commented Dec 5, 2023

Hi @mmckerns, could you please tag a 0.3.8 version? I'm considering forking dill just to get this version released, but I think it would be much preferable to have the official release. 😄

@mmckerns
Copy link
Member

mmckerns commented Dec 5, 2023

It's currently scheduled for this Friday, but do what you need to do.

@NiklasRosenstein
Copy link
Author

Any news on the release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants