can't pickle _abc_data objects defined in __main__

[NiklasRosenstein]

This is a follow up to #332

It appears it doesn't work when the base class is defined in the __main__ module:

import abc, dill

class Base(abc.ABC):
    pass

dill.dumps(Base())  # TypeError: cannot pickle '_abc._abc_data' object

vs.

# tmp.py
import abc, dill

class Base(abc.ABC):
    pass

if __name__ == '__main__':a
    from tmp import Base

dill.dumps(Base())  # Ok!

Tested in CPython 3.10 and 3.11

I suppose it is okay if that is simply not supported, but given that in the general case pickling ABCs is supposed to work, the error message is not very helpful. It would be good to include a reason for why the pickling failed.

I'm currently trying to debug an issue where this same error is raised but I don't yet see yet how this particular reproduction of the issue maps to my case. I was stuck at this reproduction for a while, not understanding why people claim it works in #332.

What's even more interesting is that in the thing I'm trying to debug, Dill 0.3.5.1 works without issues but Dill 3.6 raises the above error.

[NiklasRosenstein]

FWIW it's working well with the latest version of Dill (both, the example above without the from tmp import Base as well as the issue I was trying to debug). After running a bisect, I can confirm that #580 fixed the issue. The MR has been merged ~ 1.5 months ago. Any chance we can get a release for the latest Dill version soon?

[NiklasRosenstein]

It also appears to fix another issue I have encountered with Dill when it needs to serialize an object from a PEP 420 namespace package (kraken-build/kraken#1). Unsure how related this is to the ABC serialization fix, though.

[anivegesana]

There are no releases that have the ABC functionality yet. If you try the example with the master branch, it works:

pip install git+https://github.com/uqfoundation/dill

[NiklasRosenstein]

Thanks @anivegesana! I figured that bit out already ;-) Hence my question about the plans for a new release with the ABC functionality.

[mmckerns]

A release is imminent.

[NiklasRosenstein]

Hey @mmckerns, do you have an update on the next release?

[chamini2]

Hey, @mmckerns, we just hit this and we really need a new dill release. Is there a timeline anywhere?

[mmckerns]

It's on my schedule for tomorrow. It was today, but it got bumped. It's been bumped a lot... but it should happen in the next day or so. I've been dealing with some health issues for the past year and half that have thrown my schedule way off. Anyway, I expect to cut a release tomorrow.

[chamini2]

@mmckerns , thanks for taking the time to reply. Hope you get better soon. Let me know if you need any help with the project to keep going forward, our company would love to support development of such a great project.

[mmckerns]

Thanks for the kind words and the offer. I think I'm finally headed toward a recovery. I'm still a few hours away from a release, but t'll get it put to bed shortly.

[mmckerns]

Python 3.8.17 (default, Jun 11 2023, 01:54:00) 
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import abc, dill
>>> 
>>> class Base(abc.ABC):
...     pass
... 
>>> dill.dumps(Base())
b'\x80\x04\x95\xcc\x00\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x0c_create_type\x94\x93\x94(\x8c\x03abc\x94\x8c\x07ABCMeta\x94\x93\x94\x8c\x04Base\x94h\x03\x8c\x03ABC\x94\x93\x94\x85\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x07__doc__\x94N\x8c\x13__abstractmethods__\x94(\x91\x94\x8c\r__slotnames__\x94]\x94ut\x94R\x94\x8c\x08builtins\x94\x8c\x07setattr\x94\x93\x94h\x13\x8c\x0c__qualname__\x94h\x06\x87\x94R0)\x81\x94.'
>>> dill.__version__
'0.3.8.dev0'

this works, so I'm closing it.

[NiklasRosenstein]

Is there any chance we could get a patch release earlier than before the entire 0.3.8 milestone is completed? There are 8 issues still open in that milestone. It would be good to release non-breaking changes that fix issues consumers are facing sooner than later?

Thanks,

[mmckerns]

The plan is to cut a release sometime around the python 3.12.0 final: Monday, 2023-10-02 with formal python 3.12 support.

[NiklasRosenstein]

Hi @mmckerns 👋 Python 3.12 is available for a few weeks now, is dill ready to cut a 0.3.8 release? :)

[mmckerns]

yes. 3.12 support is ready to go. 3.13.0a1 support is nearly complete. I've been a bit swamped with end of FY closeout activities, but should be out asap.

[NiklasRosenstein]

Hi @mmckerns, could you please tag a 0.3.8 version? I'm considering forking dill just to get this version released, but I think it would be much preferable to have the official release. 😄

[mmckerns]

It's currently scheduled for this Friday, but do what you need to do.

[NiklasRosenstein]

Any news on the release?