[BUG] Dill dumps the python implementation by-value

[apmorton]

OS version: Ubuntu 22.04.3 LTS
Python3 version: Python 3.11.6 | packaged by conda-forge | (main, Oct 3 2023, 10:40:35) [GCC 12.3.0]

In [7]: dill.dumps(frozendict.frozendict(a=1))
Out[7]: b'\x80\x04\x95\xe6b\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x0c_create_type\x94\x93\x94(h\x00\x8c\n_load_type\x94\x93\x94\x8c\x04type\x94\x85\x94R\x94\x8c\nfrozendict\x94h\x04\x8c\x04dict\x94\x85\x94R\x94\x85\x94}\x94(\x8c\n__module__\x94\x8c\x0ffrozendict.core\x94\x8c\x07__doc__\x94\x8c\xac\n    A simple immutable dictionary.\n    \n    The API is the same as `dict`, without methods that can change the \n    immutability. In addition, it supports __hash__().\n    \x94\x8c\t__slots__\x94\x8c\x05_hash\x94\x85\x94\x8c\x08fromkeys\x94h\x04\x8c\x0bclassmethod\x94\x85\x94R\x94h\x00\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(C\x02\x02\x06\x94K\x01K\x00K\x00K\x03K\x07K\x0fC6\x97\x00\x02\x00|\x00t\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00j\x01\x00\x00\x00\x00\x00\x00\x00\x00|\x01i\x00|\x02\xa4\x01\x8e\x01\xa6\x01\x00\x00\xab\x01\x00\x00\x00\x00\x00\x00\x00\x00S\x00\x94\x8c/\n        Identical to dict.fromkeys().\n        \x94\x85\x94h\th\x15\x86\x94\x8c\x03cls\x94\x8c\x04args\x94\x8c\x06kwargs\x94\x87\x94\x8c-/home/amorton/aq/research3/frozendict/core.py\x94h\x15\x8c\x13frozendict.fromkeys\x94K\x16C#\x80 <truncated>

This is because dill is attempting to locate the module member and cannot, so it assumes it must also send the type itself over the wire.

Two possible solutions are:

• The same fix as proposed in [BUG] Python implementation pickles differently than native implementation #87
• Removing del core from __init__.py

Expected output (depending on solution):

In [3]: dill.dumps(frozendict.frozendict(a=1))
Out[3]: b'\x80\x04\x95\x1f\x00\x00\x00\x00\x00\x00\x00\x8c\nfrozendict\x94h\x00\x93\x94}\x94\x8c\x01a\x94K\x01s\x85\x94R\x94.'

or

In [3]: dill.dumps(frozendict.frozendict(a=1))
Out[3]: b'\x80\x04\x95/\x00\x00\x00\x00\x00\x00\x00\x8c\x0ffrozendict.core\x94\x8c\nfrozendict\x94\x93\x94}\x94\x8c\x01a\x94K\x01s\x85\x94R\x94.'

[Marco-Sulla]

Well, currently #87 is marked as wontfix.

The second solution is easy to do, but I don't like to pollute the namespace. Before considering this, I would know a couple of things:

First: you said

dill is attempting to locate the module member

why dill needs the submodule and pickle doesn't?

Secondly, frozendict is pickable. For what I know, dill is a good alternative to pickle for not pickable objects, but it's slower than pickle. Why use dill for frozendict?

[apmorton]

why dill needs the submodule and pickle doesn't?

Dill has some special logic for detecting cases where it should also serialize the type itself (like if you dill a class that is defined in a module named __main__)- and for whatever reason they implement that in a way that gets tripped up by the frozendict implementation details.

Why use dill for frozendict?

What if my frozendict contains things which cannot themselves be pickled, but dill handles them correctly?
Or if my frozendict is in an object that contains other non-picklealble object.

[Marco-Sulla]

What if my frozendict contains things which cannot themselves be pickled?

This is a really good point, but...

Dill has some special logic for detecting cases where it should also serialize the type itself (like if you dill a class that is defined in a module named __main__)- and for whatever reason they implement that in a way that gets tripped up by the frozendict implementation details.

I failed to understand your explaining, sorry. Why a class defined in a __main__.py should be a special case? Furthermore, why dill does not try to pickle before and then, if it fails, do its job?

[Marco-Sulla]

PS #87 reopened, so first solution could be valid (and preferred).

[Marco-Sulla]

#91 merged

[Marco-Sulla]

I want to say a big Thank you to you, @apmorton . You found very obscure bugs and you fixed them with care. Thank you for your time, your code and your patience.

[apmorton]

Thanks for maintaining this library - it has been very helpful!