Pickled Functions seem to be incompatible between python3.5 and python3.6+ even when using pickle.DEFAULT_PROTOCOL

[Ark-kun]

Even an empty function has different 3-byte code representation in python3.5 and python3.6+.
This causes either "SystemError: unknown opcode" when going up or "Segmentation fault" when going down.

I understand that the support for this is not implied, but I think the incompatibility should be stated explicitly. "cloudpickle uses pickle.HIGHEST_PROTOCOL by default: it is meant to send objects between processes running the same version of Python." is not explicit here since it only talks about "cloudpickle useing pickle.HIGHEST_PROTOCOL by default". Which implies that using the more compatible protocol allows passing objects between different python versions.

Summary of my results:

testing python3.5 -> python3.6, 3.7
XXX lineno: 10, opcode: 0
Traceback (most recent call last):
  File "<string>", line 9, in <module>
  File "<string>", line 10, in func
SystemError: unknown opcode

testing python3.6,3.7 -> python3.5
Segmentation fault

PoC demonstrating code difference:

def func():
    pass
print(func.__code__.co_code)
'

On python3.5:
b'd\x00\x00S'

On python3.6:
b'd\x00S\x00'

Pickle structure diff: https://www.diffchecker.com/aC5PuVHy

Testing code:

function test_compatibility() {
v1=$1
v2=$2

echo testing "$v1 -> $v2"

"$v1" -c '
import pickle
import cloudpickle
import sys

def func():# -> str:
    pass

pickled_func = cloudpickle.dumps(func, pickle.DEFAULT_PROTOCOL)
sys.stdout.buffer.write(pickled_func)
' | "$v2" -c '
import pickle
import sys

pickled_func = sys.stdin.buffer.read()

func = pickle.loads(pickled_func)

func()
'

}


test_compatibility python3.5 python3.5
test_compatibility python3.5 python3.6
test_compatibility python3.5 python3.7

test_compatibility python3.6 python3.5
test_compatibility python3.6 python3.6
test_compatibility python3.6 python3.7

test_compatibility python3.7 python3.5
test_compatibility python3.7 python3.6
test_compatibility python3.7 python3.7

[hongye-sun]

Have you tried changing the protocol to lower version like 0?

[llllllllll]

This isn't going to be solved by changing the protocol version. To serialize arbitrary lambdas we need to be able to send the bytecode from one machine to another. Python bytecode can change pretty radically between minor versions, and instructions could be added or removed with no clear translation making it impossible to convert between versions.

I think the best we can do is strengthen the claim in the readme/docs to say, "Cloudpickle can only be used to send objects between the exact same version of Python."

[ogrisel]

Done in #294. I also added a security notice. Let me know what you think.

[Ark-kun]

Let me know what you think.

Sounds good.

[Ark-kun]

Python bytecode can change pretty radically between minor versions, and instructions could be added or removed with no clear translation making it impossible to convert between versions.

Is there any place I can read more about those breaking changes?

Do you know some other python code format (e.g. pre-compiled code) that's more stable? Something similar to .Net's CIL which has pretty good backwards compatibility.

[ogrisel]

Is there any place I can read more about those breaking changes?

The generated byte code is an implementation detail of CPython. It's not part of the public API in anyway as far as I know. It's possible to grep "bytecode" in the changelog though: https://docs.python.org/3/whatsnew/changelog.html

Do you know some other python code format (e.g. pre-compiled code) that's more stable? Something similar to .Net's CIL which has pretty good backwards compatibility.

Python is not compiled so there is nothing. Saving the source code would be an optional but that would render cloudpickle much more brittle and less efficient IMO.

[AlJohri]

is it possible to store the python version with the pickled file and check it at load time? I was hoping to implement this application side but it would necessitate a metadata file which I was hoping to avoid