error on numpy types due to ignored dill.settings in dump_session

[Phyks]

Hi,

When trying to dump_session in a Jupyter notebook, with numpy items, I get a huge traceback:

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-197-b3be130d6bea> in <module>()
----> 1 dill.dump_session()

/home/phyks/.local/share/virtualenvs/physique/lib/python3.5/site-packages/dill/dill.py in dump_session(filename, main, byref)
    331         pickler._recurse = False # disable pickling recursion for globals
    332         pickler._session = True  # is best indicator of when pickling a session
--> 333         pickler.dump(main)
    334     finally:
    335         f.close()

/usr/lib64/python3.5/pickle.py in dump(self, obj)
    406         if self.proto >= 4:
    407             self.framer.start_framing()
--> 408         self.save(obj)
    409         self.write(STOP)
    410         self.framer.end_framing()

/usr/lib64/python3.5/pickle.py in save(self, obj, save_persistent_id)
    473         f = self.dispatch.get(t)
    474         if f is not None:
--> 475             f(self, obj) # Call unbound method with explicit self
    476             return
    477 

/home/phyks/.local/share/virtualenvs/physique/lib/python3.5/site-packages/dill/dill.py in save_module(pickler, obj)
   1166                 + ["__builtins__", "__loader__"]]
   1167             pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
-> 1168                                 state=_main_dict)
   1169             log.info("# M1")
   1170         else:

/usr/lib64/python3.5/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    621 
    622         if state is not None:
--> 623             save(state)
    624             write(BUILD)
    625 

/usr/lib64/python3.5/pickle.py in save(self, obj, save_persistent_id)
    473         f = self.dispatch.get(t)
    474         if f is not None:
--> 475             f(self, obj) # Call unbound method with explicit self
    476             return
    477 

/home/phyks/.local/share/virtualenvs/physique/lib/python3.5/site-packages/dill/dill.py in save_module_dict(pickler, obj)
    833             # we only care about session the first pass thru
    834             pickler._session = False
--> 835         StockPickler.save_dict(pickler, obj)
    836         log.info("# D2")
    837     return

/usr/lib64/python3.5/pickle.py in save_dict(self, obj)
    808 
    809         self.memoize(obj)
--> 810         self._batch_setitems(obj.items())
    811 
    812     dispatch[dict] = save_dict

/usr/lib64/python3.5/pickle.py in _batch_setitems(self, items)
    834                 for k, v in tmp:
    835                     save(k)
--> 836                     save(v)
    837                 write(SETITEMS)
    838             elif n:

/usr/lib64/python3.5/pickle.py in save(self, obj, save_persistent_id)
    473         f = self.dispatch.get(t)
    474         if f is not None:
--> 475             f(self, obj) # Call unbound method with explicit self
    476             return
    477 

/home/phyks/.local/share/virtualenvs/physique/lib/python3.5/site-packages/dill/dill.py in save_function(pickler, obj)
    791                                 globs, obj.__name__,
    792                                 obj.__defaults__, obj.__closure__,
--> 793                                 obj.__dict__), obj=obj)
    794         else:
    795             pickler.save_reduce(_create_function, (obj.func_code,

/usr/lib64/python3.5/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    597         else:
    598             save(func)
--> 599             save(args)
    600             write(REDUCE)
    601 

/usr/lib64/python3.5/pickle.py in save(self, obj, save_persistent_id)
    473         f = self.dispatch.get(t)
    474         if f is not None:
--> 475             f(self, obj) # Call unbound method with explicit self
    476             return
    477 

/usr/lib64/python3.5/pickle.py in save_tuple(self, obj)
    738         write(MARK)
    739         for element in obj:
--> 740             save(element)
    741 
    742         if id(obj) in memo:

/usr/lib64/python3.5/pickle.py in save(self, obj, save_persistent_id)
    473         f = self.dispatch.get(t)
    474         if f is not None:
--> 475             f(self, obj) # Call unbound method with explicit self
    476             return
    477 

/home/phyks/.local/share/virtualenvs/physique/lib/python3.5/site-packages/dill/dill.py in save_module_dict(pickler, obj)
    833             # we only care about session the first pass thru
    834             pickler._session = False
--> 835         StockPickler.save_dict(pickler, obj)
    836         log.info("# D2")
    837     return

/usr/lib64/python3.5/pickle.py in save_dict(self, obj)
    808 
    809         self.memoize(obj)
--> 810         self._batch_setitems(obj.items())
    811 
    812     dispatch[dict] = save_dict

/usr/lib64/python3.5/pickle.py in _batch_setitems(self, items)
    834                 for k, v in tmp:
    835                     save(k)
--> 836                     save(v)
    837                 write(SETITEMS)
    838             elif n:

/usr/lib64/python3.5/pickle.py in save(self, obj, save_persistent_id)
    473         f = self.dispatch.get(t)
    474         if f is not None:
--> 475             f(self, obj) # Call unbound method with explicit self
    476             return
    477 

/usr/lib64/python3.5/pickle.py in save_tuple(self, obj)
    738         write(MARK)
    739         for element in obj:
--> 740             save(element)
    741 
    742         if id(obj) in memo:

/usr/lib64/python3.5/pickle.py in save(self, obj, save_persistent_id)
    473         f = self.dispatch.get(t)
    474         if f is not None:
--> 475             f(self, obj) # Call unbound method with explicit self
    476             return
    477 

/home/phyks/.local/share/virtualenvs/physique/lib/python3.5/site-packages/dill/dill.py in save_type(pickler, obj)
   1229        #print ("%s\n%s" % (type(obj), obj.__name__))
   1230        #print ("%s\n%s" % (obj.__bases__, obj.__dict__))
-> 1231         StockPickler.save_global(pickler, obj)
   1232         log.info("# T4")
   1233     return

/usr/lib64/python3.5/pickle.py in save_global(self, obj, name)
    914                 raise PicklingError(
    915                     "Can't pickle %r: it's not the same object as %s.%s" %
--> 916                     (obj, module_name, name))
    917 
    918         if self.proto >= 2:

PicklingError: Can't pickle <class 'numpy.uint64'>: it's not the same object as numpy.uint64

Found #79 but it should not be the same issue, as I am importing numpy as import numpy as np.

Thanks

[mmckerns]

Ok, I see the traceback, but can you post a minimal bit of code that can reproduce the traceback? That way, it can be debugged much easier if I can reproduce it for myself.

In the abstract, you might want to try to toggle some of the settings in dill.settings, particularly byref.

[Phyks]

Sure, had some troubles to narrow done the problem, but this one should reproduce the problem:

import dill
from sympy import Symbol, lambdify
x = Symbol("x")
H = lambdify(x, x, "numpy")
dill.dump_session()

Thanks

[mmckerns]

I'm fairly sure that this is due to the issues from #79 and #78. You can see that the error comes from the failed picking of the int64 object, I think for reasons as noted in the above-referenced tickets. Note that it's also been fixed if you change the settings to use 'recurse' mode.

>>> import dill
>>> from sympy import Symbol, lambdify
>>> x = Symbol("x")
>>> H = lambdify(x, x, "numpy")
>>> dill.detect.errors(H)
PicklingError("Can't pickle <type 'numpy.int64'>: it's not the same object as numpy.int64",)
>>> dill.detect.errors(x)
>>> dill.settings['recurse'] = True
>>> dill.detect.errors(H)
>>> _H = dill.loads(dill.dumps(H))
>>> _H
<function <lambda> at 0x110db9320>

Looking at the source code, dump_session has 'recurse' mode hard-wired to be False. I'm not sure why (I'll have to check), but it's done in such a way that it was obviously intentional, so I'm sure there was a good reason for it.

At this point, I'm not yet sure if this is a duplicate issue, or a new one. Depends on why dump_session has the settings hard-wired as they are.

In short, I'll need to look into it further. Let me know if you can confirm by changing settings, it works as expected in dump \ dumps, but not dump_session.

[Phyks]

@mmckerns I confirm it works using dum / dumps but not dump_session. Issue #79 was closed, so I guessed it was a new bug, as I am using an up-to-date pip version of dill.

I will follow this issue concerning dump_session.

[mmckerns]

@Phyks: we will use this ticket for extending #79 to dump_session.

[tmastny]

Any progress on extending recurse to dump_session? I still see the error with @Phyks's example, and I encountered a similar error in my own work. Thanks for all the work on dill!

[TUCMath]

I ran the code suggested above, but still get the same error as before. I also checked to have all my packages up-to-date. It seems lambdify(...,'numpy') cannot be dumped whatsoever.

[mmckerns]

@sajad-872797, @tmastny: I believe the settings['recurse'] mode has not been extended to dump_session yet.

[TUCMath]

@mmckerns I actually did dill.loads(dill.dumps(H)). I ran this:

import dill
from sympy import Symbol, lambdify
x = Symbol("x")
H = lambdify(x, x, "numpy")
dill.settings['recurse'] = True
_H = dill.loads(dill.dumps(H))

And I got this:

PicklingError: Can't pickle <class 'numpy.int32'>: it's not the same object as numpy.int32

[matthew-hsr]

Wondering if there's any updates on sajad's post? I got a similar error, which cannot be solved by

dill.settings['recurse']= True;

[mmckerns]

@matthew-hsr: the ability to pickle a lambidfy(..., 'numpy')... no, I don't think it has yet..