Pickle MagicMocks?

[max-sixty]

Is this a realistic goal?
Or given that a MagicMock tries to look like something else, is this impossible?

In [27]: dill.dumps(mock.MagicMock())
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-27-bf4237d3f9f2> in <module>()
----> 1 dill.dumps(mock.MagicMock())

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dill/dill.pyc in dumps(obj, protocol, byref)
    141     """pickle an object to a string"""
    142     file = StringIO()
--> 143     dump(obj, file, protocol, byref)
    144     return file.getvalue()
    145 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dill/dill.pyc in dump(obj, file, protocol, byref)
    134     _byref = pik._byref
    135     pik._byref = bool(byref)
--> 136     pik.dump(obj)
    137     pik._byref = _byref
    138     return

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in dump(self, obj)
    222         if self.proto >= 2:
    223             self.write(PROTO + chr(self.proto))
--> 224         self.save(obj)
    225         self.write(STOP)
    226 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    394                     "args[0] from __newobj__ args has the wrong class")
    395             args = args[1:]
--> 396             save(cls)
    397             save(args)
    398             write(NEWOBJ)

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dill/dill.pyc in save_type(pickler, obj)
    814        #print ("%s\n%s" % (type(obj), obj.__name__))
    815        #print ("%s\n%s" % (obj.__bases__, obj.__dict__))
--> 816         StockPickler.save_global(pickler, obj)
    817     return
    818 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in save_global(self, obj, name, pack)
    751                 raise PicklingError(
    752                     "Can't pickle %r: it's not the same object as %s.%s" %
--> 753                     (obj, module, name))
    754 
    755         if self.proto >= 2:

PicklingError: Can't pickle <class 'mock.MagicMock'>: it's not the same object as mock.MagicMock

[mmckerns]

It's reasonable to think you can do it. I don't know exactly what the mock.MagicMock class does to mock another class, but generally, it's possible to pickle a class except in certain cases. You might want to try changing one of the settings:

>>> import dill
>>> dill.settings
{'recurse': False, 'byref': False, 'protocol': 2, 'fmode': 0}

Setting 'recurse' to True will change how dill attempts to pickle the globals dict, and by setting 'byref' to True, it will pickle the class by reference instead of storing the class object itself. Trying again with combinations of the above may find success for you.

You can also turn on pickle 'tracing', to see what dill is doing. That often helps identity the "bad" objects within the object you want to serialize. "F1: …" and similar, is an object starting the process of being dumped. The corresponding #F1 is the object finishing the dump. You can see dill pickles recursively. This trace will change depending on the settings you use (see above).

>>> dill.detect.trace(True)
>>> dill.dumps(lambda x:x)
F1: <function <lambda> at 0x1036c1668>
F2: <function _create_function at 0x10359f230>
# F2
Co: <code object <lambda> at 0x103345630, file "<stdin>", line 1>
T1: <type 'code'>
F2: <function _load_type at 0x10359f140>
# F2
# T1
# Co
D1: <dict object at 0x103080168>
# D1
D2: <dict object at 0x10367e168>
# D2
# F1
'\x80\x02cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01U\x08CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x01K\x01KCU\x04|\x00\x00Sq\x05N\x85q\x06)U\x01xq\x07\x85q\x08U\x07<stdin>q\tU\x08<lambda>q\nK\x01U\x00q\x0b))tq\x0cRq\rc__builtin__\n__main__\nh\nNN}q\x0etq\x0fRq\x10.'

Lastly, there are some decent diagnostic tools in dill.detect. I use them when the trace is not sufficient to understand what's going on. Feel free to post the "pickle trace" here.

[mmckerns]

no activity. reopen ticket if there's more to add

[stdavis]

I'm having this same issue. I tried changing the settings suggested by @mmckerns but have no luck. Here's a sample:

Python 3.5.3 |Continuum Analytics, Inc.| (default, May 15 2017, 10:43:23) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.settings
{'protocol': 3, 'byref': False, 'recurse': False, 'fmode': 0}
>>> dill.detect.trace(True)
>>> from mock import Mock
>>> dill.dumps(Mock)
T4: <class 'mock.mock.Mock'>
# T4
b'\x80\x03cmock.mock\nMock\nq\x00.'
>>> import os
>>> os.path = Mock()
>>> os.path()
<Mock name='mock()' id='5139880704'>
>>> dill.dumps(os.path)
T4: <class 'mock.mock.Mock'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 281, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 274, in dump
    pik.dump(obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 408, in dump
    self.save(obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 598, in save_reduce
    save(cls)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 1299, in save_type
    StockPickler.save_global(pickler, obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 920, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'mock.mock.Mock'>: it's not the same object as mock.mock.Mock
>>> dill.settings['recursive'] = True
>>> dill.dumps(os.path)
T4: <class 'mock.mock.Mock'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 281, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 274, in dump
    pik.dump(obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 408, in dump
    self.save(obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 598, in save_reduce
    save(cls)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 1299, in save_type
    StockPickler.save_global(pickler, obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 920, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'mock.mock.Mock'>: it's not the same object as mock.mock.Mock
>>> dill.settings['byref'] = True
>>> dill.dumps(os.path)
T4: <class 'mock.mock.Mock'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 281, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 274, in dump
    pik.dump(obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 408, in dump
    self.save(obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 598, in save_reduce
    save(cls)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\dill\dill.py", line 1299, in save_type
    StockPickler.save_global(pickler, obj)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\pickle.py", line 920, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'mock.mock.Mock'>: it's not the same object as mock.mock.Mock
>>>

@MaximilianR Did you ever figure out a solution?

[max-sixty]

@stdavis no

The basic problem is that Mocks try and look like another class, and so when you need to know their class (i.e. when pickling them), you don't get the correct result.

Would be interested if anyone has any creative approaches. For the moment we run a separate set of tests on concrete objects to ensure they pickle correctly

[mmckerns]

I'm going to agree with @MaximilianR. If that's the case, then ultimately this is the same issue as #56. Basically, the derived class needs to have a way of informing the pickler what it's actual fully qualified name is instead of what is automatically generated -- using currying and many other factory methods break name-spacing for serializers in general. it's a long-standing issue.

[stdavis]

OK. Thanks for the info.