Pickling instances of nested classes fails in `unittest.TestCase` for Python 2.7 and 3.4 #262

aleneum · 2018-04-13T16:59:27Z

The code below throws a

_pickle.PicklingError: Can't pickle <class 'test_test.TestDill.DummyModel'>: it's not found as test_test.DummyModel

for Python 2.7.14_3 und 3.4.8 but not for 3.5.5 and 3.6.4 (3.3.x untested).
Interestingly, the error appears only in unit tests executed with nostetests. When DillTest is derived from object and setUp and test_pickle_model are executed manually, there is no issue.

dill version is 0.2.7.1 installed via pip.

from unittest import TestCase
import dill


class Machine(object):

    def __init__(self, model):
        self.models = [model]


class Model(object):

    def __init__(self, model):
        self.machine = Machine(model)


class TestDill(TestCase):

    class DummyModel(object):
        pass

    def setUp(self):
        self.stuff = Model(self.DummyModel())

    def test_pickle_model(self):
        dump = dill.dumps(self.stuff)
        model2 = dill.loads(dump)

trace

-------------------- >> begin captured logging << --------------------
dill: INFO: T4: <class 'test_test.Model'>
dill: INFO: # T4
dill: INFO: D2: <dict object at 0x101bafc58>
dill: INFO: T4: <class 'test_test.Machine'>
dill: INFO: # T4
dill: INFO: D2: <dict object at 0x101bafb40>
dill: INFO: T4: <class 'test_test.DummyModel'>
--------------------- >> end captured logging << ---------------------

Update 16.04.2018: Simplified a bit.

The text was updated successfully, but these errors were encountered:

blazs · 2018-04-16T12:41:49Z

I have a similar issue. I am consider using dill specifically to avoid problems with nested classes.

import dill


class Type(enum.Enum):
    class Subtype(enum.Enum):
        D = 4
        E = 54
        F = 4
    A = 0
    B = 1
    C = 3


class SomeClass:
    def __init__(self):
        self.type = Type.Subtype.value.E


sc = SomeClass()

with open('scsc', 'wb') as f:
    dill.dump(sc, f)

kurt-rhee · 2021-10-21T19:58:02Z

dill.dumps on the sc SomeClass object from the above comment gives me the following traceback

Traceback (most recent call last):
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
dill.dumps(sc)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 304, in dumps
dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 276, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 498, in dump
StockPickler.dump(self, obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 487, in dump
self.save(obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 687, in save_reduce
save(cls)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 1422, in save_type
pickler.save_reduce(_create_type, (type(obj), obj.name,
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 692, in save_reduce
save(args)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 901, in save_tuple
save(element)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 990, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 1493, in save_function
pickler.save_reduce(_create_function, (obj.code,
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 692, in save_reduce
save(args)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 901, in save_tuple
save(element)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 990, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 1176, in save_instancemethod0
pickler.save_reduce(MethodType, (obj.func, obj.self), obj=obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 692, in save_reduce
save(args)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 886, in save_tuple
save(element)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 717, in save_reduce
save(state)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 990, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 990, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 717, in save_reduce
save(state)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 990, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/site-packages/dill/_dill.py", line 990, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/home/skrhee/miniconda3/envs/hdmodeler/lib/python3.8/pickle.py", line 578, in save
rv = reduce(self.proto)
TypeError: cannot pickle 'sqlite3.Connection' object

mmckerns · 2021-10-21T22:39:20Z

@kurt-rhee: please see my comment in #435 with regard to your traceback

mmckerns · 2021-10-21T22:47:12Z

@aleneum: Not sure why this issue was ignored... but I am seeing the same behavior for nosetests on python2.7 (which will see EOL support this year), and not when run manually. Python3.4 is no longer supported.

mmckerns · 2021-10-21T22:50:52Z

@blazs: Your code seems to be falling victim to issue #250, due to the Enum objects. I have a 95% of a fix for that issue, and hopefully it will get merged soon.

aleneum · 2021-10-25T13:45:31Z

@aleneum: Not sure why this issue was ignored... but I am seeing the same behavior for nosetests on python2.7 (which will see EOL support this year), and not when run manually. Python3.4 is no longer supported.

No worries, I dealt with it quite a while ago. This issue can be closed if there is no objection.

mmckerns · 2021-10-25T21:32:10Z

closing. Other comments posted in this issue should be resolved in the other above-noted issues.

aleneum mentioned this issue Apr 16, 2018

Not able to pickle GraphMachine pytransitions/transitions#292

Closed

mmckerns added the wontfix label Oct 25, 2021

mmckerns closed this as completed Oct 25, 2021

mmckerns added this to the dill-0.3.5 milestone Apr 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pickling instances of nested classes fails in `unittest.TestCase` for Python 2.7 and 3.4 #262

Pickling instances of nested classes fails in `unittest.TestCase` for Python 2.7 and 3.4 #262

aleneum commented Apr 13, 2018 •

edited

Loading

blazs commented Apr 16, 2018

kurt-rhee commented Oct 21, 2021

mmckerns commented Oct 21, 2021

mmckerns commented Oct 21, 2021 •

edited

Loading

mmckerns commented Oct 21, 2021

aleneum commented Oct 25, 2021

mmckerns commented Oct 25, 2021

Pickling instances of nested classes fails in unittest.TestCase for Python 2.7 and 3.4 #262

Pickling instances of nested classes fails in unittest.TestCase for Python 2.7 and 3.4 #262

Comments

aleneum commented Apr 13, 2018 • edited Loading

blazs commented Apr 16, 2018

kurt-rhee commented Oct 21, 2021

mmckerns commented Oct 21, 2021

mmckerns commented Oct 21, 2021 • edited Loading

mmckerns commented Oct 21, 2021

aleneum commented Oct 25, 2021

mmckerns commented Oct 25, 2021

Pickling instances of nested classes fails in `unittest.TestCase` for Python 2.7 and 3.4 #262

Pickling instances of nested classes fails in `unittest.TestCase` for Python 2.7 and 3.4 #262

aleneum commented Apr 13, 2018 •

edited

Loading

mmckerns commented Oct 21, 2021 •

edited

Loading