Maximum recursion with dictionary and pickle

[tclick]

While attempting to straddle Py2/3, I wanted to save a dictionary using pickle, but I receive an error when I try to load the pickled data. The error occurs if I dump in Py2 but load in Py3. I am using Anaconda Python 2.7.13 and Python 3.6.1 on macOS Sierra 10.12.5 with six 1.10.0 and future 0.16.0.

from __future__ import (absolute_import, division, print_function, unicode_literals)
from future.builtins import *
from six.moves import cPickle

d = dict(hello=1, goodbye=2)
with open('test.db', 'wb') as dbf:
    cPickle.dump(d, db, protocol=cPickle.HIGHEST_PROTOCOL)

with open('test.db', 'rb') as dbf:
    d = cPickle.load(db)
d.keys()
---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
/opt/homebrew/anaconda3/envs/mda-py3/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj)
    691                 type_pprinters=self.type_printers,
    692                 deferred_pprinters=self.deferred_printers)
--> 693             printer.pretty(obj)
    694             printer.flush()
    695             return stream.getvalue()

/opt/homebrew/anaconda3/envs/mda-py3/lib/python3.6/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    363                 if cls in self.type_pprinters:
    364                     # printer registered in self.type_pprinters
--> 365                     return self.type_pprinters[cls](obj, self, cycle)
    366                 else:
    367                     # deferred printer

/opt/homebrew/anaconda3/envs/mda-py3/lib/python3.6/site-packages/IPython/lib/pretty.py in inner(obj, p, cycle)
    600         step = len(start)
    601         p.begin_group(step, start)
--> 602         keys = obj.keys()
    603         # if dict isn't large enough to be truncated, sort keys before displaying
    604         if not (p.max_seq_length and len(obj) >= p.max_seq_length):

/opt/homebrew/anaconda3/envs/mda-py3/lib/python3.6/site-packages/future/types/newdict.py in keys(self)
     65             return self.iterkeys()
     66         elif ver >= (3, 0):
---> 67             return self.keys()
     68
     69     def values(self):

... last 1 frames repeated, from the frame below ...

/opt/homebrew/anaconda3/envs/mda-py3/lib/python3.6/site-packages/future/types/newdict.py in keys(self)
     65             return self.iterkeys()
     66         elif ver >= (3, 0):
---> 67             return self.keys()
     68
     69     def values(self):

RecursionError: maximum recursion depth exceeded in comparison

[imankulov]

I came across this issue and experimented a bit.

The problematic code is in the newdict's implementation of items(), keys() and values() that calls itself recursively in Python 3: ref.

It should rather be:

        elif ver >= (3, 0):
            return super(newdict, self).keys()

I guess that people don't come across that issue very often, because nobody creates newdict objects explicitly. When you unpickle an object from Python 2, you make the unpickler machine construct exactly what's written in the pickle definition.

[imankulov]

It looks like the PR #294 fixes the issue.

$ git checkout master
Already on 'master'
Your branch is up to date with 'origin/master'.

$ python
Python 3.8.0 (default, Nov  1 2019, 14:31:53)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
>>> from future.types.newdict import newdict
>>> newdict(a=1, b=2).keys()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "python-future/src/future/types/newdict.py", line 67, in keys
    return self.keys()
  File "python-future/src/future/types/newdict.py", line 67, in keys
    return self.keys()
  File "python-future/src/future/types/newdict.py", line 67, in keys
    return self.keys()
  [Previous line repeated 994 more times]
  File "python-future/src/future/types/newdict.py", line 62, in keys
    if ver == (2, 7):
RecursionError: maximum recursion depth exceeded in comparison

$ gh pr checkout 294
Switched to branch 'newdict-fix'

$ python
Python 3.8.0 (default, Nov  1 2019, 14:31:53)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
>>> from future.types.newdict import newdict
>>> newdict(a=1, b=2).keys()
dict_keys(['a', 'b'])