Support pickling slots in subclasses of common classes

[serhiy-storchaka]

BPO	26579
Nosy	@rhettinger, @pitrou, @vstinner, @avassalotti, @bitdancer, @berkerpeksag, @serhiy-storchaka
PRs	bpo-26579: Add object.__getstate__(). #2821
Files	copyreg_getstate.patch copyreg_getstate2.patch object_getstate.patch: Add object.getstate

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-04-06.17:02:02.310>
created_at = <Date 2016-03-17.08:43:06.421>
labels = ['extension-modules', 'type-feature', 'library', '3.11']
title = 'Support pickling slots in subclasses of common classes'
updated_at = <Date 2022-04-07.14:57:28.953>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2022-04-07.14:57:28.953>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-04-06.17:02:02.310>
closer = 'serhiy.storchaka'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2016-03-17.08:43:06.421>
creator = 'serhiy.storchaka'
dependencies = []
files = ['42183', '43462', '43716']
hgrepos = []
issue_num = 26579
keywords = ['patch']
message_count = 5.0
messages = ['261903', '268830', '270403', '416888', '416931']
nosy_count = 7.0
nosy_names = ['rhettinger', 'pitrou', 'vstinner', 'alexandre.vassalotti', 'r.david.murray', 'berker.peksag', 'serhiy.storchaka']
pr_nums = ['2821']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26579'
versions = ['Python 3.11']

Linked PRs

• gh-70766: Mention the object getstate caveat in 3.11 What's new. #108379
• [3.12] gh-70766: Mention the object getstate caveat in 3.11 What's new. (GH-108379) #108384
• [3.11] gh-70766: Mention the object getstate caveat in 3.11 What's new. (GH-108379) #108385

[serhiy-storchaka]

Pickling and copying instances of subclasses of some basic classes pickles and copies instance attributes. Example:

>>> class BA(bytearray):
...     pass
... 
>>> b = BA(b'abc')
>>> b.x = 10
>>> c = copy.copy(b)
>>> c.x
10
>>> c = pickle.loads(pickle.dumps(b))
>>> c.x
10

But this doesn't work if attributes are saved not in instance dictionary, but in slots.

>>> class BA(bytearray):
...     __slots__ = ('x',)
... 
>>> b = BA(b'abc')
>>> b.x = 10
>>> c = copy.copy(b)
>>> c.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: x
>>> c = pickle.loads(pickle.dumps(b))
>>> c.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: x

Since using __slots__ is implementation detail, this failure can be considered as a bug.

Proposed patch adds support of pickling and copying slots in subclasses of all classes that already support pickling and copying non-slot attributes. It is backward compatible, classes with slots can be unpickled on older Python versions without slots. Affected classes: bytearray, set, frozenset, weakref.WeakSet, collections.OrderedDict, collections.deque, datetime.tzinfo.

The patch adds the copyreg._getstate() function for Python classes and exposes the _PyObject_GetState() function for extension classes. An alternative (and simpler for end user) solution would be to add default implementation as object.__getstate__(). But this is not easy to reject non-pickleable classes (bpo-22995) in this case, since __getstate__ is looked up as instance attribute, not as other special methods.

[serhiy-storchaka]

Synchronized with current sources.

[serhiy-storchaka]

An alternative way is to expose a default state as object.__getstate__(). It is more efficient since it is implemented in C. Following patch implements this approach.

[serhiy-storchaka]

New changeset 884eba3 by Serhiy Storchaka in branch 'main':
bpo-26579: Add object.__getstate__(). (GH-2821)
884eba3

[vstinner]

bpo-26579: Add object.__getstate__(). (GH-2821)
884eba3

This change introduced reference leaks: see bpo-47250.

[gpshead]

This caused object.__getstate__ to exist where it hadn't in the past, causing trouble for CLIF generated extension modules.
See https://github.com/google/clif/pull/64/files#diff-de65cd2a0ef58411075450cf769b38dd6d53803108d536309289cda5810a379aR23 for an example of what we're unfortunately likely to have to do with CLIF for its generated extension modules to not be broken by this behavior change.

It'd be nice if the 3.11 What's New documentation covered this API change. https://docs.python.org/3/whatsnew/3.11.html#other-language-changes only extolls the change's virtues and does not mention anything about the behavior change consequences and workarounds at the Python and C API levels that people may need to adopt to retain compatibility.

[serhiy-storchaka]

I don't know what can be suggested as a general workaround. Each problem requires an individual solution, depending on why that code was broken by that change.

[lwinkler]

This also creates a bug in pyyaml and ruamel.yaml. It seems that for an empty class, the new getstate method returns None. This causes the serialization to crash.

@serhiy-storchaka : Is there a good reason why getstate returns None for an empty class instead of an empty directory ?

Minimal code:

class Empty:
    pass

empty = Empty()
print("getstate", empty.__getstate__())

prints: getstate None

[serhiy-storchaka]

@serhiy-storchaka : Is there a good reason why getstate returns None for an empty class instead of an empty directory ?

Because pickle.dumps() creates more compact pickle if __getstate__() returns None. Also, it saves time in __getstate__() for creating an empty dict and saves time in pickle.loads() for calling __setstate__().

Add def __getstate__(self): return {} to an empty class and compare the result of pickle.dumps().

[nitzmahone]

Accounting for this in PyYAML shouldn't be a big deal- IIUC None has always been a documented valid response from __getstate__() anyway, so we need to be able to handle and preserve that properly.