cPickleCache will not build on Python 3.9 without trace refs

[tschorr]

cPickleCache is relying on

persistent/persistent/cPickleCache.c

Line 664 in d4f1204

_Py_ForgetReference(v);

which is no longer available on Python 3.9 without Py_TRACE_REFS being defined, s. python/cpython#18332

As a result, cPickleCache can't be built + imported:

from persistent import cPickleCache
*** ImportError: /home/thomas/devel/plone/persistent/.tox/py39/lib/python3.9/site-packages/persistent/cPickleCache.cpython-39-x86_64-linux-gnu.so: undefined symbol: _Py_ForgetReference

Tests silently work around this.

It is still possible to import cPersistence though, so code in the package that does import persistence and then uses persistent.Persistent, e.g.

persistent/persistent/mapping.py

Line 33 in d4f1204

class PersistentMapping(IterableUserDict, persistent.Persistent):

will be fine. But client code that uses the items defined in init.py:

persistent/persistent/__init__.py

Line 40 in d4f1204

try:

will only ever see the Python implementation of both Persistent and the pickle cache.

-> This is causing trouble in the Persistence package, ticket coming there.

[PythonLinks]

I wonder if we could use the ArrangoDB serialization protocol, VelocityPack, instead of Python Pickle.
https://github.com/arangodb/velocypack

covers all of JSON plus dates, integers, binary data and arbitrary precision numbers
All this gives us the possibility to use the same byte sequence of data for transport, storage and (read-only) work. Using a single data format not only eliminates a lot of conversions but ?can also reduce runtime memory usage, as data does only need a single in-memory representation.

The other popular formats we looked at have all some deficiency with respect to the above list.

After reading more about it, it sounds more interesting than I initially thought.
What problems am I missing? I am curious what other people think of this idea?

[icemac]

@tschorr Thank you for your analysis and @jamadden thank you for fixing this issue.