Exceptions (ignored) in finalizers running during interpreter shutdown

[projectgus]

Hi!

I think under some circumstances weakref callbacks from blinker 1.8.2 signals raise exceptions during interpreter shutdown, leading to "Exception ignored" errors right on exit. For example:

Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x72f2bde718a0>
Traceback (most recent call last):
  File "/usr/lib/python3.12/weakref.py", line 57, in _cb
AttributeError: 'NoneType' object has no attribute '_alive'

Context

I'm sorry that I don't have a reproducer for this. I've tried, but as it seems to depend on order of finalizers during shutdown I can't seem to figure out a sequence that makes it trigger. I see it building my blog as a static site using the Pelican static site generator, and it's been reported as a bug against the Pelican sitemap plugin here: pelican-plugins/sitemap#35

Running pelican under pdb I can see that the only weakrefs created are from blinker. The sitemap plugin connects up three signals, everything works as expected, and then three ignored exceptions are logged when exiting pdm (as this is when the Python interpreter exits):

Click for pdm session log breaking on all new weakrefs and showing the errors

(Pdb) b weakref.py:46
Breakpoint 1 at /usr/lib/python3.12/weakref.py:46
(Pdb) b weakref.py:53
Breakpoint 2 at /usr/lib/python3.12/weakref.py:53
(Pdb) r
> /usr/lib/python3.12/weakref.py(53)__new__()
-> def _cb(arg):
(Pdb) bt
  /usr/lib/python3.12/bdb.py(606)run()
-> exec(cmd, globals, locals)
  <string>(1)<module>()
  /home/gus/dev/projectgus/site/.venv/bin/pelican(8)<module>()
-> sys.exit(main())
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(648)main()
-> pelican, settings = get_instance(args)
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(563)get_instance()
-> return cls(settings), settings
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(65)__init__()
-> self.init_plugins()
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(79)init_plugins()
-> plugin.register()
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/plugins/sitemap/sitemap.py(226)register()
-> signals.get_generators.connect(generator.init)
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/blinker/base.py(114)connect()
-> self.receivers[receiver_id] = make_ref(
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/blinker/_utilities.py(62)make_ref()
-> return WeakMethod(obj, callback)  # type: ignore[arg-type, return-value]
> /usr/lib/python3.12/weakref.py(53)__new__()
-> def _cb(arg):
(Pdb) x
*** NameError: name 'x' is not defined
(Pdb) c
> /usr/lib/python3.12/weakref.py(53)__new__()
-> def _cb(arg):
(Pdb) bt
  /usr/lib/python3.12/bdb.py(606)run()
-> exec(cmd, globals, locals)
  <string>(1)<module>()
  /home/gus/dev/projectgus/site/.venv/bin/pelican(8)<module>()
-> sys.exit(main())
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(648)main()
-> pelican, settings = get_instance(args)
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(563)get_instance()
-> return cls(settings), settings
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(65)__init__()
-> self.init_plugins()
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(79)init_plugins()
-> plugin.register()
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/plugins/sitemap/sitemap.py(227)register()
-> signals.content_written.connect(generator.queue_page)
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/blinker/base.py(114)connect()
-> self.receivers[receiver_id] = make_ref(
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/blinker/_utilities.py(62)make_ref()
-> return WeakMethod(obj, callback)  # type: ignore[arg-type, return-value]
> /usr/lib/python3.12/weakref.py(53)__new__()
-> def _cb(arg):
(Pdb) c
> /usr/lib/python3.12/weakref.py(53)__new__()
-> def _cb(arg):
(Pdb) bt
  /usr/lib/python3.12/bdb.py(606)run()
-> exec(cmd, globals, locals)
  <string>(1)<module>()
  /home/gus/dev/projectgus/site/.venv/bin/pelican(8)<module>()
-> sys.exit(main())
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(648)main()
-> pelican, settings = get_instance(args)
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(563)get_instance()
-> return cls(settings), settings
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(65)__init__()
-> self.init_plugins()
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/__init__.py(79)init_plugins()
-> plugin.register()
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/pelican/plugins/sitemap/sitemap.py(228)register()
-> signals.finalized.connect(generator.finalize)
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/blinker/base.py(114)connect()
-> self.receivers[receiver_id] = make_ref(
  /home/gus/dev/projectgus/site/.venv/lib/python3.12/site-packages/blinker/_utilities.py(62)make_ref()
-> return WeakMethod(obj, callback)  # type: ignore[arg-type, return-value]
> /usr/lib/python3.12/weakref.py(53)__new__()
-> def _cb(arg):
(Pdb) c
Done: Processed 68 articles, 1 draft, 0 hidden articles, 1 page, 1 hidden page and 0 draft
pages in 12.51 seconds.
The program exited via sys.exit(). Exit status: 
> /home/gus/dev/projectgus/site/.venv/bin/pelican(3)<module>()
-> import re
(Pdb) q
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x770cca40f1a0>
Traceback (most recent call last):
  File "/usr/lib/python3.12/weakref.py", line 57, in _cb
AttributeError: 'NoneType' object has no attribute '_alive'
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x770cca40f600>
Traceback (most recent call last):
  File "/usr/lib/python3.12/weakref.py", line 57, in _cb
AttributeError: 'NoneType' object has no attribute '_alive'
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x770cca40fa60>
Traceback (most recent call last):
  File "/usr/lib/python3.12/weakref.py", line 57, in _cb
AttributeError: 'NoneType' object has no attribute '_alive'

Possible fix

• Rolling back to blinker 1.7.0, the problem no longer appears.
• If I apply the following patch, adapted from Don't try to cleanup CallbackRegistry during interpreter shutdown. matplotlib/matplotlib#14843, then the problem no longer appears:

iff --git i/src/blinker/base.py w/src/blinker/base.py
index 728063b..b3420b3 100644
--- i/src/blinker/base.py
+++ w/src/blinker/base.py
@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import collections.abc as c
+import sys
 import typing as t
 import weakref
 from collections import defaultdict
@@ -403,7 +404,14 @@ class Signal:
         receiver when it is garbage collected.
         """
 
+        # Keep a reference to sys.is_finalizing, as sys may have been cleared out
+        # at that point.
+        _is_finalizing=sys.is_finalizing
+
         def cleanup(ref: weakref.ref[c.Callable[..., t.Any]]) -> None:
+            if _is_finalizing():
+                # Weakrefs can't be properly torn down at that point anymore.
+                return
             self._disconnect(receiver_id, ANY_ID)
 
         return cleanup

Happy to submit the patch as a PR if that's helpful for you, but I'm not certain it's the right fix.

Thanks for your time, please let me know if I can provide any other information.

Python version: 3.12
Blinker version: 1.8.2

[projectgus]

 # Keep a reference to sys.is_finalizing, as sys may have been cleared out
+        # at that point.
+        _is_finalizing=sys.is_finalizing

FWIW I didn't confirm this is necessary, it's copied from the linked PR. It seems like since Python 3.4 and PEP-442 global variables aren't set to None during shutdown any more, so the module may still be accessible the normal way.

[davidism]

This explanation and fix makes sense, can you submit a PR?

I didn't confirm this is necessary

Can you try leaving it out of your patch and running your build a bunch to see if you still get the error? I'd prefer not including patterns that are no longer needed.

[projectgus]

This explanation and fix makes sense, can you submit a PR?

Done!

Can you try leaving it out of your patch and running your build a bunch to see if you still get the error? I'd prefer not including patterns that are no longer needed.

Good idea. It seems not necessary here (whereas without the check I see the "Exception ignored" messages 100% of the time.)