_DeadlockError on import of django.test.signals

[caseybrichardson]

Hey y'all, I've been intermittently receiving an error that looks like this:

  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File ".../venv/lib/python3.7/site-packages/rest_framework/templatetags/rest_framework.py", line 12, in <module>
    from rest_framework.renderers import HTMLFormRenderer
  File ".../venv/lib/python3.7/site-packages/rest_framework/renderers.py", line 19, in <module>
    from django.test.client import encode_multipart
  File ".../venv/lib/python3.7/site-packages/django/test/client.py", line 23, in <module>
    from django.test import signals
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 149, in __enter__
  File "<frozen importlib._bootstrap>", line 94, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('django.test.signals') at 4512885264

(Ignore the ... just removing user paths for privacy)

I've tried for awhile to build a test case to make this easier to track down but was unsuccessful. I was able though to successfully make this go away. Here it seems that the django.test.client.encode_multipart function is being used within renderers.MultiPartRenderer.

Now re-using code seems fine to me, but I'm a bit hesitant to say that using code designed purely for testing purposes is something that should be done within production code. I've confirmed that copying that function and moving it elsewhere fixes this issue for me (and I've found that I'm not necessarily the only one suffering from this issue as seen here.

Looking at your contribution guide I know it says to talk about the expected behavior rather than code, but it seemed unavoidable to me to make this about the code causing the problem. Would anyone have any suggestions for getting this fixed seeing as it seems kind of tough to reproduce? As an outsider of this project I'd say though that not relying on a function designed for use in testing would be optimal.

I am running django-channels 2.2.0 with Django 2.2.5 at the moment if that helps narrow it down. I mention this because django-channels does replace the default runserver command with its own.

[xordoquy]

Hi,

Thanks for your detailed report.
It looks a lot like https://code.djangoproject.com/ticket/30921 or https://code.djangoproject.com/ticket/30352 which seem to be an issue Django related.

[caseybrichardson]

Hi, I know I didn't follow the contribution guide to a T, but would it be possible to have more of a discussion around this? Even if it is an issue with Django, wouldn't mitigating the problem for users of DRF be worth looking into? If this simply isn't possible I'll drop the issue. I'm only concerned for the experience of users of DRF as the error this problem induces is kind of hard to track down and can be frustrating when debugging as you have to keep restarting the debug server.

The recommended fix I saw in those threads of removing .pyc files doesn't even work for me. The only fix that works (at least for me through empirical testing) is removing the dependency on django.test.client.encode_multipart from the renderers.py file and importing the function from somewhere else (in my testing I simply in-lined the function inside the renderers.py file).

[xordoquy]

What if that import was moved inside this function ?

Does the issue still happen after that ?

[caseybrichardson]

Moving the import inside the function that it's used in does seem to fix the issue as well.

[rpkilby]

Hi, I know I didn't follow the contribution guide to a T

Just a note - I assume Tom closed the issue assuming it's an upstream Django bug. Bug reports aren't closed just because they don't precisely follow the contribution guide.

If you can't provide a test case, can you provide an example failing project?

[caseybrichardson]

I can't provide the exact repository, but I should have some time in the next few days to try and assemble a minimal one. I'll update this issue once I've done so. I figured the fact that it's probably a Django bug prompted the closure which I understand.

[xordoquy]

Meanwhile, we may mitigate the issue by moving the import. Sounds reasonable to me to accept a PR about it

[tomchristie]

I closed the issue because it seemed unrelated to REST framework itself, particularly given that there were referenced Django tickets.

Why is there a deadlock error with making an import from Django? That seems 😬.

[xordoquy]

@tomchristie I agree, but is it sensible enough to move the import so most use cases will presumably work.

[tomchristie]

Indeed.

[tomchristie]

Closed in #7083