After update to 20.3 can't use pycrypto

[namevic]

I'm using docker image with python tag: python:3.7-alpine.
This image received an update of pip from 20.2.4 to 20.3.

After geting the new image i see errors in my logs

  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 422, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 744, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 782, in save_base
    force_update, using, update_fields,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 911, in _do_insert
    using=using, raw=raw)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1376, in execute_sql
    for sql, params in self.as_sql():
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1320, in as_sql
    for obj in self.query.objs
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1320, in <listcomp>
    for obj in self.query.objs
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1319, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1260, in prepare_value
    value = field.get_db_prep_save(value, connection=self.connection)
  File "/usr/local/lib/python3.7/site-packages/encrypted_fields/fields.py", line 117, in get_db_prep_save
    encrypted_value = self.encrypt(value)
  File "/usr/local/lib/python3.7/site-packages/encrypted_fields/fields.py", line 76, in encrypt
    cipher = AES.new(bytes.fromhex(self.keys[0]), AES.MODE_GCM)
AttributeError: module 'Crypto.Cipher.AES' has no attribute 'MODE_GCM'

When reverted to 20.2.4 all working.
Using pycrypto==2.6.1

[brainwane]

Hello and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us.

Could you try upgrading to pip 20.3.1 and then choosing the old resolver behavior using the flag --use-deprecated=legacy-resolver and seeing whether that temporarily resolves the problem?

pip 20.3 has a new dependency resolver and I wonder whether that is related.

[namevic]

With pip 20.3.1 and legacy-resolver the problem is solved

[namevic]

Tryed to work through the guide you attached

bash-5.0# python -m pip install --upgrade pip
Collecting pip
  Downloading pip-20.3.1-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 12.9 MB/s 
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.4
    Uninstalling pip-20.2.4:
      Successfully uninstalled pip-20.2.4
Successfully installed pip-20.3.1
bash-5.0# pip check
No broken requirements found.

pip check not see any problems

[terencehonles]

This error is correct MODE_GCM has not been released and is only on master see pycrypto/pycrypto@5d7ab24 which is after [email protected]

Yes, there hasn't been a release in 7 years...

[terencehonles]

You'll probably want to make sure encrypted_fields specifies its dependencies correctly and also make sure it knows it's depending on unreleased changes which would need to be installed from GitHub instead of PyPI.

[namevic]

This error is correct MODE_GCM has not been released and is only on master see pycrypto/pycrypto@5d7ab24 which is after [email protected]

Yes, there hasn't been a release in 7 years...

bash-5.0# pip freeze | grep cryp
cryptography==3.2.1
django-searchable-encrypted-fields==0.1.9
pycrypto==2.6.1
pycryptodome==3.9.9

As you can see I have pycryptodome installed also, and this package gives me this MODE_GCM.

With legacy-resolver it's working, I think the issue is there...

[uranusjr]

Can you provide a step-to-step reproduction of the issue? You said when reverted to 20.2.4 all working, but I cannot reproduce it with the following Dockerfile:

FROM python:3.7-alpine

ARG PIP_VERSION

RUN apk add gcc musl-dev libffi-dev openssl-dev

RUN pip install pip==${PIP_VERSION}

RUN pip install \
 cryptography==3.2.1 \
 django-searchable-encrypted-fields==0.1.9 \
 pycrypto==2.6.1 \
 pycryptodome==3.9.9

RUN python -c 'import Crypto.Cipher; print(Crypto.Cipher.AES)'

Both docker build . --build-arg PIP_VERSION=20.2.4 and docker build . --build-arg PIP_VERSION=20.3.1 give me the same error

 > [5/5] RUN python -c 'import Crypto.Cipher; print(Crypto.Cipher.AES)':                                                                                                          
#8 0.272 Traceback (most recent call last):                                                                                                                                       
#8 0.272   File "<string>", line 1, in <module>                                                                                                                                   
#8 0.272 AttributeError: module 'Crypto.Cipher' has no attribute 'AES'

From the information currently available, pip (and the resolver) does not seem to be related to your issue.

[namevic]

FROM python:3.7-alpine
ENV PYTHONUNBUFFERED 1
RUN apk add --no-cache --virtual .build-deps gcc musl-dev postgresql-dev libffi-dev zlib-dev jpeg-dev libxslt-dev postgresql-libs bash && pip3 install lxml --no-cache-dir

WORKDIR /code/
ARG PIP_VERSION
RUN pip3 install pip==${PIP_VERSION} && pip3 install pycrypto==2.6.* django-searchable-encrypted-fields==0.1.8 --no-cache-dir
RUN python -c 'from Crypto.Cipher import AES; print(AES.MODE_GCM)'

docker build . --build-arg PIP_VERSION=20.3.1

Step 8/8 : RUN python -c 'from Crypto.Cipher import AES; print(AES.MODE_GCM)'
 ---> Running in f44ad1ac1e4e
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'Crypto.Cipher.AES' has no attribute 'MODE_GCM'

docker build . --build-arg PIP_VERSION=20.2.4

Step 8/8 : RUN python -c 'from Crypto.Cipher import AES; print(AES.MODE_GCM)'
 ---> Running in 1b67577116cd
11
Removing intermediate container 1b67577116cd
 ---> d52cbdf6dbe9
Successfully built d52cbdf6dbe9

[pfmoore]

pycryptodome seems to do some weird things in its setup.py to detect if pycrypto exists. I wonder if we're seeing a "legacy" (non-PEP517) install and a dependency on whether pycrypto or pycryptodome gets installed first? We don't guarantee any particular order of installs, so that could easily have changed between the old and new resolver.

[pradyunsg]

We don't guarantee any particular order of installs, so that could easily have changed between the old and new resolver.

We do make a tiny commitment: https://pip.pypa.io/en/stable/reference/pip_install/#installation-order

As of v6.1.0, pip installs dependencies before their dependents, i.e. in “topological order.” This is the only commitment pip currently makes related to order. While it may be coincidentally true that pip will install things in the order of the install arguments or in the order of the items in a requirements file, this is not a promise.

In the event of a dependency cycle (aka “circular dependency”), the current implementation (which might possibly change later) has it such that the first encountered member of the cycle is installed last.

[pfmoore]

True, but pycryptodome replaces pycrypto, as I understand it, so there's no dependency relationship in this case.

I may well be wrong, though, as building via wheels didn't seem to help.

I'm pretty sure this isn't a pip issue, though, but rather something in the (extremely complicated) setup.py code for one or both of pycrypto and pycryptodome that's making an unsafe assumption. But it's hard to tell what as I don't understand the code at all 🙁

[uranusjr]

The issue is indeed that pycryptodome replaces pycrypto. They both install the module Crypto, but the two pip versions order the installation differently.

# 20.2.4
Installing collected packages: pycrypto, pytz, sqlparse, asgiref, Django, pycryptodome, django-searchable-encrypted-fields

# 20.3.1
Installing collected packages: sqlparse, pytz, asgiref, pycryptodome, Django, pycrypto, django-searchable-encrypted-fields

20.3.1 installs pycryptodome first and pycrypto overwrites it, while pycryptodome “wins” in 20.2.4. So it is a pip issue (pip shouldn’t let a package overwrite another), but not resolver-related. The previous version accidentally works, and 20.3 unfortunately fails.

This issue is already tracked in #4625, so I’ll close this as a duplicate.