Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Airflow constraints file constraints-1.10.7/constraints-3.7 incorrect constraints preventing install #13045

Closed
ranamit112 opened this issue Dec 13, 2020 · 6 comments
Labels
kind:bug This is a clearly a bug won't fix

Comments

@ranamit112
Copy link

Apache Airflow version: 1.10.7

Kubernetes version (if you are using kubernetes) (use kubectl version): N.A.

Environment:

  • Cloud provider or hardware configuration: N.A.
  • OS (e.g. from /etc/os-release): Red Hat Enterprise Linux Server 7.6 (Maipo)
  • Kernel (e.g. uname -a): 3.10.0-957.35.2.el7.x86_64
  • Install tools: pip, python3.7
  • Others:

What happened: Could not install airflow 1.10.7 using constraint file for 1.10.7.

ERROR: Could not find a version that satisfies the requirement future<0.17,>=0.16.0 (from apache-airflow[celery,postgres,statsd])
ERROR: No matching distribution found for future<0.17,>=0.16.0

What you expected to happen: pip install success

The 1.10.7 constraint file (that was created using 1.10.10 dependencies) includes a constraint for future==0.18.2 while the setup.py for that version includes the requirement that future>=0.16.0, <0.17, which is incompatible.
This can be considered part of #12838, as this happens only when using the new pip 20.3 resolver, but I think pip is behaving correctly, and the real problem is in the constraint file.

How to reproduce it:

python3.7 -m venv venv_python37_airflow
cd venv_python37_airflow
. ./bin/activate
python3.7 -m pip install --upgrade pip
python3.7 -m pip install wheel
AIRFLOW_VERSION=1.10.7
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
python -m pip install "apache-airflow[celery,postgres,statsd]==${AIRFLOW_VERSION}" --constraint ${CONSTRAINT_URL}

Anything else we need to know:
From my testing removing the constraint for future==0.18.2 solved this meaning it's the only non-valid constraint for this version.
I believe this happens in other earlier versions, but I did not test version earlier than 1.10.7.

@ranamit112 ranamit112 added the kind:bug This is a clearly a bug label Dec 13, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 13, 2020

Thanks for opening your first issue here! Be sure to follow the issue template!

@potiuk
Copy link
Member

potiuk commented Dec 13, 2020

Thanks for reporting this. I will leave it for reference to anone that will try to install Airflow 1.10.7, however, our recommendation is to move forward with 1.10.14 (bridge release for 2.0) with the goal of moving to 2.0 as soon as possible. This is the release where constraints have been finally solved once and for all (also in the 2.0 version) and we highly recommend migrating to 2.0. 2.0.0rc2 is being voted and it is very likely the official 2.0.0 of Airflow will be released Monday/Tuesday. 1.10.14 is already released last week.

@ranamit112
Copy link
Author

  1. I think this raises a good case to remove the tag contraints-1.10.7 (and earlier, probably), as the constraint files in them will not work when using pip 20.3, but it's your call.

  2. Commenting a workaround for posterity, for anyone like me who needs to have a more verified version of 1.10.7 for CI or migration:

curl ${CONSTRAINT_URL} > constraints.txt
grep -v "future==" constraints.txt > constraints_good.txt  # remove bad constraint
python -m pip install "apache-airflow[celery,postgres,statsd]==${AIRFLOW_VERSION}" --constraint constraints_good.txt

@potiuk
Copy link
Member

potiuk commented Dec 14, 2020

  1. I think this raises a good case to remove the tag contraints-1.10.7 (and earlier, probably), as the constraint files in them will not work when using pip 20.3, but it's your call.

Airflow installation does not work in general using PIP 20.3. This is a known issue, very prominently displayed on our site. #12838 - if you tried pip 20.3, then that explains the problem.

We raised the issue (rather loudly) to the PIP team, They are going to release 20.3.2 today that fixes some of the problems we (and others) raised. I recommend to repeat the check once they release 20.3.2 or downgrade to 20.2.4 (or use legacy resolver) as this is officially the only version of PIP that Airflow works with.

  1. Commenting a workaround for posterity, for anyone like me who needs to have a more verified version of 1.10.7 for CI or migration:
curl ${CONSTRAINT_URL} > constraints.txt
grep -v "future==" constraints.txt > constraints_good.txt  # remove bad constraint
python -m pip install "apache-airflow[celery,postgres,statsd]==${AIRFLOW_VERSION}" --constraint constraints_good.txt

There are many other packages which have problems with PIP 20.3. Until those problems are solved, I heartily recommend to downgrade PIP to 20.2.4 and keep it this way.

@ranamit112
Copy link
Author

ranamit112 commented Dec 14, 2020

While I don't doubt that there are several problems with pip 20.3, as you documented in the attached issues, I don't think this one falls under them.
The constraints file for 1.10.7 is plainly and simply wrong, which is not surprising since it was created for 1.10.10.
As I wrote above it includes a constraint for future==0.18.2 while the setup.py for that version includes the requirement for future>=0.16.0, <0.17, so any good resolver would indicate that there is a mismatch.

@potiuk
Copy link
Member

potiuk commented Dec 14, 2020

While I don't doubt that there are several problems with pip 20.3, as you documented in the attached issues, I don't think this one falls under them.
The constraints file for 1.10.7 is plainly and simply wrong, which is not surprising since it was created for 1.10.10.
As I wrote above it includes a constraint for future==0.18.2 while the setup.py for that version includes the requirement for future>=0.16.0, <0.17, so any good resolver would indicate that there is a mismatch.

The problem is that before 20.3 PIP was happily accepting and resolving this anyway.

But I think there is nothing to fix now anyway. 1.10.7 is prehistory and we are recommending everyone to migrate to 1.10.14 which has good constraints (those constraints BTW do not work with pip 20.3 anyway). So similarly like with any other problems with previous releases - 1.10.14 + PIP 20.2.4 (for now) solves the problems we know about (including installation problems). I don't think any other action is needed at this stage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:bug This is a clearly a bug won't fix
Projects
None yet
Development

No branches or pull requests

2 participants