Improve Cosmos test environment Airflow installation

[jbandoro]

Opening up this issue after #806 was merged where more pinning specific to Airflow version dependencies had to be done in overrides and a workaround to uninstall a provider that is not available in earlier Airflow versions.

A possible better way to install Airflow is to use the recommended constraint file, this way we wouldn't have to keep track of pinning different Airflow dependencies for different versions. Installing with a constraint file can be done like the below where it depends on a specific airflow version and python version:

pip install "apache-airflow==2.8.1" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.8.txt"

After the work @tatiana did in #806 by adding a hatch post-install-commands, I think we could use the pre-install-commands to install airflow with a constraint file. The pyproject.toml would like:

######################################
# TESTING
######################################

[tool.hatch.envs.tests]
dependencies = [
    "astronomer-cosmos[tests]",
    "types-PyYAML",
    "types-attrs",
    "types-requests",
    "types-python-dateutil",
    "Werkzeug<3.0.0",
    "apache-airflow-providers-cncf-kubernetes>=5.1.1",
    "apache-airflow-providers-docker>=3.5.0",
]
pre-install-commands = [
    "pip install 'apache-airflow=={matrix:airflow}' --constraint 'https://raw.githubusercontent.com/apache/airflow/constraints-{matrix:airflow}/constraints-{matrix:python}.txt'"
]

[[tool.hatch.envs.tests.matrix]]
python = ["3.8", "3.9", "3.10"]
airflow = ["2.3.4", "2.4.3", "2.5.3", "2.6.3", "2.7.3", "2.8.1"]


[tool.hatch.envs.tests.scripts]
...

The update above would remove the need to have the [tool.hatch.envs.tests.overrides] section. However the matrix does change a bit where the Airflow patch version is required.

The GH workflow would also need to be updated so the matrix matches.

An advantage of this is that we would not need to worry about newer releases of Airflow breaking the CI tests, but we would need to increment the patch version above when a latest version of Airflow is released.

[dosubot]

🤖

Hey @jbandoro! Great to see you back here with another insightful suggestion. I'm currently diving into the details of your issue. Hang tight, I'll provide you with a comprehensive response shortly!

[tatiana]

I really like this, @jbandoro , it looks much cleaner!

[dosubot]

🤖

It's great to hear that you're enjoying the cleaner look! If you have any questions or need assistance with anything related to the project, feel free to ask. I'm here to help!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.