Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix compatibility with apache-airflow-providers-cncf-kubernetes>=10.0.0 #311

Merged
merged 2 commits into from
Dec 5, 2024

Conversation

tatiana
Copy link
Collaborator

@tatiana tatiana commented Dec 5, 2024

Users were not able to use DAG Factory with apache-airflow-providers-cncf-kubernetes>=10.0.0, which was released on 18 November 2024.

When attempting to install a local dev environment using:

python3.10 -m venv venv
source venv/bin/activate  
pip install '.[tests]'

If we attempted to run:

ln -s dev/dags dags 
AIRFLOW_HOME=`pwd` airflow dags list-import-errors

All DAG factory example DAGs would error with messages like:

filepath                                                       | error                                                        
===============================================================+==============================================================
/Users/tati/Code/dag-factory/dags/example_da | Traceback (most recent call last):                           
g_factory.py                                                   |   File                                                       
                                                               | "/Users/tati/Code/dag-factory/dagfactory/da
                                                               | gbuilder.py", line 69, in <module>                           
                                                               |     from                                                     
                                                               | airflow.providers.cncf.kubernetes.operators.kubernetes_pod   
                                                               | import KubernetesPodOperator                                 
                                                               | ModuleNotFoundError: No module named                         
                                                               | 'airflow.providers.cncf.kubernetes.operators.kubernetes_pod' 
                                                               |                                                              
                                                               | During handling of the above exception, another exception    
                                                               | occurred:                                                    
                                                               |                                                              
                                                               | Traceback (most recent call last):                           
                                                               |   File                                                       
                                                               | "/Users/tati/Code/dag-factory/dagfactory/da
                                                               | gfactory.py", line 13, in <module>                           
                                                               |     from dagfactory.dagbuilder import DagBuilder             
                                                               |   File                                                       
                                                               | "/Users/tati/Code/dag-factory/dagfactory/da
                                                               | gbuilder.py", line 71, in <module>                           
                                                               |     from airflow.contrib.kubernetes.pod import Port          
                                                               | ModuleNotFoundError: No module named                         
                                                               | 'airflow.contrib.kubernetes'                                 
                                                               |                                                              
/Users/tati/Code/dag-factory/dags/example_ta | Traceback (most recent call last):                           
sk_group.py                                                    |   File                                                       
                                                               | "/Users/tati/Code/dag-factory/dagfactory/da
                                                               | gbuilder.py", line 69, in <module>                           
                                                               |     from                                                     
                                                               | airflow.providers.cncf.kubernetes.operators.kubernetes_pod   
                                                               | import KubernetesPodOperator                                 
                                                               | ModuleNotFoundError: No module named                         
                                                               | 'airflow.providers.cncf.kubernetes.operators.kubernetes_pod' 
                                                               |                                                              
                                                               | During handling of the above exception, another exception    
                                                               | occurred:                                                    
                                                               |                                                              
                                                               | Traceback (most recent call last):                           
                                                               |   File                                                       
                                                               | "/Users/tati/Code/dag-factory/dagfactory/da
                                                               | gfactory.py", line 13, in <module>                           
                                                               |     from dagfactory.dagbuilder import DagBuilder             
                                                               |   File                                                       
                                                               | "/Users/tati/Code/dag-factory/dagfactory/da
                                                               | gbuilder.py", line 71, in <module>                           
                                                               |     from airflow.contrib.kubernetes.pod import Port          
                                                               | ModuleNotFoundError: No module named                         
                                                               | 'airflow.contrib.kubernetes'

This was due to a breaking change introduced in apache-airflow-providers-cncf-kubernetes 10:
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/changelog.html

Remove airflow.providers.cncf.kubernetes.operators.kubernetes_pod. Use airflow.providers.cncf.kubernetes.operators.pod instead.

@tatiana tatiana requested a review from a team as a code owner December 5, 2024 11:55
@codecov-commenter
Copy link

codecov-commenter commented Dec 5, 2024

Codecov Report

Attention: Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.45%. Comparing base (72bc85b) to head (4ef3539).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
dagfactory/dagbuilder.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #311      +/-   ##
==========================================
- Coverage   92.57%   92.45%   -0.13%     
==========================================
  Files          10       10              
  Lines         700      702       +2     
==========================================
+ Hits          648      649       +1     
- Misses         52       53       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana merged commit 254eac1 into main Dec 5, 2024
67 checks passed
@tatiana tatiana deleted the fix-latest-k8s-provider branch December 5, 2024 12:49
@tatiana tatiana mentioned this pull request Dec 6, 2024
tatiana added a commit that referenced this pull request Dec 6, 2024
### Added

- Add support to TaskFlow and improve dynamic task mapping support by
@tatiana in #314
- Render YML DAG config as DAG Docs by @pankajastro #305
- Support building DAGs out of topologically unsorted YAML files by
@tatiana in #307
- Add support for nested task groups by @glazunov996 and @pankajastro in
#292
- Add support for templating `on_failure_callback` by @jroach-astronomer
#252

### Fixed

- Fix compatibility with
apache-airflow-providers-cncf-kubernetes>=10.0.0 by @tatiana in #311
- Refactor telemetry to collect events during DAG run and not during DAG
parsing by @pankajastro #300

### Docs

- Fix reference for HttpSensor in README.md by @pankajastro in #277
- Add example DAG for task group by @pankajastro in #293
- Add CODEOWNERS by @pankajkoti in #270
- Update CODEOWNERS to track all files by @pankajkoti in #276
- Modified Status badge in README by @jaejun #298

### Others

- Refactor dynamic task mapping implementation by @tatiana in #313
- Remove pytest durations from tests by @tatiana in #309
- Remove DAG retries check since many DAGs have different retry values
by @tatiana in #310
- Lint fixes after running `pre-commit run --all-files` by @tatiana in
#312
- Remove redundant exception code by @pankajastro #294
- Add GitHub issue template for bug reports and feature requests by
@pankajkoti in #269

Closes: #223
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants