Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAG/plugins/data path is different from GCP Cloud Composer #2

Closed
dan0dbfe opened this issue Dec 15, 2022 · 1 comment
Closed

DAG/plugins/data path is different from GCP Cloud Composer #2

dan0dbfe opened this issue Dec 15, 2022 · 1 comment

Comments

@dan0dbfe
Copy link

dan0dbfe commented Dec 15, 2022

First of all, thanks for creating this tool. The local environments will come in very useful for local testing, shortening the feedback loop.

I noticed that my DAGs were failing because:

  • Data path in the local Airflow environment: /home/airflow/airflow/data
  • Data path in GCP Cloud Composer: /home/airflow/gcs/data

Please consider making the paths the same so that absolute path references work.

This is useful for BashOperator's cwd argument, which is non-templated and needs an absolute path.

Workaround

ln -s /home/airflow/airflow/dags /home/airflow/gcs
ln -s /home/airflow/airflow/plugins /home/airflow/gcs
ln -s /home/airflow/airflow/data /home/airflow/gcs

You may modify the CLI to add this in entrypoint.sh too.

Thank you!

@piotrwrotniak
Copy link
Collaborator

Since we're in 0.9.x version we've made a decision to align the paths so the workaround is not needed anymore.

Thanks for reporting this discrepancy :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants