Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(BSR) refactor(qpi): remove useless tasks in import dag #3389

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

vbusson-pass
Copy link
Collaborator

DE PR

Describe your changes

Please include a summary of the changes:

  • This PR [adds/removes/fixes/replaces] the [feature/bug/etc].
  • Tag a reviewer if necessacy @github/username

Jira ticket number and/or notion link

JIRA-ticket_number

Type of change

  • Fix (non-breaking change which corrects expected behavior)
  • New fields (non-breaking change)
  • New table (non-breaking change)
  • Concept change (potentially breaking change which modifies fields according to new or evolving business concepts)
  • Table deletion (potentially breaking change which adds functionality/ table)

Checklist before requesting a review

  • I have performed a self-review of my code

  • My code passes CI/CD tests

  • I updated README.md

  • I have updated the dag

  • If my changes concern incremental table, I have altered their schema to accomodate with field's creation/deletion

  • I have made corresponding changes to the tables documentation

  • I have made corresponding changes to the fields glossary

  • I will create a review on slack and ensure to specify the duration of the review task: short (<10min), medium (<30min), long (>30min)

Added tests?

  • 👍 yes
  • 🙅 no, because they aren't needed
  • 🙋 no, because I need help
  • ⏰ no, but I created a ticket

PR title format (except for MEP)

There is a linter on the PR title format. Please respect the following format:

(ticket) type(topic): comment
  • ticket surrounded by parenthesis, with optionnaly a hyphen followed by one or more digits (e.g., -1234). The first part must be one of the following strings:

    • DA
    • DE
    • AE
    • DS
    • HF
    • BSR
    • PC
  • type :
    The second part to specify the type of change one of the following :

    • build
    • lint
    • ci
    • docs
    • feat
    • fix
    • perf
    • refactor
    • test
    • core
    • dbt
  • topic within parenthesis: 1 word e.g., (dag)

  • comment: tell us your life

examples:

  • ✅ (DE-124) refactor(firebase): update source field
  • ❌ (DE-124) refactor (firebase): update source field (space between type and topic)
  • ❌ (DE-124) airflow(firebase): update source fiedd in DAG (wrong type)
  • ❌ (DE-124) (DE-124) refactor(firebase refacto): update source field (topic in two words)
  • ✅ (BSR) docs(github): add PR title valid format in template

@vbusson-pass vbusson-pass changed the title (DE) BSR(dag): remove useless tasks in qpi import (BSR) refactor(qpi): remove useless tasks in import dag Nov 4, 2024
@@ -56,9 +56,9 @@ def verify_folder():
for s in stats:
blob_list.append(s)
if len(blob_list) > 0:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok pour moi. Juste un petit renommage et minirefacto de la fonction ça serait plus clair. (même si je sais que ce n'est pas toi qui vient de mettre ce code)

  • verify_folder => check_folder_is_not_empty
  • blob_list = [ file for file in bucket.list_blobs(prefix=name)]

Et enfin, ça pourrait être pas mal de créer des constantes pour les task_id qui sont liés à des branch_operator

genre
END_TASK_ID = "end"
IMPORT_ANSWERS_TO_BQ_TASK_ID=import_historical_answers_to_bigquery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants