-
Notifications
You must be signed in to change notification settings - Fork 54
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎉
Tested Smithsonian and Stocksnap just to be thorough. Great!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Tested the WordPress Photo Directory script and it loads rows in the image table 👍
## `tsv_to_postgres_loader` | ||
|
||
#### Database Loader DAG | ||
**DB Loader Apache Airflow DAG** (directed acyclic graph) takes the media data saved | ||
locally in TSV files, cleans it using an intermediate database table, and saves |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was about to ask if we don't need to conserve this documentation but I see we have it where it was moved :)
#### Database Loader DAG | |
**DB Loader Apache Airflow DAG** (directed acyclic graph) takes the media data saved | |
locally in TSV files, cleans it using an intermediate database table, and saves | |
the cleaned-up data into the main database (also called upstream or Openledger). | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made sure to check that as well 😄 Should have made a note in the description, apologies!
8c6b3da
to
d3652e5
Compare
Fixes
Fixes WordPress/openverse#1542 by @stacimc
Description
This PR retires the
tsv_to_postgres_loader
DAG, since it is no longer used in any of our ingestion workflows 🎉 I've also removed some no-obsolete code related to TSV pathing - this is covered by the DAG's XComs in the way our current pipelines are set up.Testing Instructions
just test
tsv_to_postgres_workflow
doesn't show up in your DAGs list locally.Checklist
Update index.md
).main
) or a parent feature branch.Developer Certificate of Origin
Developer Certificate of Origin