This repository has been archived by the owner on Aug 4, 2023. It is now read-only.
Handle the upper case licenses in the add_license_dag #1049
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
openverse-bot
added
the
🚦 status: awaiting triage
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
obulat
force-pushed
the
update/add_license_url_dag
branch
from
67fb983
to
9709355
Compare
obulat
added
🛠 goal: fix
AetherUnbound
approved these changes
Mar 16, 2023
Signed-off-by: Olga Bulat <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
After the last run of the
add_license_urlDAG, we still have more than 8000 records withNULLinmeta_data. Most of them are due to the license name being upper case (CC0 instead of cc0).This PR converts the license name to lower case before using it to create the URL. It also updates both the
meta_dataand thelicense.To make this DAG faster, I've removed the first step, which basically just repeats the first part of the second task selecting the items where meta_data is NULL.
This DAG also saves the data (identifier, license, license_version) for invalid license pairs that cannot be fixed to S3 for further investigation.