Update Flickr to use new time delineated ingester class #995

stacimc · 2023-02-11T01:57:24Z

Fixes

Fixes WordPress/openverse#1789 by @obulat

Description

This PR is largely a refactor and has very little new logic. The goal is to update the Flickr DAG to dynamically break its ingestion day down into time intervals depending on the distribution of data throughout the day. To do so, I've pulled existing logic out of the Finnish Museums DAG into a reusable class to be used by both.

Summary:

Creates a subclass of the ProviderDataIngester, TimeDelineatedProviderDataIngester, which adds in all the logic for generating timestamp intervals, iterating over them, and detecting some common bugs
Refactors Finnish and Flickr to use this base class

To use the new class all you have to do is define a few variables and implement an abstract method for getting the record count for a particular interval. When ingest_records is called, timestamp pairs will be dynamically generated according to the configured variables, and iterated over automatically. The bounds of the current time interval will start_ts and end_ts will be available as kwargs in get_query_params.

The behavior of Finnish Museums should be totally unchanged. Flickr now gets for free the dynamic timeslice generation that Finnish was already working with. It will attempt to generate intervals with less than 4,000 records per slice, as the Flickr documentation reports that maximum 4k unique records are returned for any given query.

Notes about how the timeslice intervals were chosen

When experimenting locally, I found that the benefit of decreasing the interval size rapidly diminishes for Flickr (ie, after a certain point, you start getting far more duplicates when the intervals decrease in size). I suspect that the API isn't actually meant to be used for very small intervals, so the range is 'fuzzy' and causes a lot of overlap/duplicates when these small intervals are used.

For reference, I tried running ingestion using the date 2023-02-08 using (1) 1-minute time slices for the entire day, and (2) the dynamic interval generation used in this PR.

For the first, that resulted in 1440 intervals and took almost 1.5 hours to pull data. We received 37,115 unique records and a whopping 709,199 duplicates.

For the latter, only 35 intervals were generated and it took just under 3 minutes to pull (1/30 of the time). We got 34,768 records but only 18,295 duplicates.

So, for the cost of a relatively small difference in the number of records ingested we are able to run ingestion in a fraction of the time and deal with many, many fewer duplicates. It is also worth pointing out that neither approach guarantees that all unique results are collected, so we are making a concession either way. While we pursue other approaches to Flickr, I think we should go with the much faster approach, and perhaps tweak the reingestion schedule for Flickr to be more aggressive.

Testing Instructions

Run tests locally.

Manually test the Flickr and Finnish PRs. For Flickr, you should be able to simply enable it locally and let it run for a few days -- it should be pretty quick! For Finnish you may need to let it run for awhile to make sure that it hits an ingestion day with a reasonable amount of data, as that DAG is quite sparse.

Checklist

My pull request has a descriptive title (not a vague title like
Update index.md).
My pull request targets the default branch of the repository (main) or
a parent feature branch.
My commit messages follow best practices.
My code follows the established code style of the repository.
I added or updated tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no visible
errors.
I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

…ration

AetherUnbound

This is excellent! What a great idea to re-use these components, I'm sure we'll have APIs in the future that will benefit from this. I have a few comments, namely on the interface for the class 🙂

openverse_catalog/dags/providers/provider_api_scripts/time_delineated_provider_data_ingester.py

tests/dags/providers/provider_api_scripts/test_flickr.py

AetherUnbound · 2023-02-17T00:46:45Z

tests/dags/providers/provider_api_scripts/test_time_delineated_provider_data_ingester.py

+        )
+
+
+def test_ts_pairs_and_kwargs_are_available_in_get_next_query_params():


Great thing to test!

AetherUnbound · 2023-02-17T00:47:31Z

tests/dags/providers/provider_api_scripts/test_time_delineated_provider_data_ingester.py

+        # Assert that attempting to ingest records raises an exception when
+        # `should_raise_error` is enabled
+        with (pytest.raises(AirflowException, match=expected_error_string)):
+            ingester.ingest_records()
+
+        # get_mock should have been called 4 times, twice for each batch (once in `get_batch`
+        # and a second time in `get_should_continue`). Each time, it returned 2 'new' records,
+        # even though the resultCount indicated there should only be 2 records total.
+        assert get_mock.call_count == 4
+        # process_mock should only have been called once. We raise an error when getting the
+        # second batch as soon as it is detected that we've processed too many records.
+        assert process_mock.call_count == 1


Thank you for these excellent comments!

AetherUnbound · 2023-02-17T00:48:42Z

openverse_catalog/dags/providers/provider_api_scripts/finnish_museums.py

-            pairs_list.extend(minute_slices)
-
-        return pairs_list
+    max_records = 10_000


The simplification of this file is awesome ✨

openverse_catalog/dags/providers/provider_api_scripts/flickr.py

AetherUnbound

Tested locally and it worked like a charm for both DAGs!

openverse-bot · 2023-02-22T00:00:11Z

Based on the high urgency of this PR, the following reviewers are being gently reminded to review this PR:

@krysal
This reminder is being automatically generated due to the urgency configuration.

Excluding weekend¹ days, this PR was updated 2 day(s) ago. PRs labelled with high urgency are expected to be reviewed within 2 weekday(s)².

@stacimc, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings.

Specifically, Saturday and Sunday. ↩
For the purpose of these reminders we treat Monday - Friday as weekdays. Please note that the that generates these reminders runs at midnight UTC on Monday - Friday. This means that depending on your timezone, you may be pinged outside of the expected range. ↩

krysal

It´s incredible how many details are involved in these time-delineated kinds of DAGs. Comments on each step and test are very appreciated!

stacimc added 3 commits February 10, 2023 17:41

Create an ingester subclass that includes the timestamp interval gene…

33234b2

…ration

Pull logic out of Finnish and use the new base class

56ca25f

Pull logic out of Flickr and use base class

3d8e935

stacimc added 🟧 priority: high Stalls work on the project or its dependents ✨ goal: improvement Improvement to an existing user-facing feature 💻 aspect: code Concerns the software code in the repository labels Feb 11, 2023

stacimc self-assigned this Feb 11, 2023

stacimc added 3 commits February 16, 2023 10:50

Move tests to tests for new base class

0240b01

Add new tests for base class

f70c30f

Update comments

1c6f7d4

stacimc changed the title ~~Add/time delineated ingester class~~ Update Flickr to use new time delineated ingester class Feb 16, 2023

Raise error if an interval has more than 4k records

709d85b

stacimc marked this pull request as ready for review February 16, 2023 20:12

stacimc requested a review from a team as a code owner February 16, 2023 20:12

stacimc requested review from krysal and AetherUnbound February 16, 2023 20:12

AetherUnbound reviewed Feb 17, 2023

View reviewed changes

Raise ValueError instead of AssertionError, enforce type in child class

702f697

AetherUnbound approved these changes Feb 17, 2023

View reviewed changes

stacimc mentioned this pull request Apr 17, 2023

Adjust Flickr reingestion schedule WordPress/openverse#1285

Open

krysal approved these changes Feb 23, 2023

View reviewed changes

stacimc merged commit e4f39bb into main Feb 23, 2023

stacimc deleted the add/time-delineated-ingester-class branch February 23, 2023 19:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Flickr to use new time delineated ingester class #995

Update Flickr to use new time delineated ingester class #995

stacimc commented Feb 11, 2023 •

edited

Loading

AetherUnbound left a comment

AetherUnbound Feb 17, 2023

AetherUnbound Feb 17, 2023

AetherUnbound Feb 17, 2023

AetherUnbound left a comment

openverse-bot commented Feb 22, 2023

krysal left a comment

		)


		def test_ts_pairs_and_kwargs_are_available_in_get_next_query_params():

Update Flickr to use new time delineated ingester class #995

Update Flickr to use new time delineated ingester class #995

Conversation

stacimc commented Feb 11, 2023 • edited Loading

Fixes

Description

Notes about how the timeslice intervals were chosen

Testing Instructions

Checklist

Developer Certificate of Origin

AetherUnbound left a comment

Choose a reason for hiding this comment

AetherUnbound Feb 17, 2023

Choose a reason for hiding this comment

AetherUnbound Feb 17, 2023

Choose a reason for hiding this comment

AetherUnbound Feb 17, 2023

Choose a reason for hiding this comment

AetherUnbound left a comment

Choose a reason for hiding this comment

openverse-bot commented Feb 22, 2023

Footnotes

krysal left a comment

Choose a reason for hiding this comment

stacimc commented Feb 11, 2023 •

edited

Loading