Optimize secondary task recollection #2800

ABrain7710 · 2024-05-17T02:46:48Z

Description

Overall

Optimized pull_request_files, pull_request_commits, and pull_request_reviews when they are recollected

Implementation Details

When building list of repos to collect make the list contain a full_collection boolean flag so start_data_collection knows what to pass
In AugurTaskRoutine.start_data_collection iterate through the tuples and pass the repo_git and full_collection flag to each phase
Add full_collection flag to every phase's method arguments. This had to be done since AugurTaskRoutine.start_data_collection passes the same arguments for every phase
Define database methods to get_secondary_data_last_collected and get prs that have been updated since a date
Updated pull_request_files to only get pr numbers for updated prs if full_collection flag is false
Updated pull_request_commits to only get pr urls for updated prs if full_collection flag is false
Updated pull_request_reviews to only get pr numbers for updated prs if full_collection flag is false

Notes for Reviewers
I have not tested this yet, I will change it from a draft pr to a pr when it is test

Signed commits

Yes, I signed my commits.

Signed-off-by: Andrew Brain <[email protected]>

github-actions · 2024-05-17T17:22:28Z

augur/application/db/lib.py

@@ -1,7 +1,7 @@
 import sqlalchemy as s


[pylint] _{reported by reviewdog 🐶}
C0114: Missing module docstring (missing-module-docstring)

github-actions · 2024-05-17T17:22:28Z

augur/tasks/data_analysis/__init__.py

@@ -1,7 +1,7 @@
 from celery import chain


[pylint] _{reported by reviewdog 🐶}
C0114: Missing module docstring (missing-module-docstring)

github-actions · 2024-05-17T17:22:28Z

augur/tasks/data_analysis/__init__.py

@@ -1,7 +1,7 @@
 from celery import chain
 import logging 

-def machine_learning_phase(repo_git):
+def machine_learning_phase(repo_git, full_collection):
    from augur.tasks.data_analysis.clustering_worker.tasks import clustering_task


[pylint] _{reported by reviewdog 🐶}
C0415: Import outside toplevel (augur.tasks.data_analysis.clustering_worker.tasks.clustering_task) (import-outside-toplevel)

github-actions · 2024-05-17T17:22:28Z

augur/tasks/data_analysis/__init__.py

@@ -1,7 +1,7 @@
 from celery import chain
 import logging 

-def machine_learning_phase(repo_git):
+def machine_learning_phase(repo_git, full_collection):
    from augur.tasks.data_analysis.clustering_worker.tasks import clustering_task
    from augur.tasks.data_analysis.discourse_analysis.tasks import discourse_analysis_task


[pylint] _{reported by reviewdog 🐶}
C0415: Import outside toplevel (augur.tasks.data_analysis.discourse_analysis.tasks.discourse_analysis_task) (import-outside-toplevel)

github-actions · 2024-05-17T17:22:28Z

augur/tasks/data_analysis/__init__.py

@@ -1,7 +1,7 @@
 from celery import chain
 import logging 

-def machine_learning_phase(repo_git):
+def machine_learning_phase(repo_git, full_collection):
    from augur.tasks.data_analysis.clustering_worker.tasks import clustering_task
    from augur.tasks.data_analysis.discourse_analysis.tasks import discourse_analysis_task
    from augur.tasks.data_analysis.insight_worker.tasks import insight_task


[pylint] _{reported by reviewdog 🐶}
C0415: Import outside toplevel (augur.tasks.data_analysis.insight_worker.tasks.insight_task) (import-outside-toplevel)

github-actions · 2024-05-17T17:22:29Z

augur/tasks/start_tasks.py

@@ -166,7 +166,7 @@ def build_primary_repo_collect_request(session,enabled_phase_names, days_until_c
    primary_gitlab_enabled_phases.append(primary_repo_collect_phase_gitlab)

    #task success is scheduled no matter what the config says.
-    def core_task_success_util_gen(repo_git):
+    def core_task_success_util_gen(repo_git, full_collection):


[pylint] _{reported by reviewdog 🐶}
W0613: Unused argument 'full_collection' (unused-argument)

github-actions · 2024-05-17T17:22:29Z

augur/tasks/start_tasks.py

@@ -186,7 +186,7 @@ def build_secondary_repo_collect_request(session,enabled_phase_names, days_until

    secondary_enabled_phases.append(secondary_repo_collect_phase)

-    def secondary_task_success_util_gen(repo_git):
+    def secondary_task_success_util_gen(repo_git, full_collection):


[pylint] _{reported by reviewdog 🐶}
W0613: Unused argument 'full_collection' (unused-argument)

github-actions · 2024-05-17T17:22:29Z

augur/tasks/start_tasks.py

@@ -202,12 +202,12 @@ def build_facade_repo_collect_request(session,enabled_phase_names, days_until_co

    facade_enabled_phases.append(facade_phase)

-    def facade_task_success_util_gen(repo_git):
+    def facade_task_success_util_gen(repo_git, full_collection):


[pylint] _{reported by reviewdog 🐶}
W0613: Unused argument 'full_collection' (unused-argument)

github-actions · 2024-05-17T17:22:30Z

augur/tasks/start_tasks.py

        return facade_task_success_util.si(repo_git)

    facade_enabled_phases.append(facade_task_success_util_gen)

-    def facade_task_update_weight_util_gen(repo_git):
+    def facade_task_update_weight_util_gen(repo_git, full_collection):


[pylint] _{reported by reviewdog 🐶}
W0613: Unused argument 'full_collection' (unused-argument)

github-actions · 2024-05-17T17:22:30Z

augur/tasks/start_tasks.py

@@ -222,7 +222,7 @@ def build_ml_repo_collect_request(session,enabled_phase_names, days_until_collec

    ml_enabled_phases.append(machine_learning_phase)

-    def ml_task_success_util_gen(repo_git):
+    def ml_task_success_util_gen(repo_git, full_collection):


[pylint] _{reported by reviewdog 🐶}
W0613: Unused argument 'full_collection' (unused-argument)

sgoggins · 2024-06-11T16:44:18Z

@ABrain7710 : What's the status of this one with regards to testing? Should we merge it now that we have done a release.

ABrain7710 · 2024-06-11T17:06:13Z

This is complete, but I haven't had a chance to test it yet. It is a bit of a difficult test.It is a very small change though so if it was reviewed throughly it could be probably be merged

Signed-off-by: Andrew Brain <[email protected]>

github-actions · 2024-06-17T03:49:41Z

augur/tasks/github/pull_requests/commits_model/core.py

@@ -53,6 +73,7 @@ def pull_request_commits_model(repo,logger, key_auth):
        logger.info(f"{task_name}: Inserting {len(all_data)} rows")
        pr_commits_natural_keys = ["pull_request_id", "repo_id", "pr_cmt_sha"]
        bulk_insert_dicts(logger, all_data,PullRequestCommit,pr_commits_natural_keys)


[pylint] _{reported by reviewdog 🐶}
E0602: Undefined variable 'bulk_insert_dicts' (undefined-variable)

github-actions · 2024-06-17T03:49:41Z

augur/tasks/github/pull_requests/tasks.py

@@ -328,7 +326,7 @@ def collect_pull_request_review_comments(repo_git: str) -> None:


 @celery.task(base=AugurSecondaryRepoCollectionTask)
-def collect_pull_request_reviews(repo_git: str) -> None:
+def collect_pull_request_reviews(repo_git: str, full_collection: bool) -> None:


[pylint] _{reported by reviewdog 🐶}
R0912: Too many branches (16/12) (too-many-branches)

github-actions · 2024-06-17T03:49:41Z

augur/tasks/github/pull_requests/tasks.py

@@ -328,7 +326,7 @@ def collect_pull_request_review_comments(repo_git: str) -> None:


 @celery.task(base=AugurSecondaryRepoCollectionTask)
-def collect_pull_request_reviews(repo_git: str) -> None:
+def collect_pull_request_reviews(repo_git: str, full_collection: bool) -> None:


[pylint] _{reported by reviewdog 🐶}
R0915: Too many statements (54/50) (too-many-statements)

Signed-off-by: Andrew Brain <[email protected]>

github-actions · 2024-06-17T04:01:25Z

augur/tasks/github/pull_requests/commits_model/tasks.py

@@ -2,18 +2,18 @@
 from augur.tasks.github.pull_requests.commits_model.core import *


[pylint] _{reported by reviewdog 🐶}
W0401: Wildcard import augur.tasks.github.pull_requests.commits_model.core (wildcard-import)

github-actions · 2024-06-17T04:01:26Z

augur/tasks/github/pull_requests/tasks.py

+                        element.decode('utf-8').replace('\x00', ' ') if isinstance(element, bytes) else element
+                        for element in page_data
+                    ]
+                    logger.info(f"NUL characters were found in PR Reviews and replaced with spaces.") 


[pylint] _{reported by reviewdog 🐶}
W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)

github-actions · 2024-06-17T04:01:26Z

augur/tasks/github/pull_requests/tasks.py

+                    logger.info(f"NUL characters were found in PR Reviews and replaced with spaces.") 
+                elif isinstance(page_data, bytes):
+                    page_data = page_data.decode('utf-8').replace('\x00', ' ')
+                    logger.info(f"NUL characters were found in PR Reviews and replaced with spaces.") 


[pylint] _{reported by reviewdog 🐶}
W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)

github-actions · 2024-06-17T04:01:26Z

augur/tasks/github/pull_requests/tasks.py

-    contributors = []
-    for pull_request_id in all_pr_reviews.keys():
+        contributors = []
+        for pull_request_id in all_pr_reviews.keys():


[pylint] _{reported by reviewdog 🐶}
C0206: Consider iterating with .items() (consider-using-dict-items)

github-actions · 2024-06-17T04:01:26Z

augur/tasks/github/pull_requests/tasks.py

-    contributors = []
-    for pull_request_id in all_pr_reviews.keys():
+        contributors = []
+        for pull_request_id in all_pr_reviews.keys():


[pylint] _{reported by reviewdog 🐶}
C0201: Consider iterating the dictionary directly instead of calling .keys() (consider-iterating-dictionary)

github-actions · 2024-06-17T04:01:26Z

augur/tasks/github/pull_requests/tasks.py

-    pr_reviews = []
-    for pull_request_id in all_pr_reviews.keys():
+        pr_reviews = []
+        for pull_request_id in all_pr_reviews.keys():


[pylint] _{reported by reviewdog 🐶}
C0206: Consider iterating with .items() (consider-using-dict-items)

github-actions · 2024-06-17T04:01:26Z

augur/tasks/github/pull_requests/tasks.py

-    pr_reviews = []
-    for pull_request_id in all_pr_reviews.keys():
+        pr_reviews = []
+        for pull_request_id in all_pr_reviews.keys():


[pylint] _{reported by reviewdog 🐶}
C0201: Consider iterating the dictionary directly instead of calling .keys() (consider-iterating-dictionary)

github-actions · 2024-06-17T04:01:26Z

augur/tasks/github/pull_requests/tasks.py

+
+            pr_reviews = []
+            pr_reviews_generator = GithubPaginator(pr_review_url, manifest.key_auth, logger)
+            for page_data, page in pr_reviews_generator.iter_pages():


[pylint] _{reported by reviewdog 🐶}
W0612: Unused variable 'page' (unused-variable)

sgoggins

Ready for testing!

ABrain7710 added 2 commits May 16, 2024 21:35

Optimize secondary task recollection

9167238

Signed-off-by: Andrew Brain <[email protected]>

Pass full collection flag to phases

0e47f00

Signed-off-by: Andrew Brain <[email protected]>

ABrain7710 changed the base branch from main to dev May 17, 2024 02:46

ABrain7710 requested review from IsaacMilarky, sgoggins and Ulincsys May 17, 2024 02:47

Filter updated prs by repo_id

4d04633

Signed-off-by: Andrew Brain <[email protected]>

github-actions bot reviewed May 17, 2024

View reviewed changes

Merge branch 'dev' into optimize-secondary-recollection

2af7ef1

ABrain7710 added 3 commits June 16, 2024 19:06

Don't use tuple constructor

2a76f9c

Signed-off-by: Andrew Brain <[email protected]>

Fix syntax errors

6730a11

Signed-off-by: Andrew Brain <[email protected]>

Merge branch 'dev' into optimize-secondary-recollection

39f552b

github-actions bot reviewed Jun 17, 2024

View reviewed changes

fixes after merging dev

6035218

Signed-off-by: Andrew Brain <[email protected]>

ABrain7710 marked this pull request as ready for review June 17, 2024 04:01

github-actions bot reviewed Jun 17, 2024

View reviewed changes

sgoggins approved these changes Jun 17, 2024

View reviewed changes

sgoggins merged commit 6f931ee into dev Jun 17, 2024
9 checks passed

ABrain7710 deleted the optimize-secondary-recollection branch June 25, 2024 00:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize secondary task recollection #2800

Optimize secondary task recollection #2800

ABrain7710 commented May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

github-actions bot May 17, 2024

sgoggins commented Jun 11, 2024

ABrain7710 commented Jun 11, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

github-actions bot Jun 17, 2024

sgoggins left a comment

		@@ -2,18 +2,18 @@
		from augur.tasks.github.pull_requests.commits_model.core import *

Optimize secondary task recollection #2800

Optimize secondary task recollection #2800

Conversation

ABrain7710 commented May 17, 2024

Description

Overall

Implementation Details

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

sgoggins commented Jun 11, 2024

ABrain7710 commented Jun 11, 2024

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot Jun 17, 2024

Choose a reason for hiding this comment

sgoggins left a comment

Choose a reason for hiding this comment