New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize issue and pr recollection #2835

Merged

sgoggins merged 10 commits into dev from optimize-core-recollection-2

Jun 28, 2024

Contributor

ABrain7710 commented Jun 25, 2024 •

edited

Loading

Description

Optimize the recollection of issues and prs
Migrate issues task to use github_data_access since I was already changing the task

Signed commits

Yes, I signed my commits.

ABrain7710 added 3 commits

June 24, 2024 19:00


          Define method to get core_data_last_collected

523e135

Signed-off-by: Andrew Brain <[email protected]>


          Define method to get updated issues

f0cd7f7

Signed-off-by: Andrew Brain <[email protected]>


          Optimize prs and issues

d852a60

Signed-off-by: Andrew Brain <[email protected]>

ABrain7710 requested review from sgoggins, Ulincsys and IsaacMilarky

June 25, 2024 00:58

ABrain7710 changed the title ~~Optimize core recollection~~ Optimize issue and pr recollection

github-actions bot reviewed

View reviewed changes

augur/tasks/github/issues/tasks.py Outdated

+                  if full_collection:
+                      core_data_last_collected = None
+                  else:
+                      core_data_last_collected = get_core_data_last_collected().date()

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
E1120: No value for argument 'repo_id' in function call (no-value-for-parameter)

augur/tasks/github/issues/tasks.py Outdated

                   key_auth = GithubRandomKeyAuth(logger)
                   logger.info(f'this is the manifest.key_auth value: {str(key_auth)}')
                   try:
-                      issue_data = retrieve_all_issue_data(repo_git, logger, key_auth)
+                      issue_data = retrieve_all_issue_data(repo_git, logger, key_auth, core_data_last_collected)
                       if issue_data:

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)

augur/tasks/github/pull_requests/tasks.py

		@@ -61,13 +67,13 @@ def collect_pull_requests(repo_git: str) -> int:

		# TODO: Rename pull_request_reviewers table to pull_request_requested_reviewers

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
W0511: TODO: Rename pull_request_reviewers table to pull_request_requested_reviewers (fixme)

augur/tasks/github/pull_requests/tasks.py

		@@ -61,13 +67,13 @@ def collect_pull_requests(repo_git: str) -> int:

		# TODO: Rename pull_request_reviewers table to pull_request_requested_reviewers
		# TODO: Fix column names in pull request labels table

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
W0511: TODO: Fix column names in pull request labels table (fixme)

augur/tasks/github/pull_requests/tasks.py

		@@ -1,4 +1,5 @@
		import logging

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
C0114: Missing module docstring (missing-module-docstring)

augur/tasks/github/pull_requests/tasks.py Outdated

+                      if full_collection:
+                          core_data_last_collected = None
+                      else:
+                          core_data_last_collected = get_core_data_last_collected().date()

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
E1120: No value for argument 'repo_id' in function call (no-value-for-parameter)

augur/tasks/github/pull_requests/tasks.py Outdated

+                  yield data
+                  # return if last pr on the page was updated before the since date
+                  if since and datetime.fromisoformat(data["updated_at"].replace("Z", "+00:00")) < since:

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
E1136: Value 'data' is unsubscriptable (unsubscriptable-object)

augur/tasks/github/pull_requests/tasks.py

@@ @@ -61,13 +67,13 @@ def collect_pull_requests(repo_git: str) -> int: @@
               # TODO: Rename pull_request_reviewers table to pull_request_requested_reviewers
               # TODO: Fix column names in pull request labels table
-              def retrieve_all_pr_data(repo_git: str, logger, key_auth): #-> Generator[List[Dict]]:
+              def retrieve_all_pr_data(repo_git: str, logger, key_auth, since): #-> Generator[List[Dict]]:

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
R1711: Useless return at end of function or method (useless-return)

augur/tasks/github/pull_requests/tasks.py

+                  # return if last pr on the page was updated before the since date
+                  if since and datetime.fromisoformat(data["updated_at"].replace("Z", "+00:00")) < since:
+                      return
               def process_pull_requests(pull_requests, task_name, repo_id, logger, augur_db):

github-actions bot Jun 25, 2024

[pylint] _{reported by reviewdog 🐶}
R0914: Too many local variables (32/30) (too-many-locals)


          Pass full collection flag to tasks

64b82a2

Signed-off-by: Andrew Brain <[email protected]>

github-actions bot reviewed

View reviewed changes

augur/tasks/start_tasks.py

		@@ -63,8 +63,8 @@ def primary_repo_collect_phase(repo_git, full_collection):

github-actions bot Jun 26, 2024

[pylint] _{reported by reviewdog 🐶}
W0612: Unused variable 'logger' (unused-variable)

ABrain7710 added 2 commits

June 25, 2024 19:50


          Pass repo_id

3ee5e48

Signed-off-by: Andrew Brain <[email protected]>


          Fix issues based on linter

cd2c620

Signed-off-by: Andrew Brain <[email protected]>

ABrain7710 marked this pull request as ready for review

June 26, 2024 00:58


          Update issues task to use github data access

9ceb97f

Signed-off-by: Andrew Brain <[email protected]>

github-actions bot reviewed

View reviewed changes

augur/tasks/github/issues/tasks.py

@@ @@ -7,81 +7,72 @@ @@
               from augur.tasks.init.celery_app import celery_app as celery
               from augur.tasks.init.celery_app import AugurCoreRepoCollectionTask
               from augur.application.db.data_parse import *

github-actions bot Jun 26, 2024

[pylint] _{reported by reviewdog 🐶}
W0401: Wildcard import augur.application.db.data_parse (wildcard-import)

augur/tasks/github/issues/tasks.py

-                  return all_data
+                  return list(issues_paginator)
               def process_issues(issues, task_name, repo_id, logger) -> None:

github-actions bot Jun 26, 2024

[pylint] _{reported by reviewdog 🐶}
R0914: Too many local variables (32/30) (too-many-locals)

ABrain7710 added 3 commits

June 25, 2024 20:14


          Compare datetime with datetime

Signed-off-by: Andrew Brain <[email protected]>


          Merge branch 'dev' into optimize-core-recollection-2

9235aa8


          Fixes to datetime handling

636ef8c

Signed-off-by: Andrew Brain <[email protected]>

github-actions bot reviewed

View reviewed changes

augur/tasks/github/issues/tasks.py

		@@ -1,87 +1,80 @@
		import logging

github-actions bot Jun 26, 2024

[pylint] _{reported by reviewdog 🐶}
C0114: Missing module docstring (missing-module-docstring)

sgoggins approved these changes

View reviewed changes

Member

sgoggins left a comment

LGTM

sgoggins merged commit ec84836 into dev

9 checks passed

ABrain7710 deleted the optimize-core-recollection-2 branch

June 29, 2024 15:00

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet