Database improvements #2809

ABrain7710 · 2024-06-03T23:19:24Z

Signed commits

Yes, I signed my commits.

Signed-off-by: Andrew Brain <[email protected]>

…tabase-improvements-2

Signed-off-by: Andrew Brain <[email protected]>

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

pylint

augur/tasks/data_analysis/message_insights/tasks.py|33| R0915: Too many statements (169/50) (too-many-statements)
augur/tasks/db/refresh_materialized_views.py|92 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|98 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|104 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|110 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|116 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|122 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|128 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|134 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|140 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|146 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|152 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|158 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|164 col 8| W0107: Unnecessary pass statement (unnecessary-pass)
augur/tasks/db/refresh_materialized_views.py|9| R0912: Too many branches (13/12) (too-many-branches)
augur/tasks/db/refresh_materialized_views.py|9| R0915: Too many statements (81/50) (too-many-statements)
augur/tasks/git/dependency_libyear_tasks/core.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/git/dependency_libyear_tasks/core.py|2| W0401: Wildcard import augur.application.db.models (wildcard-import)
augur/tasks/git/dependency_libyear_tasks/tasks.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/git/dependency_libyear_tasks/tasks.py|2| W0401: Wildcard import augur.tasks.git.dependency_libyear_tasks.core (wildcard-import)
augur/tasks/git/dependency_libyear_tasks/tasks.py|7 col 39| W0613: Unused argument 'self' (unused-argument)
augur/tasks/git/dependency_tasks/core.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/git/dependency_tasks/core.py|3| W0401: Wildcard import augur.application.db.models (wildcard-import)
augur/tasks/git/dependency_tasks/core.py|51| W0105: String statement has no effect (pointless-string-statement)
augur/tasks/git/dependency_tasks/core.py|91 col 28| E0602: Undefined variable 'required_output' (undefined-variable)
augur/tasks/git/dependency_tasks/core.py|93 col 11| E0602: Undefined variable 'required_output' (undefined-variable)
augur/tasks/git/dependency_tasks/core.py|113 col 17| E0602: Undefined variable 'required_output' (undefined-variable)
augur/tasks/git/dependency_tasks/core.py|77 col 4| W0612: Unused variable 'command' (unused-variable)
augur/tasks/git/dependency_tasks/core.py|80 col 4| W0612: Unused variable 'path_to_scorecard' (unused-variable)
augur/tasks/git/dependency_tasks/core.py|84 col 26| W0612: Unused variable 'session' (unused-variable)
augur/tasks/git/dependency_tasks/tasks.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/git/dependency_tasks/tasks.py|3| W0401: Wildcard import augur.tasks.git.dependency_tasks.core (wildcard-import)
augur/tasks/git/facade_tasks.py|113| C0325: Unnecessary parens after '=' keyword (superfluous-parens)
augur/tasks/git/facade_tasks.py|185| C0325: Unnecessary parens after '=' keyword (superfluous-parens)
augur/tasks/git/facade_tasks.py|207| C0325: Unnecessary parens after '=' keyword (superfluous-parens)
augur/tasks/git/facade_tasks.py|16| W0401: Wildcard import augur.tasks.github.facade_github.tasks (wildcard-import)
augur/tasks/git/facade_tasks.py|65 col 38| W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)
augur/tasks/git/facade_tasks.py|59 col 37| W0613: Unused argument 'repo_git' (unused-argument)
augur/tasks/git/facade_tasks.py|199 col 7| C1802: Do not use len(SEQUENCE) without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)
augur/tasks/git/facade_tasks.py|205 col 16| W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)
augur/tasks/git/facade_tasks.py|224 col 11| C1802: Do not use len(SEQUENCE) without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)
augur/tasks/git/facade_tasks.py|369 col 4| W0612: Unused variable 'start_date' (unused-variable)
augur/tasks/git/facade_tasks.py|391 col 34| W0613: Unused argument 'logger' (unused-argument)
augur/tasks/git/facade_tasks.py|391 col 51| W0613: Unused argument 'session' (unused-argument)
augur/tasks/git/facade_tasks.py|393 col 4| W0612: Unused variable 'contributor_sequence' (unused-variable)
augur/tasks/git/facade_tasks.py|421 col 4| W0612: Unused variable 'start_date' (unused-variable)
augur/tasks/git/facade_tasks.py|475 col 4| W0612: Unused variable 'delete_marked_repos' (unused-variable)
augur/tasks/git/facade_tasks.py|476 col 4| W0612: Unused variable 'pull_repos' (unused-variable)
augur/tasks/git/facade_tasks.py|478 col 4| W0612: Unused variable 'check_updates' (unused-variable)
augur/tasks/git/facade_tasks.py|480 col 4| W0612: Unused variable 'run_analysis' (unused-variable)
augur/tasks/git/facade_tasks.py|490 col 4| W0612: Unused variable 'create_xlsx_summary_files' (unused-variable)
augur/tasks/git/facade_tasks.py|491 col 4| W0612: Unused variable 'multithreaded' (unused-variable)
augur/tasks/git/scc_value_tasks/core.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/git/scc_value_tasks/core.py|3| W0401: Wildcard import augur.application.db.models (wildcard-import)
augur/tasks/git/scc_value_tasks/tasks.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/git/scc_value_tasks/tasks.py|3| W0401: Wildcard import augur.tasks.git.scc_value_tasks.core (wildcard-import)
augur/tasks/github/contributors/tasks.py|7| W0401: Wildcard import augur.tasks.github.facade_github.tasks (wildcard-import)
augur/tasks/github/contributors/tasks.py|65 col 4| E1120: No value for argument 'natural_keys' in function call (no-value-for-parameter)
augur/tasks/github/contributors/tasks.py|21 col 4| W0612: Unused variable 'tool_version' (unused-variable)
augur/tasks/github/contributors/tasks.py|111 col 4| W0612: Unused variable 'engine' (unused-variable)
augur/tasks/github/detect_move/core.py|29 col 4| W0612: Unused variable 'result' (unused-variable)
augur/tasks/github/detect_move/core.py|47| R1711: Useless return at end of function or method (useless-return)
augur/tasks/github/detect_move/core.py|47 col 62| W0613: Unused argument 'collection_hook' (unused-argument)
augur/tasks/github/detect_move/tasks.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/github/detect_move/tasks.py|3| W0401: Wildcard import augur.tasks.github.detect_move.core (wildcard-import)
augur/tasks/github/events/tasks.py|170 col 5| W0511: TODO: Could replace this with "id" but it isn't stored on the table for some reason (fixme)
augur/tasks/github/events/tasks.py|179 col 1| W0511: TODO: Should we skip an event if there is no contributor to resolve it o (fixme)
augur/tasks/github/events/tasks.py|7| W0401: Wildcard import augur.application.db.data_parse (wildcard-import)
augur/tasks/github/events/tasks.py|75| R0914: Too many local variables (31/30) (too-many-locals)
augur/tasks/github/events/tasks.py|75| R0915: Too many statements (56/50) (too-many-statements)
augur/tasks/github/events/tasks.py|180 col 38| W0613: Unused argument 'logger' (unused-argument)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|12 col 1| W0511: #TODO: maybe have a TaskSession class that holds information about the database, logger, config, etc. (fixme)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|180 col 5| W0511: TODO: It might be better to have the canonical_email allowed to be NUll because right now it has a null constraint. (fixme)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|378 col 5| W0511: TODO: here. (fixme)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|52 col 8| R1723: Unnecessary "elif" after "break", remove the leading "el" from "elif" (no-else-break)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|52 col 11| C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|65 col 13| C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|68 col 13| C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|27| R0912: Too many branches (14/12) (too-many-branches)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|155| R1711: Useless return at end of function or method (useless-return)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|219 col 8| W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)
augur/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py|221| W0105: String statement has no effect (pointless-string-statement)
augur/tasks/github/facade_github/core.py|56| C0325: Unnecessary parens after '=' keyword (superfluous-parens)
augur/tasks/github/facade_github/core.py|4| W0401: Wildcard import augur.tasks.github.util.github_paginator (wildcard-import)
augur/tasks/github/facade_github/core.py|5| W0401: Wildcard import augur.application.db.models (wildcard-import)
augur/tasks/github/facade_github/core.py|125 col 25| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/github/facade_github/core.py|126 col 25| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/github/facade_github/core.py|39 col 4| W0612: Unused variable 'duplicate_col_map' (unused-variable)
augur/tasks/github/facade_github/core.py|63 col 25| W0612: Unused variable 'result' (unused-variable)
augur/tasks/github/facade_github/core.py|131 col 52| W0613: Unused argument 'platform' (unused-argument)
augur/tasks/github/facade_github/tasks.py|9| W0401: Wildcard import augur.tasks.github.facade_github.core (wildcard-import)
augur/tasks/github/facade_github/tasks.py|11| W0401: Wildcard import augur.tasks.git.util.facade_worker.facade_worker.facade00mainprogram (wildcard-import)
augur/tasks/github/facade_github/tasks.py|56 col 11| R1714: Consider merging these comparisons with 'in' by using 'login in (None, '')'. Use a set instead if elements are hashable. (consider-using-in)
augur/tasks/github/facade_github/tasks.py|56 col 11| C0121: Comparison 'login == None' should be 'login is None' (singleton-comparison)
augur/tasks/github/facade_github/tasks.py|59 col 11| R1714: Consider merging these comparisons with 'in' by using 'login in (None, '')'. Use a set instead if elements are hashable. (consider-using-in)
augur/tasks/github/facade_github/tasks.py|59 col 11| C0121: Comparison 'login == None' should be 'login is None' (singleton-comparison)
augur/tasks/github/facade_github/tasks.py|64 col 11| R1714: Consider merging these comparisons with 'in' by using 'login in (None, '')'. Use a set instead if elements are hashable. (consider-using-in)
augur/tasks/github/facade_github/tasks.py|64 col 11| C0121: Comparison 'login == None' should be 'login is None' (singleton-comparison)
augur/tasks/github/facade_github/tasks.py|14| R0915: Too many statements (52/50) (too-many-statements)
augur/tasks/github/facade_github/tasks.py|168| R1711: Useless return at end of function or method (useless-return)
augur/tasks/github/facade_github/tasks.py|207 col 4| W1202: Use lazy % formatting in logging functions (logging-format-interpolation)
augur/tasks/github/facade_github/tasks.py|207 col 4| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/github/facade_github/tasks.py|193| R1711: Useless return at end of function or method (useless-return)
augur/tasks/github/facade_github/tasks.py|193 col 37| W0613: Unused argument 'repo_git' (unused-argument)
augur/tasks/github/facade_github/tasks.py|198 col 4| W0612: Unused variable 'engine' (unused-variable)
augur/tasks/github/issues/tasks.py|9| W0401: Wildcard import augur.application.db.data_parse (wildcard-import)
augur/tasks/github/issues/tasks.py|37 col 8| R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
augur/tasks/github/issues/tasks.py|86| R0914: Too many local variables (32/30) (too-many-locals)
augur/tasks/github/messages/tasks.py|6| W0401: Wildcard import augur.application.db.data_parse (wildcard-import)
augur/tasks/github/messages/tasks.py|27 col 12| E0602: Undefined variable 'Repo' (undefined-variable)
augur/tasks/github/messages/tasks.py|133 col 8| E1121: Too many positional arguments for function call (too-many-function-args)
augur/tasks/github/messages/tasks.py|136| R0914: Too many local variables (38/30) (too-many-locals)
augur/tasks/github/messages/tasks.py|136| R0912: Too many branches (18/12) (too-many-branches)
augur/tasks/github/messages/tasks.py|136| R0915: Too many statements (78/50) (too-many-statements)
augur/tasks/github/pull_requests/commits_model/core.py|3| W0401: Wildcard import augur.application.db.models (wildcard-import)
augur/tasks/github/pull_requests/commits_model/tasks.py|2| W0401: Wildcard import augur.tasks.github.pull_requests.commits_model.core (wildcard-import)
augur/tasks/github/pull_requests/core.py|5| W0401: Wildcard import augur.application.db.data_parse (wildcard-import)
augur/tasks/github/pull_requests/core.py|227 col 4| E1120: No value for argument 'natural_keys' in function call (no-value-for-parameter)
augur/tasks/github/pull_requests/core.py|215 col 41| W0613: Unused argument 'logger' (unused-argument)
augur/tasks/github/pull_requests/core.py|257 col 4| E1120: No value for argument 'natural_keys' in function call (no-value-for-parameter)
augur/tasks/github/pull_requests/core.py|245 col 47| W0613: Unused argument 'logger' (unused-argument)
augur/tasks/github/pull_requests/files_model/core.py|3| W0401: Wildcard import augur.application.db.models (wildcard-import)
augur/tasks/github/pull_requests/files_model/tasks.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/github/pull_requests/files_model/tasks.py|2| W0401: Wildcard import augur.tasks.github.pull_requests.files_model.core (wildcard-import)
augur/tasks/github/pull_requests/files_model/tasks.py|13 col 4| W0612: Unused variable 'engine' (unused-variable)
augur/tasks/github/pull_requests/tasks.py|6| W0401: Wildcard import augur.application.db.data_parse (wildcard-import)
augur/tasks/github/pull_requests/tasks.py|86| R0914: Too many local variables (31/30) (too-many-locals)
augur/tasks/github/pull_requests/tasks.py|268 col 8| W0612: Unused variable 'index' (unused-variable)
augur/tasks/github/pull_requests/tasks.py|374 col 4| C0206: Consider iterating with .items() (consider-using-dict-items)
augur/tasks/github/pull_requests/tasks.py|374 col 27| C0201: Consider iterating the dictionary directly instead of calling .keys() (consider-iterating-dictionary)
augur/tasks/github/pull_requests/tasks.py|387 col 4| C0206: Consider iterating with .items() (consider-using-dict-items)
augur/tasks/github/pull_requests/tasks.py|387 col 27| C0201: Consider iterating the dictionary directly instead of calling .keys() (consider-iterating-dictionary)
augur/tasks/github/pull_requests/tasks.py|356 col 23| W0612: Unused variable 'page' (unused-variable)
augur/tasks/github/releases/core.py|67| R1711: Useless return at end of function or method (useless-return)
augur/tasks/github/releases/core.py|187 col 32| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/github/releases/core.py|200 col 40| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/github/releases/core.py|170| R0912: Too many branches (16/12) (too-many-branches)
augur/tasks/github/releases/tasks.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/github/releases/tasks.py|3| W0401: Wildcard import augur.tasks.github.releases.core (wildcard-import)
augur/tasks/github/repo_info/core.py|8| W0401: Wildcard import augur.application.db.models (wildcard-import)
augur/tasks/github/repo_info/core.py|10| W0401: Wildcard import augur.tasks.github.util.github_task_session (wildcard-import)
augur/tasks/github/repo_info/tasks.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/github/repo_info/tasks.py|4| W0401: Wildcard import augur.tasks.github.repo_info.core (wildcard-import)
augur/tasks/github/traffic/tasks.py|30 col 8| E1120: No value for argument 'logger' in function call (no-value-for-parameter)
augur/tasks/github/traffic/tasks.py|65 col 25| E1121: Too many positional arguments for function call (too-many-function-args)
augur/tasks/github/util/github_api_key_handler.py|10| C0412: Imports from package sqlalchemy are not grouped (ungrouped-imports)
augur/tasks/gitlab/issues_task.py|39 col 8| R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
augur/tasks/gitlab/issues_task.py|86| R0914: Too many local variables (31/30) (too-many-locals)
augur/tasks/gitlab/issues_task.py|86| R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
augur/tasks/gitlab/merge_request_task.py|401 col 5| W0511: TODO: Need to add unique key with pull_request_id and cntrb_id to insert gitlab reviewers (fixme)
augur/tasks/gitlab/merge_request_task.py|35 col 4| R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
augur/tasks/gitlab/merge_request_task.py|182 col 10| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|293 col 10| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|358 col 10| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|425 col 10| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|493 col 10| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/start_tasks.py|327 col 5| W0511: TODO: Isaac needs to normalize the status's to be abstract in the (fixme)
augur/tasks/start_tasks.py|27| C0413: Import "from augur.tasks.util.collection_state import CollectionState" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|28| W0401: Wildcard import augur.tasks.util.collection_util (wildcard-import)
augur/tasks/start_tasks.py|28| C0413: Import "from augur.tasks.util.collection_util import *" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|29| C0413: Import "from augur.tasks.git.util.facade_worker.facade_worker.utilitymethods import get_facade_weight_time_factor" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|30| C0413: Import "from augur.application.db.lib import execute_sql, get_session" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|157 col 68| W0613: Unused argument 'days_until_collect_again' (unused-argument)
augur/tasks/start_tasks.py|180 col 70| W0613: Unused argument 'days_until_collect_again' (unused-argument)
augur/tasks/start_tasks.py|200 col 46| W0613: Unused argument 'enabled_phase_names' (unused-argument)
augur/tasks/start_tasks.py|200 col 67| W0613: Unused argument 'days_until_collect_again' (unused-argument)
augur/tasks/start_tasks.py|221 col 42| W0613: Unused argument 'enabled_phase_names' (unused-argument)
augur/tasks/start_tasks.py|221 col 63| W0613: Unused argument 'days_until_collect_again' (unused-argument)
augur/tasks/start_tasks.py|285 col 74| C0121: Comparison 'CollectionStatus.core_weight != None' should be 'CollectionStatus.core_weight is not None' (singleton-comparison)
augur/tasks/start_tasks.py|362 col 4| W0612: Unused variable 'engine' (unused-variable)
augur/tasks/util/collection_util.py|304 col 30| R1704: Redefining argument with the local name 'session' (redefined-argument-from-local)
augur/tasks/util/collection_util.py|408 col 4| W0612: Unused variable 'engine' (unused-variable)
augur/tasks/util/collection_util.py|591 col 8| W0612: Unused variable 'augur_collection_list' (unused-variable)
tests/test_tasks/test_task_utlities/test_key_handler/test_github_api_key_handler.py|28 col 24| W0621: Redefining name 'key_handler' from outer scope (line 22) (redefined-outer-name)

github-actions · 2024-06-03T23:33:12Z

augur/application/config.py

@@ -288,6 +288,7 @@ def add_or_update_settings(self, settings: List[dict]):
            query = self.session.query(Config).filter(and_(Config.section_name == setting["section_name"],Config.setting_name == setting["setting_name"]) )

            if execute_session_query(query, 'first') is None:
+                # TODO: Update to use bulk insert dicts so config doesn't require database session


[pylint] _{reported by reviewdog 🐶}
W0511: TODO: Update to use bulk insert dicts so config doesn't require database session (fixme)

github-actions · 2024-06-03T23:33:12Z

augur/application/db/lib.py

@@ -1,9 +1,19 @@
-import sqlalchemy as s
+import time


[pylint] _{reported by reviewdog 🐶}
C0114: Missing module docstring (missing-module-docstring)

github-actions · 2024-06-03T23:33:12Z

augur/application/db/lib.py

+
+    try:
+        working_commits = fetchall_data_from_sql_text(query)
+    except:


[pylint] _{reported by reviewdog 🐶}
W0702: No exception type(s) specified (bare-except)

github-actions · 2024-06-03T23:33:12Z

augur/application/db/lib.py

+        return session.query(CollectionStatus).filter(getattr(CollectionStatus,f"{collection_type}_status" ) == CollectionState.COLLECTING.value).count()
+
+
+def facade_bulk_insert_commits(logger, records):


[pylint] _{reported by reviewdog 🐶}
W0621: Redefining name 'logger' from outer scope (line 18) (redefined-outer-name)

github-actions · 2024-06-03T23:33:12Z

augur/application/db/lib.py

+                firsthalfRecords = records[:len(records)//2]
+                secondhalfRecords = records[len(records)//2:]
+
+                facade_bulk_insert_commits(logger, session,firsthalfRecords)


[pylint] _{reported by reviewdog 🐶}
E1121: Too many positional arguments for function call (too-many-function-args)

github-actions · 2024-06-03T23:33:14Z

augur/tasks/data_analysis/insight_worker/tasks.py



-def insight_model(repo_git: str,logger,engine,session) -> None:
+def insight_model(repo_git: str,logger,engine) -> None:


[pylint] _{reported by reviewdog 🐶}
R0912: Too many branches (15/12) (too-many-branches)

github-actions · 2024-06-03T23:33:14Z

augur/tasks/data_analysis/insight_worker/tasks.py



-def insight_model(repo_git: str,logger,engine,session) -> None:
+def insight_model(repo_git: str,logger,engine) -> None:


[pylint] _{reported by reviewdog 🐶}
R0915: Too many statements (128/50) (too-many-statements)

github-actions · 2024-06-03T23:33:14Z

augur/tasks/data_analysis/insight_worker/tasks.py



-def insight_model(repo_git: str,logger,engine,session) -> None:
+def insight_model(repo_git: str,logger,engine) -> None:
    refresh = True


[pylint] _{reported by reviewdog 🐶}
W0612: Unused variable 'refresh' (unused-variable)

github-actions · 2024-06-03T23:33:14Z

augur/tasks/data_analysis/message_insights/tasks.py




-def message_insight_model(repo_git: str,logger,engine, session) -> None:
+def message_insight_model(repo_git: str,logger,engine) -> None:


[pylint] _{reported by reviewdog 🐶}
R0914: Too many local variables (56/30) (too-many-locals)

github-actions · 2024-06-03T23:33:14Z

augur/tasks/data_analysis/message_insights/tasks.py




-def message_insight_model(repo_git: str,logger,engine, session) -> None:
+def message_insight_model(repo_git: str,logger,engine) -> None:


[pylint] _{reported by reviewdog 🐶}
R0912: Too many branches (20/12) (too-many-branches)

sgoggins

Lets test it!

ABrain7710 added 30 commits April 1, 2024 18:24

Remove facade dependency on GithubTaskSession

c010c2c

Signed-off-by: Andrew Brain <[email protected]>

remove facade session dependence on AugurConfig

003ed25

Signed-off-by: Andrew Brain <[email protected]>

Define database lib methods and use in commit analysis

cc42353

Signed-off-by: Andrew Brain <[email protected]>

Update trim commits to use db lib

a070e71

Signed-off-by: Andrew Brain <[email protected]>

Reduce dependency on session logger

32af57e

Signed-off-by: Andrew Brain <[email protected]>

Update facade session to not depend on database session

812e635

Signed-off-by: Andrew Brain <[email protected]>

Rename FacadeSession to FacadeHelper

15d788b

Signed-off-by: Andrew Brain <[email protected]>

Add repo methods to lib

3258568

Signed-off-by: Andrew Brain <[email protected]>

Use get repo git

b31baf3

Signed-off-by: Andrew Brain <[email protected]>

Add more facade lib methods

5781e90

Signed-off-by: Andrew Brain <[email protected]>

Move facade insert commits method to lib

0dcbb91

Signed-off-by: Andrew Brain <[email protected]>

Define bulk insert dicts to remove database session dependence

25638bd

Signed-off-by: Andrew Brain <[email protected]>

Reduce dependence on database session using lib methods

b1ee4bd

Signed-off-by: Andrew Brain <[email protected]>

Fix bulk upsert syntax issues

3772c23

Signed-off-by: Andrew Brain <[email protected]>

Clean up lib year task session management

995fa6e

Signed-off-by: Andrew Brain <[email protected]>

Clean up db usage in dependency tasks

a4602c5

Signed-off-by: Andrew Brain <[email protected]>

Clean up value model db usage

688cee2

Signed-off-by: Andrew Brain <[email protected]>

Update value task with new db stuff

9a62832

Signed-off-by: Andrew Brain <[email protected]>

Update repofetch to use new method

d982c44

Signed-off-by: Andrew Brain <[email protected]>

Facade db improvements

ee833a1

Signed-off-by: Andrew Brain <[email protected]>

Imporve database handling in ml workers

7f35800

Signed-off-by: Andrew Brain <[email protected]>

Remove github key handling dependence from a session

e5b001c

Signed-off-by: Andrew Brain <[email protected]>

Remove git repo clean up

6c8d590

Signed-off-by: Andrew Brain <[email protected]>

Decouple gitlab api key handler from session

2ff53b1

Signed-off-by: Andrew Brain <[email protected]>

Fix issues in branch

62ac0be

Signed-off-by: Andrew Brain <[email protected]>

Merge branch 'dev' into database-improvements

3959056

Start reducing query usage in github tasks

effdd41

Signed-off-by: Andrew Brain <[email protected]>

Continue improving database handling in github tasks

78368a5

Signed-off-by: Andrew Brain <[email protected]>

Simplify facade github logic

93d5a29

Signed-off-by: Andrew Brain <[email protected]>

Reduce dependence on task manifest db

7a3a50d

Signed-off-by: Andrew Brain <[email protected]>

ABrain7710 added 22 commits April 13, 2024 09:55

Remove references to execute_sql and insert_data

9406688

Signed-off-by: Andrew Brain <[email protected]>

Remove references of session.logger

c5687e7

Signed-off-by: Andrew Brain <[email protected]>

Reduce usage of GithubTaskManifest where only key auth is needed

7683067

Signed-off-by: Andrew Brain <[email protected]>

Use get_repo_by_repo_git in gitlab tasks

3e8a6e5

Signed-off-by: Andrew Brain <[email protected]>

Database improvements

7196b78

Signed-off-by: Andrew Brain <[email protected]>

get good changes

d02c440

Merge remote-tracking branch 'origin/database-improvements-2' into da…

8a32e62

…tabase-improvements-2

Fix circular import

4d0baaa

Signed-off-by: Andrew Brain <[email protected]>

Reduce use of github task manifest

6df1bbf

Signed-off-by: Andrew Brain <[email protected]>

Reduce use of GithubTaskManifest

4f50ffa

Signed-off-by: Andrew Brain <[email protected]>

Reduce usage of github task manifest

674a5a7

Signed-off-by: Andrew Brain <[email protected]>

Continue decoupling code from github task manifest

8e29c8e

Signed-off-by: Andrew Brain <[email protected]>

Remove more references to session in domain

cce031d

Signed-off-by: Andrew Brain <[email protected]>

Remove more database

25b6adc

Signed-off-by: Andrew Brain <[email protected]>

Remove references to github task manifest

2153305

Signed-off-by: Andrew Brain <[email protected]>

Continue reducing instances of DatabaseSession

8878959

Signed-off-by: Andrew Brain <[email protected]>

Remove GitlabTaskManifest

59d5ee4

Signed-off-by: Andrew Brain <[email protected]>

Fix some syntax errors and comments

a3a5406

Signed-off-by: Andrew Brain <[email protected]>

Fix multiprocessing issue

1495ef2

Signed-off-by: Andrew Brain <[email protected]>

Merge branch 'main' into database-improvements

7a24ffc

Merge branch 'dev' into database-improvements

f2f7b87

Fix syntax error

511593b

Signed-off-by: Andrew Brain <[email protected]>

ABrain7710 requested a review from sgoggins as a code owner June 3, 2024 23:19

Merge branch 'dev' into database-improvements

12631f6

github-actions bot reviewed Jun 3, 2024

View reviewed changes

ABrain7710 requested a review from IsaacMilarky June 3, 2024 23:35

sgoggins approved these changes Jun 3, 2024

View reviewed changes

IsaacMilarky approved these changes Jun 3, 2024

View reviewed changes

sgoggins merged commit 7ec8c35 into dev Jun 3, 2024
6 of 7 checks passed

sgoggins deleted the database-improvements branch June 11, 2024 16:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Database improvements #2809

Database improvements #2809

ABrain7710 commented Jun 3, 2024

github-actions bot left a comment

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

github-actions bot Jun 3, 2024

sgoggins left a comment

		return session.query(CollectionStatus).filter(getattr(CollectionStatus,f"{collection_type}_status" ) == CollectionState.COLLECTING.value).count()


		def facade_bulk_insert_commits(logger, records):



		def insight_model(repo_git: str,logger,engine,session) -> None:
		def insight_model(repo_git: str,logger,engine) -> None:




		def message_insight_model(repo_git: str,logger,engine, session) -> None:
		def message_insight_model(repo_git: str,logger,engine) -> None:

Database improvements #2809

Database improvements #2809

Conversation

ABrain7710 commented Jun 3, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

github-actions bot Jun 3, 2024

Choose a reason for hiding this comment

sgoggins left a comment

Choose a reason for hiding this comment