Jumpstart #2902

Ulincsys · 2024-08-27T22:23:55Z

Description

Add jumpstart subroutine for more robust process management
Fix logging inconsistencies
Fix augur api start and augur collection start commands
Add Textual dependency

In preparation for changes coming in the Admin Console release, it is desirable to have a proper "process manager" in place that can communicate with external clients and control Augur processes from a central perspective. This change enables more granular control over Augur's various worker types, and in the future more features are possible.

Signed commits

Yes, I signed my commits.

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/api.py

@@ -14,15 +14,16 @@

 from augur.application.db.session import DatabaseSession


[pylint] _{reported by reviewdog 🐶}
W0611: Unused DatabaseSession imported from augur.application.db.session (unused-import)

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/backend.py

+    """
+    Stop collection tasks if they are running, block until complete
+    """
+    processes = get_augur_processes()


[pylint] _{reported by reviewdog 🐶}
W0621: Redefining name 'processes' from outer scope (line 443) (redefined-outer-name)

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/backend.py

+            stopped.append(p)
+            p.terminate()
+
+    if not len(stopped):


[pylint] _{reported by reviewdog 🐶}
C1802: Do not use len(SEQUENCE) without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/backend.py

+
+    killed = []
+    while True:
+        for i in range(len(alive)):


[pylint] _{reported by reviewdog 🐶}
C0200: Consider using enumerate instead of iterating with range and len (consider-using-enumerate)

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/backend.py

+        for i in reversed(killed):
+            alive.pop(i)
+
+        if not len(alive):


[pylint] _{reported by reviewdog 🐶}
C1802: Do not use len(SEQUENCE) without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)

github-actions · 2024-08-27T22:24:32Z

jumpstart/tui.py

+            if command["name"] == cmd:
+                packet["cmd"] = cmd
+                for arg in command["args"]:
+                    if arg["required"] and not len(args):


[pylint] _{reported by reviewdog 🐶}
C1802: Do not use len(SEQUENCE) without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)

github-actions · 2024-08-27T22:24:32Z

jumpstart/tui.py

+
+        for command in self.spec["commands"]:
+            help_str += f'\t> [bold yellow]{command["name"]}'
+            for arg in command["args"]:


[pylint] _{reported by reviewdog 🐶}
W0621: Redefining name 'arg' from outer scope (line 388) (redefined-outer-name)

github-actions · 2024-08-27T22:24:32Z

jumpstart/tui.py

+                log(e)
+                log(msg)
+
+def run_app(socket_file = Path("jumpstart.sock"), ctx = None):


[pylint] _{reported by reviewdog 🐶}
W0621: Redefining name 'socket_file' from outer scope (line 387) (redefined-outer-name)

github-actions · 2024-08-27T22:24:32Z

jumpstart/tui.py

+
+import json
+import asyncio
+import logging


[pylint] _{reported by reviewdog 🐶}
W0611: Unused import logging (unused-import)

github-actions · 2024-08-27T22:24:33Z

jumpstart/tui.py

+from textual.app import App, on
+from textual.binding import Binding
+from textual.message import Message
+from textual.events import Load, Mount


[pylint] _{reported by reviewdog 🐶}
W0611: Unused Load imported from textual.events (unused-import)

sgoggins

I think @ABrain7710 needs to check the backend.py edits.

sgoggins · 2024-08-27T22:26:22Z

augur/api/view/init.py

@@ -91,4 +91,4 @@ def write_settings(current_settings):
 # Initialize logging
 def init_logging():
    global logger
-    logger = AugurLogger("augur_view", reset_logfiles=True).get_logger()
+    logger = AugurLogger("augur_view", reset_logfiles=False).get_logger()


Does this only affect view?

No, it previously would reset all log files globally. So now the log files will continue to exist after starting Augur

sgoggins · 2024-08-27T22:28:51Z

augur/application/cli/collection.py

@@ -22,14 +22,15 @@
 from augur.application.db.session import DatabaseSession
 from augur.application.logs import AugurLogger
 from augur.application.db.lib import get_value
-from augur.application.cli import test_connection, test_db_connection, with_database
+from augur.application.cli import test_connection, test_db_connection, with_database, DatabaseContext


Is this one just incorporating the latest from @ABrain7710 ?

No, the changes are unrelated

Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

…g to understand event API limits Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

…in the data returned from the events API. It DOES look like what is returned may be a partially complte set in some cases. Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

…taException class (I think) Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

…e to the API Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

…empty repos in repo_info Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Traceback (most recent call last): File '/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/billiard/pool.py', line 366, in workloop put((READY, (job, i, result, inqW_fd))) File '/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/billiard/queues.py', line 366, in put self.send_payload(ForkingPickler.dumps(obj)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File '/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/billiard/reduction.py', line 56, in dumps cls(buf, protocol).dump(obj) billiard.pool.MaybeEncodingError: Error sending result: ''(1, <ExceptionInfo: MetadataException('\'checks\' | Additional metadata: required_output: {}')>, None)''. Reason: ''PicklingError('Can\'t pickle <class \'augur.tasks.util.metadata_exception.MetadataException\'>: it\'s not the same object as augur.tasks.util.metadata_exception.MetadataException')''. Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Ulincsys <[email protected]>

Signed-off-by: Sean P. Goggins <[email protected]>

Ulincsys requested a review from sgoggins as a code owner August 27, 2024 22:23

github-actions bot reviewed Aug 27, 2024

View reviewed changes

sgoggins previously approved these changes Aug 27, 2024

View reviewed changes

Ulincsys dismissed sgoggins’s stale review via c2e128d August 27, 2024 23:03

Ulincsys and others added 26 commits August 27, 2024 18:06

Pull relevant mods from admin-changes

36ae56c

Signed-off-by: Ulincsys <[email protected]>

initial work on jumpstart project

f0e27a3

Signed-off-by: Ulincsys <[email protected]>

Implement decoding and blocking

709e771

Signed-off-by: Ulincsys <[email protected]>

add TUI, expand server basis

d7c6a87

Signed-off-by: Ulincsys <[email protected]>

update encoding

3bac827

Signed-off-by: Ulincsys <[email protected]>

continue work on concurrency

cf6c9ec

Signed-off-by: Ulincsys <[email protected]>

mvp

121ee03

Signed-off-by: Ulincsys <[email protected]>

Fix improper client shutdown on some platforms

761cc50

Signed-off-by: Ulincsys <[email protected]>

Add DB check before parsing start command

e920890

Signed-off-by: Ulincsys <[email protected]>

Add github graphql data access'

2067d44

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Improve paginate_resource logic

140ea06

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Remove unused method

5577ce0

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Move repo_info to new graphql endpoint

06d2c06

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Migrate pr files to github_graphql_data_access

fde0ec4

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Indent client'

06bc536

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Remove strip

2e05618

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Define keys as list

ec71977

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Return data properly

034333e

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Add self

93577d7

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Fix syntax error

078bc8a

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Fix syntax error

358b146

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Fixes

a96b380

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Raise from original exception

97af39b

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Make annoying info log a debug

ecb7b57

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

updating version

46438df

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Address release bugs

f3cb92a

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

ABrain7710 and others added 26 commits August 27, 2024 18:06

Handle url not found in contributor breadth worker

cf99b23

Signed-off-by: Andrew Brain <[email protected]> Signed-off-by: Ulincsys <[email protected]>

docker fun

44b1c62

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

failed docker config

456b0f5

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

saving failed config

07c2c38

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Docker compose script now workinggit add docker/

f0cad14

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

removed old file

6a6ba73

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

debugging a couple errors. One with a missing import, the other tryin…

56808aa

…g to understand event API limits Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

possible fix for commit messages

ca0d4b2

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

changed page_count > 300 to a warning instead of an exception

f512443

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

updated unterminated string error

457b03d

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Fixing NPM

d7d435b

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

events page count seems to top out at 1,000, not 300

21660e1

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

update to events to use correct column name

e665aec

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

update to augur event collection

b80a480

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

logging event so we know what's supposed to be there

c35f449

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

fixing pickling error caused by not consistently importing our Metada…

de320b2

…taException class (I think) Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

additional update for when issue is not an object in an event respons…

dbdb7e3

…e to the API Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Trying to fix issue when the committer count API returns no value on …

4c45d8f

…empty repos in repo_info Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

exception handling

ed50be2

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

trying to fix the empty committer count issue

f510da4

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

added some comments explaining the query_committers_count method changes

82b9159

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

Making repo GONE log message more clear.

8967212

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

formatting exception for OpenSSF Scorecard pickling error more typically

97fee7e

Signed-off-by: Sean P. Goggins <[email protected]> Signed-off-by: Ulincsys <[email protected]>

fix for missing comma, fix misspelling

ebca962

Signed-off-by: Ulincsys <[email protected]>

Ulincsys force-pushed the jumpstart branch from c2e128d to ebca962 Compare August 27, 2024 23:06

Merge branch 'dev' into jumpstart

491b688

Signed-off-by: Sean P. Goggins <[email protected]>

sgoggins merged commit aae2f19 into dev Aug 27, 2024
6 of 7 checks passed

Ulincsys deleted the jumpstart branch August 27, 2024 23:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jumpstart #2902

Jumpstart #2902

Ulincsys commented Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

sgoggins left a comment

sgoggins Aug 27, 2024

Ulincsys Aug 27, 2024

sgoggins Aug 27, 2024

Ulincsys Aug 27, 2024

		@@ -14,15 +14,16 @@

		from augur.application.db.session import DatabaseSession

Jumpstart #2902

Jumpstart #2902

Conversation

Ulincsys commented Aug 27, 2024

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

sgoggins left a comment

Choose a reason for hiding this comment

sgoggins Aug 27, 2024

Choose a reason for hiding this comment

Ulincsys Aug 27, 2024

Choose a reason for hiding this comment

sgoggins Aug 27, 2024

Choose a reason for hiding this comment

Ulincsys Aug 27, 2024

Choose a reason for hiding this comment