PowerMon/StructuredLogging support #607

geeksville · 2024-06-22T17:29:06Z

Mostly done but I'd like to request review/merge soonish (because I've got too many open branches with changes and tire of merging ;-) ) - see bottom comment.

Fixes #603

CLAassistant · 2024-06-22T17:34:27Z

All committers have signed the CLA.

CLAassistant · 2024-06-22T17:34:27Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ geeksville
❌ github-actions

github-actions seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

geeksville · 2024-06-22T18:31:13Z

@ianmcorvidae and anyone else who is appropriate: How critical is it to keep python 3.8 support? I'd like to use the pandas data processing library for the 'analysis' apis/reporting/visualization (as opposed to device APIs). But that lib requires min python of 3.9.

I've checked and even my raspberry pi could do that (raspbian is on 3.11 now).

I would kinda rather not making a new 'meshtastic-analysis' python project. instead I'd like to have some new classes/code in meshtastic.analysis of this project.

thebentern · 2024-06-22T18:49:46Z

Python 3.8 is 5 years old now. I think that's fair to dump it, honestly.

ianmcorvidae · 2024-06-22T19:08:48Z

Yeah -- as I was saying in discord the other day, 3.8 is hitting formal end of life in October anyway, so I'm happy to drop support for it as part of this. Once we do so, I'll make sure to create one last 3.8-supporting version that'll warn users about it, just in case there's folks lingering on the old version.

geeksville · 2024-06-22T19:13:33Z

okay coolbeans @ianmcorvidae - I'm going to make this powermon/structured logging branch assume/require 3.8 and it can go in after that release.

…match

# Conflicts: # poetry.lock # pyproject.toml

…in works # Conflicts: # bin/regen-protobufs.sh

Including in the Poetry changes because it touches the same lines and I want to avoid hand merging ;-)

…ports. Initially only RAK4631 and heltec tracker are listed

(backwards/forwards compatible) with old firmware

# Conflicts: # poetry.lock # pyproject.toml

# Conflicts: # .vscode/launch.json # meshtastic/protobuf/config_pb2.py # meshtastic/protobuf/mesh_pb2.py # meshtastic/protobuf/mesh_pb2.pyi

# Conflicts: # .vscode/launch.json

ianmcorvidae · 2024-07-02T03:14:31Z

I hear from another PR that you're out hiking so no rush in responding here 🙂, but:

I was wondering if you'd be fine with me grabbing your LogRecord work from here after #627 goes in and amending both the code here and that code to send a consistent pubsub message that includes the full LogRecord so we can handle them consistently (and make the log lines available to library users, too). I don't want to introduce unnecessary conflicts for you to have to undo but given that stuff's making it into firmware it might be nice to get it in sooner/before the PowerMon stuff is finished. Thoughts?

# Conflicts: # meshtastic/ble_interface.py # meshtastic/protobuf/mesh_pb2.py # meshtastic/protobuf/powermon_pb2.py # meshtastic/protobuf/powermon_pb2.pyi

geeksville · 2024-07-03T16:24:04Z

Hi hi,

Does this change look okay for ya'll to go in? I'd kinda like for this to to in now even though I still have a few more changes coming (just localized to the new PowerStress class). The runtime impact of powermon & powerstress is a small amount of CODE size and a tiny about of RAM. If the corresponding config variable is not set, the code change at runtime is minimal.

If modules are being excluded it is completely removed from the build.

I'm submitting for review now because I had to touch a fair # of files and I've got too many open WIP branches ;-).

For early docs see meshtastic/firmware#4136. However after I'm finished with this feature (a weekish?) I'll move much of that text to become a new doc page and tutorial about how to do power measurements on boards. The main remaining work is not on the firmware or python client side but rather the analysis reporting chain (which in in a new meshtastic.analysis submodule in the python code).

(note: the linter test for min/max is buggy so disabled)

geeksville · 2024-07-03T17:47:25Z

Hi @ianmcorvidae Do you have any advice on how to make the code coverage tool happy with this? These new classes are mostly about hardware but I'm happy to add new meshtastic/tests that use my SimPowerMeter. Are the existing tests mocked enough that I can use a MeshInterface to talk to something and get through the config message flow / reach the later PowerStress stuff?

I assume I'll need to somehow mock up the PowerStress module/agent that is running on the (sim) UUT.

Can I sign up to do this 'later'? ;-) Because I kinda dread doing a bunch of mocking on PowerStress and then having to keep updating that mocking while I'm still developing/changing the feature.

ianmcorvidae · 2024-07-03T17:56:05Z

I'll try to give this a review later tonight or tomorrow and see if we can just get it in. Then I can do some reorganization of log stuff anyway, make sure this and BLE are consistent and the new stuff is accessible to library users without too much pain.

As far as the code coverage, yeah, signing up to do it later is fine with me; our mocks and stuff are not in a terribly convenient place and this power stuff is still in flux even if we merge this now. #579 (if it can get done, since I haven't had much time to devote to planning it) would hopefully make testing in general easier as well, fingers crossed.

ianmcorvidae

Looks generally good to me. Got a few questions, particularly about potentially passing LogRecords in full to pubsub instead of just the messages, so we can be a little more consistent with that stuff. I don't know as much about the actual power monitoring stuff, but as far as I'm aware it seems fine, and I know you're planning to keep updating/changing that, so I think we can let that happen and make any other changes we need to after it settles.

Since this will drop our 3.8 support and also since it depends on 2.3.15 protobufs, I'll make a release before merging it that's 2.3.14 and mark it as the last 3.8 release.

ianmcorvidae · 2024-07-04T01:23:27Z

.vscode/launch.json

@@ -6,11 +6,11 @@
  "configurations": [
    {
      "name": "meshtastic BLE",
-      "type": "python",


(comment at random line in file)

I know basically nothing about vscode so I'm just gonna trust that these settings make sense. If someone else who actually uses vscode wants to validate, feel free.

yah - these are just vscode warned me that "python" is deprecated and the new recommended string is pydebug...

ianmcorvidae · 2024-07-04T01:26:58Z

meshtastic/__init__.py

@@ -31,6 +31,7 @@
 - meshtastic.receive.user(packet)
 - meshtastic.receive.data.portnum(packet) (where portnum is an integer or well known PortNum enum)
 - meshtastic.node.updated(node = NodeInfo) - published when a node in the DB changes (appears, location changed, username changed, etc...)
+- meshtastic.log.line(line) - a raw unparsed log line from the radio


I was mentioning before, I think I might change this, if that's okay (would do after merge, and I'd update the spot you've used it as well) -- I'd be changing it to pass the whole LogRecord as a dict, so that folks can pull out the message, level, and time/source (if those are provided) separately and do with them as they will. This will also mean I can have ble_interface output this same message and move all the log printing/colorizing stuff to one place, hopefully.

yes - I agree, LogRecord is what should be passed (using the automatic KnownProtocol translation). IMO coloring should be done at the end and only for the stdout printer when it is going to an interactive terminal.

ianmcorvidae · 2024-07-04T01:34:42Z

meshtastic/mesh_interface.py

+    def _printLogLine(line, interface):
+        """Print a line of log output."""
+        if interface.debugOut == sys.stdout:
+            # this isn't quite correct (could cause false positives), but currently our formatting differs between different log representations


I think we can fix this by passing the whole LogRecord down to here and then looking at the log_record.DEBUG etc. as the BLE code was doing -- I'm happy to change that around but was wondering if there's a reason you didn't.

yeah - I did this as a temporary hack kinda expecting that LogRecords are coming and the location for coloring will be moving. (I was just merging in recent changes from master and trying to keep em working)

ianmcorvidae · 2024-07-04T01:38:36Z

meshtastic/protobuf/config_pb2.pyi

@@ -56,6 +56,7 @@ class Config(google.protobuf.message.Message):
            ROUTER_CLIENT: Config.DeviceConfig._Role.ValueType  # 3
            """
            Description: Combination of both ROUTER and CLIENT. Not for mobile devices.
+            Deprecated in v2.3.15 because improper usage is impacting public meshes: Use ROUTER or CLIENT instead.


Presumably you needed the v2.3.15 protobufs for this, given that's where the powermon stuff gets merged? I suppose I'll use 2.3.14 as a final-version-that-supports-3.8, I think, before merging this, and then I can just pull in protobufs v2.3.15 outright.

no idea - that string came from meshtastic/protobufs and I just merged it with my changes.

ianmcorvidae · 2024-07-04T01:40:20Z

meshtastic/slog/slog.py

+        )
+
+        # We need a closure here because the subscription API is very strict about exact arg matching
+        def listen_glue(line, interface):  # pylint: disable=unused-argument


(mostly noting to myself -- another spot that'll need updating in order to pass a LogRecord instead of just the message via pubsub)

ianmcorvidae · 2024-07-04T01:43:26Z

meshtastic/stream_interface.py

+        if utf == "\r":
+            pass    # ignore
+        elif utf == "\n":
+            self._handleLogLine(self.cur_log_line)


I guess this is where we're handling log stuff that doesn't have a full LogRecord attached to it. I guess maybe the thing to do will just be to construct one with an "unset" log level. Over time more of this will move into proper protobufs anyway, I suppose

ianmcorvidae · 2024-07-04T01:45:14Z

meshtastic/util.py


 """Some devices are highly likely to be meshtastic.
 0x239a RAK4631
 0x303a Heltec tracker"""
-whitelistVids = dict.fromkeys([0x239a, 0x303a])
+whitelistVids = dict.fromkeys([0x239a, 0x303a, 0x04b4])


What device is 0x04b4? Probably worth documenting so we don't lose what these are, I think

heh. I don't know I'll look around. I had the same thought when I added my second vid ;-). I just tried plugging in a few different boards and didn't see it I'll check the commit history and hopefully I had a comment. ;-)

geeksville · 2024-07-04T02:42:29Z

Looks generally good to me. Got a few questions, particularly about potentially passing LogRecords in full to pubsub instead of just the messages, so we can be a little more consistent with that stuff.

Yeah - my plan (sound okay to you @thebentern) is that the singleton Logger instance will do the formatting at the beginning and then be dispatching LogRecords to all of the observers who want to are handing out logs to their clients (serial, TCP, BLE)

I don't know as much about the actual power monitoring stuff, but as far as I'm aware it seems fine, and I know you're planning to keep updating/changing that, so I think we can let that happen and make any other changes we need to after it settles.

yah - my hope is those changes (powermon/powerstress) will be pretty localized.

Since this will drop our 3.8 support and also since it depends on 2.3.15 protobufs, I'll make a release before merging it that's 2.3.14 and mark it as the last 3.8 release.

smart!

geeksville · 2024-07-04T02:54:54Z

I hear from another PR that you're out hiking so no rush in responding here 🙂, but:

I was wondering if you'd be fine with me grabbing your LogRecord work from here after #627 goes in and amending both the code here and that code to send a consistent pubsub message that includes the full LogRecord so we can handle them consistently (and make the log lines available to library users, too). I don't want to introduce unnecessary conflicts for you to have to undo but given that stuff's making it into firmware it might be nice to get it in sooner/before the PowerMon stuff is finished. Thoughts?

Oh yes if you want to do that it would be awesome! It will allow me to focus on my power goo stuff anyways!

ianmcorvidae · 2024-07-04T03:06:54Z

Looks generally good to me. Got a few questions, particularly about potentially passing LogRecords in full to pubsub instead of just the messages, so we can be a little more consistent with that stuff.

Yeah - my plan (sound okay to you @thebentern) is that the singleton Logger instance will do the formatting at the beginning and then be dispatching LogRecords to all of the observers who want to are handing out logs to their clients (serial, TCP, BLE)

Oh -- yeah, I only mean here on the python side of things, but from what little I understand of C++ that stuff sounds good on that end too. I just want to change pub.sendMessage("meshtastic.log.line", line=line, interface=self) to pass a dict (with message, log level, etc.) instead of a string (line). Which it sounds like we're all in agreement about, haha, just don't want to accidentally volunteer myself for any C++ changes or anything either, given I'd need to spend a few weeks learning it first to even begin 😅

0x04b4 is cypress semi but commonly used in Chinese oscopes (like mine). So it was supposed to be a blacklist not a whitelist!

geeksville · 2024-07-04T03:12:34Z

ok - I think all good now (except for the codecoverage which I can ignore for now?)

geeksville · 2024-07-07T00:47:16Z

okay for this to go in @ianmcorvidae? I have a few more (much more localized) changes coming (finishing the power mon work items) - which I could either add to this PR or start a new one. Which do you prefer?

ianmcorvidae · 2024-07-08T02:47:42Z

Yeah, I'm sorry. Got slammed with a busy week and weekend and didn't get back to merging this, but I'll make a 2.3.14 release shortly and then merge this to master for 2.3.15, and update the protobufs.

geeksville · 2024-07-08T15:49:12Z

no rush - just checking!

geeksville marked this pull request as draft June 22, 2024 17:29

Merge branch 'pr-poetry' into powermon

b41cb7d

geeksville force-pushed the pr-powermon branch from 04597a1 to 8c8f5b8 Compare June 22, 2024 17:50

geeksville added 21 commits June 25, 2024 10:21

didn't mean to check in built protobufs

a3462e0

minor cleanup on observable

338f00a

run linter as part of prerelease tests

21e5601

add typing hints

a1f86a3

begin support for multiple power meter types

cc60f3e

remove python 3.8

362c1f3

fix linter warnings

be74c3e

generalize the powermon stuff to become structured logging

0e45637

make slog nicely add new rows

79c65c1

switch to latest protobufs

9c657c6

The github action for building protobufs is using 0.4.6 so we should …

51c6c2c

…match

move mypy and type info into dev-time only dependencies thx @njh

67e1e7c

# Conflicts: # poetry.lock # pyproject.toml

For poetry change: need to put venv in our path so mypy protobuf plug…

46edd78

…in works # Conflicts: # bin/regen-protobufs.sh

remove observable - switch because we are already using pubsub elsewhere

7b18fd5

document why using python 3.9

b7f7a40

debugging config tweaks

6c0e978

fix meshtastic#610: bump nanopb to 0.4.8

8b781d3

Including in the Poetry changes because it touches the same lines and I want to avoid hand merging ;-)

Add a whitelist of known meshtastic USB VIDs to use a default serial …

1add293

…ports. Initially only RAK4631 and heltec tracker are listed

add NordicSemi Power Profiler Kit 2 device to the USB blacklist

5ff4025

Add support for NRF PPK2 power testing board.

ea18057

ppk2 tweaks

26a672e

geeksville added 7 commits June 28, 2024 09:41

handle the new LogRecord protobufs

542f99b

(backwards/forwards compatible) with old firmware

remvoe unneeded paren

821d3e9

Merge branch 'ble-logging' into pr-powermon

42236f2

# Conflicts: # poetry.lock # pyproject.toml

move @thebentern spiffy logging so it is shared with !ble log sources

1da687c

debug launch configs

13ca8fd

Merge branch 'ble-logging' into pr-powermon

8f98878

# Conflicts: # .vscode/launch.json # meshtastic/protobuf/config_pb2.py # meshtastic/protobuf/mesh_pb2.py # meshtastic/protobuf/mesh_pb2.pyi

Merge branch 'pr-bletweak' into pr-powermon

6da04f7

# Conflicts: # .vscode/launch.json

ianmcorvidae mentioned this pull request Jul 2, 2024

New LogRecord protobuf BLE logging #627

Merged

Merge remote-tracking branch 'root/master' into pr-powermon

2f5a736

# Conflicts: # meshtastic/ble_interface.py # meshtastic/protobuf/mesh_pb2.py # meshtastic/protobuf/powermon_pb2.py # meshtastic/protobuf/powermon_pb2.pyi

geeksville changed the title ~~PowerMon/StructuredLogging support (do not merge)~~ PowerMon/StructuredLogging support Jul 3, 2024

fix linter warnings

ae2ef78

(note: the linter test for min/max is buggy so disabled)

geeksville force-pushed the pr-powermon branch from 12ee54d to ae2ef78 Compare July 3, 2024 16:57

geeksville marked this pull request as ready for review July 3, 2024 16:59

geeksville requested review from ianmcorvidae and thebentern July 3, 2024 16:59

change --slog to use nargs

5695ec7

ianmcorvidae reviewed Jul 4, 2024

View reviewed changes

fix incorrect Vid - thanks @ianmcorvidae for notixing.

6332798

0x04b4 is cypress semi but commonly used in Chinese oscopes (like mine). So it was supposed to be a blacklist not a whitelist!

ianmcorvidae merged commit c049433 into meshtastic:master Jul 8, 2024
7 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PowerMon/StructuredLogging support #607

PowerMon/StructuredLogging support #607

geeksville commented Jun 22, 2024 •

edited

Loading

CLAassistant commented Jun 22, 2024 •

edited

Loading

CLAassistant commented Jun 22, 2024

geeksville commented Jun 22, 2024 •

edited

Loading

thebentern commented Jun 22, 2024

ianmcorvidae commented Jun 22, 2024

geeksville commented Jun 22, 2024

ianmcorvidae commented Jul 2, 2024

geeksville commented Jul 3, 2024

geeksville commented Jul 3, 2024 •

edited

Loading

ianmcorvidae commented Jul 3, 2024

ianmcorvidae left a comment

ianmcorvidae Jul 4, 2024

geeksville Jul 4, 2024

ianmcorvidae Jul 4, 2024

geeksville Jul 4, 2024 •

edited

Loading

ianmcorvidae Jul 4, 2024

geeksville Jul 4, 2024

ianmcorvidae Jul 4, 2024

geeksville Jul 4, 2024

ianmcorvidae Jul 4, 2024

ianmcorvidae Jul 4, 2024

ianmcorvidae Jul 4, 2024

geeksville Jul 4, 2024

geeksville commented Jul 4, 2024

geeksville commented Jul 4, 2024

ianmcorvidae commented Jul 4, 2024

geeksville commented Jul 4, 2024 •

edited

Loading

geeksville commented Jul 7, 2024

ianmcorvidae commented Jul 8, 2024

geeksville commented Jul 8, 2024

PowerMon/StructuredLogging support #607

PowerMon/StructuredLogging support #607

Conversation

geeksville commented Jun 22, 2024 • edited Loading

CLAassistant commented Jun 22, 2024 • edited Loading

CLAassistant commented Jun 22, 2024

geeksville commented Jun 22, 2024 • edited Loading

thebentern commented Jun 22, 2024

ianmcorvidae commented Jun 22, 2024

geeksville commented Jun 22, 2024

ianmcorvidae commented Jul 2, 2024

geeksville commented Jul 3, 2024

geeksville commented Jul 3, 2024 • edited Loading

ianmcorvidae commented Jul 3, 2024

ianmcorvidae left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

geeksville Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

geeksville commented Jul 4, 2024

geeksville commented Jul 4, 2024

ianmcorvidae commented Jul 4, 2024

geeksville commented Jul 4, 2024 • edited Loading

geeksville commented Jul 7, 2024

ianmcorvidae commented Jul 8, 2024

geeksville commented Jul 8, 2024

geeksville commented Jun 22, 2024 •

edited

Loading

CLAassistant commented Jun 22, 2024 •

edited

Loading

geeksville commented Jun 22, 2024 •

edited

Loading

geeksville commented Jul 3, 2024 •

edited

Loading

geeksville Jul 4, 2024 •

edited

Loading

geeksville commented Jul 4, 2024 •

edited

Loading