[Bug]: some kind of buffer overflow in MQTT?

[jointerCZ]

Category

Other

Hardware

T-Beam S3

Firmware Version

2.5.7

Description

Hello,
I have 2 Lilygo Supreme devices. When I configure WiFi and enable MQTT, the devices randomly reboots.
One device is very close to AP one more far away and based on logs they do not drop connection to wifi.
But I found this in the middle of the serial log, it seems it is randomly loosing connection and reconnects:
INFO | 08:08:44 863 [Router] MQTT not connected, queueing packet

ERROR | 08:08:44 863 [RadioIf] ignoring received packet due to error=-7

DEBUG | 08:08:44 863 [RadioIf] Packet received (noise?) : 170ms

INFO | 08:08:44 863 [mqtt] Using non-TLS-encrypted session

INFO | 08:08:44 863 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats

INFO | 08:08:47 867 [mqtt] MQTT connected

INFO | 08:08:47 867 [mqtt] Subscribing to msh/CZ/2/e/redacted/+

INFO | 08:08:47 867 [mqtt] Subscribing to msh/CZ/2/e/PKI/+

DEBUG | 08:08:47 867 [mqtt] Publishing enqueued MQTT message

However eventually I get a panic due to buffer being full, see log.
It seems this situation is not handled well.

Can I avoid this somehow and keep MQTT? If I connect it to Linux meshtasticd and maybe process MQTT there somehow?
Or is it public server issue?

Relevant log output

INFO | 07:34:56 2085 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats

INFO | 07:34:57 2086 [mqtt] MQTT connected

DEBUG | 07:34:57 2086 [mqtt] Publishing enqueued MQTT message

ERROR | 07:34:57 2086 [mqtt] Panic: can't encode protobuf reason='bytes size exceeded'

[jointerCZ]

Subjective observation is that during night the device was without reboot, but now it started rebooting again.
Feels like some kind of traffic or node can be causing it (speculation).

INFO | 10:13:59 1201 [DeviceTelemetryModule] (Sending local stats): uptime=1201, channel_utilization=12.259999, air_util_tx=0.785444, num_online_nodes=43, num_total_nodes=100

INFO | 10:13:59 1201 [DeviceTelemetryModule] num_packets_tx=137, num_packets_rx=234, num_packets_rx_bad=68

DEBUG | 10:13:59 1201 [DeviceTelemetryModule] Partially randomized packet id 2424609705

ERROR | 10:13:59 1201 [RadioIf] ignoring received packet due to error=-7

DEBUG | 10:13:59 1201 [RadioIf] Packet received (noise?) : 207ms

INFO | 10:13:59 1201 [mqtt] Using non-TLS-encrypted session

INFO | 10:13:59 1201 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats

INFO | 10:14:00 1202 [mqtt] MQTT connected

DEBUG | 10:14:00 1202 [mqtt] Publishing enqueued MQTT message

ERROR | 10:14:00 1202 [mqtt] Panic: can't encode protobuf reason='bytes size exceeded'

[jointerCZ]

I have to confirm with 2.5.8 it still remains (as expected).

[esev]

I might have spotted something related here. Looks like a potential use-after-free bug.

firmware/src/mqtt/MQTT.cpp

Line 611 in 827553f

env->packet = (meshtastic_MeshPacket *)&mp_encrypted;

I'm not sure that type cast is safe when queuing env, and I suspect the original const meshtastic_MeshPacket &mp_encrypted reference has gone out of scope in the calling function by the time the item is pulled from the mqttQueue.

Callers:

• firmware/src/mesh/Router.cpp

  Line 281 in 827553f

  mqtt->onSend(*p, *p_decoded, chIndex);

• firmware/src/mesh/Router.cpp

  Line 630 in 827553f

  mqtt->onSend(*p_encrypted, *p, p->channel);

I've been digging around in the MQTT code recently. I'll send a PR to fix. Might be a few days though.