docs(network): report MAVLink UDP read path stall under sustained ingest

[krital]

What this is

A bug report, not a fix — filing it as a PR so the investigation lands in the
repo next to the code. Happy for it to be converted to an issue instead if that
suits triage better.

The problem

After 20–40 minutes of sustained MAVLink ingest (~360 msg/s over UDP), the server
stops reading its MAVLink UDP socket. Datagrams pile up unread in the kernel
receive queue while the broker stays otherwise healthy — REST, MQTT and the web
UIs keep serving normally. It never recovers on its own; restarting the sender
does nothing, only restarting the broker clears it.

Operationally this shows up as "all vehicles vanish from the map while the server
looks fine", because MAVLink stops flowing and the twins then pass
retentionTimeoutMillis.

Evidence (captured while stalled)

• ss -u -a shows Recv-Q pinned at 170–211 KB across three consecutive
  samples a minute apart (healthy value is 0)
• kill -3 shows every SelectorThread idle in EPoll.wait — nothing blocked
  on a lock, the store, or I/O; no deadlock
• Host idle — load 0.10, ~95% CPU idle, no iowait
• No exception logged anywhere near the stall
• Heap ~150 MB of 2560 MB, ~730 FDs, disk <15% — nothing exhausted

Already ruled out

Hypothesis	Result
Store back-pressure (Partition)	Ruled out — identical with type: memory
Store write failure	Ruled out — zero "Failed to store" lines
Disk / IO saturation	Ruled out — host idle
Heap / GC death-spiral	Ruled out — heap ~150 MB of 2560 MB
FD exhaustion	Ruled out — ~730 of 1,000,000
Sender fault	Ruled out — stable single socket; kernel queue keeps growing, so datagrams are arriving

Remaining suspicion is that the UDP endpoint's selector registration is lost or
cancelled without the endpoint being torn down — socket stays open and buffered
but is never selected for read again.

One possibly-relevant detail: selectorThreadCount: "{processors}/2" resolves to
1 on the 2-vCPU host this was seen on. Whether a larger explicit selector pool
avoids it is untested.

Workaround currently running

A supervisor samples Recv-Q every 60s and restarts the broker after three
consecutive readings >100 KB. Confirmed working — ~30s outage, re-armed 4s after
firing.

Two secondary observations

1. Swallowed exception — MavlinkProtocol logs
   Failed to store MAVLink message on topic '<topic>': {}; the {} is never
   substituted so the underlying exception is lost. This cost real debugging time.
2. Log severity — routine lifecycle events log at ERROR/WARN
   (ERROR … Destination /mavlink/3/TIMESYNC created,
   ERROR … anonymous successfully logged off, WARN … Created Protocol WS),
   so a healthy server's log reads as though it is failing.

Full detail, including the reproduction and the raw thread dump, is in
docs/review/MAVLINK-UDP-READ-STALL.md.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5b613c21-bc4d-40d6-9cf0-ea41bde360fc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[krital]

Update: 24.6 h of continuous observation — this is frequent

The environment has now run unattended for a full day, with the supervisor
logging every detection. Pushed the numbers into the report; summary here.

Faults detected	19 in 24.6 h
Mean time between faults	82 min
Median	56 min
Shortest / longest gap	22 min / 288 min
Recv-Q at detection	163–214 KB (median 202 KB)
Total downtime	~10 min (0.6 %, only because the supervisor restarts it)

Intervals in minutes, in order:

171, 37, 288, 55, 43, 68, 32, 62, 53, 228, 66, 69, 47, 55, 72, 49, 22, 56

What the spread rules out

The interval doesn't correlate with anything I can see:

• Not load — replay rate is constant at ~360 msg/s throughout
• Not subscriber activity — it reproduces with zero external clients
  connected; most of the 24 h had none
• Not the replay's capture-loop boundary — a full pass is ~13 min at this
  speed, so several complete inside even the shortest 22 min gap

The clustering of Recv-Q at detection (163–214 KB) is just an artefact of the
supervisor sampling every 60 s and firing on the third consecutive reading — it
isn't a ceiling. Left alone the queue keeps growing.

Why this matters beyond the demo

Without the supervisor this deployment would have been unusable for ~23 of the
24 hours: each fault is terminal until the broker restarts. Any long-running
MAVLink ingest would hit the same thing.

Happy to run a targeted experiment if it helps narrow it — the obvious one being
an explicit selectorThreadCount (it resolves to 1 on this 2-vCPU host) to
see whether a larger selector pool changes the rate or removes it. Say the word
and I'll run it and report back with the same measurements.