Sustained CPU in sync: full manifest exchange per peer per pass (size x mutation rate), unswept bus tombstones, Content-buffered Entry

[myobie]

The daemon spends sustained CPU decoding sync manifests. Observed on Silber 2026-08-14: PID 11655 at ~44.7% CPU on an otherwise idle machine, with no active dial handler. A sample run places the work in sync_once → serde_json::from_slice::<ReplyHeader> → Manifest → BTreeMap<String, Entry> → Entry decode.

The numbers

Entry	Entries	manifest.json	state.json
st2-bus-default	23,407 (11,636 present + 11,771 tombstones)	19.4 MB	33.5 MB
st2-declarations-default	66	69 KB	140 KB

The bus manifest is ~355× the declarations manifest by entry count. All of the cost is the bus entry.

Three compounding causes

1. Tombstones are never swept

SyncPolicy::Bus declares sweep_tombstones: true (src/sync/config.rs:62), but the flag has no consumer in production code — the only read in the tree is a test assertion (src/sync/config.rs:321). The policy doc already admits it: "not yet fully implemented" (src/sync/config.rs:35-36).

So bus tombstones accumulate permanently. They are currently half the manifest (11,771 of 23,407), and they are decoded on every pass forever. This grows without bound with every message and status write the bus has ever carried.

2. Entry decodes through serde's Content buffer

Entry is an internally-tagged enum (#[serde(tag = "kind")], src/sync/manifest.rs:105-107). Serde buffers every internally-tagged value into an intermediate Content tree and re-deserializes from it, which is its documented slow path. The sample confirms this is where the time goes and shows nothing else: TaggedContentVisitor, ContentDeserializer, __deserialize_content_v1.

This is the same representation that caused the u128 decode failure in PR 36 — ControlResponse is internally tagged, so it routed through Content, which has no u128. One representation choice, a correctness bug then and a performance bug now.

3. The whole manifest is decoded per peer, per pass

sync_once clones the local manifest and reconciles against each peer (src/sync/engine.rs:751-792); the reply header carries the peer's entire manifest and is decoded with from_slice. With two peers, a pass on every mutation signal plus every 30s, that is a 19.4 MB JSON decode each time.

Fix directions, in the order I would take them

1. Implement the tombstone sweep for bus policy. It is already declared, already documented, and removes about half the input. Bounded and it needs no wire change. Sweeping must be a normal versioned change so peers converge rather than resurrect — a swept tombstone must not read as "never seen".
2. Stop re-decoding an unchanged manifest. Both sides already track a mutation generation. A pass where neither side has changed should not pay a full decode; a cheap generation or digest exchange before the manifest body would remove most passes entirely.
3. Change the Entry representation last, and treat it as a wire change. Adjacently tagged, or a hand-written Deserialize, avoids the Content buffer. This changes bytes on the wire, so it needs the same care as any manifest format change: peers on older builds must still parse, or it must be staged behind a version. Do not do this first just because the profile points at it.

Test notes

• Measure before and after on a manifest of realistic size. A microbenchmark on a handful of entries will not show any of this.
• For the sweep, prove convergence across three nodes, not just locally: the failure mode is a swept tombstone reappearing from a peer that still holds it.
• Prove any regression test fails against the current code before trusting it.

Not caused by the catalog backup directory

Ruled out by measurement. The six leftover .hetz-backup-1786658066 declarations are 6 entries of 23,473 (0.026%) and ~12.7 KB of ~33.7 MB (0.04%). Removing them is worth doing for other reasons (#50) but will not change the CPU.

---

Correction and measurement, 2026-08-24

Cause 3 is misnamed above, and the title was too. It said full decode per
pass. My own profile disproved that framing on 2026-08-14 and it was never
corrected here. Cause 3 is full manifest EXCHANGE per peer, per pass.

Live cost is manifest size MULTIPLIED BY mutation rate

The lab ran at 2.6% of a core against 60% live, on the same corpus, the
same code and the same manifest size. The only difference: the lab corpus is
static and the live bus is not, because it is a message bus.

So shrinking the manifest attacks one factor of a product. The other factor
is how often we send it, and we send it in full on every pass, and passes are
driven by mutations we generate ourselves. Every status write, every message and
every archive is a pass that re-sends the whole manifest to every peer.

Sweeping tombstones (cause 1) attacks size only. It cannot fix a product.

Measured directly, 2026-08-24

sync_once was instrumented with per-phase counters (#68). Silber, on
0.2.0+0ca1bd8, 420 s window taken after a 240 s settle, under live fleet load:

phase	ms per pass	share
reconcile	4057	90.8%
scan	197	4.4%
materialize	111	2.5%
persist	101	2.3%

This is CPU, not network waiting. Phase time is wall inside the phase, so a
phase parked in an await reads identically to one doing work. Daemon CPU over
the same window was 66.2 s against 72.5 s of measured phase wall. If
reconcile were mostly blocking, CPU would fall far below phase wall; it does
not, and no other phase is large enough to hold 66 s of CPU. Peer round trip is
45–52 ms, which rules out latency independently.

Two measurements, ten days apart, by different methods, agree on the same cause.

Current size, for comparison with the table above

st2-bus-default is now 30,924 entries (17,404 present + 13,520 tombstones),
manifest.json 26 MB, state.json 46 MB — describing a watched tree of 16.6
MB across 17,175 files. The state on disk is over four times the size of the
data it describes.

What the walk is worth, so nobody re-targets it

4.4% of a pass. A full lstat pass over the tree costs ~36 ms and reading
and hashing every byte would cost ~20 ms more. #66 already removed one redundant
stat per file. The walk is not the problem and cannot be made into one.

Direction

Digest-first: exchange a hash, fall through to a manifest only when the
hashes differ. The common case is a peer that already has what we are about to
send, and that exchange should cost a hash rather than a manifest.

It is load-bearing rather than an optimisation, and this is the trade worth
stating out loud: presence needs exactly one mutation per refresh to be visible,
and this issue says every mutation costs a full manifest exchange with every
peer. We deliberately made the bus busier in order to make it honest. That
is the right trade, but it only stays affordable if a presence write costs a
changed hash and one small entry rather than a whole manifest.

Not started. It is a design conversation first.

[myobie]

Correcting this issue: the cause was wrong, and so was the magnitude

I filed this. Both headline claims are wrong and I want them corrected in place rather than left for the next person to inherit.

The magnitude: 60% of a core, not ~45%

This issue says "PID 11655 at ~44.7% CPU". That came from ps, and ps %CPU is a decaying average, not a rate. In one 63-second window it read 32.4%, then 45.7%, then 28.4%, while the truth was flat.

Measure with a cputime delta instead — the difference between two cumulative readings over a known wall interval:

ps -o cputime= -p <pid>     # 21:50:16Z  556:18.05
ps -o cputime= -p <pid>     # 21:51:19Z  556:55.81
# 37.76 CPU seconds in 63 wall seconds = 60% of one core

The real steady-state baseline is ~60% of one core, and it is steady. Repeated over the next 63 seconds it gave 54.7%. The number we have been quoting all evening was a smeared average of a constant.

This generalises past this issue: a percentile, an average, and a total all describe the past. Only a difference between two samples describes the present. reconnect_p50=30.0s on #51 misled me the same way in the same hour — I read a percentile of past durations as a present-tense frequency.

The cause: decode is 12.6% of the work, not the driver

I wrote that a sample run "places the work in sync_once → serde_json::from_slice". It does show those frames. What I did not do is check their share, and the arithmetic alone should have stopped me: one decode measures 38.7 ms, and against a 30-second periodic tick that is 0.13% of a core. That can never be 45%.

A 10-second sample of the live daemon, bucketed by top-of-stack with idle waits excluded (7,552 work samples):

bucket	share
network send/recv	45.5%
filesystem syscalls	16.3%
json + manifest	12.6%
malloc/memory	7.6%
blake3 hashing	5.8%
quic crypto	1.8%

Read the "Sort by top of stack" section of the sample file for this, not the call graph — I first hand-walked the tree, ran past the end of the call graph into the process-wide summary, and confidently attributed a hashing total to the fsevents thread. It was not there.

Decode is real, but it is 12.6% and it is fourth. The driver is sending bytes.

What that means for the three causes

Cause 1, unswept tombstones — still correct, still first, but for a different reason. The flag genuinely has no consumer and tombstones genuinely grow without bound; that part stands. But its value is not a decode saving. wire.rs puts the full manifest in both the hello header and the reply header, so both sides send their entire manifest on every session. Sweeping halves the entries and takes the manifest from 7.4 MB to 4.7 MB compact, so its value is 36% fewer bytes per exchange. PR #53 implements it, off by default.

Cause 2, the Entry Content buffer — withdrawn. Do not build this. It targets the smallest of the three buckets and it breaks the wire format, which would mean two machines disagreeing about what a manifest is. The cost/benefit is bad in both directions.

Cause 3 — this is the real one, and its name here is wrong. Not "the whole manifest is decoded per peer, per pass" but the whole manifest is EXCHANGED per peer, per pass. That is where the 45.5% lives. It needs a design conversation, not a patch: some form of incremental or digest-first reconcile so a converged pair stops re-sending a manifest neither side has changed.

Not the cause: #51

I also claimed the reconnect leak was contaminating this baseline. It is not. Session counters were byte-identical across samples 63 seconds apart and across four scheduled sweeps hours apart, so #51 is dormant. The 60% is genuinely the sync path, on a quiet daemon, with nothing reconnecting.

[myobie]

The measurement, and a reframing of cause 3

Measured end to end in a two-node lab on the real corpus, seeded with the real live sync state so the tombstones are genuinely old. PR #53 has the full method and the safety proof.

	manifest entries	tombstones	CPU per node
sweep off	23,811	11,892	2.6% of a core
sweep on (TTL 1d)	12,187	268	1.1% of a core

57% less CPU, 49% fewer entries. That ratio will not transfer to the live 60% baseline, and the reason is the important part.

Live cost is manifest size × mutation rate

Lab baseline is 2.6% of a core. Live is 60%. Same corpus, same code, same manifest size. The difference: the lab corpus is static and the live bus is not.

A converged idle pair exchanges manifests on the 30-second periodic tick — that is the 2.6%. The live bus mutates constantly, because it is a message bus, and every mutation triggers another pass that re-sends the whole manifest to every peer.

So the cost is a product of two factors, and this issue has only ever discussed one of them. Sweeping shrinks the manifest. It does nothing about how often we send it.

A worked example of the second factor: seven agents writing presence every five minutes is 2,016 mutations a day, each triggering a full manifest exchange with two peers. Correct liveness makes this term worse. That is a fair trade, but it should be a known one.

So cause 3 is misnamed in the issue body

I wrote it as "the whole manifest is decoded per peer, per pass". The profile disproved the decode part — decode is 12.6% of work, network send/recv is 45.5%. The right name is:

The whole manifest is EXCHANGED per peer, per pass.

wire.rs puts the full manifest in both the hello header and the reply header, so both sides send everything on every session, including the overwhelmingly common case where neither side has changed anything since the last pass.

The fix is digest-first: exchange a hash, and only fall through to a manifest when the hashes differ. The common case is a peer that already has what we are about to send, and that case should cost a hash rather than 7.4 MB. This is now the main structural piece, ahead of anything else in this issue. It needs a design conversation before code.

Instrument note: ps %CPU misled us three times in one evening

Adding to the method note above, because this was the instrument in most of tonight's wrong numbers:

• It read 32.4% / 45.7% / 28.4% across one 63-second window while the true rate was a flat 60% of a core.
• It read 0.0% for two lab daemons that a cputime delta showed were using 9% of a core each.

ps %CPU is a decaying average since process start. It cannot answer "what is happening now". Use two ps -o cputime= readings divided by the elapsed wall seconds.

[myobie]

Cause 3 is weakened by measurement. I filed it and I am correcting it.

This issue names "full decode per pass" as cause 3. I profiled the live daemon
on 2026-08-19 and the profile does not support that as the live cost.

In a 15 second profile of PID 11655, version 0.2.0+fff659e:

• Manifest::diff_from: 1 sample
• every serde_json frame together: 12 samples
• blake3_hash_many_neon: 1,403 samples, 14.22 percent of one core

Manifest decode was not the cost. Content hashing was, and the cause was a scan
storm feeding it:

• materialize_tracked re-reads and re-hashes every present file on every pass, ignoring the scan cache #56 (merged): materialize_tracked re-read and re-hashed every present
  file each pass, ignoring the scan cache.
• A nested entry root makes the declarations entry full-scan a converged tree twice a second #57 (merged): the declarations entry full-scanned a converged tree about
  twice a second, because the bus root is nested inside it and st2 writes there
  continuously.

Together those accounted for the sustained CPU this issue was opened about.

What this does NOT settle

Cause 1, unswept tombstones, is real and unchanged. The bus manifest holds
26,851 entries and 12,604 of them are tombstones, which is 46.9 percent. The
sweep from #53 is merged but not enabled, and the running daemon predates it.

I also recorded a related risk while measuring: bluey answered 14 liveness
probes and failed 699 on the day I looked. An ack-gated sweep would freeze for
every entry a permanently unreachable configured peer has not acked.

Cause 2 stays withdrawn.

Status

Neither #56 nor #57 is deployed. The fleet runs 0.2.0+fff659e. The measured
effect of both is still owed and waits on a deploy, which is not mine to make.