Same-content rewrites do not propagate mtime, making synced st2 status falsely stale

[myobie]

Summary

Fabric records mtime_secs/mtime_nanos in FileMeta, but a rewrite whose bytes are unchanged does not advance the manifest and receivers do not apply the manifest mtime. Consumers that use file mtime as an activity signal therefore see stale, asymmetric state even while the Fabric route and sync service are live.

This is visible in the documented st2 bus recipe because st2 derives an agent's lastActivity from its status file mtime. Rewriting available\n as available\n refreshes the origin's presence, but Fabric treats it as a content no-op, so other hosts eventually classify that live agent as unknown.

Production evidence

Read-only comparison across a healthy Silber↔Hetz route:

• fabric exec hetz -- ... returned immediately; both daemons reported 0.2.0+16dd81e.
• Path: agents/hetz/evals/status.
• Both copies contained available\n with SHA-256 ed01b1557bd7a1e70f90d8838f38ae0bb6b8367122b3dca7e5935bafa440a45a.
• Hetz origin file mtime: 1785447578; Silber replica mtime: 1785445466 (about 35 minutes stale).
• Hetz st2 agents --json --enrich reported lastActivity=1785447578865.727 and status=available.
• Silber reported lastActivity=1785445466332.6655 and status=unknown for the same live agent.
• Both Fabric manifests had the same entry: hash above, version=313, mtime_secs=1785445463. The fresh same-byte origin rewrite never became a logical update.
• Each host reported drift=false for st2-bus-default, so local logical/content drift did not expose the stale metadata.

The asymmetry disappears temporarily when status bytes change (for example available → busy), because that content change advances the logical version and materializes a fresh destination file.

Root cause boundary

• SyncNode::local_write returns early when the content hash matches, regardless of a changed mtime (src/sync/node.rs).
• The durable observed receipt is path -> ContentHash, so the scan path cannot distinguish a same-byte metadata update from an unchanged file (src/sync/engine.rs).
• materialize_tracked skips a path when the existing content hash matches and never applies FileMeta.mtime_*; write_atomic also leaves the receiver's local write time.
• This conflicts with the FileMeta contract that modification time is preserved metadata (while correctly not being used for conflict ordering).

Classification

This is distinct from:

• fabric sync ls masks manifest/observed folder divergence #19, which addressed manifest/observed count visibility;
• PR Suppress delayed sync self-events #26, which suppresses delayed daemon-authored watcher events;
• PR docs: describe partition-isolated peer presence #23's Fabric peer-reachability/last_seen documentation, which does not address application file metadata replication.

Expected behavior

Define and implement an echo-free metadata contract. Given the existing FileMeta promise, a same-content local mtime change should become a causally newer logical update, and the receiver should apply the winning mtime without creating a cross-peer metadata ping-pong.

If Fabric intentionally does not replicate metadata-only changes, the FileMeta preservation claim and the documented st2 status-sync recipe need an explicit incompatible-consumer boundary, and st2 must encode heartbeat time in file bytes instead. The current half-contract is unsafe because the manifest carries mtime but neither observes nor materializes it consistently.

Acceptance criteria

• Add a two-real-folder regression: sync identical bytes, advance only the origin mtime, and prove the destination mtime converges within filesystem precision.
• The metadata-only change receives a higher logical version and remains echo-free after repeated bidirectional reconciles.
• Persisted state/restart retains the metadata receipt needed to avoid replay or false local edits.
• Content updates, bus tombstones, catalog semantics, and exact self-event suppression remain unchanged.
• fabric sync ls or another machine-readable surface can distinguish metadata drift if it can affect declared sync semantics.
• Document the metadata contract and the st2 liveness consequence.

[myobie]

Constraint from #34: the obvious fix here caused a permanent sync divergence

Flagging this before anyone implements the second half of this issue, because the direct approach is now known to be unsafe.

This issue asks, in part, that receivers apply the manifest mtime. That was briefly implemented (a5bfde2) and has just been deliberately reverted in #34, merged as 00c528f.

Why

Applying the origin's mtime on materialization makes mtime a value shared across nodes rather than a local write timestamp. scan_folder reuses a recorded hash whenever size and both mtime components match, so two nodes contending on one key can produce equal-size entries carrying equal mtimes with different content. The scan then reports a hash the file does not hold; the republish branch reads the real bytes, sees a mismatch, concludes a user edited the file, and republishes stale content under local authorship. Both nodes do it and versions leapfrog without bound.

Measured on Linux CI run 30814462024 attempt 4: twenty-four consecutive alternating republishes, v44 → v67, never converging.

What this means for fixing #27

Propagating mtime to receivers is not safe on its own. It needs a content-identity mechanism so the scan cache cannot mistake equal size + equal mtime for equal content — deliberately deferred in #34 rather than attempted alongside a data-integrity fix.

#34 also asserts that residual risk in a passing test: with a collision already present (however it arrives — a restore resetting mtimes, for instance) the cache still trusts it. So that hazard exists independently of whether mtime is propagated.

The first half of this issue — that a same-content rewrite does not advance the manifest at all — is untouched by #34 and remains open on its own terms.