Catalog entries have no supported way to remove a path

[myobie]

A catalog entry has no supported way to remove a path. Every removal so far has needed a hand-patched state.json on every host, with all daemons stopped. This is the third occurrence.

Why a removal is hard today

Catalog policy is union-of-presence and never deletes. That is deliberate and correct for a job catalog. The consequence is that a path, once included, is permanent:

• local_remove records nothing when the policy does not propagate deletes (src/sync/node.rs:157, src/sync/config.rs:56).
• materialize_tracked rewrites every present manifest path whose content the node holds, and its delete-to-tombstone branch is gated on propagate_deletes (src/sync/engine.rs:1415-1460).

So an operator rm is a no-op that the next materialize undoes. Deleting on every host at the same moment does not help, because each node restores from its own manifest with no peer involved.

Seeding a tombstone by hand does not work either. Since 16dd81e a scan promotes a tombstoned path back to Present at a higher version when unchanged physical bytes survive with a matching observed receipt (src/sync/engine.rs:1289-1311, pinned by authoritative_tombstone_with_stale_observed_file_respects_policy). That promotion is correct for its own purpose: it converges an inherited tombstone. It also means a single-host tombstone loses to any host still holding the bytes, and the resulting higher Present re-infects the fleet.

Flipping the entry to bus is not an escape. It makes every existing catalog-preserved tombstone authoritative, which deletes unrelated files that were intentionally retained.

The three occurrences

1. plans/vrs-pilot/st2/requirements.md.root-1420970.tmp - a publisher temp file.
2. agents/Silber/personas/agent.kdl - a retired declaration.
3. agents/.hetz-backup-1786658066/**/agent.kdl - a backup directory copied inside the catalog, which produced six duplicate agent IDs on three machines.

Each needed the same manual work: stop the daemons, edit authoritative state on every host, delete the files, restart, then soak. Every one of those steps is a chance to damage unrelated state, and the procedure is only safe because it is done rarely and slowly.

What to build

A path-scoped retire command, for example fabric sync retire <entry> <path>..., that performs the removal as one coordinated operation instead of an operator ritual:

• Set the path to a tombstone whose version exceeds any present version.
• Clear the path from observed and from scan_cache, so no later scan re-promotes it.
• Remove the physical file.
• Do all of the above on every peer of the entry, and make the operation refuse rather than half-apply when a peer is unreachable.

The ordering requirement is the whole point. A node that scans while it still holds the bytes and already holds the tombstone will promote the path back to Present. The command has to close that window; an operator running steps by hand on three machines cannot reliably close it.

Notes for whoever builds it

• Do not weaken the promotion at engine.rs:1289-1311. It exists to converge inherited tombstones and has a test.
• The retire path must be distinguishable from that inherited-tombstone case. An intentional retire and a stale inherited tombstone look identical in the manifest today, which is exactly why the promotion cannot tell them apart.
• Preserve the author and use one deleted_secs across all hosts, matching the two repairs already performed.
• manifest.json is a projection. It must be daemon-regenerated, never written by the command directly.
• Test against the real failure: prove the pre-fix behaviour resurrects the path, so the regression test can actually fail.

Prevention already shipped

The documented rule from 721e4a9 stands and is unchanged by this: stage temp, backup and partial files outside the configured sync folder, because a watcher-visible included name is a durable catalog key. This issue is about the case where something is already inside and has to come out.