libra add: LBR-IO-002 object-index repair lock timeout on batched invocation (0.22.10)

[genedna]

Summary

libra add fails with LBR-IO-002 when a single invocation is given many
paths at once, reporting a timeout on the object-index repair generation
lock and suggesting that "another Libra process may be stalled". No Libra
process was running at the time, and the named lock file was a 12-hour-old
zero-byte leftover. Adding the same files one at a time succeeded for all
33 of them on the first try, which points at internal concurrency
contention within a single batched invocation rather than a genuinely held
lock.

Recording this so there is a reference point if batched libra add starts
failing repeatedly.

Environment

• libra 0.22.10
• A libra-managed working copy (no .git), ~35 tracked files modified
• Linux 7.1.9-arch1-2, repository on an external mount

What happened

A single libra add invocation carrying 33 modified paths (via xargs)
failed partway through:

fatal: failed to store object for 'docs/refactoring/config.md': stored object
81a624766e0260156670a3f65d7409bc59564459, but failed to register its cloud
object-index repair marker: timed out waiting for object-index repair lock
'.libra/object-index-repair-locks/object-index-repair-generation.lock' for
repair-marker generation; another Libra process may be stalled
Error-Code: LBR-IO-002

Note the object itself was stored successfully; only the repair-marker
registration timed out. Nothing was staged — libra status afterwards still
listed all 33 files as unstaged.

Investigation

• pgrep -a libra → no Libra process running.
• fuser over .libra/object-index-repair-locks/*.lock → no holders.
• .libra/object-index-repair-locks/object-index-repair-generation.lock is
  a zero-byte file dated ~12 hours earlier (Sep 4 09:53), i.e. from around
  the previous commit on this working copy. The directory held 132 lock
  files in total: most dated 09:54–09:55 (previous commit), a handful
  dated 22:15 (the failed run).
• The 09:53 generation lock predates a commit that completed successfully at
  09:55, so mere existence of the file cannot be what blocks acquisition —
  which is what makes the "another Libra process may be stalled" diagnosis
  look misleading here.

No files under .libra/ were modified or removed during the investigation.

Workaround

Adding the files one at a time:

while read -r f; do libra add "$f"; done < paths.txt

All 33 succeeded on the first attempt, no retries needed. The subsequent
libra commit / libra rebase / libra push all worked normally.

Why this looks like contention rather than a stale lock

1. No process held the lock, by two independent checks.
2. Sequential adds of the exact same paths, moments later, all succeeded.
3. The failure surfaced only in the batched invocation.

If the batch path fans out concurrent object stores that each try to take
the generation lock, they would contend with one another and the timeout
would be self-inflicted. That is a guess about the internals, not a verified
root cause.

Suggested follow-ups

• Confirm whether batched libra add performs concurrent object stores that
  each acquire the object-index repair generation lock.
• If so, either take the generation lock once per invocation, or make the
  repair-marker registration tolerant of contention within one process.
• Reword LBR-IO-002 so it does not assert "another Libra process may be
  stalled" when the contention can be intra-process; the current wording
  points investigation in the wrong direction.
• Consider whether a failed repair-marker registration should abandon the
  whole add when the object itself was already stored.

Reproduction

Not reproduced deterministically. Shape that triggered it: a libra-managed
repo with ~130 pre-existing entries in .libra/object-index-repair-locks/,
33 modified tracked files passed to one libra add invocation.