[Direction]: Long-horizon benchmark evidence and mechanism research

[huangruiteng]

Outcome

Build reproducible evidence for long-horizon agent work while using benchmark
tasks as controlled research environments. This direction must distinguish two
lanes:

• capability evidence: whether LoopX improves benchmark-native outcomes,
  efficiency, or recovery under matched conditions;
• mechanism research: what stride, evidence delivery, replan, memory, and
  human-attention mechanisms explain an observed change.

The canonical research contract is the
Long-Horizon Harness Benchmark and Research Program RFC.
ALE, LHTB, and DeepSWE form a complementary portfolio; LoopX does not replace
their native results with a synthetic aggregate score.

Current stage

Active research program. The claim ladder runs from adapter fidelity (C0)
through production-promotion evidence (C4). A benchmark result is not a product
claim unless the required treatment-integrity, comparison, and evidence gates
for that claim have passed.

Contributor-ready cuts

• deterministic fake-fixture coverage for adapter fidelity and lifecycle
  receipts;
• typed treatment-integrity and configuration-drift checks;
• public-safe ledger, reducer, and analysis improvements;
• synthetic setup, recovery, termination, and attribution cases;
• documentation that makes benchmark-native metrics and claim level explicit.

Maintainer boundary

Live cases, raw tasks, trajectories, logs, verifier tails, uploads, official
scoring, leaderboard or submission behavior, and unpublished comparison
evidence remain maintainer-owned. Ask for a public synthetic slice instead of
starting or duplicating a live run.

How to contribute

Propose one bounded issue or PR, link this tracker, and state the intended
claim level. The public task board remains the source for currently claimable
work; this issue records direction and material decisions rather than every
individual experiment.

[huangruiteng]

Direction governance is now merged in
#3248.

• Canonical map: English · 中文
• Pinned community projection: Discussion #2851
• Claimable work: Contributor Task Board

This issue remains the long-lived direction and decision tracker. Concrete
implementation should use a separate bounded issue or task-board row with an
explicit base branch, smallest slice, non-goals, and validation plan.