Move coding-agent tracing into the Rust daemon

[Stephen Belanger (Qard)]

Dependency

Depends on braintrustdata/braintrust-sdk-rust#86. The daemon pins commit d33e806bf6ab9548d37355f6a5098a971ef150aa as an exact Git dependency.

Summary

This adds the shared Rust coding-agent tracing daemon that will be embedded in the bt CLI. The daemon owns event ordering, auth-redacted journaling, restart recovery, session state, span hierarchy construction, Braintrust delivery, credential isolation, experiment routing, custom parent attachment, and span-origin metadata.

The journal is the daemon's recovery WAL. On restart, historical hook events rebuild the translator's in-memory correlation state before new events are processed. Rows produced during that rebuild may be resubmitted to repair interrupted delivery, but every logical span uses the same deterministic span ID and backend row ID, so those writes merge into the existing row rather than creating duplicate spans. The restart test verifies that all historical IDs recur unchanged and only a genuinely new hook event receives a new span ID.

Explicit import is a separate historical workflow. import <source> <session-id> locates a native Codex rollout or Claude Code transcript in that agent's standard session directories, snapshots it at invocation time, synthesizes the lifecycle events recoverable from that snapshot, and passes them through the regular agent translator and sink. Hook-only facts that are not present in a native transcript are not invented. Codex imports stream the snapshot once, preserve native turn IDs, replace conversation state from native compaction records, and drain delivery in small bounded operation batches so very large sessions cannot overflow the SDK queue or backend row limits and collapse into a partial trace. Claude imports use physical record checkpoints rather than assuming file-order timestamps are monotonic, keeping every reconstructed LLM under its native turn even when Claude appends older continuation records after newer queue bookkeeping. Native Claude API-error markers populate turn and LLM errors, including the HTTP status in LLM metadata. Translator-level import tests cover multi-turn Codex and Claude sessions with LLM and tool spans.

The daemon is one Rust crate with a feature-gated standalone development binary. Linux and macOS use Unix-domain sockets; Windows uses per-user byte-mode named pipes with exclusive ownership, busy-instance retry, detached spawn, status, shutdown, and version handover through the same JSON-RPC pipeline.

The daemon supports one shared coding-agent configuration file. It controls trace enablement, project, turn-end flushing, and additional metadata with file-over-environment precedence. Authentication, credentials, organization selection, and backend URLs remain owned and resolved by bt; parent/root span attachment remains per invocation.

This PR intentionally leaves src/ exactly unchanged from main. The currently published Codex TypeScript runtime and Claude shell runtime therefore continue to build, test, and ship while the daemon and its bt integration land. The plugin launchers and legacy-runtime removal remain in stacked follow-up PR #3 and must not land until a daemon-capable bt release is available.

Real-agent coverage remains in this PR without depending on that future plugin cutover. The test harness generates isolated Codex and Claude plugins whose hooks call bt trace hook directly, then runs the latest unpinned agent CLIs through deterministic inference and ingest scenarios. CI builds, tests, and lints the daemon on Ubuntu, macOS, and Windows and cancels superseded runs for the same PR or branch.

Validation

• cargo test --manifest-path bt-daemon/Cargo.toml --all-features
  • 60 tests passed; the two opt-in live-agent tests were ignored without explicit executable paths.
  • Includes session-ID discovery, native multi-turn Codex and Claude imports, non-monotonic Claude record ordering and API errors, compaction-state replacement, turn-checkpoint delivery, and duplicate-free restart recovery identity checks.
• cargo fmt --manifest-path bt-daemon/Cargo.toml --check
• cargo check --manifest-path bt-daemon/Cargo.toml --all-features
• make test
• Real installed Codex and Claude Code sessions through generated direct-hook plugins completed against deterministic mock inference and satisfied their mock ingest trace scenarios.
• A real three-turn Claude transcript with non-monotonic record timestamps was imported into Braintrust; all three LLM spans were parented to their corresponding turns, native API failures were marked as errors, and re-import retained the same seven span IDs.

[Abhijeet Prasad (AbhiPrasad)]

I was hoping more of an architecture where we extract out the state machine to be re-usable between the different translators

   struct ClaudeTranslator {
       trace: TraceMachine,

       // Claude-specific parsing state
       transcript_cursors: HashMap<String, TranscriptCursor>,
       main_history: Vec<Value>,
       pending_skills: Vec<String>,
   }

   struct CodexTranslator {
       trace: TraceMachine,

       // Codex-specific parsing state
       scopes: HashMap<String, CodexScope>,
       compaction_triggers: HashMap<String, String>,
       spawn_turn_by_agent_id: HashMap<String, String>,
   }

but we can think about this in a follow up given these are all internals.

[Stephen Belanger (Qard)]

I held off on that for now specifically because I don't have a clear enough intuition for what that would look like across all agent types. I didn't want to be defining that with only the common two agents covered and then find none of the others could fit.

At the least, I want to get the JS-based ones like Pi and OpenCode in first, then maybe after we have a bit broader coverage we can think about what unifying the internal state machine would look like.