Add deterministic agent inference test harness - #2
Merged
Stephen Belanger (Qard) merged 17 commits intoJul 29, 2026
Merged
Conversation
Add separate OpenAI Responses and Anthropic Messages mock servers with closure-driven responses, errors, request capture, and protocol-valid SSE streams. Run real Codex and Claude Code sessions through deterministic tool and provider-error scenarios, then verify the daemon delivers the resulting traces to a mock Braintrust backend. Install the latest unpinned agents in dedicated CI so upstream compatibility breaks surface immediately. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Run the latest real Codex and Claude Code integrations inside the existing Linux, macOS, and Windows daemon matrix instead of a separate Linux-only job. Support both deterministic mock inference with exact scenario assertions and live inference with stable trace-invariant assertions through the same agent runner. Keep the complete programmable inference endpoints self-contained and independent of daemon orchestration so they can later be extracted as a reusable mock-inference crate. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Have inference and ingest own only their protocol-specific Axum routers and captured state, while the upper-level tests host both through one generic server container. Expose an extensionless Windows test wrapper for hooks launched through Git Bash and make native command hooks prefer executable or command wrappers explicitly. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Add named row-shape expectations over the captured ingest stream, matched as an ordered subsequence independently of HTTP batching and unrelated update rows. Use the same scenario mechanism for deterministic deep trace assertions and live invariant-only assertions, with a focused router and ordering test. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Keep both agents unpinned at latest while using a per-runner temporary npm cache and skipping audit/funding requests, avoiding hosted Windows global-cache stalls before the test suite. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Install the latest Codex and Claude Code packages into a runner-temporary npm prefix on every platform and pass their exact executable paths into the shared tests, avoiding Windows global-install state. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Resolve runner-temporary executable paths only after the matrix runner exists, avoiding workflow validation failure while retaining exact cross-platform agent paths. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Print the real Codex process output on Windows so hook-selection failures are visible in the cross-platform integration job. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Stephen Belanger (Qard)
merged commit Jul 29, 2026
03928b9
into
agent/daemon-tracing-migration
4 checks passed
Stephen Belanger (Qard)
added a commit
that referenced
this pull request
Jul 31, 2026
* Move agent tracing into the Rust daemon Port the Codex and Claude trace state machines, journal recovery, Braintrust delivery, and hook configuration into the shared bt-daemon crate. Replace the legacy TypeScript and shell implementations with fail-open launchers.\n\nAdd Unix-socket and Windows named-pipe transports, detached lifecycle management, cross-platform pipeline tests, and Linux/macOS/Windows CI. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Add deterministic agent inference test harness (#2) * Add deterministic agent inference test harness Add separate OpenAI Responses and Anthropic Messages mock servers with closure-driven responses, errors, request capture, and protocol-valid SSE streams. Run real Codex and Claude Code sessions through deterministic tool and provider-error scenarios, then verify the daemon delivers the resulting traces to a mock Braintrust backend. Install the latest unpinned agents in dedicated CI so upstream compatibility breaks surface immediately. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Make agent integration tests core and cross-platform Run the latest real Codex and Claude Code integrations inside the existing Linux, macOS, and Windows daemon matrix instead of a separate Linux-only job. Support both deterministic mock inference with exact scenario assertions and live inference with stable trace-invariant assertions through the same agent runner. Keep the complete programmable inference endpoints self-contained and independent of daemon orchestration so they can later be extracted as a reusable mock-inference crate. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Separate protocol routers from test hosting Have inference and ingest own only their protocol-specific Axum routers and captured state, while the upper-level tests host both through one generic server container. Expose an extensionless Windows test wrapper for hooks launched through Git Bash and make native command hooks prefer executable or command wrappers explicitly. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Add ordered ingest trace scenarios Add named row-shape expectations over the captured ingest stream, matched as an ordered subsequence independently of HTTP batching and unrelated update rows. Use the same scenario mechanism for deterministic deep trace assertions and live invariant-only assertions, with a focused router and ordering test. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Isolate coding-agent install cache in CI Keep both agents unpinned at latest while using a per-runner temporary npm cache and skipping audit/funding requests, avoiding hosted Windows global-cache stalls before the test suite. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Install CI agents into an isolated prefix Install the latest Codex and Claude Code packages into a runner-temporary npm prefix on every platform and pass their exact executable paths into the shared tests, avoiding Windows global-install state. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Scope agent executable paths to the test step Resolve runner-temporary executable paths only after the matrix runner exists, avoiding workflow validation failure while retaining exact cross-platform agent paths. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Expose Windows Codex hook diagnostics Print the real Codex process output on Windows so hook-selection failures are visible in the cross-platform integration job. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Extract reusable coding agent test adapters Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Support Codex shell command tool in tests Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Return from Windows bt hook preflight Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Decouple agent adapters from test world Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Unify live and deterministic agent scenarios Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Remove unreleased test mode alias Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Layer deterministic trace expectations over baseline checks Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Separate live assertions from mock scenarios Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Use direct mode checks in agent tests Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> --------- Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Keep existing plugin runtimes during daemon rollout Restore src exactly to main so the currently published Codex and Claude tracing implementations remain in place while the Rust daemon and bt integration land independently. The plugin cutover will follow in a stacked change after a daemon-capable bt release is available. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Test daemon through direct agent hook plugins Keep real Codex and Claude integration coverage in the daemon PR without depending on the production plugin cutover. Generate isolated test plugins whose hooks call bt agents hook directly; the stacked plugin PR replaces these fixtures with the simplified production plugins. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Log daemon request handling Report JSON-RPC request receipt and completion while the daemon runs in the foreground. Event logs include only source, event, and session identifiers, with explicit accepted or rejected outcomes; payloads and credentials remain excluded. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Use bt trace in agent test plugins Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Separate transcript replay from journal recovery Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Fix trace namespace in agent test wrapper Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Import agent sessions by source and id Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Preserve Codex turns during transcript import Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Restore Codex history after compaction Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Batch transcript import delivery Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Stream and bound transcript imports Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> * Fix Claude transcript turn correlation Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com> --------- Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is stacked on #1 and deliberately keeps the agent integration test
harness separate from the daemon migration until it has proven stable. It
should be reviewed and merged after the daemon PR.
The test support is split into reusable layers.
serverowns the generic Axumrouter lifecycle.
inferenceimplements programmable OpenAI Responses andAnthropic Messages routers with captured provider requests.
ingestimplementsthe mock Braintrust API and ordered span/log shape scenarios.
agent_processowns the daemon world and selects inference and ingest backends independently.
Finally,
agentscontains reusable Codex and Claude Code adapters that hideinstallation, isolated configuration, standard CLI flags, mock endpoint
routing, and process output. Run builders still accept extra arguments and
environment variables so future scenarios can cover attachments and other
agent-specific inputs without duplicating setup.
Each agent has one test body for live and mocked modes. Ordinary assertions
always validate stable process behavior and trace delivery. When rows are
locally captured, ordinary assertions also validate stable origin metadata.
IngestScenariois separate and is constructed only when both inference andingest are mocked; it validates the additional deterministic request,
tool-result, output, injected-error, and ordered trace-shape expectations.
Inference and ingest modes are orthogonal.
BT_AGENT_INFERENCE_MODEselectsmock or live model inference, while
BT_AGENT_INGEST_MODEselects capturedlocal ingest or the normal Braintrust backend. This permits deterministic mock
inference with live Braintrust reporting and no model cost. Mock ingest makes
rows locally available; live ingest waits for emitted spans and fails if the
daemon records a sink error.
CI installs
@openai/codex@latestand@anthropic-ai/claude-code@latestwithout version pins. The real-agentscenarios are part of the core daemon matrix on Linux, macOS, and Windows.
This also fixes native Windows Codex compatibility by selecting the
shell_commandtool it advertises and by ensuring the.cmdhook shims returnfrom a
bt.cmdcapability preflight before forwarding hook input.Validation:
cargo fmt --manifest-path bt-daemon/Cargo.toml -- --checkcargo test --manifest-path bt-daemon/Cargo.toml --all-featurescargo test --manifest-path bt-daemon/Cargo.toml --all-features --test agent_integration -- --ignored --test-threads=1cargo clippy --manifest-path bt-daemon/Cargo.toml --all-targets --all-features -- -D warningsmake test