Skip to content

Add deterministic agent inference test harness - #2

Merged
Stephen Belanger (Qard) merged 17 commits into
agent/daemon-tracing-migrationfrom
inference-test-harness
Jul 29, 2026
Merged

Add deterministic agent inference test harness#2
Stephen Belanger (Qard) merged 17 commits into
agent/daemon-tracing-migrationfrom
inference-test-harness

Conversation

@Qard

@Qard Stephen Belanger (Qard) commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

This is stacked on #1 and deliberately keeps the agent integration test
harness separate from the daemon migration until it has proven stable. It
should be reviewed and merged after the daemon PR.

The test support is split into reusable layers. server owns the generic Axum
router lifecycle. inference implements programmable OpenAI Responses and
Anthropic Messages routers with captured provider requests. ingest implements
the mock Braintrust API and ordered span/log shape scenarios. agent_process
owns the daemon world and selects inference and ingest backends independently.
Finally, agents contains reusable Codex and Claude Code adapters that hide
installation, isolated configuration, standard CLI flags, mock endpoint
routing, and process output. Run builders still accept extra arguments and
environment variables so future scenarios can cover attachments and other
agent-specific inputs without duplicating setup.

Each agent has one test body for live and mocked modes. Ordinary assertions
always validate stable process behavior and trace delivery. When rows are
locally captured, ordinary assertions also validate stable origin metadata.
IngestScenario is separate and is constructed only when both inference and
ingest are mocked; it validates the additional deterministic request,
tool-result, output, injected-error, and ordered trace-shape expectations.

Inference and ingest modes are orthogonal. BT_AGENT_INFERENCE_MODE selects
mock or live model inference, while BT_AGENT_INGEST_MODE selects captured
local ingest or the normal Braintrust backend. This permits deterministic mock
inference with live Braintrust reporting and no model cost. Mock ingest makes
rows locally available; live ingest waits for emitted spans and fails if the
daemon records a sink error.

CI installs @openai/codex@latest and
@anthropic-ai/claude-code@latest without version pins. The real-agent
scenarios are part of the core daemon matrix on Linux, macOS, and Windows.
This also fixes native Windows Codex compatibility by selecting the
shell_command tool it advertises and by ensuring the .cmd hook shims return
from a bt.cmd capability preflight before forwarding hook input.

Validation:

  • cargo fmt --manifest-path bt-daemon/Cargo.toml -- --check
  • cargo test --manifest-path bt-daemon/Cargo.toml --all-features
  • cargo test --manifest-path bt-daemon/Cargo.toml --all-features --test agent_integration -- --ignored --test-threads=1
  • cargo clippy --manifest-path bt-daemon/Cargo.toml --all-targets --all-features -- -D warnings
  • make test
  • CI runs the daemon and real-agent suite on Ubuntu, macOS, and Windows

Add separate OpenAI Responses and Anthropic Messages mock servers with closure-driven responses, errors, request capture, and protocol-valid SSE streams.

Run real Codex and Claude Code sessions through deterministic tool and provider-error scenarios, then verify the daemon delivers the resulting traces to a mock Braintrust backend. Install the latest unpinned agents in dedicated CI so upstream compatibility breaks surface immediately.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Run the latest real Codex and Claude Code integrations inside the existing Linux, macOS, and Windows daemon matrix instead of a separate Linux-only job.

Support both deterministic mock inference with exact scenario assertions and live inference with stable trace-invariant assertions through the same agent runner. Keep the complete programmable inference endpoints self-contained and independent of daemon orchestration so they can later be extracted as a reusable mock-inference crate.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Have inference and ingest own only their protocol-specific Axum routers and captured state, while the upper-level tests host both through one generic server container.

Expose an extensionless Windows test wrapper for hooks launched through Git Bash and make native command hooks prefer executable or command wrappers explicitly.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Add named row-shape expectations over the captured ingest stream, matched as an ordered subsequence independently of HTTP batching and unrelated update rows.

Use the same scenario mechanism for deterministic deep trace assertions and live invariant-only assertions, with a focused router and ordering test.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Keep both agents unpinned at latest while using a per-runner temporary npm cache and skipping audit/funding requests, avoiding hosted Windows global-cache stalls before the test suite.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Install the latest Codex and Claude Code packages into a runner-temporary npm prefix on every platform and pass their exact executable paths into the shared tests, avoiding Windows global-install state.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Resolve runner-temporary executable paths only after the matrix runner exists, avoiding workflow validation failure while retaining exact cross-platform agent paths.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Print the real Codex process output on Windows so hook-selection failures are visible in the cross-platform integration job.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
@Qard
Stephen Belanger (Qard) merged commit 03928b9 into agent/daemon-tracing-migration Jul 29, 2026
4 checks passed
@Qard
Stephen Belanger (Qard) deleted the inference-test-harness branch July 29, 2026 17:30
Stephen Belanger (Qard) added a commit that referenced this pull request Jul 31, 2026
* Move agent tracing into the Rust daemon

Port the Codex and Claude trace state machines, journal recovery, Braintrust delivery, and hook configuration into the shared bt-daemon crate. Replace the legacy TypeScript and shell implementations with fail-open launchers.\n\nAdd Unix-socket and Windows named-pipe transports, detached lifecycle management, cross-platform pipeline tests, and Linux/macOS/Windows CI.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Add deterministic agent inference test harness (#2)

* Add deterministic agent inference test harness

Add separate OpenAI Responses and Anthropic Messages mock servers with closure-driven responses, errors, request capture, and protocol-valid SSE streams.

Run real Codex and Claude Code sessions through deterministic tool and provider-error scenarios, then verify the daemon delivers the resulting traces to a mock Braintrust backend. Install the latest unpinned agents in dedicated CI so upstream compatibility breaks surface immediately.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Make agent integration tests core and cross-platform

Run the latest real Codex and Claude Code integrations inside the existing Linux, macOS, and Windows daemon matrix instead of a separate Linux-only job.

Support both deterministic mock inference with exact scenario assertions and live inference with stable trace-invariant assertions through the same agent runner. Keep the complete programmable inference endpoints self-contained and independent of daemon orchestration so they can later be extracted as a reusable mock-inference crate.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Separate protocol routers from test hosting

Have inference and ingest own only their protocol-specific Axum routers and captured state, while the upper-level tests host both through one generic server container.

Expose an extensionless Windows test wrapper for hooks launched through Git Bash and make native command hooks prefer executable or command wrappers explicitly.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Add ordered ingest trace scenarios

Add named row-shape expectations over the captured ingest stream, matched as an ordered subsequence independently of HTTP batching and unrelated update rows.

Use the same scenario mechanism for deterministic deep trace assertions and live invariant-only assertions, with a focused router and ordering test.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Isolate coding-agent install cache in CI

Keep both agents unpinned at latest while using a per-runner temporary npm cache and skipping audit/funding requests, avoiding hosted Windows global-cache stalls before the test suite.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Install CI agents into an isolated prefix

Install the latest Codex and Claude Code packages into a runner-temporary npm prefix on every platform and pass their exact executable paths into the shared tests, avoiding Windows global-install state.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Scope agent executable paths to the test step

Resolve runner-temporary executable paths only after the matrix runner exists, avoiding workflow validation failure while retaining exact cross-platform agent paths.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Expose Windows Codex hook diagnostics

Print the real Codex process output on Windows so hook-selection failures are visible in the cross-platform integration job.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Extract reusable coding agent test adapters

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Support Codex shell command tool in tests

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Return from Windows bt hook preflight

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Decouple agent adapters from test world

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Unify live and deterministic agent scenarios

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Remove unreleased test mode alias

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Layer deterministic trace expectations over baseline checks

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Separate live assertions from mock scenarios

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Use direct mode checks in agent tests

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

---------

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Keep existing plugin runtimes during daemon rollout

Restore src exactly to main so the currently published Codex and Claude tracing implementations remain in place while the Rust daemon and bt integration land independently. The plugin cutover will follow in a stacked change after a daemon-capable bt release is available.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Test daemon through direct agent hook plugins

Keep real Codex and Claude integration coverage in the daemon PR without depending on the production plugin cutover. Generate isolated test plugins whose hooks call bt agents hook directly; the stacked plugin PR replaces these fixtures with the simplified production plugins.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Log daemon request handling

Report JSON-RPC request receipt and completion while the daemon runs in the foreground. Event logs include only source, event, and session identifiers, with explicit accepted or rejected outcomes; payloads and credentials remain excluded.

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Use bt trace in agent test plugins

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Separate transcript replay from journal recovery

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Fix trace namespace in agent test wrapper

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Import agent sessions by source and id

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Preserve Codex turns during transcript import

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Restore Codex history after compaction

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Batch transcript import delivery

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Stream and bound transcript imports

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

* Fix Claude transcript turn correlation

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>

---------

Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant