replay: four configuration axes — typed frontends, context policies, ordered admission, native token backends - #19
Closed
serendipity-zk wants to merge 17 commits into
Closed
Conversation
… metrics
Replace the VibeSim-specific request frontend with a generic
independent-request frontend, and add the capabilities that share its types:
- independent frontend: TraceFormat::Vibesim -> Independent, VibeSimRequest ->
IndependentRequest, request ids are now `independent_{id}`
- session context policies: --session-context-policy {trace-reported,monotonic}.
monotonic carries the full prior prompt plus the server's exact output ids,
resets only on major compaction, and fails a round rather than continuing
from re-tokenized text
- vllm-tokens backend: native token-in/token-out /inference/v1/generate,
selected with --backend vllm-tokens
- token-delivery TTFT/TPOT: first_token_id_ms, token_delivery_tpot_ms and
terminal_tail_ms measured from token-id events; the historical
completion-amortized figures are retained as an audit metric
- context-limit skip: --skip-when-reaching-limit, reserving one token of
headroom, with --skip-on-context-limit / --fail-on-context-overflow aliases
- client runtime diagnostics: peak Tokio global queue depth and
arrival_release_lag_ms
These land as one commit because they cannot be separated without breaking the
build. The elapsed_ms(SystemTime -> Instant) signature, the TraceFormat and
VibeSimRequest renames, StepLog::vibesim_request -> independent_request (which
also gains a parameter), and context_overflow_result ->
context_limit_skip_result each have call sites in files that carry more than
one of the changes above.
cargo test: 23 passed, 0 failed.
The intro mixed trace format, session context policy, arrival control and wire backend into a single flat capability table, so a reader could not tell which choices were independent of which. Replace it with an explicit four-axis section, each axis stating what it decides, plus the cross-axis constraints the runner actually enforces. Add an engine-side setup guide covering the vLLM launch flags the measurement contract depends on, and the distinction between vLLM API processes, TraceLab concurrency, and Tokio workers. Correct statements that did not match the code, and fill documented gaps: - monotonic with --trace-format independent is rejected at startup, not ignored - --max-items keeps the lexicographically smallest session ids, or the first CSV rows for the independent frontend -- not the earliest arrivals - name the vLLM flags the preflight requires (--enable-prompt-tokens-details) - the throughput window spans every attempted request while both numerators count successes only - the prefix-cache preflight probes the tail of the token pool, not the head, so it cannot warm workload unit 0 and fabricate a hit - per-request responses do record a missing cache detail as zero; only the preflight treats missing telemetry as fatal - completion_amortized_tpot anchors on first_token_ms while canonical TTFT anchors on first_token_id_ms - add a complete CLI reference, including --stop-session-on-error, which is a set-true flag defaulting to true and so cannot currently be disabled - document the per-frontend request_id shapes, which are also the x-request-id join key against server logs - complete the module map (main.rs, util.rs, executor/mod.rs, trace/mod.rs) Claims were verified against a freshly built binary: --max-items ordering, the monotonic/independent rejection, and the --max-concurrency 0 rejection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`monotonic` kept the entire accumulated context whenever the trace's target total shrank without crossing the major-compaction threshold. The live prompt then grew past the recorded workload shape: a round whose trace target was 9,500 tokens was replayed at 10,000, and the reported `derived_prefix_len` followed the client's context rather than the trace. Take the exact leading prefix truncated to `prefix_len + input_len` instead. The prefix stays token-identical, so the exact-ID carry-forward contract is untouched; only the length now honours the trace. `major_compaction` keeps its existing threshold arithmetic, so a genuine compaction still resets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`--backend sglang-tokens` drives SGLang's native `POST /generate` with `input_ids` in and `output_ids` out, so nothing in the measured path is detokenized. This is SGLang's counterpart to the existing `vllm-tokens` backend; the server must run with `--skip-tokenizer-init` and `--stream-output` (renamed `--incremental-streaming-output` in newer builds). Two guards, both fail-closed rather than silently reporting a polluted number: - Cumulative streaming. Without the delta flag SGLang resends the whole output in every chunk, which is O(n^2) wire bytes and inflates late-token latency by a position-dependent serialization cost. `restates_accumulated_output` spots a chunk that repeats everything delivered so far and fails the round naming the missing flag, instead of double-counting tokens into TPOT. - Prompt echo (sgl-project/sglang#10896). `output_ids` may lead with a segment that repeats the prompt tail. `classify_prompt_echo` uses the server's own `completion_tokens` to size the surplus, verifies it equals the corresponding prompt tail token for token, and only then trims it. An unexplained surplus fails the round; the trimmed count is logged as `echoed_prompt_tokens` so the monotonic carry-forward never silently inherits prompt IDs as output. Step-log schema goes to 7 for the new `echoed_prompt_tokens` field. The four files land together because they do not compile apart: the new `BackendKind::SglangTokens` variant makes the executor's backend match non-exhaustive until backend.rs supplies the impl, and the new outcome field breaks record.rs's struct literal plus summary.rs's test helper at the same time. Verified against SGLang 0.5.9 (Llama-3.2-1B-Instruct): 8/8 rounds succeed under both `trace-reported` and `monotonic` with planned-vs-actual prefix hit rate delta +0.0000 on every round, `echoed_prompt_tokens=0` throughout, and a server launched without the delta flag fails all 3 first rounds on the cumulative guard rather than producing readings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds an endpoint comparison table at the head of "Request backends": all backends send the prompt as token IDs and share the same prefix-cache key, so the only real difference is whether the server detokenizes the output. That makes `openai` token-in / text-out, not token-out — `force_no_detokenize` exists solely on vLLM's disagg `/inference/v1/generate` path. Also covers the new backend across axis 4, the cross-axis constraints, the CLI reference, the engine-side launch flags, and troubleshooting, and corrects the `monotonic` description to say the context is truncated to the trace target rather than retained in full. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`load_sessions` grouped rounds by `session_id` alone, but that column is not a session identity in the source. On the public trace, 16 ids span more than one session file, covering 1,216 rounds, and `(session_id, round_index)` has 514 duplicates — both documented in `artifacts/utils/DB_SCHEMA.md`. Grouping on the id alone merged unrelated conversations, interleaved their rounds by `round_index`, and then re-derived a contiguous `round_idx` over the result. The merge was therefore invisible downstream: the output looked like a well-formed session whose prefix chain never existed. Emitted sessions go from 4,265 to 4,281 on the public trace, with all 357,161 rounds preserved. The `round_idx` re-derivation itself is correct and stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A raw coding-agent trace is not replayable as reported. It records prefixes that only existed in a conversation the published data does not contain: on the public syfi trace, 3,114 of 4,281 sessions (72.7%) report a nonzero prefix on their very first round, 55.4M tokens in total. Resolving that is a decision, and until now it was made twice — once per consumer, at runtime, differently. This moves the decision upstream into one artifact both consumers read verbatim. `tracegen` reads a declared raw session CSV, applies one context policy, and emits `session-execution-v2.csv` plus a manifest and a normalized plan. In the canonical file `prefix_len` is guaranteed to exist by the time the round runs, so a consumer needs no policy and no reinterpretation — it reads two integers and builds `previous_context[..prefix_len] + fresh(input_len)`. The manifest records the source hash, the policy, the compaction thresholds, and how much of the raw trace had to be folded, because on real data that fold is the headline number, not a footnote. The canonical form is strict: rows of a session are contiguous, blocks are ordered by nondecreasing arrival, round indices are contiguous from zero, a first round declares no prefix, and milliseconds carry fixed precision. Row order is validated rather than re-derived, because it is the release order and the tie-break under equal arrivals. `tracegen` validates its own output with the same code a consumer runs. The pieces land together because they do not compile apart: - the policy arithmetic moves to a new library so the generator and the runtime share one implementation; that forces `SessionContextPolicy` out of `cli` and rewires every module that named it; - `SessionStep` gains `request_id`, which the canonical file supplies and the legacy frontend now mints in the same shape, so no existing log changes; - the session workload becomes an ordered sequence instead of a `BTreeMap<String, _>`, which had silently replaced trace order with the lexicographic order of the session id — including in `--max-items`, which kept `session-10` ahead of `session-2` regardless of arrival. Behaviour change worth calling out: `trace-reported` no longer invents prefix tokens and counts them as a cache hit. It folds an unavailable prefix into fresh input, preserving total prompt length. The old behaviour reported a large planned hit rate for a prefix the server had never seen; on the public trace that mispriced the first round of nearly three quarters of all sessions. `monotonic` is renamed `prefix-preserving` and kept as a value alias. Verified end to end on the public dataset: 4,281 sessions and 357,161 rounds materialize under both policies, the generator's fold accounting matches an independent DuckDB audit to within the 44 tokens that `convert.py`'s documented `max(...,1)` floors explain, the manifest's hash matches `sha256sum`, and the replay runtime dry-runs the canonical file and rejects both a policy flag and a non-canonical file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Source arrival times are offsets within the whole dataset. Any subset carried those absolute offsets, so a canonical file opened with a lead-in belonging to sessions it did not contain. Found by running the chain end to end rather than by reading it: a four-session smoke trace cut from the public syfi dataset began at 1,173,758 ms, and the simulator idled through 1,174 reporting ticks with `completed=0 submitted=0` before the first request existed. Two subsets of one source were also not comparable to each other, since each inherited a different origin. `tracegen` now subtracts the earliest selected arrival from every session and records it as `arrival_origin_ms`, so a row can still be traced back to the source timeline. Validation requires the first row to arrive at zero, which makes a non-rebased file a load-time error in both consumers rather than a slow simulation nobody questions. Also adds a canonical example generated from the public dataset, with its manifest and exported plan, as the fixture the cross-system plan-equality test compares against. Re-run after the fix: the dead lead-in is gone, and the smoke trace completes 20 of 20 rounds with every workload-conservation check OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Release is two independent decisions — when a unit may start, and how many may run at once — but only the second was expressible. Adds `--arrival-mode trace-timed | saturated` alongside the existing `--max-concurrency`, so "replay the recorded timeline, but never more than N conversations at once" and "saturate the server with N conversations" are both sayable. `saturated` is rejected with `--rate`, which rescales a timeline it discards. Under a cap, which unit takes a freed slot was whatever the tokio scheduler polled first, so two runs of the same trace could admit different sessions. An ordinal turn gate hands slots out in trace order, matching VibeSim's release cursor, which never releases row k+1 before row k. The gate exists only under a cap, since without one there is no contention to order. It advances on guard drop so a unit that panics cannot strand every unit behind it, and uses one Notify per unit rather than a broadcast to keep a saturated run from quadratic wake churn inside the tool that measures latency. The session-slot contract itself needed no change: `_concurrency_permit` was already bound to `run_session`'s scope, so a session holds its slot across every round and tool wait, and it is taken after the arrival wait. That is now stated in a comment rather than left as a property of where a binding happens to live. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The README listed two trace formats when the runner accepts three: `session-execution-v2` — the materialized format that is now the recommended input, and the only one whose `prefix_len` is guaranteed to exist when the round runs — was absent, along with `tracegen` that produces it. The JSONL contract was also still labelled v7 while the runner emits v8; documents `folded_prefix_tokens` and `prefix_shortfall_tokens`, and why the distinction matters (a planned departure from the source trace versus an unplanned one at run time). Adds `skills/coding-trace-replay`, the first skill covering the replay side at all. The existing five all cover the dataset pipeline, so the load generator's entire configuration surface lived only in a 1000-line README. The skill is the decision path through it, not a second copy: which value on each axis, what that commits you to, and which failures are misconfiguration rather than a real problem — including that a cap which changes nothing means the units never overlapped, which is not obvious and cost a debugging round to establish. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vLLM adopts the `x-request-id` header as the request's id, so the alignment records it emits carry the id this replay logged and the two sides join. SGLang ignores that header and generates its own rid, which broke the join outright: the alignment reader saw request timing records whose ids matched nothing in the replay log. Both engines read a body-level `rid` -- SGLang forwards it straight to GenerateReqInput, and vLLM's OpenAIBaseModel allows unknown keys and ignores it -- so the id now goes in both places and one payload works for either. The prefix-cache preflight sends `tracelab-prefix-cache-preflight` rather than a trace id, so a server log makes it obvious that row is not workload. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Closing: this crate no longer belongs in TraceLab.
It has been extracted into its own repository, carrying the full history of the 21 commits that touched The one change in this PR that is genuinely about the dataset — grouping rounds by |
serendipity-zk
added a commit
to serendipity-zk/TraceLab
that referenced
this pull request
Aug 13, 2026
The replay client was a request frontend sharing a directory with a corpus. Nothing else here referenced it: the top-level README never mentioned `replay`, `session_runner`, or `tracegen`, and its Cargo.toml already declared an empty `[workspace]` so it would not inherit a host repo. It now lives at serendipity-zk/req-frontend, carrying the full history of the 21 commits that touched this directory. The skill that documented it goes with it. PR uw-syfi#19, which would have landed this crate upstream, is closed; its one dataset-level fix is re-sent standalone as uw-syfi#20. The contract between the two repos is unchanged and unidirectional: this repo's `artifacts/trace_facts/csv_export/convert.py` exports raw session rounds, and `tracegen` over there materializes them into a canonical execution trace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns the replay client from one implicit shape — VibeSim-specific independent requests over OpenAI-compatible completions — into four independent configuration axes. Nothing in one axis implies a value in another, and the five real cross-axis constraints are enforced by the runner rather than left to the operator.
--trace-formatsession-execution-v2,session,independent--session-context-policytrace-reported(default),monotonic--arrival-mode,--rate,--max-concurrency,--max-itemstrace-timed(default),saturated--backendopenai(default),vllm-tokens,sglang-tokensWhat this enables
Typed frontends (axis 1).
sessionandindependentnow have separate schemas. An independent request is never rewritten into a session row with placeholder fields.session-execution-v2is a new already-materialized format whoseprefix_lenis guaranteed to exist when the round runs, so two different runtimes replay identical work without having to agree on a policy — this is the format to prefer for any comparison against a simulated run.Context policies (axis 2).
monotoniccarries the prior prompt plus the server's exact output token IDs, truncated or grown to the trace target, resetting only on major compaction. It fails a round rather than silently continuing from re-tokenized text. Only thesessionfrontend reads this axis;monotonicwithindependentis rejected at startup instead of ignored.Ordered admission (axis 3).
--arrival-modeis separated from--max-concurrency: one decides when a unit may start, the other how many run at once. Under a cap, units take slots in strict trace order. Without that rule the winner of a freed slot is whichever task the async runtime happened to poll first, so two runs of the same trace admit different sessions and no comparison against a simulated run means anything. A session holds its slot across all of its rounds and its tool waits — while waiting on a tool it has no request in flight but is still the unit a coding agent actually is.Native token backends (axis 4).
vllm-tokens(/inference/v1/generate) andsglang-tokens(/generate) speak token-ID deltas. All three backends send the prompt as token IDs; they differ only in whether the server detokenizes.Token-delivery metrics. TTFT/TPOT are measured from token-ID events (
first_token_id_ms,token_delivery_tpot_ms,terminal_tail_ms) with explicit, auditable boundaries. The historical completion-amortized figures are retained as an audit metric rather than redefined in place.Also included
tracegen— generates a canonicalsession-execution-v2trace from the public corpus, rebasing onto its own origin.artifacts/trace_facts/csv_export/convert.pycorrectness fix. Sessions are now keyed by(project, session_file, session_id), notsession_idalone.session_idis not unique in the public trace: 16 ids span more than one session file (1,216 rounds) and(session_id, round_index)has 514 duplicates. Grouping on the id alone merged unrelated conversations, interleaved their rounds byround_index, and hid the merge behind the contiguousround_idxthe converter re-derives — yielding a prefix chain that never existed.skills/coding-trace-replaydocuments the replay workflow.Test
cargo testinreplay/: 34 passed, 0 failed.🤖 Generated with Claude Code