Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,14 @@ jobs:
# its corpus exits 2 and fails the job — an unmeasured guard is never folded into a pass.
- name: Run every guard
run: npm run check:all

# Delivery cost is the one property no guard measures, because it is a property of a WALK
# rather than of a file: the gate walks work-package end to end and compares the characters
# delivered against scripts/fixtures/token-benchmark-baseline.json. The corpus above is the
# pinned commit, so a delta is attributable to what this tree changed.
#
# A definition change that adds delivery fails here at 1%. That is the gate working: confirm
# the increase is wanted, re-record the fixture in the same commit, and say what the corpus
# gained. Left unrun, delivery rose 31.3% in 32 days with nothing reporting it.
- name: Gate delivery cost against the recorded baseline
run: npm run --silent bench:token -- --label=ci --context-mode=fresh --gate > /dev/null
53 changes: 46 additions & 7 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,12 +213,30 @@ Stdout is one JSON object with per-activity fresh/resume characters and the aggr

[`scripts/run-token-benchmark.ts`](../scripts/run-token-benchmark.ts) measures payload-char and history/ledger cost for a fixed headless walk (`work-package` / e2e `skip-optional`), comparing `context_mode: fresh` vs `persistent` and resource reference delivery. It reuses the e2e harness/walker and probes `get_resource` for linked + hot templates (the robot walker does not call `get_resource` on its own).

By default each run compares against the frozen pre-optimisation reference
[`scripts/fixtures/token-benchmark-a0-reference.json`](../scripts/fixtures/token-benchmark-a0-reference.json)
(A0: fresh, recorded 2026-07-16 against `workflows@a1409d5b`). Stderr prints a
By default each run compares against the committed baseline
[`scripts/fixtures/token-benchmark-baseline.json`](../scripts/fixtures/token-benchmark-baseline.json)
(fresh mode, recorded 2026-08-17 against `workflows@34cd5429`). Stderr prints a
compact scorecard; stdout JSON includes `vsReference` with absolute/percent deltas
and a **deliveryCostIndex** (A0 = 100, lower is better — sum of activity + workflow
+ resource + technique chars).
and a **deliveryCostIndex** (baseline = 100, lower is better — sum of activity +
workflow + resource + technique chars).

#### The gate runs on every pull request

The [Verify](../.github/workflows/verify.yml) workflow runs `--gate` at the 1% default
against the pinned corpus. No guard can measure this, because delivery cost is a
property of a walk rather than of a file — so until the job existed, delivery rose
31.3% in 32 days with nothing reporting it.

**A definition change that adds delivery fails the gate, and that is the gate working.**
Pricing corpus growth at merge is the point. When the increase is wanted:

1. Confirm it — a new activity or a widened contract legitimately costs characters.
2. Re-record the fixture from a `--no-compare` run on the same corpus commit, in the
same commit as the change.
3. Say in the fixture's `description` what the corpus gained for the characters.

A fixture recorded against a different corpus makes ordinary authoring read as a
regression, which is how a gate stops being run at all.

#### A persistent-only comparison is not a valid ship gate

Expand All @@ -240,8 +258,8 @@ measurement of the reference-delivery win, never the gate.
WORKFLOWS_DIR=/path/to/workflows npm run --silent bench:token -- \
--label=AFTER --context-mode=fresh --gate --max-regression-pct=1

# Baseline (full redelivery) — expect deliveryCostIndex ≈ 100
npm run --silent bench:token -- --label=A0 --context-mode=fresh
# Re-record the baseline (same corpus commit as the change that moved it)
npm run --silent bench:token -- --label=baseline --context-mode=fresh --no-compare

# Supplementary: the reference-delivery win. Banner-warned as cross-mode, not a gate.
npm run --silent bench:token -- --label=opt --context-mode=persistent
Expand Down Expand Up @@ -375,6 +393,27 @@ npm run test:ci -- -u # re-baseline the walk
npm run baseline:stamp # record the corpus commit it was baselined against
```

The binding-fidelity triage carries the same coupling: its 69 verdicts are judgements about
definitions as they stood at `corpusSha`. The guard prints how far the corpus has moved since,
without failing on it — a verdict usually survives edits elsewhere, and an entry whose finding no
longer occurs is already reported by name as stale.

### Sessions in flight

A definition edit reaches the runs already walking that workflow. Their variable bags were seeded
from the declarations on disk when they opened, so a declaration added since is absent until they
resume: on resume the server seeds what the bag lacks and re-stamps the recorded version. What that
does *not* cover is a run part-way through an activity whose steps changed under it.

Count them before landing:

```bash
npm run sessions:census -- --workflow work-package --status running --list
```

Zero means the edit reaches nothing in flight. A non-zero count is the set of runs that will pick it
up, and the `--list` output names each one's folder, recorded version and current activity.

## Branch Structure

| Branch | Content | Purpose |
Expand Down
12 changes: 5 additions & 7 deletions docs/dispatch_model.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,21 +88,19 @@ The run pauses at every activity boundary, because the orchestrator owns the com

The character budget carries a headroom fraction of its own because `BUNDLE_HEADROOM_FRACTION` answers a different question — how much of one activity's window may go to inlined step techniques — and at `0.80` the arithmetic admits thirteen of the main workflow's fifteen activities into one context. The activity cap covers what a character count is blind to: the establishment the server never delivers, the code the worker reads, the artifacts it drafts, and degradation across a long walk.

**Which limit binds depends on the workflow, and both cases are wanted.** The two rest on different evidence. `npm run bench:batch` measures activity payloads only — it never fetches a technique or resource lazily — so its 161,027 characters for the three-activity analysis run is the *eager floor*, not what a batch really accumulates. Read off 112 worker contexts in the sealed session records, one activity costs a median 74,109 characters once its lazy fetches are counted, with a 90th percentile of 182,642 and a maximum of 261,827. The lazy half is usually the larger one.
**Which limit binds depends on the workflow, and both cases are wanted.** The two rest on different evidence. `npm run bench:batch` measures activity payloads only — it never fetches a technique or resource lazily — so its figure for the three-activity analysis run is the *eager floor*, not what a batch really accumulates. The lazy half is usually the larger one.

At a 200,000-token window, giving a 280,000-character budget:
At a 200,000-token window, giving a 280,000-character budget, **the cap binds first on measured content**. The benchmark's three activities cost 159,093 characters batched — 78,128, then 58,588, then 22,377 — which is 57% of budget, because a batch's second and later activities collapse the invariant blocks and the ancestor contract their techniques share (see [Reference Delivery](resource_resolution_model.md#11-reference-delivery)). Standalone, the same three cost 232,954. Reaching the budget takes roughly seven activities of that weight, and a worker declaring a smaller window is bounded proportionally: the budget takes over below roughly 114,000 declared tokens on this workload.

- **On the main workflow the budget binds first** — two real runs reach it after two activities. That is the mechanism working: three heavy activities would put over half the declared window into workflow content before a line of code is read.
- **On the setup sequence the cap binds first**, its activities costing 33,000 to 154,000. That sequence is batching's first user, and a character budget alone would admit more of it than a context should hold.
- **A smaller declared window is bounded proportionally**, and where the third activity is refused depends on what the first two cost. On the median activity the budget binds before the cap below roughly 106,000 declared tokens; on the 90th percentile, below roughly 261,000 — so on heavy content the budget is the binding limit at any window worth declaring. The lighter run the benchmark walks puts the crossover near 99,000.
Admission is checked *before* a delivery rather than after, so the admitted activity can carry a batch past the budget by up to one heavy activity. Refusing after composing would pay the composition and still not un-deliver it.

Admission is checked *before* a delivery rather than after, so the admitted activity can carry a batch past the budget — by up to one heavy activity, 261,827 characters on measured content. Refusing after composing would pay the composition and still not un-deliver it.
**Revising either value needs evidence a byte count cannot supply.** The cap covers the context establishment the server never delivers, the code the worker reads and the artifacts it drafts — so `batch_refused` counts and per-activity usage rows over real runs are what a revision rests on, not a benchmark that only sees payloads.

Both limits count each delivery once. An `activity_dispatched` size is the whole `get_activity` response, so the techniques and resources it bundled eagerly are already inside it and their own observability events are not added again; what counts on top is only what the worker went back for lazily. Counting the bundled entries twice inflated one activity of the main workflow by 48% and a run of three by 70%, which made a nominal 280,000-character budget bind at 164,540.

`get_activity` reports where a context stands in `_meta.batch` (`activities`, `max_activities`, `delivered_chars`, `budget_chars`, `may_continue`), so the ordinary end of a batch is the worker stopping. Asking past the bound is refused with the payload undelivered and a `batch_refused` history event naming the limit — recorded once per scope, activity and limit, so the tally counts how often a limit bound rather than how often a worker retried. That tally is what the starting settings are revised from.

`may_continue` is answered as of that delivery, and the worker then fetches techniques and resources lazily while it runs the activity, drawing down the same budget. So a batch reported as having room can still be refused at the next boundarythe delivered and budget counts on the same response are what a reader compares to see how close it was. The refusal is an expected outcome rather than an error, and the orchestrator handles it by releasing the identity and dispatching a replacement — which must carry a **new** `agent_id`, since the bound is keyed on the identity and a fresh context under a used one would receive markers for content it does not hold.
`may_continue` on `get_activity` is answered as of that delivery, and the worker then fetches techniques and resources lazily while it runs the activity, drawing down the same budget. So a batch reported as having room can still be refused at the next boundary. `next_activity` answers the same question at the boundary instead — pass the exiting worker's `agent_id` and `context_tokens` and its `_meta.batch` counts those lazy fetches, which is the reading a continue-or-respawn decision wants. The refusal is an expected outcome rather than an error, and the orchestrator handles it by releasing the identity and dispatching a replacement — which must carry a **new** `agent_id`, since the bound is keyed on the identity and a fresh context under a used one would receive markers for content it does not hold.

Three carve-outs keep the bound aimed at what it is for:

Expand Down
Loading
Loading