Rectify: Recipe Delivery Budget Reconciliation (#4557) - #4564
Merged
Trecek merged 81 commits intoAug 13, 2026
Merged
Conversation
Trecek
force-pushed
the
impl-rectify-recipe-delivery-budget-reconciliation-20260811-100109
branch
from
August 12, 2026 15:20
4dd29ec to
c515c5e
Compare
- Add meta=response_backstop_tool_meta('get_recipe_section') to the
@mcp.tool decorator, attaching the 195,000-char annotation waiver
that was registered but never wired (incident root cause).
- New bidirectional AST parity test (tests/arch/test_response_backstop_parity.py)
modeled on test_all_mcp_tools_are_registered: asserts set equality between
RESPONSE_BACKSTOP_EXEMPTION_REGISTRY keys and meta= attachment sites.
- New runtime metadata parity test (tests/server/test_response_backstop_metadata.py)
at both Layer 2 (mcp.list_tools) and Layer 3 (Client wire output): verifies
anthropic/maxResultSizeChars, measurement_id, and max_utf8_bytes match the
registry for every exempted tool.
- Convert enumerated-subset tests in test_pretty_output_recipe.py from hardcoded
tuples to RESPONSE_BACKSTOP_EXEMPTION_REGISTRY iteration — future registry
additions are automatically covered.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add tools_recipe.py to _LINE_LIMIT_EXEMPTIONS (1050 lines) — the meta= decorator expansion from 1 line to 4 lines pushed past 1000. - Wrap _transforms.clear() calls in test_response_backstop_metadata.py with try/finally guards to satisfy test_transforms_hygiene. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ersion policies - Export RECIPE_RESPONSE_MAX_UTF8_BYTES (195,000) and RECIPE_RESPONSE_DEFAULT_BYTES (90,000) as public core constants. Config dataclass and tools_recipe fallback now import from core. - New reconciled resolver resolve_recipe_section_response_bound in core/_delivery_bounds.py: an override is an input to reconciliation (clamped by exemption ceiling), never a bypass. - resolve_recipe_section_bound_bytes delegates to core resolver; call sites in _recipe_delivery.py and tools_recipe.py pass their surface's exemption ceiling. - Named conversion policies in core/types/_type_dimensions.py: CLIENT_CHARS_PER_TOKEN_POLICY (4:1, client heuristic) replaces bare * 4 in _response_budget.py; CONSERVATIVE_ADMISSION_POLICY (1:1) replaces inline byte-count return in _conservative_token_upper_bound. - OutputBudgetConfig.__post_init__ validates page_max_bytes upper bound. - New arch guards: literal-uniqueness (195K/90K), conversion-authority (token-limit × numeric outside core). - New tests: reconciled-bound unit tests, config↔YAML parity for OutputBudgetConfig, hook path-copy sync guard. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Hoist deferred imports to module level in _recipe_delivery.py and _recipe_section_pagination.py (IL-3 cross-layer import guard). - Move BackendCapabilities() default access to module-level constant _DEFAULT_CONSERVATIVE_LIMIT in tools_recipe.py (construction-site guard). - Add _type_dimensions to SINGLETON_ALLOWED_MODULES for named conversion policies (CLIENT_CHARS_PER_TOKEN_POLICY, etc). - Update type_constants_split_completeness expected count to 149 (added RECIPE_RESPONSE_MAX_UTF8_BYTES + RECIPE_RESPONSE_DEFAULT_BYTES). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…pe.py Use a literal _DEFAULT_CONSERVATIVE_LIMIT=10_000 with derivation comment instead of constructing BackendCapabilities() — the construction-site arch guard requires all fields as kwargs for full construction sites. Drift guarded by test_resolve_general_output_token_limit_per_backend. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…otify reconciliation Address audit findings REQ-002, REQ-004, REQ-005, REQ-006: - AST parity test now detects stray response_backstop_tool_meta() calls outside decorator meta= kwargs (REQ-002). - Export CONSERVATIVE_RESULT_TOKEN_FLOOR from core; tools_recipe.py and _notify.py consume the named constant (REQ-005). - _notify.py routes get_recipe_section token limit through resolve_recipe_section_bound_bytes with the exemption ceiling, using the reconciled resolver instead of independently deriving from page_max_bytes (REQ-006). - New bound-consumption provenance guard (tests/arch/test_delivery_bound_provenance.py) pins .page_max_bytes/.response_max_bytes read sites to resolver call sites (REQ-004 partial). - New default-config four-call efficiency pin (tests/contracts/test_default_config_four_call_efficiency.py) replaces the runtime invariant as the non-negotiable budget guarantee (REQ-004 partial). - compile_bounded_page_plan gains output_budget parameter for conservative-fallback testing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test 6 from the plan: Claude backend + page_max_bytes=None must produce a working ENVELOPE plan for the implementation recipe. Currently xfail (strict=True) because _MAX_PAGES_PER_INITIALIZATION_SECTION=1 rejects multi-page plans — will go green when Step 15's fail-degrade planning is implemented. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…re cleanup Stage C — Budget at client-measured layer, typed units: - SerializedChars frozen dataclass + client_serialized_char_len measurement function in core/types/_type_dimensions.py - Exemption admission checks both byte and char ceilings - Post-mutation final-form char check in get_recipe_section - Unit-mixing mypy rejection test + measurement function tests Stage D — Flat delivery encoding: - _render_candidate flattens json-array-page content to structured lists (errors, flow_records, warnings) — −44% wire size - Format-aware verification identity check in _verification.py - Updated Codex conformance test + pull test + pagination test + helpers - New flat-array-encoding contract test Stage E — Annotation-aware one-call inline + host attestation: - Client-gate constants (25K/50K/500K) single-sourced in core - HostClientAttestation frozen type for launcher-sourced capabilities - resolve_recipe_delivery_decision gains annotation-aware ORDINARY_INLINE path with conservative (None) defaults preserving Codex compatibility - _backend_cmd_builder_base derives injected token value from core constant - Decision-level attestation tests (4 cases) Stage F — Ship only what's consumed: - finalized_recipe_projection removed from inline surface_payload (retained in persisted artifact) - Removed from _fmt_recipe.py initialization fields - Removed from LoadRecipeResult + OpenKitchenResult TypedDicts - Wire-schema contract test asserting absence from inline delivery Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Monkeypatches client_serialized_char_len to force an oversized char measurement on a real get_recipe_section pull, asserting the recipe_section_bound_too_small failure fires. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix ruff I001 import sorting in core/__init__.pyi - Bump _recipe_section_pagination.py line limit to 1030 (char-ceiling plumbing + json-array-page flattening) - Skip consolidate-health-reports in envelope-only contract tests when recipe resolves ORDINARY_INLINE (payload fits unnegotiated limit) - Add Stage C post-mutation char-ceiling test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
REQ-B5: Replace fixed pagination limits with bound-derived liveness. Removed _MAX_PAGES_PER_INITIALIZATION_SECTION; validate only rejects non-terminating plans (0-page sections). Conservative-fallback liveness test now passes (xfail removed). CALIBRATED_PAGES_PER_SECTION retained as a default-config calibration target. REQ-C2/D3: Thread char_ceiling through production planning. _initialization_requirements, get_or_build_recipe_section_page_plan, and get_recipe_section all pass the surface's max_chars through to verify_finalized_recipe_section_plan. Cache key includes char_ceiling. REQ-C3: Type character margins with SerializedChars. _exemption_admitted_chars accepts/returns SerializedChars. REQ-D4: Pin implementation flow_records to <=115K compiled bytes. REQ-E1: Wire host_client_attestation into finalize_recipe_delivery. resolve_recipe_delivery_decision receives attestation, serialized chars, and annotation ceiling from the delivery finalizer. REQ-E2: Document unannotated regime gate derivation. REQ-E3: Derive 46,500 from CLAUDE_INJECTED_CLIENT_RESULT_TOKENS * 93%. Test sites import the derived value instead of hardcoding. REQ-E4: Attestation transport via launcher env vars. AUTOSKILLIT_ATTESTED_CLIENT_GATE_TOKENS and _META_SUPPORT injected in SHARED_BASELINE_ENV, read by _resolve_host_client_attestation() in _recipe_delivery.py, stripped from IDE env, required in env symmetry. REQ-E6: Delivery-mode ledger (recipe × backend → mode). REQ-F2: Consumed-field wire schema contract test. REQ-F3: Live e2e delivery verification tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codex safety: attestation env vars moved from SHARED_BASELINE_ENV to Claude-only injection in claude.py. Decision branch scoped with budget-is-None guard + 500K cap check. #4399 ad-hoc exemption override removed — annotation-aware inline branch replaces it. Config: nullable response/page relationship validation. Request state: separate recipe_section_bound_chars field. Final-form check: uses char ceiling instead of byte bound. ADR-0005: updated for fail-degrade pagination semantics. ADR-0004: documented attested one-call inline path. Stage F: exact consumed-field assertion, persisted-artifact projection retention test, pinned delivery-mode ledger with attestation. Literal guards: scoped 25K/50K/500K uniqueness checks. Host attestation: lru_cached resolution, Claude-only env transport. Array reserialization: documented as envelope-only (content stays parsed). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Unit-mixing test: removed type:ignore suppression, assert specific arg-type diagnostic on the accept_chars call - Liveness test: assert multipage delivery (max_pages > 1) - Flow-size pin: use compiled_bytes as conservative char proxy - E2E test: require claude_code_inline mode + exactly 1 call - Attestation parsing: malformed meta_support (not '0'/'1') → None; non-positive gate tokens → None - Counter updates: targeted regex replacement instead of json.loads/dumps round-trip — eliminates array-content reserialization - Decision: use attested_client_gate_tokens for unannotated regime when present; fall back to CLAUDE_DEFAULT_CLIENT_RESULT_TOKENS (25K) when absent for Claude; Codex retains its static capability Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert effective_unannotated_limit change — the original ordinary_limit logic was correct for the unannotated fast path. Attested gate flows through annotation-aware path, not here. - Remove lru_cache from _resolve_host_client_attestation — it poisoned tests that monkeypatch env vars across test cases. - Remove functools import (no longer used). - Remove unused CLAUDE_DEFAULT_CLIENT_RESULT_TOKENS import. - Counter update: use json.loads/dumps (not regex) — the content field passes through unchanged as a Python list, not re-encoded. - E2E test: set attestation env vars via monkeypatch.setenv to simulate real Claude launcher behavior. - Unit-mixing test: verify runtime TypeError on min()/comparison instead of subprocess mypy invocation (mypy not available in test venv). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The implementation recipe's payload exceeds the 195K annotation ceiling even after Stage F's projection removal, so it correctly resolves to bounded ENVELOPE. Added a separate test for a small recipe (consolidate-health-reports) that proves the one-call inline path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Config defaults: explicit core-constant binding assertions - Wire schema: exact consumed-field check (equality, not intersection) with serialized-size measurement - Flat-array: nested JSON-string rejection test, fragment string preservation test - Decision: two-regime invariant chain test (annotated char-gated independently from unannotated token-gated) - Unit mixing: runtime TypeError test replaces mypy subprocess test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove 'warnings' from _CONSUMED_INLINE_FIELDS: the warnings field is handler-injected by open_kitchen/load_recipe tool handlers, not by the delivery pipeline. The test fixture bypasses handlers entirely via load_and_validate + build_open_kitchen_recipe_payload, so warnings is never present in the fixture output for any recipe. - Increase fragment test bound from 300 to 2048 bytes with a 4000-char element: the page envelope overhead (~1475 bytes of descriptor/SHA fields) makes 300 bytes insufficient to fit even a single-char fragment page. The larger element + bound reliably triggers multi-page fragmentation. - Convert client-facing fitness assertions from byte-based to client_serialized_char_len: envelope fit at line 217, raw spill projection at line 283, and _generic_backstop_bound_bytes helper via CLIENT_CHARS_PER_TOKEN_POLICY. Byte-based assertions for enforce_response_budget output remain correct (the function internally budgets in UTF-8 bytes). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The static CLAUDE_CODE_CAPABILITIES.unnegotiated_tool_result_token_limit must derive from the conservative (unattested) 25,000-token client gate, not the attested 50,000 value injected at runtime: 25,000 × 93% = 23,250 (was: 50,000 × 93% = 46,500) The attested value (50K) flows only through host attestation at runtime via AUTOSKILLIT_ATTESTED_CLIENT_GATE_TOKENS, never through this static capability table. Recipes exceeding the 23,250 unattested threshold but fitting the exemption ceiling inline via the annotation-aware path when attestation is present. Also revert fitness assertion at line 283 back to byte-based: the enforce_response_budget function budgets in UTF-8 bytes, so its output is correctly measured in bytes, not serialized chars. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add test_regimes_are_independent_no_cross_unit_comparison: proves unannotated regime outcome is unaffected by char-domain inputs, and annotated regime outcome is unaffected by varying the token limit. - Enhance flow-record size pin to measure the actual envelope as client-serialized chars via client_serialized_char_len, not just the compiled_bytes proxy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Read host client attestation once in make_context() and store it on ToolContext.host_client_attestation. finalize_recipe_delivery reads from the context first, falling back to env-read only for test SimpleNamespace contexts that don't carry the field. This satisfies Stage E's requirement that the composition root owns attestation intake. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert ToolContext.host_client_attestation addition: ToolContext fields must use Protocol types per architectural contract, and HostClientAttestation is a frozen dataclass. The attestation env-read in _recipe_delivery.py's _resolve_host_client_attestation() is fast (os.environ) and correct. - Change _RECIPE_ORDINARY from 'full-audit' to 'promote-to-main-wrapper' in test_attestation_delivery_reachability: full-audit exceeds the new 23,250-token unattested limit. - Set BACKEND_REGISTRY['claude-code']() on mock_ctx.backend in test_tools_kitchen_envelope smoke tests: backend=None falls back to CLAUDE_CODE_CAPABILITIES which now has the lower limit. - Reduce test payload in test_decorated_run_skill_preserves_routing to 15K+15K with response_max_bytes=20K: the 30K+30K payload exceeds the 23,250-token delivery bound, triggering irreducible_shape failure. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The smoke-test recipe is ~1.6KB — well within any token limit. Setting a real ClaudeCodeBackend changes the open_kitchen handler's delivery path in ways that affect field passthrough (ingredients_table dropped from the surface payload). backend=None with the CLAUDE_CODE_CAPABILITIES fallback correctly exercises these ingredient-resolution tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The smoke-test recipe's rendered inline payload (~25KB with flow records) exceeds the conservative 1:1 admission policy at 23,250 tokens, routing to ENVELOPE and stripping ingredients_table. These tests exercise ingredient resolution, not delivery mode — provide a mock backend with a 46,500-token limit so the inline path is exercised. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…provenance Address all five audit findings from the second audit pass: Finding 1 (typed config boundaries): Add settings.py coercion branch for Utf8ByteLimit wrappers, named CONSERVATIVE_GATE_HEADROOM constants (93/100), CLAUDE_ANNOTATION_SUPPORT_MIN_VERSION constant, and char-margin helper (_recipe_exemption_admitted_chars) with shared 90% headroom factor. Finding 2 (delivered-form flattening): Eliminate redundant json.loads in get_recipe_section by guarding the reparse with else-branch — when the InitializingRecipe branch already produced rendered_payload, skip the unconditional reparse. Add delivered-form 115K flow_records size proof test. Finding 3 (attestation regime chain): Validate attested_client_gate_tokens against CLAUDE_INJECTED_CLIENT_RESULT_TOKENS before trusting attestation. Derive annotation support from the version probe in ensure_pre_launch() instead of hardcoding it. Use attested gate tokens (with headroom) for the unannotated regime when attestation is valid and budget is None (Claude only). Finding 4 (wire schema exact equality): Assert exact equality of inline payload fields against consumed + handler-injected sets, rejecting unexpected wire fields. Finding 5 (doc/provenance tests): Import real constants in doc-pin test (test_output_budget_protocol_decision.py). Add provenance guards for attestation gate vs annotation ceiling independence and gate validation. Non-blocking note: Document the 7% margin authority via named constants CONSERVATIVE_GATE_HEADROOM_NUMERATOR/DENOMINATOR used in both the static capability derivation and the attested gate computation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tation, wire reduction Typed config boundaries (Finding 1): - Convert Utf8ByteLimit, TokenLimit, SerializedChars from frozen dataclasses to int subclasses — backward-compatible with all arithmetic/comparisons while providing mypy type discrimination. Bool and float rejected. - Retype OutputBudgetConfig.response_max_bytes/page_max_bytes as Utf8ByteLimit with auto-coercion from bare int via int subclass inheritance. - Fix strict type(x) is int check in RecipeSectionRequestStateBase to isinstance(x, int) so Utf8ByteLimit values pass validation. - settings.py Utf8ByteLimit coercion branch retained as the single config-loader construction point. Flow-record object parsing (Finding 2): - Parse flow_records elements from canonical JSON strings to dicts in _render_candidate (defensive: only parses strings that decode to dicts). - Update verification content_matches to expect parsed objects for flow_records. Context-owned attestation (Finding 3): - Add initialize_host_client_attestation() / get_context_host_client_attestation() to _recipe_delivery.py — reads env once, caches result. - Call from make_context() (composition root) so finalize_recipe_delivery consumes the cached value instead of rereading os.environ per call. - E2e test verifies attested delivery completes with usable content. Wire reduction pin (Finding 4): - Add test_implementation_wire_reduction_from_projection_removal asserting >= 150KB reduction between persisted artifact and wire payload. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Trecek
force-pushed
the
impl-rectify-recipe-delivery-budget-reconciliation-20260811-100109
branch
from
August 13, 2026 02:31
2c285cd to
825df6e
Compare
Trecek
deleted the
impl-rectify-recipe-delivery-budget-reconciliation-20260811-100109
branch
August 13, 2026 03:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
open_kitchenon the Claude backend is forced into ENVELOPE delivery, and the resultingget_recipe_sectionpage (189,773 client-measured chars) spilled the Claude Code client's MCP output gate to disk. Four structural weaknesses compound: (1)get_recipe_sectionhas aRESPONSE_BACKSTOP_EXEMPTION_REGISTRYentry granting a 195,000-charanthropic/maxResultSizeCharswaiver that its decorator never attaches; (2) four-plus unreconciled limit systems with at least five independent bytes↔tokens conversion authorities and no single resolver; (3) the server budgets in compiled UTF-8 bytes while the client gates in JSON-serialized chars (+41.4% escaping inflation); (4) the 451,507-byte payload carries three parallel representations of one recipe, the largest of which (finalized_recipe_projection, 205KB) has zero programmatic readers.This plan makes the architecture innately immune in six sequenced stages (A–F): registry↔decorator parity, single-source limits with a reconciled resolver, client-measured typed units, flat delivery encoding, annotation-aware one-call inline with host attestation, and ship-only-what's-consumed wire schema.
Stage A — Registry↔decorator parity + spill stop
meta=response_backstop_tool_meta("get_recipe_section")— the 195,000-char annotation waiver that was registered but never wiredStage B — Single-source limits, reconciled resolver, conversion policies
RECIPE_RESPONSE_MAX_UTF8_BYTES(195K),RECIPE_RESPONSE_DEFAULT_BYTES(90K),CONSERVATIVE_RESULT_TOKEN_FLOOR(10K) exported from coreresolve_recipe_section_response_bound— override is reconciliation input, never bypass* 4arithmeticStage C — Budget at client-measured layer, typed units
SerializedCharstype +client_serialized_char_lenmeasurement functionrecipe_section_bound_charsin request stateStage D — Flat delivery encoding
json-array-pagecontent flattened to structured lists at_render_candidateStage E — Annotation-aware one-call inline + host attestation
HostClientAttestationtype with Claude-only env injection (not Codex)ORDINARY_INLINEdecision scoped to unprotected backends with 500K capCLAUDE_INJECTED_CLIENT_RESULT_TOKENS * 93%Stage F — Ship only what's consumed
finalized_recipe_projectionremoved from inline wire payload (retained in persisted artifact)Closes #4557
Implementation Plan
Plan file:
.autoskillit/temp/rectify/rectify_recipe_delivery_budget_reconciliation_2026-08-11_073208.md🤖 Generated with Claude Code via AutoSkillit