feat(elision): a statement is a smaller thing than a function body

[ojassug]

Follows DECISIONS §48, which made --target-reduction-ratio bind and recorded that adherence was partial, and the limit structural: elision's smallest unit was a whole function body, files typically have one dominant region — 58%, 61%, 83% measured — and a body cannot be taken in part. This divides the region into its statements.

Result — per-row A/B, 576 rows, one frozen corpus, both routes, target 0.3

	baseline	after
rows above 50% achieved	34	18
rows reducing	95	99
new fallbacks	—	0
rows that stopped reducing	—	0
moved closer to target / further	—	39 / 11

522 of 576 rows are byte-identical, and every changed row is TypeScript or Python under a ceiling. Subdivision is confined to the ceiling path on purpose: with no target the stage still takes regions whole — one marker per body rather than nine — and that is also what makes the A/B mean anything.

628 tests, typecheck, lint and build green.

The precondition was measured before the feature was written

BM25 and MMR were both cancelled for failing their preconditions after their specs existed, so the question was asked first: do dominant regions decompose at all?

bucket	files	dominant >50% of file	median sub-spans	indivisible
python	44	5	9	0
typescript	55	22	9	1

The instrument was wrong first, and the corpus is what said so

The first probe reported 1 sub-span covering 100% of the dominant region for 44 of 44 Python files — too uniform to be real. scanPythonDefBodies returns a region starting after the first body line's indentation, so region text is dedented on line 1 and indented afterwards; taking the minimum indent across all lines yields 0 and matches only line 1.

The self-test had passed, because its fixture began with a newline and therefore did not have the shape the scanner emits. A validated instrument is only validated against the inputs it was shown. Every Python fixture in test/unit/sub-region-elision.test.ts now starts mid-line for that reason, and the production splitter carries the warning.

Separately: a source comment in an earlier revision asserted a threshold sweep that had not been run. When it was run it contradicted the comment. The real table is now in the source.

What a span may be

Depth-0 boundaries only, so every span is bracket- and quote-balanced and removing one cannot cut a statement in half — a ; at depth 0 or the } returning depth to 0 for TypeScript; a line at base indentation with no bracket open and no triple-quoted string in progress for Python. A nested if block is one span, not several. elideRegions would refuse an unbalanced span rather than ship it, but a refusal is a 0% run and adherence is the point.

Python spans start after the line's indentation, inheriting scanPythonDefBodies' boundary: the marker must hold the body's column or PythonValidator reports AST_INDENTATION_ERROR, and the indentation must stay outside the replaced bytes or rehydration is not byte-identical. isSubstantiveRegion runs per span — a body can be substantive overall while one statement is nothing but a docstring, and eliding that alone is HumanEval/0 at finer granularity.

The guard, and the trade deliberately not taken

Statements below the marker floor are dropped, which in a body of short lines can be nearly all of them — leaving the caller only the survivors and no way to reach for the region as a whole. One file went 38.9% → 6.6% against a 30% target. A division now stands only if what survives still covers most of its region.

The threshold was swept, and the sweep is in the source because it shows a trade rather than an optimum:

coverage	rows >50%	new fallbacks	fallbacks fixed	closer to 0.3	further
0.25 (≈ no guard)	8	2	9	48	23
0.50	12	2	4	44	21
0.75	18	0	4	39	11
0.90	33	0	5	13	3

Dividing aggressively controls overshoot best but converts two rows that were reducing into fallbacks. The cause is not the splitter: a finer span is likelier to contain a comment carrying an imperative, cleanup:constraint-preservation refuses to lose one, and on a single-item bundle Phase 1c has no other item to keep — so the refusal is a whole-file fallback. pip/_internal/commands/cache.py goes 34.1% → 0% on a comment reading "normalized to underscores (_), meaning hyphens can never occur".

0.75 regresses nothing and was chosen on that basis. Buying a better headline with two working files is the trade this project keeps having to un-make. §50 records the condition under which the better setting becomes free: multi-item bundles, where that constraint failure names its item and Phase 1c reverts only that one.

Still open

18 rows exceed 50% because a single statement is itself dominant — one 83%-of-file span in python-validator.ts. Dividing that needs elision inside a control-flow block, which is a different question from dividing a body and is not attempted here.

🤖 Generated with Claude Code