Phase 10 ground truth - #3
Merged
Merged
Conversation
version is now required and validated non-empty on OCROutput, since an all-blank batch previously collapsed to one (model_name, version) pair and passed aggregate()'s mixed-version check silently — defeating rule #9. Extends the same guard to the verifier arm's identity in run_harness/aggregate, and adds version_source to Runner so the contract test can cross-check each registered runner's version against the real library's __version__ instead of trusting a hand-typed value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The single definition of what gt.csv looks like: the 12-column spec (GTToken with bbox flattened), a collect-all validator, and the loader returning run_harness's dict[str, list[GTToken]] shape. PHI-free — no pixels, no DICOM, no token inspection. Reports are PHI-free by construction: GTError has no free-text field, only a code, a 1-based row index, and a column name, with fixed explanations from a lookup table. A malformed row puts values in the wrong columns, so no cell is ever echoed. Resolved: ground_truth/ becomes a real package (hence __init__.py + pyproject); image_sizes is required so the in-bounds check cannot be silently skipped; PHI/KEEP imported from harness.metrics rather than redefined; validate_gt() collects while load_gt() raises; header must match COLUMNS including order; stdlib csv, not pandas (pandas NA-filters "NA"/"NULL" token_text even with dtype=str). 10b-10e not started. README's Phase 10 box stays unticked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…0.2, D-10.8) ground_truth/render.py turns local DICOM series into one PNG per series of the middle frame — the exact file both the annotator (10d) and every engine read. A human runs it; the agent never reads a .dcm, a render, or the back-map. Frame 0 is skipped by design: it is typically a banner/title screen, so scoring it measures the wrong thing. Two passes, so the id check is a real gate. A header-only pass (stop_before_pixels) resolves the instance, the frame and the image_id; the collision gate then runs over the whole set AND against the existing back-map; only then are pixels decoded and written, re-using the exact member path pass 1 chose. A collision raises before a single byte lands, names the shared truncated id plus both full digests, and says why re-running cannot help. Three artifacts, one PHI split: the PNGs (gitignored, Read-denied), a PHI-free manifest of image_id/frame_idx/w/h/sha256/fallback_used — the only file the agent may read — and a separate image_id -> series_uid/SOPInstanceUID back-map that is gitignored and hook-denied by rules added here (add-only; nothing loosened). Also fixes a .gitignore rule that a trailing inline comment had silently disabled. D-10.1 (windowing) is now settled and implemented in voi_to_uint8(), decided from the per-stratum tag counts (40 series/stratum, 384 sampled) rather than from expectation: raise on a VOI LUT Sequence and on a non-LINEAR VOILUTFunction (1 image each — one mapping produced every scored pixel); modality LUT before windowing (rescale on 37/37 ct_axial, 36/36 ct_scout, 27 signed images); index 0 for a multi-valued window (51/384), which keeps the rule global so the manifest needs no seventh column; the PS3.3 C.11.2.1.2 LINEAR function evaluated straight into 0-255 rather than apply_voi_lut, whose output range is y_min..y_max shifted by rescale (≈ -1024..64511 for a typical CT, which clipped into 8 bits renders near-black and loses the burned-in text); and a per-image min/max stretch when neither tag is present, guarded against max == min. Colour never reaches the policy — windowing is grayscale-only. Also resolved: D-10.2 (8-char id, --id-len 12 recovery), forced RGB output, D-10.8 (explicit input list; manifest.csv untouched), frame selection with recorded fallback provenance, and a raise for a multi-instance series whose selected instance is multi-frame. Pixels are decoded with pydicom's raw=False path, which already performs YBR -> RGB (including the JPEG component-id cases); converting again corrupted every colour frame. 68 synthetic tests, no real DICOM, none skipped. The dataset remains pseudonymized PHI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs LOCAL Tesseract over the 10b renders to pre-fill boxes + strings as a
seed the human corrects in 10d. Tesseract only: local (no egress, no BAA),
neutral (never a bake-off candidate, so seeding cannot bias the comparison),
and its errors are obvious garbage rather than plausible near-misses.
- D-10c.1 one JSON per image at <out-dir>/<image_id>.json
- D-10c.2 every seed token defaults to label=PHI; nothing implies review state
- D-10c.2a the PHI/KEEP hint comes from a later separate allowlist step (not 10c)
- D-10c.2b tessdata variant settled as `best`; still a required, never-guessed flag
- D-10c.3 w/h kept: bbox in-bounds asserts, 2x buffer check, 10d canvas sizing
- D-10c.4 pinned config + 16-field provenance in every seed file; NO conf
threshold and NO minimum length - a dropped faint token means no box in 10d
- D-10c.5 pytesseract as a pinned optional extra (wrapper only; the binary,
Leptonica and eng.traineddata are system-level and are what fix the seed)
- D-10c.6 manifest-driven + bidirectional disk sweep + re-hash preflight,
reusing render.read_manifest (never pandas: it would coerce an all-digit
image_id to int64 and break the PNG-stem join)
- D-10c.7 --on-existing {skip,overwrite}, prompt when ambiguous, non-TTY fails
loud, provenance mismatch aborts rather than mixing two provenances
- D-10c.8 PHI-free seed_summary.json outside the seed dir
- D-10c.9 a failed image gets no file (so skip retries it) and is recorded as
{image_id, exception_type} only
Safety rails, ADD-only and scoped to ground_truth/seed/ so the PHI-free
seed_summary.json stays readable: .gitignore, block_phi_read.py, settings.json
permissions.deny, and .githooks/pre-commit.
Tests are synthetic-only. The positive control needs the tesseract binary,
which is not installed here, so it SKIPS with a loud reason; set
SEED_REQUIRE_TESSERACT=1 to make it a hard failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
600s; re-seed gt_v2 (199/199)
ground_truth/review_gt.py + review_ui.html: the human verification pass between the Tesseract seed (10c) and gt.csv (10e). Stdlib http.server bound to a constant 127.0.0.1:8765 (no --host/--port/env: the page needs no auth BECAUSE it cannot be reached off-box), one vanilla-JS page, zero outbound requests of any kind. - Multiple --set <renders>:<seed> pairs, unioned. Missing seed = startup failure (D-10c.4: a silently dropped image never reaches gt.csv). Duplicate image_id dedupes when both the render sha256 AND the seed provenance match, and is a hard error naming both paths otherwise (rule #9: seeder versions are not comparable). - D-10d.A = A3: per-stratum view-time confidence gate, live per image, defaults in one config dict. Hidden means EXCLUDED from the record, never "accepted". No length floor at any stratum, ever. Unlisted stratum -> c0. conf < 0 (Tesseract's CONF_FALLBACK) and hand-drawn boxes are never gated away. - One atomic JSON per image; absence of a file means unreviewed and nothing else. Deletions are recorded explicitly so a deleted box cannot resurrect when the gate moves. round/timestamp/gate are annotation provenance only and must never become a gt.csv column. - --summary emits PHI-free counts only, pooled AND per stratum: progress, seed quality, box geometry percentiles (the number that settles iou_thr), defer breakdown, round/date coverage, gate coverage. Never a token, never the note. Safety rails widened (never narrowed) for the new PHI class: .gitignore ground_truth/review*/, hook PHI_NAME + Bash patterns for review*/ and the per-stratum seed_*/ dirs the 10c literal missed, and matching permissions.deny. /etc/claude-code/hooks/phi_guard.py needs the same branch — human action item. 20 synthetic tests (fake CMFN-style tokens only); full suite 353 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review UI + summary fixes found while actually annotating, plus ANNOTATION.md (D-10.9 conventions: tight boxes, one box per whitespace-separated run, transcribe as printed, PHI = identifies or re-links the patient). - review_gt.py: a defer now writes `deleted_seed_indexes: []`. A deferred image is undecided, so keeping the deletions made "come back later" destructive -- a sweep-delete then defer silently retired real seed boxes with no way back. - review_gt.py --summary: report the recorded PHI/KEEP label distribution per stratum. KEEP is the false-redaction and exact-match denominator, so a stratum with keep == 0 contributes nothing to the engine ranking and now says so out loud. - review_ui.html: annotation-pass usability work across canvas, list, defer and keyboard paths. PHI rails (add-only, nothing loosened): - .claude/settings.json permissions.deny: add ground_truth/seed_*/** and review_*/**. The hook already covered the per-stratum seed dirs (seed_v2, seed_pilot, seed_ct_sc, seed_ct_scout); the deny list still named seed/ only. - .gitignore: gt_pilot_*.csv. It carries series_uid, study_date and institution_name -- same Limited-Data-Set class as manifest.csv -- and the gt_sample_*.csv rule does not match that filename, so gt_pilot_v1.csv was sitting untracked-but-committable. - .gitignore: ground_truth/seed_summary*.json, extending the existing exact-name rule to the per-stratum variants (regenerable build output). tests/test_review_gt.py: 21 passed, synthetic fixtures only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ground_truth/build_gt.py turns the 10d human review records into gt.csv and freezes it. Arnav runs it; Claude reads only the PHI-free summary. gt.csv and its sidecars are produced by that run and are not in this commit. The gate: rows come from the review records and nowhere else. No seed fallback, no --force, no --skip-missing. The 10c seed is opened only by review_gt.collect() to count how much the human changed, never to produce a row. Completeness is bidirectional and runs before anything is written -- a missing record, a deferred record, an orphan record outside the scored set, or a record whose inner image_id disagrees with its filename all raise and exit non-zero. That last one matters: a copied or renamed record passes every other check and joins one image's tokens to another image's series_uid, vendor and frame_idx, with in-bounds boxes so 10a validates clean and both hashes look healthy. Two hashes, both full lowercase-hex sha256 (D-10.2): - gt.csv.sha256 pins CONTENTS; Phase 13 re-verifies it (D-13.1) - gt_set.sha256 pins SCOPE. Blank images contribute zero rows, so a build over 199 images and one over only the 102 text-bearing ones produce a byte-identical gt.csv. Without the second hash a silently shrunken set hides behind a matching content hash. Either hash moving is surfaced as a loud invalidation (rule #8), never quietly. text_presence_v2.csv (image_id,has_text) is the only record of which images were reviewed-and-blank -- 97 of 199 -- and .gitignore is negated for the text_presence_* family so that recall denominator lives in git. The summary also names any stratum with zero text-bearing images, so its recall cannot read as a saturated 100% (CLAUDE.md section 8, the ct_scout lesson). Blank controls get no stratum special-case at all: a blank record emits zero rows by itself and a ct_axial frame that turns out to have text emits normal rows through the same path, which is how it leaves the control set. A stratum filter would instead delete the one observation the negative control exists to surface. Survivors are counted, never enforced; nothing is hard-coded as a pass condition. D-10.7: with review_r2/ absent the summary prints a sentence and no number, on a separate code path -- it cannot print an agreement figure it cannot compute. A deferred round-2 record is excluded rather than counted as disagreement. D-10.8 (harness/manifest.py, the only edit outside 10e): series_uid is loaded as the DataFrame INDEX, not added to _COLUMN_MAP, so _EXPOSED_COLUMNS and the non-whitelisted-column guard are unchanged. Two keyed accessors, neither returning an identifier and nothing enumerating them: attrs_for_series -> (vendor, stratum, modality), and frame_idx_for_series, which is diagnostic-only. gt.csv takes frame_idx from 10b's render manifest -- the frame actually rendered and annotated -- and that accessor exists solely to count how often middle_frame_index disagrees with it. Duplicate/blank series_uid now fails at load; the join is 1:1 or it is wrong. Verified: 34 new tests in tests/test_build_gt.py plus 7 in tests/test_manifest.py, synthetic fixtures only. Full suite 395 passed, 1 skipped (tesseract binary absent in this VM -- pre-existing), 1 xfail (intentional, strict). ruff clean on every file touched here. Four defects found by an adversarial fresh-context review and fixed: the filename-vs-inner-image_id hole above; review_gt.collect() resolving records through a module global so --review-dir was ignored and the provenance block could describe a different batch; SetupError/ValueError escaping main() as tracebacks; and deferred round-2 records depressing the agreement figure. The plan.md D-10.8 amendment and the CLAUDE.md rule #7 amendment are on disk but not in this commit -- both files are gitignored on main as internal docs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tes, narrow deny globs near done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
built gt csv