feat(doc-index): infer real section granularity, hash sections for staleness

architecture/features/traceability-validation.md

@@ -447,25 +447,33 @@ Catches structural and traceability issues that AI agents miss or hallucinate 
 
 **Input**: Markdown file path
 
-**Output**: A structural index (headings, section line ranges, per-section summary slots) cached per file, keyed by a stat-based fingerprint
+**Output**: A structural index (every heading's line range, plus a coarser "one chunk per real section" grouping at an inferred heading level, each with a content hash and a summary slot) cached per file, keyed by a stat-based fingerprint
 
 A cached, read-once-per-file structural index for Markdown JIT retrieval (see
 constructorfabric/studio#104): parsing a file's headings/section boundaries
 happens once, not once per query, until the file's content actually changes.
 The cache-validity fingerprint is deliberately metadata-only (`mtime` + file
 size via `Path.stat()`), never a content hash — the point of the cache is to
 avoid reading the file at all on a hit, and a content hash would defeat that
-by requiring the read it's meant to save.
+by requiring the read it's meant to save. A build reads the content and
+takes that fingerprint bracketed by a stat snapshot on each side, so the
+fingerprint saved is provably the one that matches what was actually parsed
+even if a write lands in the narrow window during the read.
 
 1. [x] - `p1` - Build a fresh structural index: parse headings + line ranges from current content, compute the stat-based fingerprint - `inst-doc-index-build`
 2. [x] - `p1` - Load a cached index for a file, validated against current stat metadata (no content read on a hit); returns `None` if missing, stale, or corrupt - `inst-doc-index-load`
 3. [x] - `p1` - Persist an index to its cache location; no-ops silently outside a Studio-adapted project - `inst-doc-index-save`
 4. [x] - `p1` - Return the cached index or build-and-cache a fresh one; reports cache hit/miss for benchmarking - `inst-doc-index-get-or-build`
 5. [x] - `p1` - Attach a one-line, LLM-authored summary to a cached section by its `line_start`, for a future per-section-summary caller - `inst-doc-index-annotate`
+6. [x] - `p1` - Infer which heading level represents one retrievable section: the most-recurring level wins over a level that appears only once (however shallow), since PDF-conversion heading levels don't reliably encode true nesting depth — a fixed level assumption silently produces a degenerate mega-section on such documents - `inst-doc-index-infer-level`
+7. [x] - `p1` - Group headings at exactly the inferred level into retrieval sections (off-level headings stay inside whichever section they fall under, never split one apart); hash each section's own text for section-granularity staleness detection - `inst-doc-index-retrieval-sections`
+8. [x] - `p1` - Diff the current file against its last cached build at section granularity: which retrieval sections are unchanged vs. changed, or whether the section count itself changed (a structural change, matched by position not heading text, since duplicate titles are real) - `inst-doc-index-diff-stale`
 
 **Supporting**:
 - [x] - `p1` - Stat-based cache-validity fingerprint (`mtime_ns` + size); resolved from the file's own path, never a content hash - `inst-doc-index-etag`
 - [x] - `p1` - Resolve the cache file location within the Studio directory owning the indexed file, resolved from the file's own path (not the process's working directory) - `inst-doc-index-cache-path`
+- [x] - `p1` - Read a file's content bracketed by an etag snapshot on each side, retrying on mismatch: closes the window where a write between the read and the fingerprint could save stale headings under a fresh-looking etag - `inst-doc-index-stable-read`
+- [x] - `p1` - Re-parse a file's current content into retrieval sections for staleness comparison, and build the `(heading, line_start)` identity pair that disambiguates a duplicate heading title in a diff result - `inst-doc-index-diff-stale-helpers`
 
 ### Markdown Parsing Utilities
 

skills/studio/scripts/studio/commands/doc_index.py

@@ -48,6 +48,9 @@ def cmd_doc_index(argv: List[str]) -> int:
         "total_lines": index["total_lines"],
         "section_count": len(index["sections"]),
         "sections": index["sections"],
+        "section_level": index["section_level"],
+        "retrieval_section_count": len(index["retrieval_sections"]),
+        "retrieval_sections": index["retrieval_sections"],
     }
     ui.result(output, human_fn=_human_doc_index)
     return 0
@@ -62,3 +65,11 @@ def _human_doc_index(data: dict) -> None:
         summary = f" — {s['summary']}" if s.get("summary") else ""
         ui.substep(f"  H{s['level']} [{s['line_start']}-{s['line_end']}] {s['heading']}{summary}")
     ui.blank()
+
+    level = data["section_level"]
+    ui.step(f"Retrieval sections (level {level}, {data['retrieval_section_count']} section(s))" if level is not None
+             else "Retrieval sections (no headings — none inferred)")
+    for s in data["retrieval_sections"]:
+        summary = f" — {s['summary']}" if s.get("summary") else ""
+        ui.substep(f"  [{s['line_start']}-{s['line_end']}] {s['heading']} ({s['hash'][:12]}){summary}")
+    ui.blank()