Skip to content

feat(doc-index): infer real section granularity, hash sections for staleness - #1

Open
tkcoding wants to merge 1 commit into
jit-retrieval-doc-indexfrom
jit-retrieval-cascade
Open

feat(doc-index): infer real section granularity, hash sections for staleness#1
tkcoding wants to merge 1 commit into
jit-retrieval-doc-indexfrom
jit-retrieval-cascade

Conversation

@tkcoding

Copy link
Copy Markdown
Owner

Stacked on constructorfabric#108 (constructorfabric#108)

This branches from jit-retrieval-doc-index, the branch behind constructorfabric#108 — it needs doc_index.py/toc.py to exist first, so it's opened here (within this fork) rather than against upstream directly, since GitHub won't let a cross-repo PR base on a branch that only exists on the fork. Retarget to constructorfabric/studio:main once constructorfabric#108 merges.

Summary

Resolves two of constructorfabric#108's follow-up open questions (findings doc §13b/§13c).

  • infer_section_level(): picks which heading level represents one real retrievable section, using the level's frequency as the signal. A document's real recurring structure (its chapters) shows up as the level used most often; an occasional heading at an anomalous level — exactly what PDF-to-Markdown conversion produces, since it assigns levels by font-size heuristics, not semantic depth — is rare precisely because it's noise, not structure. Levels used only once are excluded as candidates outright.

    This is a direct, verified fix for a real failure found building feat(toc,doc-index): add JIT-retrieval readiness checks and cached doc index constructorfabric/studio#108: a real PDF-converted document put all 8 of its actual chapters on H5 and a single stray subsection on H3. Treating H3 as "the" section level (or any fixed level) turned the entire back half of the document into one fake 6,601-line "section". Re-run against that same document with this change: 12 correctly-sized real sections, not one.

  • retrieval_sections (new field on the built index): headings grouped at exactly the inferred level — off-level stray headings stay inside whichever section they geographically fall under, rather than splitting one apart — each with a SHA-256 hash of its own text.

  • diff_stale_sections(): compares a file's current content against its last cached build at this granularity and reports which sections actually changed, matched by position (not heading text — duplicate titles are real, see toc-heading-duplicate). This is the piece a future caller needs to re-summarize only what changed instead of the whole document — the whole-file etag from feat(toc,doc-index): add JIT-retrieval readiness checks and cached doc index constructorfabric/studio#108 can only say "something changed", not "what".

Existing sections/annotate_section_summary/etag behavior is untouched.

Test plan

  • pytest tests/test_doc_index.py tests/test_toc.py — 156 passed, 100% coverage on touched files
  • Full suite — 4815 passed; the 12 failures present are the same pre-existing macOS-local/flaky ones seen on feat(toc,doc-index): add JIT-retrieval readiness checks and cached doc index constructorfabric/studio#108, none in files touched here
  • pylint / vulture clean; cfs validate 0 errors; spec-coverage thresholds met
  • infer_section_level re-run for real against the actual PDF-converted document that originally exposed the bug (see commit message)

…aleness

Neither of these existed before: doc_index.py indexed every heading at
every level, with no notion of "one retrievable section", and its
staleness check was whole-file only -- any edit anywhere invalidated the
entire cached index, making a real per-section partial rebuild impossible
regardless of how small the actual edit was.

infer_section_level() picks which heading level represents one real
section, using the level's *frequency* as the signal: a document's real
recurring structure (its chapters) shows up as the level used most often,
while an occasional heading at an anomalous level -- exactly what
PDF-to-Markdown conversion produces, since it assigns levels by font-size
heuristics, not semantic depth -- is rare precisely because it's noise,
not structure. Levels used only once are excluded as candidates outright.
This is a direct, verified fix for a real failure found earlier building
this feature: a real PDF-converted document put all 8 of its actual
chapters on H5 and a single stray subsection on H3; treating H3 as "the"
section level (or any fixed level) turned the entire back half of the
document into one fake 6,601-line "section". Re-run against that same
document with this change: 12 correctly-sized real sections, not one.

build_doc_index() now also computes retrieval_sections -- headings grouped
at exactly the inferred level (off-level stray headings stay inside
whichever section they geographically fall under, rather than splitting
one apart), each with a SHA-256 hash of its own text. diff_stale_sections()
compares a file's current content against its last cached build at this
granularity and reports which sections actually changed, matched by
position (not heading text -- duplicate titles are real, see
toc-heading-duplicate) -- the piece needed for a future caller to
re-summarize only what changed instead of the whole document.

Registered the three new instructions in traceability-validation.md;
whitelisted diff_stale_sections in vulture_whitelist.py alongside
annotate_section_summary (same "future caller, exercised by tests"
situation). New code is 100% covered; existing sections/annotate/etag
behavior is untouched and still passing.

See constructorfabric#104.

Verified: pytest (test_doc_index.py + test_toc.py: 156 passed; full suite:
4815 passed, the same 12 pre-existing macOS-local/flaky failures as on
main, none in the files touched here), pylint and vulture clean, cfs
validate 0 errors, spec-coverage thresholds met, and infer_section_level
re-run against the real PDF-converted document that originally exposed
the bug.

Signed-off-by: TECK KEAT WILSON <yeow.teck.keat@constructor.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant