feat(doc-index): infer real section granularity, hash sections for staleness - #1
Open
tkcoding wants to merge 1 commit into
Open
feat(doc-index): infer real section granularity, hash sections for staleness#1tkcoding wants to merge 1 commit into
tkcoding wants to merge 1 commit into
Conversation
…aleness Neither of these existed before: doc_index.py indexed every heading at every level, with no notion of "one retrievable section", and its staleness check was whole-file only -- any edit anywhere invalidated the entire cached index, making a real per-section partial rebuild impossible regardless of how small the actual edit was. infer_section_level() picks which heading level represents one real section, using the level's *frequency* as the signal: a document's real recurring structure (its chapters) shows up as the level used most often, while an occasional heading at an anomalous level -- exactly what PDF-to-Markdown conversion produces, since it assigns levels by font-size heuristics, not semantic depth -- is rare precisely because it's noise, not structure. Levels used only once are excluded as candidates outright. This is a direct, verified fix for a real failure found earlier building this feature: a real PDF-converted document put all 8 of its actual chapters on H5 and a single stray subsection on H3; treating H3 as "the" section level (or any fixed level) turned the entire back half of the document into one fake 6,601-line "section". Re-run against that same document with this change: 12 correctly-sized real sections, not one. build_doc_index() now also computes retrieval_sections -- headings grouped at exactly the inferred level (off-level stray headings stay inside whichever section they geographically fall under, rather than splitting one apart), each with a SHA-256 hash of its own text. diff_stale_sections() compares a file's current content against its last cached build at this granularity and reports which sections actually changed, matched by position (not heading text -- duplicate titles are real, see toc-heading-duplicate) -- the piece needed for a future caller to re-summarize only what changed instead of the whole document. Registered the three new instructions in traceability-validation.md; whitelisted diff_stale_sections in vulture_whitelist.py alongside annotate_section_summary (same "future caller, exercised by tests" situation). New code is 100% covered; existing sections/annotate/etag behavior is untouched and still passing. See constructorfabric#104. Verified: pytest (test_doc_index.py + test_toc.py: 156 passed; full suite: 4815 passed, the same 12 pre-existing macOS-local/flaky failures as on main, none in the files touched here), pylint and vulture clean, cfs validate 0 errors, spec-coverage thresholds met, and infer_section_level re-run against the real PDF-converted document that originally exposed the bug. Signed-off-by: TECK KEAT WILSON <yeow.teck.keat@constructor.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on constructorfabric#108 (constructorfabric#108)
This branches from
jit-retrieval-doc-index, the branch behind constructorfabric#108 — it needsdoc_index.py/toc.pyto exist first, so it's opened here (within this fork) rather than against upstream directly, since GitHub won't let a cross-repo PR base on a branch that only exists on the fork. Retarget toconstructorfabric/studio:mainonce constructorfabric#108 merges.Summary
Resolves two of constructorfabric#108's follow-up open questions (findings doc §13b/§13c).
infer_section_level(): picks which heading level represents one real retrievable section, using the level's frequency as the signal. A document's real recurring structure (its chapters) shows up as the level used most often; an occasional heading at an anomalous level — exactly what PDF-to-Markdown conversion produces, since it assigns levels by font-size heuristics, not semantic depth — is rare precisely because it's noise, not structure. Levels used only once are excluded as candidates outright.This is a direct, verified fix for a real failure found building feat(toc,doc-index): add JIT-retrieval readiness checks and cached doc index constructorfabric/studio#108: a real PDF-converted document put all 8 of its actual chapters on H5 and a single stray subsection on H3. Treating H3 as "the" section level (or any fixed level) turned the entire back half of the document into one fake 6,601-line "section". Re-run against that same document with this change: 12 correctly-sized real sections, not one.
retrieval_sections(new field on the built index): headings grouped at exactly the inferred level — off-level stray headings stay inside whichever section they geographically fall under, rather than splitting one apart — each with a SHA-256 hash of its own text.diff_stale_sections(): compares a file's current content against its last cached build at this granularity and reports which sections actually changed, matched by position (not heading text — duplicate titles are real, seetoc-heading-duplicate). This is the piece a future caller needs to re-summarize only what changed instead of the whole document — the whole-file etag from feat(toc,doc-index): add JIT-retrieval readiness checks and cached doc index constructorfabric/studio#108 can only say "something changed", not "what".Existing
sections/annotate_section_summary/etag behavior is untouched.Test plan
pytest tests/test_doc_index.py tests/test_toc.py— 156 passed, 100% coverage on touched filespylint/vultureclean;cfs validate0 errors;spec-coveragethresholds metinfer_section_levelre-run for real against the actual PDF-converted document that originally exposed the bug (see commit message)