Make ContextSelector grammar cache compatible#2686
Merged
Conversation
Comprehensive Tier-1 jest coverage for the raw-turns -> context-vector extraction subsystem: realistic multi-turn accumulation, recency ordering, tokenization robustness (punctuation/NFKC/emoji), protected patterns, plural->singular stemming consistency, window eviction, degenerate all-glue turns, and the documented adversarial limitations (negation, quoted speech). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comprehensive Tier-1 jest coverage for the context-vector + candidates -> resolve/abstain decision subsystem via TfIdfStrategy and TfIdfScorer: clear-winner resolve, shared-token cancellation (candidate-local IDF), margin/minMass/coverage gates at their boundaries, determinism under candidate reordering, and the documented adversarial score patterns (broad-token leak, negated-word pile-up). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Suppress negated content words from the conversation context vector so "not the spreadsheet" no longer deposits its topic and misroutes a collision. New opt-in tokenize option (dropNegatedSpans) enabled by the conversation signal via a negationGuard config lever (default on); keyword extraction stays byte-identical. A negation scope opens at a cue (not/no/never/without/...) and closes at a clause boundary, a reset connector (but/instead/...), or turn end. Clause-boundary detection requires whitespace adjacency so intra-token punctuation (decimals 2.5, times 3:30, versions v3.2) does not reopen the scope. Measured on the offline benchmark: adversarial wrong-target 13 -> 8, resolution accuracy 7.1 -> 20.0 percent, retrieval share 32.7 -> 56.6 percent, zero safe-tier regressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…report fix Add a CS_NEGATION_GUARD toggle to the offline runner so the pre-guard baseline can be replayed for before/after measurement. Document the shipped negation guard and the measured-net-negative global-IDF finding in the README, and add a "Methodology & relationship to the static-collision benchmark" section relating this deterministic lexical benchmark to the LLM/embedding @collision corpus probe. Compute the adversarial wrong-target count dynamically in the generated report instead of a stale hard-coded value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ites metricRunner/measureMetrics: full head-to-head of contextSelector vs every context-blind collision strategy (first-match, score-rank, priority, user-clarify), plus the deployed routing-lift of contextSelector layered on each silent auto-resolver; rendered to console and the markdown report. reproduce.mts: one-command run-all harness (measureMetrics + cached LLM arm) with a per-child V8 heap-cap safeguard (CS_BENCH_MAX_OLD_SPACE_MB, default 2 GB) so a memory regression OOM-fails fast instead of exhausting system RAM; --out is forwarded to both arms so their reports land in one directory. Tests: contextSelectorTrigger (150 trigger/abstain collisions incl. multi-turn) and contextSelectorExtractionAccuracy (75 extraction cases) - deterministic, offline, no LLM. README documents the negation-scope guard and strategy comparison. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
All arms now write a single contextSelector-report.md at a fixed local path (next to the scripts; --out overrides), instead of scattering contextSelector-metrics.{md,json} + contextSelector-vs-llm.md to cwd.
New reportFile.mts provides the shared target: path resolution, the base-report write, and an idempotent marker-delimited LLM-section upsert - so any run order or repetition yields the metrics report followed by exactly one LLM section. measureMetrics writes the base; compareLlm upserts its section; reproduce coordinates and prints the path.
Also drops the unused machine-readable JSON sidecar (nothing in the repo consumed it) and fixes the llm-cache.json trailing-newline churn so re-runs keep it prettier-clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
…define retrieval The per-tier tables (and console) showed 'spurious / wrong-target' as a single '18 / 8' cell that reads like a ratio. Split into two separate percentage rows: Spurious (over should-abstain fixtures, e.g. 54.5% (18/33)) and Wrong-target (over should-resolve fires, e.g. 80.0% (8/10)), each with its denominator. Metric 1 now defines 'appropriately retrieve the conversation's topic' in plain language (the signal source sorts recent, recency-weighted words into three buckets - intended topic / look-alike distractor / unrelated noise - and checks most weight landed on the topic), with a one-line gloss per measure. Also adds a 'Reading the columns' note defining the combined column (the full realistic calibration corpus, with the adversarial tier excluded). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
…tric goals Readability pass so the report explains itself to a reader who has never seen it. Adds a plain-language 'What this measures' overview (the grammar-collision problem, resolve/abstain, the wrong-target=0 safety promise, the offline/deterministic nature, the corpus makeup), a 'Key terms' section with a four-outcome table (correct / safe-miss / wrong-target / spurious) and one-line metric definitions, and a 'How to regenerate this report' block with the run command. Gives Metrics 2, 3, Safety, and the threshold sweep explicit 'Goal:' statements (Metric 1 already had one), and adds the same framing plus a caching note to the LLM-comparison section. De-jargons row labels (Gate A/Gate D/WRR/B-6/L2/'vice versa' -> plain language), fixes the self-contradictory 'Safety is threshold-robust: N cells produced a wrong-target' line, and drops the now-unused easy-slice metric. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
New '## Summary of results' section near the top (after the overview, before Key terms): a plain-language bottom-line conclusion plus five headline bullets — safety (0 silent misroutes across the realistic corpus), helpfulness (yield + resolution accuracy), net routing gain over first-match, how it compares with every silent strategy, and the adversarial limitation. A reader gets the conclusion up front and can scroll down for the detailed findings. All figures are interpolated from the live metrics, so the summary stays accurate on every regeneration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
…ub.com/microsoft/TypeAgent into dev/georgeng/benchmark_contextselector
…examples Adds a 'How the benchmark is built' section to the generated report: an at-a-glance one-paragraph summary of the whole benchmark (offline, deterministic, real pipeline, corpus sizes, no LLM), the three-layer structure (foundation / corpus generators / scoring engine), the Fixture shape, a generators comparison table, and a worked example fixture for each of the four corpus generators (easy / siblings / real-pairs / dialogue). Restructures the README's flat file list into the same three layers with the Fixture shape and a generator table, so the code layout and the report tell the same story. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
The report is now a tracked artifact at docs/architecture/collision/contextSelector-report.md, but the generator defaulted to the (untracked) source folder, so the committed copy drifted. Make that docs path the default output of resolveReportPath() (still overridable with --out), so a plain 'reproduce.mts' keeps the tracked report current. reproduce.mts now runs 'prettier --write' on the report as a final step: the generator emits valid markdown but not prettier-normalized table alignment, and the tracked file must pass CI's 'prettier --check .'. This bakes in the manual format pass that previously kept it clean. Regenerates the report (now including the 'How the benchmark is built' section) into the tracked location; updates README + header comments to state the docs default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
The LLM arm is a proxy, not the production fallback: it asks a default chat model to pick between the two colliding agents (described by one-line blurbs) and is explicitly told to watch for negation/sarcasm/quoting, whereas the real fallback is full schema translation across all agents on the configured model. Document that this makes the LLM look stronger than production would — so the realistic-tier 0 regressions is a robust floor, but the adversarial regression count is a worst-case (not expected) cost — and point at an L3 live replay as the faithful measurement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
Flip the shipped default contextSelector.detect to true. It runs as a deterministic, LLM-free first pass on grammar collisions: a confident topical pick resolves the collision, otherwise it abstains and (via the unchanged defer-to-strategy fallback) hands off to first-match — strictly additive over today's routing, never a silent misroute. Backed by the offline benchmark (docs/architecture/collision/contextSelector-report.md): 0 wrong-target across the ~1690-collision realistic corpus, +29% routing lift over first-match with no per-class regression. Verified the full dispatcher offline suite still passes (78 suites / 1407 tests) with the default on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
…k_contextselector
Extract helpers to bring two functions under the CI complexity caps (cyclomatic 25 / cognitive 30) without changing behavior: - tokenize (CC 32 / Cog 34): pull option resolution, the vocabulary drop check, and per-token classification into resolveTokenizeOptions, isDroppedWord, and classifyToken. tokenize is now a thin match loop. - runMetrics (CC 26 / Cog 58): extract the per-fixture accumulation loop and the assembly into a MetricsTally plus createTally, tallyBaselines, tallyResolvable, tallyAbstainable, makeAccuracy, buildDeployedLifts, and buildStrategyComparison. Verified behavior-preserving: dispatcher build clean, harness typecheck clean, 287 contextSelector specs pass, and the generated benchmark report is byte-identical (apart from its wall-clock timestamp). Ratchet now OK (0 functions over either cap).
… drop dead var and any-casts - lintReport.ts: add the contextSelector benchmark/validation harness (tsx-run CLI scripts that print reports to stdout) to CONSOLE_ALLOWED_GLOBS, matching the existing tools/scripts no-console exemption. - proveKeywordUse.mts: remove the unused listText read and two superfluous \�s any\ casts (b.decision already narrows on .kind, same as line 145). Ratchet now OK (base 7 -> head 7, +0).
…t schemas to use real schemas instead of authored hardcoded ones
…ub.com/microsoft/TypeAgent into dev/georgeng/benchmark_contextselector
Convert compareLlm's LLM call to a TypeChat createJsonTranslator with a typed CollisionVerdict schema (createTypeScriptJsonValidator), replacing the hand-rolled parseChoice JSON parsing. Add a 'C'/neither verdict so the model can decline both colliding agents (distinct from 'unclear'); both score as no-commit. Rename the module-directory const HERE -> MODULE_DIR across the harness for clarity. Regenerate llm-cache.json and the consolidated report, and refresh README numbers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 37258b23-68ac-410b-9f8c-33ba5d74234d
…ub.com/microsoft/TypeAgent into dev/georgeng/benchmark_contextselector
…t cache scenarios
Resolved 2 conflicts: - docs/architecture/collision/context-weighted-collision-resolution-design.md (section 13.3): kept the implemented registry-first/contextSelector bullets (HEAD) over main's stale "open TODO" bullet describing the same gap. - packages/defaultAgentProvider/README.AUTOGEN.md: took main's newer AI-generated output (regenerated by the daily docs pipeline; the feature commit never touched this file). Verified post-merge: pnpm install --frozen-lockfile, full workspace build (264/264), and the dispatcher spec suite (80 suites / 1449 tests) all pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 039b2e98-263c-4011-b869-3705d2c12615
GeorgeNgMsft
marked this pull request as ready for review
July 17, 2026 07:16
GeorgeNgMsft
force-pushed
the
dev/georgeng/contextselector_cache
branch
from
July 17, 2026 18:23
c53a5f5 to
0c594f1
Compare
Collaborator
|
We are going to have trouble assessing the quality/usefulness of this until we actually have real users with real behaviors and preferences. |
Contributor
Author
I agree, I implemented this though so that it could be compatible with grammar caching, which would quickly skip it otherwise |
robgruen
approved these changes
Jul 17, 2026
GeorgeNgMsft
enabled auto-merge
July 17, 2026 21:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make contextSelector work with the construction cache (registry-first Tier 1.5)
Summary
Today, when a phrase could belong to two agents but the construction cache has already learned it for just one, the router sees a single confident answer and stops. The topic-based resolver ( contextSelector ) never gets to weigh in, so it can't fix the routing even when recent conversation makes the right agent obvious. This PR lets it step in for those "cache-hidden" collisions. (Before, you had to turn the cache off with @const builtin off just to see the problem.)
Why
The cache returns one match for an ambiguous phrase, but contextSelector needs at least two options to choose between. So the cached answer always wins by default. The fix: use the neighborhood registry to notice the hidden ambiguity, then let contextSelector pick the right agent from it.
What changed
• New step (Tier 1.5) in resolveGrammarRegistryFirst ( matchCollision.ts ): when the registry marks a single cache match as ambiguous, it pulls in the related "sibling" agents and lets contextSelector choose among them. It runs after an explicit user choice (Tiers 0 and 1), so it never overrides one, and before asking the user to clarify (Tier 2) — so when intent is clear, it just routes automatically instead of prompting.
• Scorer cleanup ( matchContextSelector.ts ): a shared scoreAndDecide core, plus a new resolveContextSelectorMembers entry point that can score the registry siblings (which don't carry a MatchResult ) next to the cache matches.
• Routing: if a cache match wins, it routes with no LLM call. If a sibling wins, it saves a one-turn hint ( pendingTopicalRoute , a new field) that pickInitialSchema reads to lock in the agent and show the ↪ routed to … — recent topic note — only once the route is final, and it works even with embeddings off. This replaces an older hint that lived too long, so the note can't be wrong or carry over into the next turn.
• Safety limits: siblings are limited to agents active this turn (so it never routes to a disabled one), and it scores every real match plus the registry siblings together (so a genuine two-way match never loses a valid option).
Gating
Off by default and does nothing unless you opt in. It only runs with preference.registryFirst: on and a loaded preference.registryPath (both off/empty by default) and contextSelector.detect (on by default). Default config is unchanged.
Accepted trade-off
With registryFirst on and a real two-way match, scoring includes the registry siblings too — so a sibling the cache didn't match, but that fits the recent topic well, can win. This can change the outcome compared to scoring only the cache matches. That's on purpose: the cache isn't the final word, and a strong topic signal should be able to correct it. It stays safe because the evidence bar leans toward asking the user — more contenders means it's more likely to clarify, not less.