Skip to content

Cache negative match results so sibling fragment files skip the full pipeline - #44

Open
sebclark wants to merge 3 commits into
Chaptarr:developfrom
sebclark:negative-match-cache
Open

Cache negative match results so sibling fragment files skip the full pipeline#44
sebclark wants to merge 3 commits into
Chaptarr:developfrom
sebclark:negative-match-cache

Conversation

@sebclark

@sebclark sebclark commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Problem

When a folder's files can't be matched, every sibling file re-runs the entire staged matching pipeline to reach the identical NO MATCH conclusion. Trace-level profiling on a large library (six-figure file count, Postgres) measured:

  • ~11.6s average per no-match evaluation (max ~26s) — full staged FTS, containment validation, and fallbacks per file
  • a 40-fragment unmatchable book burns ~8 minutes of CPU concluding "no" 40 times
  • in one 4-minute window, 25 no-match pipelines consumed 290s of pipeline time; ContainmentValidator alone re-validated the same author against the same fields dozens of times (94% of trace volume)

Sibling fragments share folder + album/artist evidence, so one verdict covers them all.

Fix

A short-TTL negative-result cache in EvaluateHolyGrailMatchFileInternal: when a file exhausts all fallbacks with no match, record it keyed by folder + mediaType + restrictToAuthorId + unscoped + disablePathFallback + album + artist tags. Sibling files hitting the same key within 10 minutes short-circuit to unmatched with a debug log line.

Safety properties

  • consulted only after the identifier short-circuit — a file with matching ISBN/ASIN evidence never touches the cache path
  • scope changes (e.g. an author-scoped rescan after an auto-add) produce different keys → cache miss → full evaluation
  • 10-minute TTL bounds staleness against catalog growth (new author/edition arrivals)
  • a cache hit can only yield "unmatched", never a wrong match — worst case a file waits for a later pass

Results

On the same library, retry commands over unmatchable folders went from tens of minutes to seconds; chunks with genuine matches were unaffected (their pipeline is untouched).

Found while profiling why large-library retry matching was slow — happy to adjust key composition or TTL if you'd prefer different trade-offs.

When a folder's files exhaust the full matching pipeline with no result,
every sibling fragment re-ran the identical ~10s staged-FTS pipeline to the
same NO MATCH conclusion. Cache the negative verdict per folder+scope+album/
artist for 10 minutes so siblings short-circuit. Identifier-bearing files
bypass the cache; scoped rescans use different keys; hits only ever yield
'unmatched', never a match.
@sebclark

Copy link
Copy Markdown
Contributor Author

Pushed two hardening commits after testing this branch against the latest develop test suite:

  • Instance-scoped the cache (was static). The service is a singleton in production so behaviour is unchanged, but static state leaked between test fixtures that build their own service instances and reuse similar folder paths — 14 matching tests failed when run together and passed in isolation.
  • Suppress the cache during manual preview (new MatchingContext.SuppressNegativeUnitCache, flow-scoped via AsyncLocal). PerFileMatchingIdentitySplitFixture.manual_preview_should_not_fast_path_unsampled_meaningful_leftover_into_short_title correctly requires an exact verdict per file in manual preview; sibling files sharing a cached negative is the right trade for bulk scans but not there.

Full MediaFiles + import suite: 715/715 passing with both commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant