Cache negative match results so sibling fragment files skip the full pipeline - #44
Open
sebclark wants to merge 3 commits into
Open
Cache negative match results so sibling fragment files skip the full pipeline#44sebclark wants to merge 3 commits into
sebclark wants to merge 3 commits into
Conversation
When a folder's files exhaust the full matching pipeline with no result, every sibling fragment re-ran the identical ~10s staged-FTS pipeline to the same NO MATCH conclusion. Cache the negative verdict per folder+scope+album/ artist for 10 minutes so siblings short-circuit. Identifier-bearing files bypass the cache; scoped rescans use different keys; hits only ever yield 'unmatched', never a match.
Contributor
Author
|
Pushed two hardening commits after testing this branch against the latest develop test suite:
Full MediaFiles + import suite: 715/715 passing with both commits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a folder's files can't be matched, every sibling file re-runs the entire staged matching pipeline to reach the identical NO MATCH conclusion. Trace-level profiling on a large library (six-figure file count, Postgres) measured:
ContainmentValidatoralone re-validated the same author against the same fields dozens of times (94% of trace volume)Sibling fragments share folder + album/artist evidence, so one verdict covers them all.
Fix
A short-TTL negative-result cache in
EvaluateHolyGrailMatchFileInternal: when a file exhausts all fallbacks with no match, record it keyed by folder + mediaType + restrictToAuthorId + unscoped + disablePathFallback + album + artist tags. Sibling files hitting the same key within 10 minutes short-circuit to unmatched with a debug log line.Safety properties
Results
On the same library, retry commands over unmatchable folders went from tens of minutes to seconds; chunks with genuine matches were unaffected (their pipeline is untouched).
Found while profiling why large-library retry matching was slow — happy to adjust key composition or TTL if you'd prefer different trade-offs.