Feat/Add GPCA (Grace-Period Confidence-Cost Aware) NEW eviction policy - #100
Feat/Add GPCA (Grace-Period Confidence-Cost Aware) NEW eviction policy#100Safaael25 wants to merge 2 commits into
Conversation
SCUEvictionPolicy treats any item without a confirmed t_prime (fewer than 6 real observations) as maximally evictable, indistinguishable from a genuinely bad match. GPCA replaces that with a grace period: unconfirmed items are protected in proportion to generation cost and access frequency instead of being evicted first; confirmed items are protected by SCU-style confidence blended with frequency. Benchmarked against LRU/FIFO/SCU/CostAware/CostAwareSCU on real LmArena data: shows a substantial hit-rate and hit-precision improvement under tight-to-medium cache pressure (up to ~6x at 20-40MB on a real repeated- query workload), but a reproducible weakness protecting infrequently- reused expensive items across long reuse gaps, and no measurable edge once the cache is generous relative to the working set. Not a universal improvement -- a targeted one, documented as such.
- Unit tests covering cold-start cost/frequency protection, the confirmed- item confidence+frequency blend, and the specific SCU behavior GPCA fixes (an unconfirmed-but-expensive item is protected instead of always evicted first). - README section documenting the protection-score formula and disclosing the empirical results honestly: a real hit-rate/precision improvement under tight-to-medium cache pressure, and a real weakness protecting expensive items across long reuse gaps. - Export GPCAEvictionPolicy from the strategies package, matching the other eviction policies.
| cold_start_floor: float = 0.2, | ||
| cold_start_freq_weight: float = 0.6, | ||
| confirmed_base: float = 0.3, | ||
| confirmed_confidence_weight: float = 0.7, | ||
| confirmed_freq_weight: float = 0.3, |
There was a problem hiding this comment.
How is a user supposed to determine all those threshold/heuristic values?
There was a problem hiding this comment.
These values weren't tuned via a formal sweep—they're reasonable starting points based on the intuition behind the formula, and they held up well across the benchmark suite (LRU/FIFO/SCU/CostAware comparison on real conversational data), providing solid hit-rate and precision gains under tight-to-medium cache pressure.
At the moment, these values are fixed as internal defaults. The user does not have access to modify or tune them, and there is no need to interact with these parameters during normal use. If needed in the future, they can be exposed as configurable settings or documented in the README as tunable parameters.
There was a problem hiding this comment.
Can you please share quantitative benchmarking results for this approach?
2 participants
What this does
GPCA is a new eviction policy I designed and implemented myself. It fixes
a specific gap in the existing SCU policy: SCU treats any cached item that
hasn't yet accumulated enough observations to have a confirmed similarity
threshold as maximally evictable — indistinguishable from a proven bad
match, regardless of how expensive or how frequently accessed it is.
GPCA replaces that with a "grace period": while an item is still
unconfirmed, it's protected in proportion to its generation cost and how
often it's already been accessed, instead of being the first thing
evicted. Once confirmed, protection blends SCU-style confidence with
access frequency.
Motivation
I noticed SCU's blind spot specifically punishes expensive,
infrequently-accessed items the hardest — exactly the items a cost-aware
policy should be protecting. GPCA is my attempt to close that gap while
keeping the confidence-based reasoning SCU already does well.
Results
Benchmarked against LRU, FIFO, SCU, and CostAware on real conversational
data: GPCA shows a real, substantial hit-rate and hit-precision
improvement under tight-to-medium cache pressure on workloads with
genuine repeated queries. This is a targeted improvement, not a universal
one — the advantage fades once the cache is generous relative to the
working set, and GPCA has a known weakness protecting rarely-reused
expensive items over long gaps, which I've documented in the README
rather than hidden.
Testing
4 new unit tests covering cold-start cost/frequency protection, the
confirmed-item blend, and the specific SCU behavior this fixes.