fix(ingest): exponential backoff on price-service negative cache - #463
Open
matheus1lva wants to merge 3 commits into
Open
fix(ingest): exponential backoff on price-service negative cache#463matheus1lva wants to merge 3 commits into
matheus1lva wants to merge 3 commits into
Conversation
Negative day-cache ttl was fixed at 2m vs 24h for hits, so a failing key retried 720x/day against prices.yearn.dev. Now each repeat failure doubles the ttl (2m base, 6h cap, ~13 attempts/day); attempt counter kept under a separate 24h key so it survives marker expiry.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
An escalated negative marker (up to 6h) turned a replay run inside the backoff window into a silent no-op: the cached 'unavailable' returned before any fetch and tvl re-wrote the NULL row. Replays now drop the negative markers and attempt counters first, so recovery heals NULL days on the first run; cached prices are kept.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Retry loop against prices.yearn.dev: kong re-requests the same failing historical prices every ~2 minutes, each attempt costing ~14 metered drpc calls (~1.5M/day vs ~700 baseline). #462 bounded the loop; this stops the spin.
Root cause is a cache-ttl asymmetry in
packages/ingest/prices.ts: a successful past-day price caches for 24h, a failed one for 2m. With the service failing ~51% of requests, every failure returned to the queue almost immediately — 720 attempts/day per failing key.How
fanout replays) now flushes the price-service negative markers and attempt counters from Redis before enqueuing, so an escalated 6h marker can't turn the replay into a no-op. Cached real prices are kept.packages/lib/cache.tsgains akeys(pattern)accessor for that flush, backed by node-redisscanIterator(non-blocking SCAN, not KEYS); deletes run in batches of 100.Tests
prices.service-cache.mock.spec.tsfor the extra attempts-key write.bun --filter ingest test: 179 passed.bun --filter lib test: 29 passed.