Skip to content

fix: batch Circles Trust logs, make Gnosis RPC and IPFS gateway confi… - #882

Merged
madlabman merged 2 commits into
lidofinance:developfrom
eenti:fix/ics-circles-batch-and-ipfs-gateway
Sep 9, 2026
Merged

fix: batch Circles Trust logs, make Gnosis RPC and IPFS gateway confi…#882
madlabman merged 2 commits into
lidofinance:developfrom
eenti:fix/ics-circles-batch-and-ipfs-gateway

Conversation

@eenti

@eenti eenti commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Description

Three reliability fixes to the ICS assessment sync, found while running round 6.

1. sync_circles now batches its log fetches. It was the only target using
the stateful filter API (events.Trust.create_filter(...).get_all_entries()
eth_newFilter + eth_getFilterLogs) and the only one bypassing _fetch_logs,
so --chunk-size had no effect on it. It asked for 6.6M blocks in a single call
and failed against rpc.gnosis.gateway.fm, which caps eth_getFilterLogs at
100k — surfacing as either that error or a misleading filter not found, since
the endpoint is load-balanced. get_event_logs gains an optional
argument_filters passthrough (only forwarded when set, so existing callers and
tests are unaffected) and sync_circles uses it.

2. GNOSIS_RPC_URL is now configurable. It was the only endpoint in the
repo with no os.getenv override, which made the batch-loading fix unusable in
practice: measured head-to-head on the real Circles Trust query over the full
range, Alchemy Gnosis with 10k chunks did 664 requests in 424s returning 48 logs
with zero failures, while rpc.gnosis.gateway.fm with 100k chunks failed
67 of 67 requests (43× -32062 Block range is too large, 24×
JSONDecodeError from malformed responses). That error is not about range at
all: the same endpoint served 6.6M, 1M, 100k and 1k fine while failing 50k, 10k
and 5k. The default is unchanged, so behaviour is identical for anyone not
setting the variable, and circles already validates the chain ID via
_target_rpcs.

3. The IPFS gateway is now configurable. ipfs.io was hardcoded in
experience/sync.py and experience/sync_hoodi.py. It currently answers
HTTP 429 with retry-after: 900 plus a sunset notice — it is moving to a
service-worker-only gateway, so programmatic GET /ipfs/<cid> is going away.
dweb.link and w3s.link share the rate limit and cloudflare-ipfs.com is
discontinued; gateway.pinata.cloud serves the same content in ~5s. Round 6
needed 147 reports totalling ~38MB and the newest Hoodi report is already
827KB, so this grows every round. IPFS_GATEWAY_URL now defaults to Pinata.

Verification

  • pytest ics_assessment/tests102 passed, no test files changed.
  • sync circles re-run at the round-5 cutoff reproduces the committed
    circle_group_members.csv byte-for-byte, both over the full range and
    with --chunk-size 100000, confirming the migration is behaviour-preserving.
  • sync circles verified end-to-end with GNOSIS_RPC_URL pointing at a
    dedicated provider and --chunk-size 10000.
  • IPFS_GATEWAY_URL verified with the default and with an override, including a
    real report download.

Not included

Two related issues were measured but left out to keep this reviewable:
_fetch_logs has no retry and Web3RPCError is not an HTTPError, so a single
bad backend response propagates through run_fetch and aborts the target; and
run_sync (sync.py:303) has no per-target isolation, so one failure kills all
eleven targets. Happy to add either here or in a follow-up.

Checklist

  • Appropriate PR labels applied
  • Test coverage maintained (just coverage)
    • No need to add/update tests
    • Tests are added/updated
  • Documentation maintained
    • No need to update
    • Updated

@eenti
eenti requested a review from a team as a code owner September 8, 2026 08:41
@eenti
eenti requested review from dgusakov and madlabman September 8, 2026 08:41
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@eenti
eenti force-pushed the fix/ics-circles-batch-and-ipfs-gateway branch from 09cda53 to 9dd1e2e Compare September 8, 2026 11:49
Comment thread ics_assessment/README.md Outdated
Comment thread ics_assessment/README.md Outdated
Comment thread ics_assessment/README.md Outdated
Comment thread ics_assessment/experience/sync.py Outdated
@eenti
eenti force-pushed the fix/ics-circles-batch-and-ipfs-gateway branch from 9dd1e2e to f9d6c75 Compare September 8, 2026 12:37
@eenti
eenti requested a review from madlabman September 8, 2026 16:08
@madlabman
madlabman merged commit b5a8452 into lidofinance:develop Sep 9, 2026
9 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants