fix: batch Circles Trust logs, make Gnosis RPC and IPFS gateway confi… - #882
Merged
madlabman merged 2 commits intoSep 9, 2026
Merged
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
eenti
force-pushed
the
fix/ics-circles-batch-and-ipfs-gateway
branch
from
September 8, 2026 11:49
09cda53 to
9dd1e2e
Compare
madlabman
requested changes
Sep 8, 2026
eenti
force-pushed
the
fix/ics-circles-batch-and-ipfs-gateway
branch
from
September 8, 2026 12:37
9dd1e2e to
f9d6c75
Compare
madlabman
approved these changes
Sep 8, 2026
dgusakov
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Three reliability fixes to the ICS assessment sync, found while running round 6.
1.
sync_circlesnow batches its log fetches. It was the only target usingthe stateful filter API (
events.Trust.create_filter(...).get_all_entries()→eth_newFilter+eth_getFilterLogs) and the only one bypassing_fetch_logs,so
--chunk-sizehad no effect on it. It asked for 6.6M blocks in a single calland failed against
rpc.gnosis.gateway.fm, which capseth_getFilterLogsat100k — surfacing as either that error or a misleading
filter not found, sincethe endpoint is load-balanced.
get_event_logsgains an optionalargument_filterspassthrough (only forwarded when set, so existing callers andtests are unaffected) and
sync_circlesuses it.2.
GNOSIS_RPC_URLis now configurable. It was the only endpoint in therepo with no
os.getenvoverride, which made the batch-loading fix unusable inpractice: measured head-to-head on the real Circles
Trustquery over the fullrange, Alchemy Gnosis with 10k chunks did 664 requests in 424s returning 48 logs
with zero failures, while
rpc.gnosis.gateway.fmwith 100k chunks failed67 of 67 requests (43×
-32062 Block range is too large, 24×JSONDecodeErrorfrom malformed responses). That error is not about range atall: the same endpoint served 6.6M, 1M, 100k and 1k fine while failing 50k, 10k
and 5k. The default is unchanged, so behaviour is identical for anyone not
setting the variable, and
circlesalready validates the chain ID via_target_rpcs.3. The IPFS gateway is now configurable.
ipfs.iowas hardcoded inexperience/sync.pyandexperience/sync_hoodi.py. It currently answersHTTP 429 with
retry-after: 900plus a sunset notice — it is moving to aservice-worker-only gateway, so programmatic
GET /ipfs/<cid>is going away.dweb.linkandw3s.linkshare the rate limit andcloudflare-ipfs.comisdiscontinued;
gateway.pinata.cloudserves the same content in ~5s. Round 6needed 147 reports totalling ~38MB and the newest Hoodi report is already
827KB, so this grows every round.
IPFS_GATEWAY_URLnow defaults to Pinata.Verification
pytest ics_assessment/tests→ 102 passed, no test files changed.sync circlesre-run at the round-5 cutoff reproduces the committedcircle_group_members.csvbyte-for-byte, both over the full range andwith
--chunk-size 100000, confirming the migration is behaviour-preserving.sync circlesverified end-to-end withGNOSIS_RPC_URLpointing at adedicated provider and
--chunk-size 10000.IPFS_GATEWAY_URLverified with the default and with an override, including areal report download.
Not included
Two related issues were measured but left out to keep this reviewable:
_fetch_logshas no retry andWeb3RPCErroris not anHTTPError, so a singlebad backend response propagates through
run_fetchand aborts the target; andrun_sync(sync.py:303) has no per-target isolation, so one failure kills alleleven targets. Happy to add either here or in a follow-up.
Checklist
just coverage)