feat: round-robin repo-enumeration checkpoint + secondary rate-limit throttle#3
Merged
Conversation
…throttle The upstream RepositoryStream re-walks its full repo list in the same endpoint-imposed order every run (by updated_at, which the "since" param can't actually filter on either the org-listing or search endpoints), so it perpetually re-verifies the same front-of-queue repos and never advances into the rest of the catalog before hitting GitHub's secondary rate limit. Most of an org's repos never get PR/commit data synced at all, despite running on schedule for months. - RepositoryStream now round-robins through organizations/searches mode partitions: resumes right after the last repo whose children fully synced last run (tracked via a `round_robin_checkpoint` state key, deliberately not nested under `progress_markers`, which singer_sdk wipes on every partition finalize), wrapping at the end. Traversal order is sorted by immutable repo `id`, not the API's volatile `updated_at` order, so reshuffling from ordinary repo activity doesn't cause redundant re-syncs of already-current repos before genuinely new ones are reached. - Logs a one-line round-robin summary at the end of each partition's enumeration (repos processed, elapsed time, full lap or not, next resume point) - fires even on a mid-run crash via a `finally` block, since a crash from child-stream syncing tears down the paused generator (GeneratorExit) rather than skipping straight to process exit. - Adds an adaptive, tap-wide throttle for the secondary (frequency) rate limit: escalates a shared inter-request delay on each hit (capped), applied to every subsequent request from any stream, and decays it back down after a run of clean requests. Previously a secondary-limit hit only paced the one retried request while everything else kept firing at the same rate that triggered it. Verified against live GitHub data (fresh + simulated mid-cycle-crash resume) and via fake-clock/logic simulations for the throttle and reshuffling-robustness properties - see PR description for details.
ttomalak-bkpr
force-pushed
the
round-robin-checkpoint
branch
from
July 17, 2026 11:16
8cc65b9 to
d7168c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the actual motivation for forking
tap-githubin the first place (seelumapps/github-automation#2571).Root cause:
RepositoryStream(theorganizations/searchesconfig modes) re-walks its full repo list in the same endpoint-imposed order every single run — sorted byupdated_at, a field thesinceparam can't actually filter on for either the org-listing or search endpoints. So every run restarts at the same front-of-queue repos, and once heavy child streams (pull_request_commits,reviews— bothfull_table, noreplication_keyat all) exhaust GitHub's secondary rate limit, the tap never advances into the rest of the catalog. Most of an org's repos have never had PR/commit data synced, at all, despite running on schedule for months.Changes
RepositoryStream._get_records_round_robin): resumes right after the last repo whose children fully synced last run, wraps at the end. Persisted under a dedicatedround_robin_checkpointstate key — deliberately not nested underprogress_markers, since singer_sdk wipes that key entirely on every partition finalize.id, not the API'supdated_atorder. Verified via simulation that using the volatileupdated_atorder (even though never incorrect — no repo is ever permanently skipped) costs ~2.8x more runs to guarantee full coverage under continuous repo activity, because reshuffling keeps pushing already-synced repos back in front of genuinely new ones. Sorting byidremoves that churn — simulation showed coverage time drop to within ~1% of the theoretical minimum.get_records()generator is torn down (GeneratorExit), which still runs ourfinallyblock.GitHubRestStream): a shared, tap-wide inter-request delay that escalates (doubles, capped at 30s) on each secondary-limit hit and applies to every subsequent request from any stream — not just the one retried call, which is all the existing sleep-and-retry did. Decays back down after 25 consecutive clean requests.Verification
finallyblock still fire with the correct partial count.Next steps
Once merged,
lumapps/dataplatform'sbi/meltano/meltano.ymltap-githubpip_urlswaps fromMeltanoLabs/tap-githubto this fork.