fix(kind): route around Docker Hub's anonymous pull-rate limit - #486
Open
yoavkatz wants to merge 2 commits into
Open
fix(kind): route around Docker Hub's anonymous pull-rate limit#486yoavkatz wants to merge 2 commits into
yoavkatz wants to merge 2 commits into
Conversation
…429s Docker Hub's anonymous pull limit (100/6h per IP) causes 429 on the node's base-image pulls (busybox:1.36 init containers -> Init:ErrImagePull) with no account to log into. Route docker.io through Google's public pull-through cache mirror.gcr.io: add containerdConfigPatches to set config_path, then write a docker.io hosts.toml drop-in into the node after create. Transparent -- image refs are unchanged, so release build-input hashing is unaffected. Document the mirror approach (node containerd + podman machine registries.conf) in the podman-on-apple-silicon guide, alongside the existing ECR one-off. Signed-off-by: Yoav Katz <katz@il.ibm.com>
…ll failure The in-cluster upstream probe pulled curlimages/curl from docker.io. Docker Hub rate-limits anonymous pulls per source IP, so behind a shared office or VPN address the node's pull fails with 429 and the probe reported a cluster-egress or private-CA fault that did not exist. Two changes: - Default PROBE_IMAGE to mirror.gcr.io/curlimages/curl:8.11.1 — Google's pull-through cache of Docker Hub, same manifest, no anonymous pull limit. Overridable for an internal mirror or an air-gapped node. Every other image this deployment uses is on ghcr.io, which has no such limit, so this was the one exposure. - Distinguish "never obtained the probe image" from "connected but the exchange never completed". Both previously surfaced as an absent __probe__ trailer and were read as an upstream fault. Report the kubelet's own ImagePullBackOff / ErrImagePull verdict as a distinct image-pull-failed sentinel, and have the caller name it as a probe-tooling problem — explicitly not a verdict on the upstream, since the host leg has already passed by then. A 429 gets the mirror-override remedy. Pull failures are not retried: the image will not appear on its own, and retrying a throttled registry deepens the throttle. The sentinel is handled before any arithmetic test, because bash's `-eq` evaluates operands as arithmetic expressions and would abort under `set -u` trying to resolve it as a variable name; the two numeric comparisons on that path move to string form for the same reason. Signed-off-by: Yoav Katz <katz@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extracted from #408. Docker Hub's anonymous pull limit (100 pulls / 6h per
source IP) was surfacing as two different unrelated-looking failures on kind:
busybox:1.36)pull straight from
docker.io. Behind a shared office/VPN IP, the nodehits
429and the pod sits inInit:ErrImagePull— with no Docker Hubaccount to log into to raise the limit.
create.sh's connectivity probe pullscurlimages/curlfromdocker.iotoo. The same 429 made the probe reporta cluster-egress or private-CA fault that didn't actually exist — the
pull never got far enough to test either.
Fix:
docker.iothroughmirror.gcr.io(Google's public pull-throughcache) for the node's own pulls:
containerdConfigPatchessetsconfig_path, and ahosts.tomldrop-in is written into the node aftercreate.shprovisions it. Transparent — image refs are unchanged, sorelease build-input hashing is unaffected.
PROBE_IMAGEtomirror.gcr.io/curlimages/curl:8.11.1— identicalmanifest to
docker.io/curlimages/curl:8.11.1, just not rate-limited.Overridable via
PROBE_IMAGEfor an internal mirror or an air-gapped node.exchange never completed" — both previously surfaced as an absent
__probe__trailer and were misread as an upstream fault. The kubelet'sown
ImagePullBackOff/ErrImagePullverdict is now reported as a distinctimage-pull-failedsentinel, named explicitly as a probe-tooling problemrather than a verdict on the upstream. Pull failures are not retried (the
image won't appear on its own, and retrying a throttled registry deepens
the throttle); a 429 gets the mirror-override remedy instead.
PROBE_IMAGEoverride indocs/guides/podman-on-apple-silicon.mdanddeploy/kind/README.md.Rules checked against
(
.agents/gateways/RULES.md,.agents/RULES.md; declared per.agents/contributing/RULES.mdrule 3).agents/edits.base-image pulls to an equivalent public mirror.
Scope
Four files:
deploy/kind/create.sh,deploy/kind/README.md,docs/guides/podman-on-apple-silicon.md.