🐛 fix(dev): select envtest assets by host OS/arch - #326
Merged
Conversation
Four envtest suites resolved their control-plane binaries by returning the
first directory under bin/k8s. os.ReadDir returns that listing sorted, so on
an Apple-silicon workstation whose bin/ also holds linux downloads the pick
was 1.35.0-linux-amd64 and every suite died in BeforeSuite with
fork/exec bin/k8s/1.35.0-linux-amd64/etcd: exec format error
with the correct 1.36.2-darwin-arm64 assets sitting right next to it. This is
a selection bug, not a procurement one: local green on internal/collect,
internal/controller, internal/pipeline and internal/webhook/v1alpha1 has been
unobtainable on a non-linux host for as long as this has been true.
internal/envtestassets now resolves the assets once, for all four suites:
KUBEBUILDER_ASSETS first (the order controller-runtime itself uses, so an
exporting harness stays authoritative), then the bin/k8s entry whose name
carries the -<goos>-<goarch> suffix. Only suffix matches are considered, which
also skips non-asset directories such as the stray bin/k8s/k8s a --bin-dir
typo leaves behind, and among matches the highest version wins compared field
by numeric field so 1.36.10 outranks 1.36.2.
CI is unaffected: hack/coverage.sh and `make test` export KUBEBUILDER_ASSETS
from `setup-envtest ... -p path`, which resolves the host platform itself, and
controller-runtime's BinPathFinder consults that variable before any
BinaryAssetsDirectory. The linux-amd64 selection is pinned by its own table
case rather than left to CI to discover.
The helper joins internal/digest and internal/errors under the arch-lint
`shared` component because all four suites, in four different components, need
the same resolution; each still passes its own base paths, since the webhook
suite sits one directory deeper than the others.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Four envtest suites each carried their own copy of the kubebuilder scaffold's "first directory under
bin/k8s" pattern for locating the control-plane binaries.os.ReadDirreturns entries sorted, so abin/k8sholding downloads for more than one platform — a checkout shared between machines, a stale download, a directory left behind by an earlier--bin-dir— resolves the lexically-first entry regardless of what host is running.On Apple silicon with both platforms present, that picks
1.35.0-linux-amd64, and every suite dies inBeforeSuitewithexec format error. The suites do not fail loudly — they run 0 specs and reportok.Reproduced on
main: 0 of 48 controller specs, 0 of 5 webhook specs. On this branch: 48 of 48 and 5 of 5.Fix
One shared resolver,
internal/envtestassets:-<goos>-<goarch>suffix, which also rules out non-asset directories;1.36.10outranks1.36.2(lexical order gets that backwards);""when there is no match, leavingBinaryAssetsDirectoryunset so controller-runtime falls back to its own default.ResolveprefersKUBEBUILDER_ASSETSwhen set, matching the order controller-runtime itself resolves in.The four suites now call the shared resolver instead of duplicating the scaffold pattern.
Why linux CI cannot regress
hack/coverage.shexportsKUBEBUILDER_ASSETSfromsetup-envtest -p path, and controller-runtime'sBinPathFinderprefers that environment variable over anenvtest.Environment'sBinaryAssetsDirectory— verified against the module source. CI's resolution is unchanged; this only affects a barego test ./internal/...on a developer machine.Verification
task coverage— 90.7% (floor 90)task coverage:race— 90.7%, no data racestask lint— 0 issues;task arch-lint— no warnings;task verify— okKUBEBUILDER_ASSETSunset against abin/k8scontaining1.35.0-linux-amd64,1.36.0-linux-amd64and1.36.2-darwin-arm64: 48/48 and 5/5docs/development/setup.mddocuments the multi-platformbin/k8scase;.go-arch-lint.ymlregisters the new component.