[ANE-3098] Read Go buildinfo from binaries during fossa analyze - #1757
Draft
saramaebee wants to merge 3 commits into
Draft
[ANE-3098] Read Go buildinfo from binaries during fossa analyze#1757saramaebee wants to merge 3 commits into
saramaebee wants to merge 3 commits into
Conversation
Go binaries embed the list of modules linked into them (the data `go version -m` prints). millhone could already read this, but only from container image layers, so Go code shipped as a binary with no manifest next to it stayed invisible to `fossa analyze`. Add an opt-in `gobinary` discovery strategy that finds those binaries on the filesystem and reports their modules as regular `go+` dependencies. Opt-in via `--enable-go-binary-analysis`, or `enableGoBinaryAnalysis` in `.fossa.yml`. `fossa analyze` models package-manager scanning, so reading binaries by default would add dependencies to existing projects without the user asking, and could newly fail builds on vulnerabilities in code the user never intended to scan. Implemented as a normal discovery strategy rather than an extension of `--experimental-enable-binary-discovery`, because `--unpack-archives` re-runs only the strategy list over extracted contents. That is what reaches a binary nested inside an AAR or JAR; the binary-deps path is invoked once on the scan root and never sees archive contents. The two flags stay independent - neither implies the other. A project is a directory rather than a single binary: source units are named after their directory, so one project per binary emitted colliding units whenever a directory held more than one Go binary. Binaries in a directory are now combined, each still visible as an origin path. millhone gains an `analyze-go-binaries` subcommand reading candidate paths from stdin (a large repo can exceed the argument-length limit). The CLI pre-filters with the existing `contentIsBinary` check and millhone applies the precise magic/size checks, so the buildinfo parser stays the single Rust implementation shared with container analysis. `DiscoveredGoBinary` moves out of the container command into the shared parser module for the same reason. Verified end to end against real Go binaries: reported bare in a tree, nested inside an AAR under `--unpack-archives` (origin path renders as `<archive>/jni/<abi>/<lib>.so`), combined when two binaries share a directory, and absent entirely when the flag is not passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VnoktDR9eVW7yH5pciJFh6
The unused import broke every build job (and the integration-test job, which builds the library first); the rest are lint-only. - Drop the now-unused Data.Text import left behind when normalizeVersion moved out of App.Fossa.Container.Sources.GoBinary. CI builds with -Werror, so this was an error there and only a warning locally. - Apply fourmolu and cabal-fmt using the toolchain image CI runs (ghc-lib-parser 9.8.4), rather than formatting by hand. - Move the dependency guard ahead of the path lookup in toProjects, per hlint. Also avoids the lookup for binaries with no usable dependency. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VnoktDR9eVW7yH5pciJFh6
Go binary analysis is now opt-in solely via --enable-go-binary-analysis. Removes the enableGoBinaryAnalysis config-file key, its parser, and the config-file fallback in collectStrategyConfig, plus the field from test config fixtures and the docs/schema entries. Also adds the missing --enable-go-binary-analysis row to the analyze subcommand flag table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjBgXmB2ZCmd23oUudRibG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Go binaries embed the module list
go version -mprints. #1740 taught millhone to read it, but only from container image layers, so Go shipped as a bare binary stayed invisible tofossa analyze.This adds an opt-in
gobinarydiscovery strategy that finds those binaries on the filesystem and reports their modules as regulargo+dependencies. Motivating case: a gomobile SDK shippingjni/<abi>/lib<name>.soinside an AAR — nogo.mod, no Go source, and a third-party notice file missing a package demonstrably linked into the shipped binary.Four decisions a reviewer can't read off the diff:
--enable-go-binary-analysis. Per #cli-and-analysis: on by default would add dependencies to existing projects unasked, and could newly fail builds on code nobody meant to scan. Notexperimental-prefixed — v4 is far enough out that a rename later costs more than a stable name now.--experimental-enable-binary-discovery.--unpack-archivesre-runsdiscoverFuncsover extracted contents; the binary-deps path runs once on the scan root and never enters archives. The motivating binary is inside an AAR, so hooking binary-deps would have missed it. The flags stay independent — a nested binary needs both.Srclib.Converter.toSourceUnit), so one project per binary emits collidingName/Manifestwhenever a directory holds two. Binaries in a directory now combine, each still visible as an origin path. The container path can name units per binary only because it hand-builds source units outside the strategy framework.analyze-go-binariessubcommand reads candidate paths from stdin (a large repo exceeds the argv limit).DiscoveredGoBinarymoved out of the container command into the shared parser module, so there's one implementation.Acceptance criteria
Without the flag, nothing changes. With it,
fossa analyzereports the modules linked into compiled Go binaries asgo+dependencies, and with--unpack-archivesreaches binaries nested inside archives.Testing plan
cargo test -p millhone— 27 pass, 3 new coveringscan_file/scan_files: a real Go buildinfo fixture on disk, and rejection of too-small files, non-binaries, directories, and missing paths. Existing container tests still pass, confirming the shared-type move is behavior-preserving.cabal run unit-tests— 20 GoBinary examples, 10 new: millhone output decoding, version normalization, and directory grouping (same dir combines, different dirs stay separate, unusable binaries dropped).End to end,
fossa analyze --outputagainst real Go binaries:--enable-go-binary-analysis, bare binary in a treego+deps--unpack-archives, binary inside an AAR<archive>/jni/arm64-v8a/<lib>.soNot run locally:
fourmolu(couldn't install in this environment), so please let CI arbitrate Haskell formatting. 31 unit tests fail in my checkout both before and after this change, all in suites needing vendored binaries (themis, lernie, berkeleydb, circe) that aren't present locally.Risks
(devel), the normal case), kept when it carries a real version, which happens forgo install module@versionbuilds. Kept for consistency with the container path — happy to drop it.Container.Typesnow importsStrategy.Go.GoBinaryto re-export the shared types. No cycle, and the direction already exists (Container.Sources.GoBinaryimportsStrategy.Go.Gomod). A neutralStrategy/Go/BuildInfo/Types.hsis a mechanical follow-up if you'd prefer.vendor/is excluded by default, so a vendored Go binary is skipped unless--include-path vendoris passed. Documented as a limitation rather than special-casing the default filters.--only-target gobinarydoes nothing without the enabling flag — consistent withbinary-depstoday.Metrics
Not tracked today. Adoption of the flag would be the number worth having; there's no hook for it in this change.
References
Checklist
docs/.docs/README.msand gave consideration to how discoverable or not my documentation is.## Unreleasedsection at the top ofChangelog.md..fossa.ymlorfossa-deps.{json.yml}, I updateddocs/references/files/*.schema.jsonAND I have updated example files used byfossa initcommand.docs/references/subcommands/<subcommand>.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01VnoktDR9eVW7yH5pciJFh6