Skip to content

Fix Linux release binaries requiring GLIBC_2.39, bump to 1.9.3 - #139

Open
Ibrahimrahhal wants to merge 8 commits into
mainfrom
cursor/fix-linux-glibc-floor-0b17
Open

Fix Linux release binaries requiring GLIBC_2.39, bump to 1.9.3#139
Ibrahimrahhal wants to merge 8 commits into
mainfrom
cursor/fix-linux-glibc-floor-0b17

Conversation

@Ibrahimrahhal

@Ibrahimrahhal Ibrahimrahhal commented Jul 29, 2026

Copy link
Copy Markdown
Member

Problem

A customer on a Harness Cloud ARM64 delegate (Ubuntu 22.04, glibc 2.35) cannot run corgea-aarch64-unknown-linux-gnu.zip:

corgea: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by corgea)
corgea: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by corgea)

They asked which version to pin. The answer is none — every published Linux -gnu binary is affected, on both architectures, going back to the first aarch64 build in v1.7.2:

Release aarch64-...-gnu x86_64-...-gnu
v1.7.2, v1.8.0, v1.8.7, 1.8.8 GLIBC_2.39 GLIBC_2.39
v1.9.0, v1.9.1, v1.9.2 GLIBC_2.39 GLIBC_2.39

x86_64 users have had a static musl fallback since v1.8.3. ARM64 users have had no working option at all, including via npm install -g @corgea/cli, because bin/corgea.js maps linux/arm64 to the gnu build.

Root cause

release-binaries.yml built the gnu targets natively on ubuntu-latest (24.04) and ubuntu-24.04-arm, both glibc 2.39. Linking against the runner's glibc records version requirements in the ELF version-need table that an older loader refuses. No floor was pinned anywhere, so the supported minimum silently tracked GitHub's runner images.

Exactly three symbols pushed the floor from 2.34 to 2.39:

  • __isoc23_strtol (GLIBC_2.38) — C code compiled against glibc 2.38+ headers (vendored OpenSSL / bundled libgit2); GCC 13 redirects strtol to the C23 variant. Hard undefined, fatal on its own.
  • pidfd_spawnp, pidfd_getpid (GLIBC_2.39) — Rust std's process-spawn path. Weak, but still recorded in verneed, which is why the loader names them too.

Changes

  • release-binaries.yml — all four Linux targets cross-build on ubuntu-latest with cargo-zigbuild, pinning the gnu targets to a workflow-level GLIBC_FLOOR of 2.17. Adds aarch64-unknown-linux-musl, so ARM64 gets a fully static option like x86_64 already had. The ubuntu-24.04-arm runner and the musl-tools step are no longer needed. Both new actions are pinned to commit SHAs, since this workflow holds contents: write.
  • scripts/check-linux-binary.sh — new guard, run on every Linux artifact before it is archived. Fails if a gnu binary needs a glibc above the floor, if a musl binary is not static, or if the ELF machine does not match the target triple. This is what stops the floor from silently drifting again.
  • Smoke test — the aarch64 artifacts are now cross-built, so nothing in CI would otherwise execute them. Each Linux artifact runs --version, ARM64 under qemu-user-static.
  • bin/corgea.js / scripts/npm/bundle-binaries.js — Linux arm64 now resolves to the musl binary, matching what x64 already does.
  • build_release.sh — routes Linux through the same zigbuild + verification path, so local builds cannot produce artifacts the pipeline would reject. Drops x86_64-pc-windows-gnu, absent from releases since v1.7.2 and the script's only remaining use of cross.
  • Cargo.toml1.9.3 — a fix in the pipeline changes nothing for users until a tag is cut, and no tag on the 1.9.x line has a working ARM64 binary. Per RELEASING.md this is the only manual version edit; PyPI and npm derive theirs from it and the tag.
  • RELEASING.md / README.md — document the floor, how to check it, and the gnu vs. musl choice for Linux.

GLIBC_2.17 covers RHEL 7 / Ubuntu 14.04 and newer, and matches the manylinux2014 floor release.yml already asserts for the PyPI wheels. It is also Rust's hard minimum for *-unknown-linux-gnu: building at .2.12 fails with undefined symbol: getauxval from std.

Verification

All evidence below uses the artifacts CI actually produced, not local builds.

The 1.9.3 candidate on an unmodified Ubuntu 22.04 ARM64 rootfs (glibc 2.35) under qemu:

corgea-aarch64-unknown-linux-gnu.zip  : corgea 1.9.3   (glibc required: GLIBC_2.17)
corgea-aarch64-unknown-linux-musl.zip : corgea 1.9.3   (statically linked)

for comparison, published v1.9.2 on the same rootfs:
  corgea: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found
  corgea: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found

glibc range, same aarch64 gnu artifact: runs on glibc 2.23 (Ubuntu 16.04) and 2.35 (Ubuntu 22.04); the x86_64 gnu artifact runs on 2.39 (Ubuntu 24.04), so the old floor did not cost forward compatibility.

TLS and DNS, a live authentication request against the Corgea API returning its 401, proving DNS, TLS handshake and CA verification all work — including through musl's resolver:

x86_64  gnu  (glibc 2.39) : Invalid token provided from parameter.
x86_64  musl (static)     : Invalid token provided from parameter.
aarch64 gnu  (glibc 2.35) : Invalid token provided from parameter.
aarch64 musl (static)     : Invalid token provided from parameter.

Real workload, corgea deps scan and corgea deps diff --base HEAD~1 on the glibc 2.35 rootfs, both producing correct findings.

Guard behaviour, including against the artifact that shipped broken, a binary with no versioned glibc references, and one carrying GLIBC_PRIVATE:

OK:   aarch64-unknown-linux-gnu   requires at most glibc 2.17 (floor 2.17)
OK:   aarch64-unknown-linux-musl  is statically linked
OK:   <static binary as gnu>      requires no versioned glibc symbols
FAIL: <shipped v1.9.2 aarch64>    requires glibc 2.39, above the 2.17 floor
FAIL: <system binary w/ PRIVATE>  requires glibc 2.38, above the 2.17 floor
                                  versions required: 2.2.5 2.3 2.3.4 2.4 2.8 2.14 2.34 2.36 2.38

Size, all artifacts got slightly smaller: aarch64 gnu 15.4 MB → 14.7 MB, x86_64 musl 17.0 MB → 15.2 MB.

npm path, bundle-binaries.js over a real asset set produces the expected vendor/ layout and the vendored arm64 musl binary runs.

./harness ci passes locally in full — strict clippy, format check, dep audit, 524 tests, coverage gate — which is the pre-tag gate in RELEASING.md.

Review feedback addressed

  • Guard aborted instead of accepting a binary with no glibc references. Under set -euo pipefail, grep's no-match exit status killed the assignment, so the branch handling that case was unreachable and the script exited 1 with no diagnostic. Reproduced by checking a static binary against the gnu triple, fixed by running grep outside the pipeline that feeds the assignment. The pattern now requires a digit, so GLIBC_PRIVATE and GLIBC_ABI are not read as versions.
  • Mutable action tags in a contents: write workflow. mlugg/setup-zig and taiki-e/install-action are pinned to commit SHAs, verified against the live tags.

Releasing

Once merged, follow RELEASING.md: gh release create v1.9.3 --target main --generate-notes. The release will carry a new asset, corgea-aarch64-unknown-linux-musl.zip.

Follow-ups, not in this PR

  • softprops/action-gh-release@v2 and actions/upload-artifact@v4 in this same workflow are still on mutable tags. Pre-existing, and out of scope here, but the same argument applies.
  • release.yml builds no aarch64 Linux wheels, so pip install corgea-cli on ARM64 still compiles from the sdist.
  • corgea scan's in-process libgit2 paths sit behind authentication and were not exercised here; worth one authenticated run on a real ARM64 machine before announcing.
Open in Web Open in Cursor 

cursoragent and others added 6 commits July 29, 2026 14:33
Reads the ELF version-need table of a built artifact and fails when a
*-gnu binary requires a glibc newer than GLIBC_FLOOR, or when a *-musl
binary is not fully static. Also checks the ELF machine matches the
target triple.

Co-authored-by: Ibrahim Rahhal <ibrahim.rahhal3636@gmail.com>
Building the *-gnu targets natively stamped the runner's glibc into the
ELF version-need table, so every release since v1.7.2 required
GLIBC_2.39 and failed to load on Ubuntu 22.04 (glibc 2.35) and older:

  corgea: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found
  corgea: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found

Cross-build all Linux targets on ubuntu-latest with cargo-zigbuild,
pinning the gnu targets to GLIBC_FLOOR, and add an aarch64 musl target
so ARM64 hosts have a libc-independent option. Each artifact is verified
before it is archived.

Co-authored-by: Ibrahim Rahhal <ibrahim.rahhal3636@gmail.com>
Linux x64 already resolves to the musl build; arm64 pointed at the gnu
build and inherited its glibc requirement. Also ignore the generated
vendor/ tree so bundled binaries cannot be committed by accident.

Co-authored-by: Ibrahim Rahhal <ibrahim.rahhal3636@gmail.com>
The local release script built the gnu targets natively, which produces
exactly the binaries the pipeline now rejects. Route Linux through
cargo-zigbuild with the same floor and verify the result. Drops the
x86_64-pc-windows-gnu target, which has not been part of a release since
v1.7.2 and was the script's only remaining use of cross.

Co-authored-by: Ibrahim Rahhal <ibrahim.rahhal3636@gmail.com>
The aarch64 artifacts are now cross-built, so no job in CI would
otherwise execute them. Running --version catches a binary that passes
the ELF header checks but fails to start.

Co-authored-by: Ibrahim Rahhal <ibrahim.rahhal3636@gmail.com>
Ships the Linux glibc floor fix. No tag on the 1.9.x line has a usable
aarch64-unknown-linux-gnu binary, so a release is the only way to
unblock ARM64 users.

Co-authored-by: Ibrahim Rahhal <ibrahim.rahhal3636@gmail.com>
@Ibrahimrahhal
Ibrahimrahhal marked this pull request as ready for review July 29, 2026 15:11
@cursor cursor Bot changed the title Fix Linux release binaries requiring GLIBC_2.39 Fix Linux release binaries requiring GLIBC_2.39, bump to 1.9.3 Jul 29, 2026
Comment thread .github/workflows/release-binaries.yml Outdated
Comment on lines +62 to +68
uses: mlugg/setup-zig@v2
with:
version: ${{ env.ZIG_VERSION }}

- name: Install cargo-zigbuild
if: runner.os == 'Linux'
uses: taiki-e/install-action@v2

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two newly added third-party actions are referenced by mutable major-version tags, while this workflow has contents: write (line 13) and runs on every push. If either tag is moved or its upstream is compromised, that code receives a write-capable GITHUB_TOKEN and can modify releases or repository contents. Pin both actions to immutable full commit SHAs; the current tag commits are shown below.

Suggested change
uses: mlugg/setup-zig@v2
with:
version: ${{ env.ZIG_VERSION }}
- name: Install cargo-zigbuild
if: runner.os == 'Linux'
uses: taiki-e/install-action@v2
uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
with:
version: ${{ env.ZIG_VERSION }}
- name: Install cargo-zigbuild
if: runner.os == 'Linux'
uses: taiki-e/install-action@18b1216eba7f8039b0f8d131d5473787f0edce68 # v2.85.3

Comment thread scripts/check-linux-binary.sh Outdated
;;

*-gnu)
needed=$(readelf -V "$binary" | grep -o 'GLIBC_[0-9.]*' | sed 's/^GLIBC_//' | sort -uV)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With set -euo pipefail, grep returning 1 for no matches aborts this assignment before the intended empty-result branch on lines 60–63 can run. I reproduced it by checking a static x86_64 ELF as the GNU triple: the script exited 1 with no diagnostic. That makes the new release gate silently reject a compatible GNU artifact with no versioned glibc symbols. Capture successful readelf output first, then tolerate only grep's no-match result; requiring a digit also avoids treating GLIBC_PRIVATE as an empty version.

Suggested change
needed=$(readelf -V "$binary" | grep -o 'GLIBC_[0-9.]*' | sed 's/^GLIBC_//' | sort -uV)
version_info=$(readelf -V "$binary")
versions=$(grep -oE 'GLIBC_[0-9]+(\.[0-9]+)*' <<<"$version_info" || true)
needed=$(printf '%s
' "$versions" | sed 's/^GLIBC_//' | sort -uV)

cursoragent and others added 2 commits July 29, 2026 16:02
set -euo pipefail made grep's no-match exit status abort the assignment,
so the branch that accepts a binary with no versioned glibc symbols was
unreachable and the script exited 1 with no diagnostic. Run grep outside
the pipeline that feeds the assignment, and require a digit so
GLIBC_PRIVATE and GLIBC_ABI are not read as versions.

Co-authored-by: Ibrahim Rahhal <ibrahim.rahhal3636@gmail.com>
This workflow holds contents: write, so a moved tag on either action
would get a write-capable GITHUB_TOKEN.

Co-authored-by: Ibrahim Rahhal <ibrahim.rahhal3636@gmail.com>

@yhoztak yhoztak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants