-
-
Notifications
You must be signed in to change notification settings - Fork 149
ci(onnx): build ONNX Runtime for macOS with the floor the app declares #595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
c0a71fa
ci(onnx): build ONNX Runtime for macOS with the floor the app declares
EtienneLescot 5a528ef
ci(onnx): stage the LICENSE, and build a cache key that is not empty
EtienneLescot 65e8f57
ci(onnx): attest the build's provenance, so the digest is not the onl…
EtienneLescot 4f1d918
ci(onnx): bound the build, stop piling up runs, and pin every action
EtienneLescot b59008c
ci(onnx): build a pinned commit, not a movable tag, and drop the token
EtienneLescot b1d6bd7
fix(onnx): give SOURCE_COMMIT a use, instead of a lint suppression
EtienneLescot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,310 @@ | ||
| name: Build ONNX Runtime (macOS 13 floor) | ||
|
|
||
| # Builds ONNX Runtime for macOS arm64 with the deployment target pinned to the | ||
| # floor this app declares, and publishes it as an artifact for a maintainer to | ||
| # attach to a release. | ||
| # | ||
| # WHY THIS EXISTS. Microsoft's own `onnxruntime-osx-arm64-*.tgz` is built for | ||
| # macOS 14. `electron-builder.json5` declares `minimumSystemVersion: "13.0"`, | ||
| # and `scripts/before-pack.cjs` refuses to package any binary that demands more | ||
| # than the floor — correctly, because the deployment target decides which | ||
| # symbols the linker resolves against the OS rather than emitting locally, so a | ||
| # too-high floor strands users on the older OS with `Symbol not found` at dyld | ||
| # time (#515). The result is that `npm run build:mac` cannot package at all. | ||
| # | ||
| # Every published release from 1.24 onward is `minos 14.0`, and every release | ||
| # before it is at least 13.3, so no prebuilt artifact has ever satisfied a 13.0 | ||
| # floor. Building it is the only way to keep both macOS 13 support and webcam | ||
| # segmentation. | ||
| # | ||
| # WHAT IS NOT BUILT. The CoreML execution provider. `segmentation.rs` builds its | ||
| # session with no explicit provider, i.e. the CPU EP, which | ||
| # technical-documentation/engineering/webcam-segmentation.md records as a | ||
| # measured decision ("Inference p50, CPU EP: 3.575 ms — the CPU is faster"). | ||
| # Dropping CoreML is what takes the library from 36.7 MB to ~22 MB. | ||
| # | ||
| # HOW IT IS PUBLISHED. This workflow does not create releases. It builds, | ||
| # verifies, and prints the exact `PINNED` entry for | ||
| # `scripts/fetch-onnxruntime.mjs` in the job summary; attaching the archive to a | ||
| # release and pasting that entry stays a deliberate human step, so the | ||
| # supply-chain posture the script documents — immutable URL, SHA-256 verified | ||
| # before the archive is opened — is preserved rather than replaced by "whatever | ||
| # CI last uploaded". | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| push: | ||
| paths: | ||
| - ".github/workflows/build-onnxruntime-macos.yml" | ||
| - "scripts/fetch-onnxruntime.mjs" | ||
|
|
||
| # One build per branch. Without this, every push to a branch that touches the paths | ||
| # above starts another 1-2 h build and none of the earlier ones stop: three ran | ||
| # concurrently on this workflow's own PR, the oldest for nearly four hours. | ||
| concurrency: | ||
| group: onnxruntime-macos-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
| # Provenance attestation. `id-token` mints the OIDC token GitHub signs with, and | ||
| # `attestations` lets the run record the result. Both are needed by | ||
| # `actions/attest-build-provenance`; neither grants write access to the repository. | ||
| id-token: write | ||
| attestations: write | ||
|
|
||
| jobs: | ||
| build: | ||
| name: macOS arm64, deployment target 13.0 | ||
| # arm64 runner: the only macOS target upstream ships, and the only one the | ||
| # app packages. There is no Intel build to match. | ||
| runs-on: macos-latest | ||
| # The default is 6 h. This build takes ~10 min on an 8-core M1 with `--parallel 4` | ||
| # and well over an hour on the runner; a cap turns "pathologically slow" into a | ||
| # failure somebody sees rather than six hours of quietly burnt minutes. | ||
| timeout-minutes: 150 | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| with: | ||
| # This job compiles third-party source in the same workspace. Leaving the | ||
| # token in .git/config would put it within reach of ONNX Runtime's own build | ||
| # scripts — and this workflow holds `attestations: write`. Same setting | ||
| # build.yml, docs.yml and nix-build.yml already use. | ||
| persist-credentials: false | ||
|
|
||
| - name: Read the pinned version from fetch-onnxruntime.mjs | ||
| id: pin | ||
| # Single source of truth. `fetch-onnxruntime.test.mjs` already cross-checks | ||
| # that VERSION satisfies the `api-NN` feature `crates/Cargo.toml` gives | ||
| # `ort`; reading it here rather than repeating it means a bump cannot leave | ||
| # this workflow building a version nothing consumes. | ||
| run: | | ||
| set -euo pipefail | ||
| VERSION="$(sed -n 's/^const VERSION = "\(.*\)";$/\1/p' scripts/fetch-onnxruntime.mjs)" | ||
| [ -n "$VERSION" ] || { echo "::error::VERSION not found in scripts/fetch-onnxruntime.mjs"; exit 1; } | ||
| COMMIT="$(sed -n 's/^const SOURCE_COMMIT = "\(.*\)";$/\1/p' scripts/fetch-onnxruntime.mjs)" | ||
| [ -n "$COMMIT" ] || { echo "::error::SOURCE_COMMIT not found in scripts/fetch-onnxruntime.mjs"; exit 1; } | ||
| echo "version=$VERSION" >> "$GITHUB_OUTPUT" | ||
| echo "commit=$COMMIT" >> "$GITHUB_OUTPUT" | ||
| echo "Building ONNX Runtime v$VERSION at $COMMIT" | ||
|
|
||
| - name: Read the deployment floor from electron-builder.json5 | ||
| id: floor | ||
| # Also single-sourced: if somebody raises `mac.minimumSystemVersion`, this | ||
| # build follows rather than silently producing a library for the old floor. | ||
| run: | | ||
| set -euo pipefail | ||
| FLOOR="$(grep -o '"minimumSystemVersion": *"[0-9.]*"' electron-builder.json5 | grep -o '[0-9][0-9.]*')" | ||
| [ -n "$FLOOR" ] || { echo "::error::minimumSystemVersion not found"; exit 1; } | ||
| echo "floor=$FLOOR" >> "$GITHUB_OUTPUT" | ||
| echo "Deployment target: $FLOOR" | ||
|
|
||
| - name: Checkout ONNX Runtime | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| with: | ||
| repository: microsoft/onnxruntime | ||
| # The COMMIT, not the tag. `v1.27.1` upstream is a lightweight tag — it points | ||
| # straight at a commit and can be moved. Building from a tag would mean | ||
| # attesting an artifact to "whatever that tag meant this morning", which is | ||
| # precisely the property this workflow exists to provide. | ||
| ref: ${{ steps.pin.outputs.commit }} | ||
| path: onnxruntime-src | ||
| submodules: recursive | ||
| fetch-depth: 1 | ||
| persist-credentials: false | ||
|
|
||
| - name: Check the pinned commit is still what the tag names | ||
| # Not fatal to the build — the commit above is what gets built either way — but a | ||
| # tag that has moved means the pin and the version string no longer describe the | ||
| # same thing, and somebody should look before adopting the artifact. | ||
| run: | | ||
| set -euo pipefail | ||
| TAGGED="$(git -C onnxruntime-src ls-remote https://github.com/microsoft/onnxruntime "refs/tags/v${{ steps.pin.outputs.version }}" | cut -f1)" | ||
| if [ "$TAGGED" != "${{ steps.pin.outputs.commit }}" ]; then | ||
| echo "::error::v${{ steps.pin.outputs.version }} now resolves to ${TAGGED:-nothing}, not the pinned ${{ steps.pin.outputs.commit }}" | ||
| exit 1 | ||
| fi | ||
| echo "v${{ steps.pin.outputs.version }} still resolves to ${{ steps.pin.outputs.commit }}" | ||
|
|
||
| - name: Read the runner image | ||
| id: image | ||
| # `${{ env.ImageOS }}` DOES NOT WORK, and it fails silently. The `env` | ||
| # expression context only carries what a workflow, job or step `env:` block | ||
| # defined; `ImageOS`/`ImageVersion` are set by the runner in its own | ||
| # environment, so the expression evaluates to the empty string and the cache | ||
| # key simply loses that component. The evidence is in this repo's own cache | ||
| # list: `build-whisper-stt.yml` builds its key the same way and the stored | ||
| # keys read `whisper-stt-build-darwin-arm64---<hash>` — three hyphens, both | ||
| # values empty. Reading them in a `run:` step, where they are ordinary shell | ||
| # variables, is what actually works. | ||
| run: | | ||
| set -euo pipefail | ||
| echo "tag=${ImageOS:-unknown}-${ImageVersion:-unknown}" >> "$GITHUB_OUTPUT" | ||
| echo "Runner image: ${ImageOS:-unknown} ${ImageVersion:-unknown}" | ||
|
|
||
| - name: Cache the CMake build tree | ||
| uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 | ||
| with: | ||
| path: onnxruntime-build | ||
| # Keyed on the version, the floor and the runner image. The image matters: | ||
| # CMake bakes absolute SDK paths into the tree, so when GitHub rolls Xcode a | ||
| # restored tree fails on paths that no longer exist. Scoping the key to the | ||
| # image busts it automatically on every roll. | ||
| key: ort-${{ steps.pin.outputs.version }}-${{ steps.floor.outputs.floor }}-${{ steps.image.outputs.tag }} | ||
|
|
||
| - name: Configure | ||
| # Driven straight at CMake rather than through upstream's `build.sh`. | ||
| # `build.sh` -> `build_args.py` uses `match`, so it needs Python 3.10+, | ||
| # and `cmake/CMakeLists.txt` asks for `find_package(Python 3.10)` — but | ||
| # that requirement is only real for the Python bindings, which are not | ||
| # built here. The one thing Python IS needed for is generating the symbol | ||
| # export list (`onnxruntime.lds`); pointing `Python_EXECUTABLE` at | ||
| # whatever the runner has is enough, and `gen_def.py` parses on 3.9. | ||
| run: | | ||
| set -euo pipefail | ||
| cmake -S onnxruntime-src/cmake -B onnxruntime-build \ | ||
| -DCMAKE_BUILD_TYPE=Release \ | ||
| -DCMAKE_OSX_DEPLOYMENT_TARGET=${{ steps.floor.outputs.floor }} \ | ||
| -DCMAKE_OSX_ARCHITECTURES=arm64 \ | ||
| -Donnxruntime_BUILD_SHARED_LIB=ON \ | ||
| -Donnxruntime_BUILD_UNIT_TESTS=OFF \ | ||
| -DPython_EXECUTABLE="$(command -v python3)" | ||
|
|
||
| - name: Build | ||
| # BOUNDED, deliberately. Bare `--parallel` means "as many jobs as cores", and | ||
| # ONNX Runtime's C++ translation units are memory-hungry: on a runner with far | ||
| # less RAM per core than the reference M1, that is how a 10-minute build becomes | ||
| # an hour of swapping. `nproc`-1 leaves the machine a core to breathe. | ||
| run: | | ||
| set -euo pipefail | ||
| JOBS="$(( $(sysctl -n hw.ncpu) > 2 ? $(sysctl -n hw.ncpu) - 1 : 1 ))" | ||
| echo "Building with $JOBS jobs on $(sysctl -n hw.ncpu) cores, $(( $(sysctl -n hw.memsize) / 1073741824 )) GiB" | ||
| cmake --build onnxruntime-build --config Release --parallel "$JOBS" | ||
|
|
||
| - name: Verify the deployment target | ||
| # The entire reason this workflow exists. A library that comes out at 14.0 | ||
| # anyway is worse than no library, because it would sail through packaging | ||
| # and strand macOS 13 users at dyld time. | ||
| run: | | ||
| set -euo pipefail | ||
| V="${{ steps.pin.outputs.version }}" | ||
| DYLIB="onnxruntime-build/libonnxruntime.${V}.dylib" | ||
| [ -f "$DYLIB" ] || { echo "::error::$DYLIB was not produced"; exit 1; } | ||
| MINOS="$(otool -l "$DYLIB" | awk '/LC_BUILD_VERSION/{f=1} f&&/minos/{print $2; exit}')" | ||
| echo "minos=$MINOS floor=${{ steps.floor.outputs.floor }}" | ||
| [ "$MINOS" = "${{ steps.floor.outputs.floor }}" ] || { | ||
| echo "::error::built for macOS $MINOS, expected ${{ steps.floor.outputs.floor }}"; exit 1; } | ||
|
|
||
| - name: Verify the ABI surface | ||
| # `ort` is wired `load-dynamic`, so it dlopens this file and calls | ||
| # `OrtGetApiBase`. The CPU provider is the one `segmentation.rs` uses. | ||
| # CoreML is deliberately absent and is NOT checked for. | ||
| run: | | ||
| set -euo pipefail | ||
| V="${{ steps.pin.outputs.version }}" | ||
| DYLIB="onnxruntime-build/libonnxruntime.${V}.dylib" | ||
| for sym in _OrtGetApiBase _OrtSessionOptionsAppendExecutionProvider_CPU; do | ||
| nm -gU "$DYLIB" | grep -q " ${sym}$" || { echo "::error::missing export ${sym}"; exit 1; } | ||
| done | ||
| lipo -info "$DYLIB" | ||
| echo "ABI surface OK" | ||
|
|
||
| - name: Package in the upstream layout | ||
| # Byte-for-byte the same shape as `onnxruntime-osx-arm64-<v>.tgz`, so | ||
| # `fetch-onnxruntime.mjs` needs no extraction change — only a URL and a | ||
| # digest. `member` there is the VERSIONED file; the two symlinks beside it | ||
| # are kept so the archive stays a drop-in for anything that expects them. | ||
| run: | | ||
| set -euo pipefail | ||
| V="${{ steps.pin.outputs.version }}" | ||
| DIR="onnxruntime-osx-arm64-${V}" | ||
| mkdir -p "stage/${DIR}/lib" "stage/${DIR}/include" | ||
| cp "onnxruntime-build/libonnxruntime.${V}.dylib" "stage/${DIR}/lib/" | ||
| ln -s "libonnxruntime.${V}.dylib" "stage/${DIR}/lib/libonnxruntime.dylib" | ||
| ln -s "libonnxruntime.${V}.dylib" "stage/${DIR}/lib/libonnxruntime.1.dylib" | ||
| cp onnxruntime-src/include/onnxruntime/core/session/*.h "stage/${DIR}/include/" || true | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| # NOT optional. `fetch-onnxruntime.mjs` refuses an archive with no LICENSE | ||
| # (`LICENSE not found inside …`) and then reads it to confirm the library | ||
| # really is MIT — "asset names are not evidence". Without this the archive | ||
| # would pass its SHA-256 and fail at vendoring, which is the worst place to | ||
| # find out. `ThirdPartyNotices.txt` rides along because upstream ships it and | ||
| # THIRD-PARTY-NOTICES.md is what carries the attribution. | ||
| cp onnxruntime-src/LICENSE "stage/${DIR}/LICENSE" | ||
| cp onnxruntime-src/ThirdPartyNotices.txt "stage/${DIR}/ThirdPartyNotices.txt" | ||
| tar -czf "${DIR}.tgz" -C stage "${DIR}" | ||
| shasum -a 256 "${DIR}.tgz" | ||
|
|
||
| - name: Attest build provenance | ||
| id: attest | ||
| # A SHA-256 in `fetch-onnxruntime.mjs` says "these are the bytes somebody | ||
| # pinned". It cannot say WHERE they came from — and once the publisher is us | ||
| # rather than Microsoft, that is the question that matters. This binds the | ||
| # archive's digest to the commit, workflow and run that produced it, signed by | ||
| # GitHub, so provenance becomes verifiable rather than asserted: | ||
| # | ||
| # gh attestation verify onnxruntime-osx-arm64-<v>.tgz --repo getopenscreen/openscreen | ||
| # | ||
| # It does not replace the digest pin, it answers a different question. Keep both. | ||
| uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be # v2.4.0 | ||
| with: | ||
| subject-path: onnxruntime-osx-arm64-*.tgz | ||
|
|
||
| - name: Upload | ||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||
| with: | ||
| name: onnxruntime-osx-arm64 | ||
| path: onnxruntime-osx-arm64-*.tgz | ||
| if-no-files-found: error | ||
| retention-days: 90 | ||
|
|
||
| - name: Workflow summary | ||
| if: always() | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| V="${{ steps.pin.outputs.version }}" | ||
| ARCHIVE="onnxruntime-osx-arm64-${V}.tgz" | ||
| { | ||
| echo "## ONNX Runtime ${V}, macOS arm64, deployment target ${{ steps.floor.outputs.floor }}" | ||
| echo "" | ||
| echo "- Source: microsoft/onnxruntime@\`${{ steps.pin.outputs.commit }}\`" | ||
| echo "" | ||
| echo "- Result: ${{ job.status }}" | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| if [ "${{ steps.attest.outcome }}" != "success" ]; then | ||
| { | ||
| echo "" | ||
| echo "> **Provenance was NOT attested** (\`${{ steps.attest.outcome }}\`)." | ||
| echo "> Do not adopt this archive: the digest below says what the bytes are," | ||
| echo "> nothing says where they came from. Re-run the workflow." | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| fi | ||
| if [ -f "$ARCHIVE" ] && [ "${{ steps.attest.outcome }}" = "success" ]; then | ||
| SHA="$(shasum -a 256 "$ARCHIVE" | cut -d' ' -f1)" | ||
| { | ||
| echo "- Archive: \`${ARCHIVE}\` ($(du -h "$ARCHIVE" | cut -f1))" | ||
| echo "- SHA-256: \`${SHA}\`" | ||
| echo "" | ||
| echo "Provenance is attested; before adopting, verify it with:" | ||
| echo "" | ||
| echo '```bash' | ||
| echo "gh attestation verify ${ARCHIVE} --repo ${{ github.repository }}" | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| echo '```' | ||
| echo "" | ||
| echo "To adopt it: attach the archive to a release, then point the" | ||
| echo "\`darwin-arm64\` entry of \`PINNED\` in \`scripts/fetch-onnxruntime.mjs\`" | ||
| echo "at that release with this digest:" | ||
| echo "" | ||
| echo '```js' | ||
| echo '"darwin-arm64": {' | ||
| echo ' slug: "osx-arm64",' | ||
| echo ' ext: "tgz",' | ||
| echo " sha256: \"${SHA}\"," | ||
| echo " member: \`libonnxruntime.\${VERSION}.dylib\`," | ||
| echo ' out: "libonnxruntime.dylib",' | ||
| echo ' baseUrl: "<the release download URL>",' | ||
| echo '},' | ||
| echo '```' | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: getopenscreen/openscreen
Length of output: 11638
🤖 get_repo_knowledge executed:
get_repo_knowledge getopenscreen/openscreen /tmp/coderabbit-repo-knowledge/getopenscreen-openscreen-086fd783/learnings /tmp/coderabbit-repo-knowledge/getopenscreen-openscreen-086fd783/conventionsLength of output: 4482
Raise the job timeout above the measured build duration. The
Buildstep can run for more than 150 minutes withJOBS="$(( $(sysctl -n hw.ncpu) > 2 ? $(sysctl -n hw.ncpu) - 1 : 1 ))". If the reported duration above 227 minutes applies, GitHub cancels the job before packaging and attestation. Set the timeout above the worst-case duration with margin.🤖 Prompt for AI Agents