feat :Testing workflow to push images - #405
Conversation
Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new bash script, scripts/push_docker_image.sh, designed to build and push the endpoints client Docker image to a registry with support for multi-platform builds, custom git refs, and registry authentication. The review feedback highlights two important shell scripting improvements: using printf instead of echo to safely pipe the registry login token, and properly quoting the BUILD_ARGS array expansion to prevent word splitting.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| LOGIN_TOKEN="${GHCR_TOKEN:-${GITHUB_TOKEN:-}}" | ||
| if [[ "$PUSH" == "1" && -n "$LOGIN_TOKEN" ]]; then | ||
| echo ">> Logging in to ${REGISTRY_HOST} as ${LOGIN_USER:-<token>}" | ||
| echo "$LOGIN_TOKEN" | docker login "$REGISTRY_HOST" -u "${LOGIN_USER:-x-access-token}" --password-stdin |
There was a problem hiding this comment.
Using echo to pipe sensitive tokens or arbitrary strings can lead to unexpected behavior if the token starts with a hyphen (e.g., -n or -e), as echo may interpret it as an option. It is safer and more robust to use printf '%s\n' instead.
| echo "$LOGIN_TOKEN" | docker login "$REGISTRY_HOST" -u "${LOGIN_USER:-x-access-token}" --password-stdin | |
| printf '%s\n' "$LOGIN_TOKEN" | docker login "$REGISTRY_HOST" -u "${LOGIN_USER:-x-access-token}" --password-stdin |
| -f "$DOCKERFILE" \ | ||
| -t "${IMAGE}:${SHORT_SHA}" \ | ||
| -t "${IMAGE}:${REF_TAG}" \ | ||
| ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"} \ |
There was a problem hiding this comment.
The array expansion ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"} is missing outer double quotes. Without outer double quotes, any array elements containing spaces or glob characters will undergo word splitting and pathname expansion when expanded. To safely preserve spaces within array elements while maintaining compatibility with set -u in older Bash versions, wrap the entire expansion in double quotes.
| ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"} \ | |
| "${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"}" \ |
arekay-nv
left a comment
arekay-nv
left a comment
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Reviewed by Codex (gpt-5.5, xhigh) + Claude. 3 inline findings below (2 medium, 1 low; no critical/high).
|
|
||
| # Building the working tree as-is (no distinct ref checked out): don't publish a | ||
| # :$SHORT_SHA tag whose '.' build context doesn't match commit $SHORT_SHA. | ||
| if [[ -z "${DID_CHECKOUT:-}" && "$ALLOW_DIRTY" != "1" ]] && tree_has_tracked_changes; then |
There was a problem hiding this comment.
[Codex + Claude] medium (build reproducibility + context): The dirty guard here uses git diff --quiet HEAD, which ignores untracked files. But scripts/Dockerfile.dev:49 does COPY src/ ./src/, so an untracked file under src/ gets baked into the image — a local --push can publish :${SHORT_SHA} whose contents don't match commit ${SHORT_SHA}.
Separately, the comment at line 124 claims ".dockerignore governs the build context", but no .dockerignore exists in the repo, so the whole working tree is uploaded as build context (docker buildx build … ., line 197), pulling in large untracked dirs (cached_datasets/, gtc_results/, dsr1_v61_seed_logs_*/) on every local run.
Fix: add a .dockerignore (exclude artifacts + anything not COPYed); for the SHA-tag correctness, also include untracked files under COPYed paths in the cleanliness check, or build from git archive.
| # Check out the ref *name* (not github.sha) so a local branch head exists | ||
| # and the script can resolve/tag it; blank input falls back to the | ||
| # dispatched branch/tag. Full history for `git rev-parse --short`. | ||
| ref: ${{ inputs.ref || github.ref_name }} |
There was a problem hiding this comment.
[Codex] medium (correctness): This checks out the dispatch ref directly, replacing the tree before the run step. Building a ref that predates this workflow means ./scripts/push_docker_image.sh won't exist and the run fails — even though the UI advertises building arbitrary refs. It also lets the target ref supply the script that runs after GHCR login.
Fix: keep the checkout on the workflow's own (trusted) ref, and pass the target via ENDPOINTS_REF so the script switches refs only for the Docker build. (Ensure the target ref is fetched — the script's origin/<ref> fallback needs the object present.)
| - name: Set up QEMU | ||
| uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4.2.0 | ||
|
|
||
| - name: Set up Docker Buildx |
There was a problem hiding this comment.
[Claude] low (design): This provisions a buildx builder that the script never uses — push_docker_image.sh always creates and selects its own endpoints-buildx (docker-container) builder (lines 173/191). buildx is preinstalled on ubuntu-latest, so this step is effectively idle. Either drop it, or have the script honor a caller-supplied builder. (The Set up QEMU step above is needed for multi-arch — keep it.)
Review Council — Multi-AI Code ReviewReviewed by: Codex (gpt-5.5, xhigh) + Claude | Depth: standard (low-severity C included on request) Found 3 issues, all posted inline. No critical/high.
All comments are neutral ( |
Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>
Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>
| --builder "$BUILDX_BUILDER" \ | ||
| --platform "$PLATFORM" \ | ||
| -f "$DOCKERFILE" \ | ||
| -t "${IMAGE}:${SHORT_SHA}" \ |
There was a problem hiding this comment.
One thing I noticed: re-running this for the same commit with a different PLATFORM/PROVISION_DSR1 re-pushes :${SHORT_SHA} and quietly swaps out the bytes a consumer may have pinned. push_image.sh in this PR already has a nice guard for exactly this (imagetools inspect + --force) — maybe reuse it here, or settle on one canonical config per SHA?
| # --------------------------------------------------------------------------- | ||
| # Ensure a docker-container builder (supports --push and multi-platform). | ||
| # --------------------------------------------------------------------------- | ||
| if ! docker buildx inspect "$BUILDX_BUILDER" >/dev/null 2>&1; then |
There was a problem hiding this comment.
Small thing: this mirrors the builder setup in livecodebench/push_image.sh, but that one also recreates a stale builder when QEMU was registered after it bootstrapped (unsupported_platforms() + recreate-once, ~L180-196). Without that, --multi-arch on an older endpoints-buildx builder fails with a raw buildx error. Worth porting that bit over — or factoring the shared builder logic into a sourced helper like _image_env.sh?
github-actions
Bot
added
the
size/normal
github-actions
Bot
added
size/large
…467) The published ghcr.io/mlcommons/endpoints:*-livecodebench image inherited zstd-compressed base layers from the dhi.io Docker Hardened Image base. enroot/ pyxis does not route the docker-namespaced zstd media type to a decompressor and hands the raw blob to tar, failing with "tar: This does not look like a tar archive" (#467). - scripts/lib_registry.sh: assert_gzip_layers — inspects the pushed manifest and fails on any non-gzip layer (walks a manifest list; docker + oci media types). - scripts/push_docker_image.sh: push via --output type=image,push=true,compression=gzip,force-compression=true so base layers are re-compressed to gzip; verify after push. - livecodebench/push_image.sh: verify layers after both the --platform and native push paths. verified a zstd build reproduces the tar error at pyxis import; the force-compression gzip build imports and runs cleanly.
6fa0f1c to
75b84bd
Compare
Signed-off-by: arekay-nv <230885705+arekay-nv@users.noreply.github.com>
…guards
Follow-up hardening for the image push/pull tooling (client + lcb-service):
- lib_registry.sh: add ref_exists_in_registry, a fail-closed 0/1/2 registry
existence probe. A two-stage match screens tooling/credential/permission
errors to "indeterminate" BEFORE matching not-found phrasings, so a missing
docker-credential helper ("executable file not found") can't be mistaken for
an absent tag (which would let a push overwrite an immutable one). Handles
GHCR "<ref>: not found" and ECR "name unknown ... does not exist".
- push_docker_image.sh: add --force; refuse to overwrite an existing :<sha>
unless --force. Blocks on any non-absent probe result (fail closed).
- push_image.sh: --no-build now verifies layers on a transient staging tag then
promotes onto the pinned tag only on success (verify-before-publish, so a
zstd/#467 image never poisons the immutable pin). Promote with
--prefer-index=false and re-assert the pin. Immutability guard reuses the
shared ref_exists_in_registry.
- _image_env.sh: always append an idempotent "-livecodebench" suffix to the LCB
tag so the image is self-identifying and never collides with the client
image's bare :<sha> in a shared registry package. Applied in the shared
push/pull helper, so pull stays symmetric (consumers still pass the SHA).
- Dockerfile.dev / lcb_serve.dockerfile: add OCI title/description LABELs so
docker inspect self-identifies each image. Client label placed last (cache);
its description is capability-neutral (PROVISION_DSR1/PROVISION_VBENCH may be 0).
- README: document the tag scheme and add a maintainer recipe to publish the
official ghcr.io/mlcommons/endpoints image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review-council findings on the image publish tooling: - lib_registry.sh: ref_exists_in_registry no longer classifies auth-hidden-as- not-found as absent. Registries that answer an unauthorized/private repo with "repository does not exist or may require 'docker login': denied" previously matched the not-found phrasing and returned "absent" (rc=1), letting a push- only / split-scope credential overwrite an immutable tag. Auth/denied/401/403 phrasings are now screened to indeterminate (rc=2) BEFORE the not-found match; GHCR "<ref>: not found" and ECR "name unknown" still resolve to absent. - publish-image.yml: add a `force` workflow_dispatch input (forwarded as a fixed --force flag via env, never interpolating the input into the command) so a SHA can be re-published (e.g. a different arch, or a failed run); document that platform/DSR1/cache are not part of the tag. - push_docker_image.sh / push_image.sh: document that the buildx-path assert_gzip_layers is a post-publish confirmation (the build forces gzip, so the pushed image is enroot-safe by construction), and that a transient inspect flake fails the job after the tag is live (re-run with --force). Reword the client guard comment: the :<sha> gate covers the whole build (both tags). - README: correct the pinned tag to :<sha>-livecodebench; add an official pull example (LCB_IMAGE_NAME=endpoints, else it resolves to a different repo); document that --no-build stages+verifies+promotes, is host-arch only, ignores --platform, and rejects (does not repair) zstd local images. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 participants
What does this PR do?
Adds a reproducible way to build and publish the endpoints client image to
ghcr.io/mlcommons/endpoints:scripts/push_docker_image.sh— buildsscripts/Dockerfile.devat a chosengit ref via
docker buildxand pushes it, tagged with both the short commit SHAand the ref name so consumers can pin either.
.github/workflows/publish-image.yml— a manually-triggered(
workflow_dispatch) workflow that runs the script, so a publish can be kickedoff from the Actions tab against any selected branch/tag/SHA without a local
Docker setup.
The workflow reuses the script (single source of truth for the tag scheme) rather
than duplicating build logic.
Behavior & design
linux/amd64(what benchmark clients run on); multi-arch is one flag/input away--multi-arch/linux/amd64,linux/arm64publishes a manifest list usable on x86 and arm:<short-sha>+:<sanitized-ref>docker-containerbuildx builder (the defaultdockerdriver can neither--pushnor build multi-platform)GHCR_USER/GHCR_TOKENif set, else assumesdocker login; workflow usesgithub.actor+ the automaticGITHUB_TOKEN:$SHAtag from a dirty tree (--allow-dirtyto override); optional ref checkout is restored on exitPROVISION_DSR11); set0for a lean client-only imageCI prerequisites (one-time, outside this PR)
permissions: packages: writeis set in the workflow; the org must allow theGITHUB_TOKENto write packages, and theendpointspackage must grant this repoWrite access (Package → Manage Actions access). Usually automatic since the
package lives under this repo's org.
Validation
--multi-arch --no-push): bothlinux/amd64(emulated)and
linux/arm64(native) compiled all stages, exit0.bash -n; the empty-arrayexpansion and tag sanitizer were unit-checked; the dirty-tree guard was exercised
across push/dry/allow-dirty/clean/checked-out cases.
ref resolution, Docker-tag sanitization beyond
/, bash-3.2 empty-array crash, andthe dirty-tree publish guard. Workflow security posture confirmed (inputs passed via
env:not inline${{ }}, actions pinned to SHAs, minimalcontents:read+packages:write).Type of change
Related issues
Testing
Checklist