CI: single content-addressed criterion for Docker image selection and rebuilds by Fedr · Pull Request #6436 · MeshInspector/MeshLib

Fedr · 2026-07-16T14:12:41Z

Problem

Which Docker tag the build jobs pull and whether prepare-images runs were decided by dorny/paths-filter lists that drifted from the inputs of scripts/devops/docker_image_source_checksum.sh: the filters exclude thirdparty/licenses/**, the checksum script hashes it. Licenses-only changes (#6429, #6422, #6434) therefore moved every Linux image checksum without triggering a rebuild, so the new source-checksum-* tags were never pushed and consumers that pin them fail with manifest unknown. Interleaved image-affecting PRs leave similar gaps: master currently has meshlib-emscripten-build-c-bindings-4.0.19-arm64 built for its content but not the 3.1.38 twin.

Fix

Drop the Linux paths-filters entirely and derive everything from one criterion — the content-addressed checksum, compared against the registry:

The prepare-config steps pick the conventional tag (branch tag on internal PRs, latest otherwise) and call scripts/devops/sync_docker_image_tags.sh <linux|linux-vcpkg> <tag>, which computes each image's source-checksum-* tag and points the conventional tag at it right in the gate, with an unconditional docker buildx imagetools create:

Conventional tag already points at the wanted image: the create is a no-op.
source-checksum-* tag exists but the conventional tag points elsewhere: the create retags it — seconds, no build matrix spun up.
source-checksum-* tag absent: the create fails, the script exits 1 and the step sets need_rebuild=true, so prepare-images builds and pushes it and points the conventional tag at it (its jobs unchanged: build-or-reuse per image, then imagetools create).

So need_rebuild now literally means "something must be built", and pull requests never move latest — they always pull their branch tag, synced the same way. Credential-less PRs (forks and Dependabot — no DOCKERHUB_TOKEN, CAN_PUSH_DOCKER_TAGS=false) skip the login and the registry entirely: they just use latest as-is. The gate also runs on schedule (the old filters were skipped there, #6403), so any gap self-heals on the nightly at the latest.

The image inventory itself is also single-sourced: the per-family matrix/docker-images-{linux,emscripten-c-bindings,linux-vcpkg}.json files are read both by the gate script (via jq) and by the compute-image-matrices job in prepare-images.yml, which derives the build matrices from them. Adding or removing an image is a one-file change; the gate and the builders cannot drift.

need_windows_vcpkg_rebuild is untouched — the Windows vcpkg image is not checksum-tagged.

Notes

The first run of this PR (before the rework) already rebuilt and pushed the checksum tags that were missing for master content as of this morning; later runs demonstrate the branch-tag path on top of them — the latest one synced all eight linux-family branch tags inline in ~9 s with need_rebuild=false, the case that previously spun up the full matrix.
The thirdparty/licenses/** checksum inputs are deliberately unchanged: narrowing them would invalidate every existing tag. With the registry as source of truth a too-broad checksum only costs an occasional no-op-content rebuild; trimming the input list can be a follow-up.

Review follow-up: matrix-builder reads the inventory files directly

Per review, matrix-builder learned to load the base matrix from a file, and compute-image-matrices now uses that instead of raw jq:

New action input, mutually exclusive with matrix: matrix-file (path of a YAML/JSON file holding the base matrix). The base matrix now also accepts the include-list form: a list of entries used verbatim. A matrix-key selector existed while the inventory was one bundled file and was removed per review once the files were split per family. dist/ rebuilt; README updated.
Each family is one matrix-builder step over its own inventory file; the linux step adds per-label exclude rules. The jq startswith("ubuntu") filter became explicit per-distro rules, so a new ubuntu distro in docker-images-linux.json needs a matching rule line (commented in the workflow).
New tests: engine coverage for the list-form base, child-process tests for the file loading and input-validation errors, and a parity suite pinning the workflow's rules to the replaced jq filter across all 8 disable-flag combinations (fixtures captured from the actual jq output).
The emscripten-c-bindings / linux-vcpkg jobs consume the flat entry lists as matrix.include, so their cell display names now include the runner field (e.g. "(3.1.38, ubuntu-latest)").

🤖 Generated with Claude Code

… rebuilds Selecting the Docker image tag and deciding whether prepare-images must run used dorny/paths-filter lists that drifted from the inputs of docker_image_source_checksum.sh (the filters exclude thirdparty/licenses/**, the checksum script does not), so licenses-only changes moved every image checksum without building the new source-checksum-* tags, breaking consumers that pin them. Interleaved image-affecting PRs left similar gaps: master merge combinations nobody's branch built (e.g. emscripten-build-c-bindings 3.1.38 vs 4.0.19 right now). Replace the filters with one criterion derived from the checksum itself: select_docker_image_tags.sh resolves each image's source-checksum-* tag and the conventional tags to their leaf manifest digests (imagetools create wraps single manifests into lists, so own digests differ) and decides: - all images reachable via latest -> use latest, skip prepare-images; - source-checksum-* tag absent -> prepare-images builds and pushes it; - present but conventional tag elsewhere -> prepare-images repoints only. Pull requests never move latest: they fall back to branch tags unless their sources already match latest. Registry query failures count as absent, i.e. fail toward rebuilding. The check also runs on schedule, so any gap self-heals on the nightly at the latest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Per review: the rebuild-selection mechanism moves to the registry-check approach of #6436, which the filter edits here would conflict with; the licenses exclusion composes with it under either merge order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…json The gate script hard-coded the (repository, distro) list that had to mirror the prepare-images build matrices by hand. Move the inventory to matrix/docker-images.json: select_docker_image_tags.sh reads it with jq, and the compute-image-matrices job (extended to serve the vcpkg and emscripten-c-bindings jobs too) derives all build matrices from the same file, so adding or removing an image is a one-file change and the gate and the builders cannot drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

oitel · 2026-07-16T18:07:35Z

+      cbindings_axis:    ${{ steps.compute.outputs.cbindings_axis }}
+      cbindings_include: ${{ steps.compute.outputs.cbindings_include }}
+      vcpkg_arch_axis: ${{ steps.compute.outputs.vcpkg_arch_axis }}
+      vcpkg_os_axis:   ${{ steps.compute.outputs.vcpkg_os_axis }}
+      vcpkg_include:   ${{ steps.compute.outputs.vcpkg_include }}


You can output the whole matrices instead.

oitel · 2026-07-17T08:00:13Z

@@ -241,12 +251,8 @@ jobs:
      fail-fast: false
      matrix:
        # The old `3.1.38` didn't have ARM images.


The comment does not belong here anymore.

oitel · 2026-07-17T08:08:37Z

Split the script into several logical blocks.

- select_docker_image_tags.sh split into docker_image_list.sh (inventory from matrix/docker-images.json) and docker_image_leaf_digests.sh (registry digest query), keeping only the selection logic. - compute-image-matrices outputs complete matrix objects consumed via a single fromJSON each, instead of separate axis/include outputs. - Drop the stale 3.1.38 comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…s create Instead of comparing digests and delegating retags to prepare-images, the gate now runs `docker buildx imagetools create` per image: a no-op when the conventional tag already points at the wanted image, a retag when it doesn't, and a failure exactly when the source-checksum-* tag is absent — which is now the only thing that sets need_rebuild=true, so prepare-images runs only when something must actually be built. PRs still resolve to `latest` read-only while it matches, so clean branches mint no tags; fork PRs (no DOCKERHUB_TOKEN) keep the read-only digest comparison. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Dependabot branches are same-repo, so the fork check alone would run the DockerHub login with an empty secret and fail prepare-config on every Dependabot PR; route them to the read-only digest fallback instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fork and Dependabot PRs cannot sync tags anyway; per review discussion they now short-circuit to image_tag=latest / need_rebuild=false instead of running the read-only digest comparison. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

With the unconditional imagetools create, the read-only digest comparison has no consumers left: PRs now always sync and pull their branch tag instead of resolving to latest first. The inventory jq folds back into select_docker_image_tags.sh, still reading the shared docker-images.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

oitel · 2026-07-17T12:39:51Z

+      # have no repository secrets; they skip the login and the gate then
+      # selects `latest` as-is without inspecting the registry.
+      - name: Login to DockerHub
+        if: ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && github.actor != 'dependabot[bot]' }}


How about

jobs: prepare-config: env: TAG_WRITES: ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && github.actor != 'dependabot[bot]' }}

and then

if: ${{ env.TAG_WRITES == 'true' }}

?

does job level if support env?

Job-level if does not see the env context, but this suggestion only needs it at step level (the login step), where env is available — implemented in f6ef688: TAG_WRITES is defined once as job env and the login step tests env.TAG_WRITES.

oitel · 2026-07-17T12:46:48Z

+          BRANCH_TAG=""
+          if [ "${{ github.event_name }}" = "pull_request" ]; then
            # https://stackoverflow.com/q/58033366/7325599
-            IMAGE_TAG=$(echo "${{ github.head_ref || github.ref_name }}" | sed -r 's/[^a-zA-Z0-9._-]+/-/g')
-          else
-            IMAGE_TAG="latest"
+            BRANCH_TAG=$(echo "${{ github.head_ref }}" | sed -r 's/[^a-zA-Z0-9._-]+/-/g')
          fi
-          echo "image_tag=${IMAGE_TAG}" >> $GITHUB_OUTPUT
+          scripts/devops/select_docker_image_tags.sh linux "${BRANCH_TAG}" >> $GITHUB_OUTPUT


if [ "${{ github.event_name }}" = "pull_request" || "${TAG_WRITES}" != "true" ]; then # https://stackoverflow.com/q/58033366/7325599 IMAGE_TAG=$(echo "${{ github.head_ref }}" | sed -r 's/[^a-zA-Z0-9._-]+/-/g') else IMAGE_TAG="latest" fi echo "image_tag=${IMAGE_TAG}" >> $GITHUB_OUTPUT if scripts/devops/select_docker_image_tags.sh linux "${IMAGE_TAG}" ; then echo "need_rebuild=false" >> $GITHUB_OUTPUT else echo "need_rebuild=true" >> $GITHUB_OUTPUT fi

And update the select_docker_image_tags.sh script accordingly.

…de protocol - TAG_WRITES defined once as prepare-config job env; the login step tests it via a step-level if (job-level if does not see the env context, but step-level does). - The steps now pick the tag name themselves (branch tag on internal PRs, latest otherwise) and map the script's exit status to need_rebuild; select_docker_image_tags.sh becomes sync_docker_image_tags.sh: point <tag> of every family image at its source-checksum-* tag, exit 0 when synced, 1 when something must be built, 2 on usage errors (escalated to a step failure rather than a spurious rebuild). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…x-key) New inputs, mutually exclusive with `matrix`: `matrix-file` names a YAML/JSON file holding the base matrix, so one inventory file can be shared between workflows and shell scripts; `matrix-key` selects one top-level key of that document for files bundling several matrices. The base matrix (from either input) now also accepts the include-list form: a list of entries used verbatim, with the axis keys being the union of the entries' keys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Replace the jq disable-label filter in compute-image-matrices with a matrix-builder step reading docker-images.json via the new matrix-file / matrix-key inputs, per review. The c-bindings and vcpkg outputs stay on jq: they are reshapes into axes + include form (keeping runner names out of the cell display names), which the action's flat entry-list output cannot express. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

oitel · 2026-07-17T16:03:19Z

  # use the ${{ needs.config.outputs.bool-param == 'true' }} construction to check boolean values
  config:
    uses: ./.github/workflows/config.yml
+    # for the DockerHub login in prepare-config (tag sync)


Redundant comment.

oitel · 2026-07-17T16:06:59Z

+    echo "${repo}:${tag} -> ${hash_tag}" >&2
+  else
+    echo "${repo}:${hash_tag} absent, needs building" >&2
+    synced=false


Why not early exit?

oitel · 2026-07-17T16:08:09Z

+    pairs=$(jq -r '."linux-vcpkg"[] | "meshlib/meshlib-\(.os)-vcpkg-\(.arch) \(.os)-vcpkg"' "${manifest}") ;;
+  *)
+    echo "unknown image family: ${family}" >&2
+    exit 2 ;;


Nothing to rebuild → does not need rebuild → exit 0.

oitel · 2026-07-17T16:08:41Z

+    exit 2 ;;
+esac
+
+here=$(dirname "$0")


Inline the variable.

oitel · 2026-07-17T16:10:13Z

+    # The label-derived booleans (full_config_build / upload_artifacts /
+    # version_namespace) are not here: they are written to $GITHUB_ENV by
+    # the live-labels step below.


Redundant comment.

oitel · 2026-07-17T16:12:12Z

+  fi
+done <<< "${pairs}"
+
+[ "${synced}" = true ]


Exit with 0 when there's nothing to rebuild.

oitel · 2026-07-17T16:13:01Z

+                - { distro: ubuntu22, arch: x64 }
+                - { distro: ubuntu24, arch: x64 }


No inline JSON.

oitel · 2026-07-17T16:16:25Z

-          fi
+          FILE=.github/workflows/matrix/docker-images.json
+          {
+            echo "cbindings_matrix=$(jq -c '{emsdk_image: [."emscripten-c-bindings"[].emsdk_image], include: ."emscripten-c-bindings"}' "$FILE")"


Why not separate matrix-builder steps with matrix-key: linux-vcpkg and matrix-key: emscripten-c-bindings?

oitel · 2026-07-17T16:18:20Z

I feel like it's more convenient to have three separate files for each kind since the data are in fact heterogeneous.

oitel · 2026-07-17T16:18:43Z

          echo "image_tag=${IMAGE_TAG}" >> $GITHUB_OUTPUT

+          NEED_REBUILD=false
+          if [ "${TAG_WRITES}" = "true" ]; then


if [ "${TAG_WRITES}" != "true" ] || ! scripts/devops/sync_docker_image_tags.sh linux-vcpkg "${IMAGE_TAG}" ; then echo "need_rebuild=false" >> $GITHUB_OUTPUT else echo "need_rebuild=true" >> $GITHUB_OUTPUT fi

…t cleanup - matrix/docker-images.json split into docker-images-{linux, emscripten-c-bindings,linux-vcpkg}.json (bare entry lists); each family gets its own matrix-builder step (matrix-file only, no matrix-key, no inline JSON in rules, jq reshape step gone; the cbindings/vcpkg jobs consume flat include lists). - sync_docker_image_tags.sh: early exit 1 on the first absent image, fall-through exit 0 when there is nothing to build (unknown family included), dirname inlined. - config.yml: TAG_WRITES renamed to CAN_PUSH_DOCKER_TAGS, redundant comments dropped, select steps reduced to a plain if/else on the script's exit status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

oitel · 2026-07-17T17:19:13Z

+|---------------|----------|---------|------------------------------------------------------------------------------------------|
+| `matrix`      | no       | `{}`    | YAML/JSON base matrix: axis map (cartesian product) or list of entries (used as-is). Mutually exclusive with `matrix-file`. |
+| `matrix-file` | no       | —       | Path of a YAML/JSON file to load the base matrix from (same forms as `matrix`). Mutually exclusive with `matrix`. |
+| `matrix-key`  | no       | —       | Top-level key to select within the `matrix-file` document, for files bundling several matrices. Requires `matrix-file`. |


The option is redundant now.

oitel · 2026-07-17T17:20:30Z

+      # Linux cells encode only `distro` and `arch` so the auto-generated
+      # cell name stays "(<distro>, <arch>)"; the runner OS and image suffix
+      # are derived from `matrix.arch` inline below. A new ubuntu distro in
+      # docker-images-linux.json must also be added to the exclude rules here.
+      # `emscripten-generate-c-bindings` (the image for generate-c-bindings)
+      # has no disable rule: that job always runs, so the image is always
+      # built. The parity tests in the action's tests/engine.test.js pin
+      # these rules to the jq filter they replaced.


Shorten the comment.

oitel · 2026-07-17T17:20:45Z

+  # Build the job matrices from the image inventory in the
+  # matrix/docker-images-*.json files (shared with
+  # scripts/devops/sync_docker_image_tags.sh), dropping linux cells the PR
+  # opted out of via `disable-build-<platform>` labels. The `matrix` context
+  # is not available in job-level `if:` for reusable workflow inputs, so
+  # per-cell skipping must happen here.


Shorten the comment.

oitel · 2026-07-17T17:21:17Z

+# Points <tag> of every image of the family at its content-addressed
+# source-checksum-* tag (docker_image_source_checksum.sh) with an
+# unconditional `docker buildx imagetools create` (requires a prior docker
+# login): a no-op when it already points there, a cheap retag when it
+# doesn't, and a failure exactly when the source-checksum-* tag is absent,
+# i.e. when the image really must be built — then this script exits 1;
+# it exits 0 when there is nothing to build. Must run from the repository
+# root. The image inventory comes from the matrix/docker-images-*.json
+# files, shared with the build matrices in
+# .github/workflows/prepare-images.yml; Docker Hub repository names are
+# derived here the same way its jobs derive them from the matrix.


Shorten the comment.

With one inventory file per family, matrix-key has no consumers: remove the input from the action (src, action.yml, README, tests; dist rebuilt, 62 tests pass). Shorten the compute-image-matrices and linux-step comments in prepare-images.yml and the sync_docker_image_tags.sh header. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fedr added disable-build-windows disable-build-macos disable-build-linux-vcpkg labels Jul 16, 2026

Fedr force-pushed the ci/registry-check-image-rebuild branch from 3ce70d4 to b2d7fc4 Compare July 16, 2026 15:13

Fedr changed the title ~~CI: rebuild Docker images when their source-checksum tags are missing from the registry~~ CI: single content-addressed criterion for Docker image selection and rebuilds Jul 16, 2026

Fedr mentioned this pull request Jul 16, 2026

Exclude thirdparty/licenses from the docker image source checksums #6437

Closed

Grantim approved these changes Jul 16, 2026

View reviewed changes

oitel reviewed Jul 17, 2026

View reviewed changes

Fedr and others added 5 commits July 17, 2026 12:07

oitel reviewed Jul 17, 2026

View reviewed changes

Fedr and others added 3 commits July 17, 2026 16:42

oitel reviewed Jul 17, 2026

View reviewed changes

Grantim approved these changes Jul 17, 2026

View reviewed changes

oitel approved these changes Jul 17, 2026

View reviewed changes

Fedr merged commit 7610ed4 into master Jul 17, 2026
34 checks passed

Fedr deleted the ci/registry-check-image-rebuild branch July 17, 2026 18:22

Grantim mentioned this pull request Jul 17, 2026

CI: exclude thirdparty/licenses from docker image source checksums #6447

Merged

		- { distro: ubuntu22, arch: x64 }
		- { distro: ubuntu24, arch: x64 }

Uh oh!

Conversation

Fedr commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Notes

Review follow-up: matrix-builder reads the inventory files directly

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fedr commented Jul 16, 2026 •

edited

Loading