Sync branch ci rollout - #327
Open
mcocdawc wants to merge 16 commits into
Open
Conversation
Migrates eckit onto the manifest-driven CI in ecmwf/ci-infrastructure, the same shape ecbuild and stack-dependencies already run. The legacy ecmwf/downstream-ci pipeline is untouched and keeps running: it moves to old_CI.yml (name: old CI) so the new pipeline can take the plain `CI` name, which is what the generated downstream orchestrators match their `workflow_run` trigger against. .ci/manifest.toml declares ecbuild (compiler-independent) and stack-deps (libaec, for FEATURE AEC) as dependencies, and eccodes as the one downstream consumer. Both build legs use platform ubuntu-24.04 so they reuse the upstream artifacts published under that slug rather than forcing rebuilds; runs-on and container are scheduling only and stay out of artifact identity. Build and test share a job and a container: the artifact is published as soon as the install step completes, so consumers are unblocked without waiting for ctest. No pre-commit job. The repo has a .pre-commit-config.yaml, but nothing has ever run it, and the shared workflow's `pre-commit run --all-files` fails today on the pre-existing workflows (yamllint --strict flags document-start and the `on:` truthy rule on every one). Enforcing that here would fail the pipeline for reasons unrelated to this migration; it is left as a follow-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the generated cross-repo-trigger.yml, and trigger-downstream.yml for the three repos that have consumers. Regenerate with the ci-infrastructure-generate CLI; never edit by hand. [[trigger-downstream]].ref stays at develop. That ref is what the orchestrator pins its `uses:` to — GHA forbids an expression there — and it selects only which workflow DEFINITION runs, never the code. Branch coordination is entirely a runtime affair and needs no help from it: the orchestrator passes branch: workflow_run.head_branch, pick-ref checks out the same-named branch in each consumer when one exists, and resolve_deps sees a current-branch matching ^sync-branch- and resolves every upstream's manifest and artifacts from it. One consequence worth knowing: `on: workflow_run` only fires for a workflow file present on the default branch, and runs that copy. So the fan-out cannot be exercised from sync-branch-ci-rollout at all — it goes live only once these files reach develop. Nothing to flip later; this is the steady-state form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci-infrastructure removed its reusable .github/workflows/pre-commit.yml (ecmwf/ci-infrastructure#14), so the comment here pointed at a workflow that no longer exists. Each repo now carries the job itself; stack-dependencies' ci.yml has the copyable version. Comment only — no job runs either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The expensive work — the actual compile — is already on arc-runner-very-large via the matrix legs. resolve was not: it sat on arc-runner-normal inside the base container, for a job that is one batched GraphQL walk of the manifest graph with no compiler and nothing to build. ubuntu-slim is the cheap 1-CPU ARC runner and needs no container, which is also what ci-infrastructure's generator picks for the identical resolve job it emits into cross-repo-trigger.yml. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These are the first expensive compiles in the new system, and artifact reuse does not help them: it avoids a rebuild only when nothing changed, so a PR touching one source file still recompiles the whole tree in a fresh container. That is exactly what sccache is for. ecbuild and stack-dependencies did without it for good reason — one compiles nothing, the other builds once and is then reused from its artifact. The steps live in the build action rather than the workflow so both entry points get them: the generated cross-repo-trigger.yml calls this action directly and never sees the workflow's own steps. It already exports SCCACHE_BUCKET; ci.yml now does too, alongside the artifact bucket. They share the object store's endpoint and credentials and differ only in bucket. Opt-in by presence of that bucket. sccache has no default bucket and a daemon that cannot reach one refuses to start, which would fail the build outright, so with SCCACHE_BUCKET unset both steps no-op, the launcher flags come out empty and the compile just runs uncached. A repo without the secret still builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts the ubuntu-slim change: it saved runner size and lost far more to a per-job venv build. ci-infrastructure is baked into the base image and advertised via CI_INFRASTRUCTURE_PYTHON, which makes ensure-infrastructure-present a true no-op — no build, no PyPI/GitHub egress. ubuntu-slim is a runner label, not an image, so with no container there is no bake and every run installs the package from scratch. It cannot even be cached between runs: the venv lives under RUNNER_TOOL_CACHE inside _work, and ARC's Kubernetes mode overlays _work with an ephemeral per-job volume, so the cache is discarded each time. resolve is short, so the container dominates its cost, not the runner size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #327 +/- ##
========================================
Coverage 67.61% 67.62%
========================================
Files 1182 1182
Lines 61697 61697
Branches 4667 4667
========================================
+ Hits 41718 41724 +6
+ Misses 19979 19973 -6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Temporary, and paired with landing the generated orchestration on the default branches ahead of the rest of the rollout. on: workflow_run only ever executes the DEFAULT BRANCH's copy of trigger-downstream.yml, so the fan-out cannot be exercised from a branch at all — a branch's own orchestrator is inert. The only way to see the full downstream CI before merging is to put the orchestrator on the default branch and have it call consumer logic that still lives on the sync branch. [[trigger-downstream]] .ref is what selects that: it pins the `uses:`, which GHA forbids being an expression, and so decides which workflow DEFINITION runs. Runtime branch coordination is unaffected and needs no pinning: the orchestrator passes branch: workflow_run.head_branch, pick-ref checks out the same-named branch in each consumer, and resolve_deps resolves upstream manifests and artifacts from it. Flip these to develop and regenerate when the rollout merges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts the temporary sync-branch pin. It only had a purpose alongside a separate plumbing branch that put the orchestrator on the default branch early; with the whole rollout landing from one branch, [[trigger-downstream]].ref should just name where the consumer workflow will actually live, and there is nothing to unwind at merge time. The consequence is worth stating: on: workflow_run only ever executes the default branch's copy of trigger-downstream.yml, so the fan-out cannot fire until this branch merges. Each repo's own CI — resolve, build, test, publish — is fully exercised on the branch; the cross-repo edges, including eckit -> eccodes, are first exercised on develop after the merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eckit built and tested inside a SLURM job, linking the atos-hpc-gnu ecbuild and stack-deps artifacts. The recipe runs ctest, so a green job proves those artifacts link and run on a compute node rather than merely that they downloaded. `needs` names the HPC kinds, never the runner ones: the two lanes are separate artifact universes distinguished by platform, and a runner rebuild must not be what re-runs this. The compiler fields are discriminators for artifact identity and dep matching, not the binaries the recipe invokes — the job loads whatever prgenv/gnu provides, and they only have to agree with what the upstreams published under. The job runs in the internal-tools container for its cluster ssh identity, and stages dep prefixes with install-python-deps: false — they are consumed on the compute node, not in this container. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Temporary, and paired with landing the generated orchestration on the default
branches ahead of the rest of the rollout.
on: workflow_run only ever executes the DEFAULT BRANCH's copy of
trigger-downstream{,-hpc}.yml, so the fan-out cannot be exercised from a branch
at all. The only way to see the full downstream CI before merging is to put the
orchestrator on the default branch and have it call consumer logic that still
lives here. [[trigger-downstream]].ref selects that: it pins the `uses:`, which
GHA forbids being an expression.
Runtime branch coordination needs no pinning and is unaffected: the orchestrator
passes branch: workflow_run.head_branch, pick-ref checks out the same-named
branch in each consumer, and resolve_deps resolves upstream manifests and
artifacts from it.
Flip to develop and regenerate when the rollout merges.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci.yml has run ctest here all along, but the generated cross-repo-trigger.yml job only built and published. So when an upstream fanned out to us, a green downstream/runner proved only that eckit still LINKED against the new ecbuild and stack-deps -- never that it still worked. Set `ctest = true` on [matrix.build] and regenerate. The step lands after the publish step, matching ci.yml: whoever consumes our artifact picks it up as soon as publish runs and never waits on these tests. Plain `ctest`, exactly what ci.yml runs. Also add .github/actionlint.yaml so the ARC scale-set labels stop reading as unknown runners. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZXmNdJJYqZkpBNTycoPdP
Publish ran before Test, so a failing suite still put its output in the store — and nothing downstream can tell the difference afterwards. resolve_deps and fetch_deps both decide with a bare object_exists, and check_artifact stops asking about the producing run the moment the object is found. The orchestrator's `conclusion == 'success'` gate does not cover this: it stops the fan-out from starting, not a rebuild-request waiter (whose poll returns as soon as Publish completes, before ctest has begun) or a later run that finds the object already present. Swapping the two steps closes it — a failing step ends the job, so nothing is uploaded. The cost is that consumers now wait for our tests. `Print dependency table` moves above `Test` in the same pass. It does not depend on the publish step, and having it run on a red suite is the point: it names the upstream shas the failing build resolved against. The generated cross-repo-trigger.yml carries the same swap, from the regenerated ci-infrastructure. HPC is unaffected: those recipes already ctest before `cmake --install`, and build-on-hpc publishes only after the job succeeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZXmNdJJYqZkpBNTycoPdP
Every suite ran one test at a time. The builds are parallel — `cmake --build
--parallel $(nproc)` on the runner, `--parallel ${SLURM_NTASKS:-8}` on HPC —
but ctest defaults to serial and nothing overrode it, so the test phase left an
arc-runner-very-large (or an 8-task SLURM allocation) idle but for one process.
Visible in every run so far as cumulative time equal to wall-clock:
ecbuild gh-hosted 48 tests 80.65 sec*proc 80.70 sec real
ecbuild HPC gnu 48 tests 126.85 sec*proc 127.10 sec real
eccodes fan-out 121 tests 74.19 sec*proc 74.26 sec real
A ratio of 1.00 is the signature of no concurrency at all.
-j $(nproc) on the runner lanes and -j ${SLURM_NTASKS:-8} on HPC, mirroring
what the build step already asks for on each. Applied to ci.yml, the manifest's
ctest-args (so the fan-out matches) and the HPC recipes.
Not applied to ecflow in the same shape: its suite starts real servers, so it
takes the -j 8 its own cd-config.yml already uses, on its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZXmNdJJYqZkpBNTycoPdP
Repoints every container reference at the renamed Harbor projects:
eccr.ecmwf.int/public-playground-ci/... -> public-ci-images/...
eccr.ecmwf.int/private-playground-ci/... -> private-ci-images/...
in .ci/manifest.toml (the `container` field on each matrix leg) and in ci.yml's
hardcoded resolve-job image. The "playground" name has outlived the playground.
Stage 3 of 3, and deliberately last: nothing may point at a project before the
images exist in it. ci-infrastructure's rename published all six public images
first (its own run is green and the repositories are visible in the registry),
then ci-container-images published internal-tools to private-ci-images. Every
image named here has been confirmed present before this commit was made.
No artifact churn: `container` is scheduling only and never enters artifact
identity, so nothing published under the old image names is invalidated and no
rebuild cascades from this.
Regenerating changes nothing, which is the expected result rather than a missed
step — the generated cross-repo-trigger workflows carry `image: ${{
matrix.container || '' }}` and resolve the image from the matrix at run time, so
the names live in the manifest alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZXmNdJJYqZkpBNTycoPdP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contributor Declaration
By opening this pull request, I affirm the following:
🌦️ >> Documentation << 🌦️
https://sites.ecmwf.int/docs/dev-section/eckit/pull-requests/PR-327