Skip to content

Add Ubuntu APT mirror fallback - #6643

Open
thomhurst wants to merge 3 commits into
mainfrom
agent/apt-mirror-fallback
Open

Add Ubuntu APT mirror fallback#6643
thomhurst wants to merge 3 commits into
mainfrom
agent/apt-mirror-fallback

Conversation

@thomhurst

@thomhurst thomhurst commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Summary

  • configure APT's native mirror+file transport on Ubuntu CI runners
  • prefer archive.ubuntu.com and fall back to azure.archive.ubuntu.com
  • apply the mirror configuration before disk cleanup and pipeline execution

Why

Ubuntu package mirrors have intermittently stalled, causing Playwright dependency installation to exceed its timeout. The timed-out install can leave apt-get holding the package-manager lock, making subsequent retries fail immediately.

Using APT's native mirror fallback reduces exposure to a single degraded mirror without changing the test pipeline.

Validation

  • parsed .github/workflows/dotnet.yml as YAML
  • validated the embedded Bash syntax
  • ran git diff --check

Summary by CodeRabbit

  • Chores
    • Added an Ubuntu workflow step to configure prioritized fallback package mirrors.
    • Improved package retrieval reliability in supported automated workflows.

@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2bdbce99-a8d4-4ced-bc02-31b85128675d

📥 Commits

Reviewing files that changed from the base of the PR and between 19249cf and 5af7f19.

📒 Files selected for processing (1)
  • .github/workflows/dotnet.yml

Included review availability: Your plan provides up to 3 included reviews per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The .NET workflow adds an Ubuntu-only step that writes three prioritized fallback APT mirror URLs to /etc/apt/apt-mirrors.txt with elevated permissions.

Changes

APT mirror configuration

Layer / File(s) Summary
Configure Ubuntu APT mirrors
.github/workflows/dotnet.yml
The workflow writes three prioritized APT mirror URLs to /etc/apt/apt-mirrors.txt on Ubuntu runners.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 5af7f

The mirror fallback change is localized and introduces no actionable merge-blocking risk; it is merge-ready after normal checks and review.

Poem

A rabbit hops where workflows run,
Three mirrors line up in the sun.
Ubuntu finds a quicker way,
With fallback paths set bright today.
Hop, hop—builds can start!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding an Ubuntu APT mirror fallback for CI.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/apt-mirror-fallback

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Aug 19, 2026

Copy link
Copy Markdown

Greptile Summary

The PR adds an Ubuntu-only setup step intended to provide prioritized APT mirrors before disk cleanup and pipeline execution.

  • Writes primary, Azure fallback, and security mirror entries to /etc/apt/apt-mirrors.txt.
  • The generated list is not connected to an active APT source, so the intended fallback is not enabled.

Confidence Score: 4/5

The PR is not yet safe to merge because the Ubuntu APT fallback remains inactive.

The workflow writes a prioritized mirror list but never references it from an active APT source, so subsequent package installations retain the single-mirror failure mode this change is intended to address.

Files Needing Attention: .github/workflows/dotnet.yml

Important Files Changed

Filename Overview
.github/workflows/dotnet.yml Adds the prioritized mirror-list file, but does not configure an APT source to consume it, leaving fallback behavior inactive.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A["Write /etc/apt/apt-mirrors.txt"] -. "no mirror+file source reference" .-> B["APT active sources"]
  B --> C["Single configured mirror"]
  C --> D["Subsequent package installation"]
Loading

Reviews (3): Last reviewed commit: "ci: update active apt mirror list" | Re-trigger Greptile

Comment thread .github/workflows/dotnet.yml Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b3f730293

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/dotnet.yml Outdated
Comment on lines +38 to +39
'http://archive.ubuntu.com/ubuntu/' '1' \
'http://azure.archive.ubuntu.com/ubuntu/' '2' |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add explicit failover instead of relying on mirror priorities

In the Configure Ubuntu mirror fallback step, priority:1 and priority:2 do not make the second URI a retry target: as documented by apt-transport-mirror(1), the transport redirects requests to mirrors selected from the list, while failures remain with the underlying HTTP transport. With Ubuntu's apt 2.8.3, an unreachable priority-1 endpoint is retried repeatedly without any request reaching a working priority-2 endpoint. Consequently, when archive.ubuntu.com stalls—the scenario this change targets—the job still never tries Azure; use an explicit retry that swaps the source or another mechanism that performs failover.

Useful? React with 👍 / 👎.

@claude

claude Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Code review

Confirmed bug — the sed replacement leaves a stray trailing slash, which will break APT on every ubuntu-latest job.

steps:
- name: Configure Ubuntu mirror fallback
if: matrix.os == 'ubuntu-latest'
run: |
printf '%s\tpriority:%s\n' \
'http://archive.ubuntu.com/ubuntu/' '1' \
'http://azure.archive.ubuntu.com/ubuntu/' '2' |
sudo tee /etc/apt/ubuntu-mirrors.txt > /dev/null
sudo sed -i \
's|http://azure.archive.ubuntu.com/ubuntu|mirror+file:/etc/apt/ubuntu-mirrors.txt|' \

ubuntu.sources (deb822 format) stores the URI with a trailing slash, e.g. URIs: http://azure.archive.ubuntu.com/ubuntu/. The sed pattern only matches http://azure.archive.ubuntu.com/ubuntu (no trailing /), so the substitution leaves the original slash in place:

$ sed -i 's|http://azure.archive.ubuntu.com/ubuntu|mirror+file:/etc/apt/ubuntu-mirrors.txt|' file
URIs: mirror+file:/etc/apt/ubuntu-mirrors.txt/

A trailing / on a mirror+file: path makes APT treat it as a directory, which fails to open since ubuntu-mirrors.txt is a regular file — so apt-get update/install would fail on every Ubuntu run, turning an intermittent mirror stall into a 100% failure rate. (Greptile flagged the same root cause independently.)

Suggested fix — match optional scheme/trailing slash instead of relying on the exact current format, so this doesn't silently regress if the base image's default URI format changes again:

sudo sed -i -E \
  's#https?://azure\.archive\.ubuntu\.com/ubuntu/?#mirror+file:/etc/apt/ubuntu-mirrors.txt#' \
  /etc/apt/sources.list.d/ubuntu.sources

Design note (optional, not blocking): hand-patching the runner image's OS-provided ubuntu.sources file with a substring sed is inherently fragile — it silently no-ops if Canonical/Microsoft ever change the default URI format on the base image, and it silently breaks (as above) if the format shifts in a way that's almost but not quite matched. Two alternatives worth considering if mirror stalls keep recurring:

  • Point APT at Ubuntu's own dynamic mirror service (mirror://mirrors.ubuntu.com/mirrors.txt) instead of hand-maintaining a 2-entry list — it already does geo-based ordering with many fallback mirrors.
  • Address the root cause directly by adding Acquire::Retries / a longer timeout to the specific apt-get/Playwright-install invocation that's timing out, rather than rewriting global APT source configuration for a problem that shows up in one step.

Everything else in the diff (the mirror+file list syntax with priority:N, the step ordering before disk cleanup, and scoping to matrix.os == 'ubuntu-latest') is correct.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review

Re-checked against the previous review comment on this PR: the trailing-slash sed bug is fixed correctly (the regex now has /? and matches the deb822 URIs: line without leaving a stray slash). However, that fix is applied to the wrong file, which produces a new, more serious problem.

Confirmed bug — the step is a silent no-op; it does not fix the mirror stall it's meant to fix

steps:
- name: Configure Ubuntu mirror fallback
if: matrix.os == 'ubuntu-latest'
run: |
printf '%s\tpriority:%s\n' \
'http://archive.ubuntu.com/ubuntu/' '1' \
'http://azure.archive.ubuntu.com/ubuntu/' '2' |
sudo tee /etc/apt/ubuntu-mirrors.txt > /dev/null
sudo sed -i -E \
's#https?://azure\.archive\.ubuntu\.com/ubuntu/?#mirror+file:/etc/apt/ubuntu-mirrors.txt#' \

On the current ubuntu-latest (24.04 "noble") image, /etc/apt/sources.list.d/ubuntu.sources already uses APT's mirror+file transport:

URIs: mirror+file:/etc/apt/apt-mirrors.txt

There is no literal http://azure.archive.ubuntu.com/ubuntu string in that file for the sed to match — I verified this directly against a real ubuntu-latest-equivalent image. Since sed with zero matches exits 0, the step goes green but changes nothing: ubuntu.sources is untouched, and the newly-written /etc/apt/ubuntu-mirrors.txt is never referenced by anything. The mirror actually used for every apt call is still governed by the pre-existing /etc/apt/apt-mirrors.txt, which this PR never touches.

Fix: edit /etc/apt/apt-mirrors.txt directly (it already exists in this exact tab-separated URI<TAB>priority:N format) instead of trying to rewrite ubuntu.sources:

http://azure.archive.ubuntu.com/ubuntu/	priority:1
https://archive.ubuntu.com/ubuntu/	priority:2
https://security.ubuntu.com/ubuntu/	priority:3

If you fix the target file, two more things to fix in the new content

  1. Priority order looks inverted. GitHub-hosted runners run on Azure, so azure.archive.ubuntu.com is the in-region mirror — that's exactly why the stock image already lists it at priority:1. The PR's replacement list puts archive.ubuntu.com (a smaller, cross-region pool — the classic source of cloud-IP stalls) at priority:1 instead. That makes the steady-state path slower/less reliable and only falls back to the fast mirror after a failure. Keep azure first; add archive as the fallback.
  2. Don't drop security.ubuntu.com. The replacement list has only 2 entries, dropping the stock image's dedicated security mirror (used by the noble-security stanza). archive/azure do carry -security too so it won't hard-break, but it needlessly shrinks the fallback set for security updates.

Design suggestions (not blocking)

  • Fail loudly instead of silently reverting. There's no set -euo pipefail, and no post-check that the substitution actually took effect. A future Ubuntu image change (e.g. a different default URI/path) would make this silently no-op again exactly as it does today, undermining the reliability goal. Consider set -euo pipefail plus a grep -q ... || exit 1 sanity check after the edit.
  • Consider whether this addresses the actual symptom. APT's mirror method fails over on acquire errors, not stalls-then-trickles. If the reported failure mode is a timeout rather than a hard connection error, Acquire::http::Timeout / Acquire::Retries tuning (via a drop-in in /etc/apt/apt.conf.d/) on the specific slow step may be the more direct fix, and is additive rather than mutating a package-owned config file that can silently diverge on image updates.
  • Scoping mutation to a package-owned file (ubuntu.sources or apt-mirrors.txt, both maintained by the base image) for one flaky step means every apt call in the job is affected by any bug here. That's probably fine at this scale (one call site today), but worth an eventual composite action if a second workflow needs the same fallback (dotnet-build-different-locale.yml also runs apt-get and could benefit).

CLAUDE.md compliance

No violations — the rules in the repo's CLAUDE.md are scoped to C# source-generator/reflection/testing concerns and don't apply to this CI-only workflow change. The PR was correctly submitted via a feature branch, per the Branch & PR Workflow section.

'http://archive.ubuntu.com/ubuntu/' '1' \
'http://azure.archive.ubuntu.com/ubuntu/' '2' \
'https://security.ubuntu.com/ubuntu/' '3' |
sudo tee /etc/apt/apt-mirrors.txt > /dev/null

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Mirror list is never consumed

When a subsequent Ubuntu step installs packages, APT continues using the runner's original source because no active source references /etc/apt/apt-mirrors.txt, causing a degraded mirror to stall dependency installation without using the configured fallback.

Knowledge Base Used: Build and CI

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5af7f192db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

'http://archive.ubuntu.com/ubuntu/' '1' \
'http://azure.archive.ubuntu.com/ubuntu/' '2' \
'https://security.ubuntu.com/ubuntu/' '3' |
sudo tee /etc/apt/apt-mirrors.txt > /dev/null

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Point APT sources at the mirror list

After the ubuntu.sources rewrite was removed, this step only creates /etc/apt/apt-mirrors.txt; APT reads such a list only when a configured source uses the documented mirror+file:/etc/apt/apt-mirrors.txt transport. Because the Ubuntu runner's existing source remains an ordinary HTTP URI and nothing else in the workflow references this file, subsequent Playwright dependency installation continues contacting the original source directly and this fallback configuration has no effect.

Useful? React with 👍 / 👎.

@claude

claude Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Code review (update)

Previous issue resolved differently than suggested, but a new blocking issue was introduced.

The trailing-slash sed bug I flagged on the first commit is gone — the latest commit (5af7f19) dropped the sed-based rewrite of ubuntu.sources entirely. But that also removed the only place that told APT to actually use the mirror list.

steps:
- name: Configure Ubuntu mirror fallback
if: matrix.os == 'ubuntu-latest'
run: |
printf '%s\tpriority:%s\n' \
'http://archive.ubuntu.com/ubuntu/' '1' \
'http://azure.archive.ubuntu.com/ubuntu/' '2' \
'https://security.ubuntu.com/ubuntu/' '3' |
sudo tee /etc/apt/apt-mirrors.txt > /dev/null

Current step only writes /etc/apt/apt-mirrors.txt:

- name: Configure Ubuntu mirror fallback
  if: matrix.os == 'ubuntu-latest'
  run: |
    printf '%s\tpriority:%s\n' \
      'http://archive.ubuntu.com/ubuntu/' '1' \
      'http://azure.archive.ubuntu.com/ubuntu/' '2' \
      'https://security.ubuntu.com/ubuntu/' '3' |
      sudo tee /etc/apt/apt-mirrors.txt > /dev/null

Nothing in the workflow points any APT source at mirror+file:/etc/apt/apt-mirrors.txtubuntu.sources still has its original, single, hardcoded URI. So this step writes a file that APT never reads: apt-get update/install will use the base image's default mirror exactly as before, and the intermittent-stall problem this PR sets out to fix is unchanged. (Greptile independently flagged the same root cause on this commit.)

Suggested fix — the mirror list is inert without wiring it into sources.list.d/ubuntu.sources (or sources.list), e.g.:

sudo sed -i -E \
  's#^URIs:.*$#URIs: mirror+file:/etc/apt/apt-mirrors.txt#' \
  /etc/apt/sources.list.d/ubuntu.sources

placed after the tee and before the step ends, so the rewrite happens once, in the same step that owns the mirror-list file.

Design note (still stands, optional): rather than hand-patching an OS-provided sources file that can shift format across base-image updates, consider pointing at Ubuntu's own dynamic mirror service (mirror://mirrors.ubuntu.com/mirrors.txt) or addressing the root cause directly with Acquire::Retries/a longer timeout on the specific step that times out. Either avoids re-deriving the exact URI-rewrite logic each time the base image changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant