regen-ci-req workflows report drift permanently: lockfile is not reproducible across checkouts

[speediedan]

🐛 Bug

The generated CI lockfile is not byte-reproducible across checkouts, so regen-ci-req-report.yml and regen-ci-req-check.yml should be reporting drift on every run regardless of whether any pin actually changed.

Expected symptoms: a regen-check-pins comment on every PR that touches the relevant paths, and a [auto] regen pinned requirements PR every two weeks (00:00 UTC on the 1st and 15th) whose diff is only the lockfile header and/or a trailing newline.

Found while porting these workflows to finetuning-scheduler, which had the same defects. Both are fixed there; this issue is to carry the fixes back.

Cause 1 — absolute --output-file path baked into the uv header

uv pip compile records its --output-file argument verbatim in the generated header. generate_lockfile() passes an absolute path (requirements/utils/lock_ci_requirements.sh:165, where output_file derives from CI_DIR at :35), so the committed lockfile contains this machine's checkout location:

# requirements/ci/requirements.txt:2
#    uv pip compile pyproject.toml --extra examples --extra lightning --group dev --group test \
#      --group profiling --output-file /home/speediedan/repos/interpretune/requirements/ci/requirements.txt ...

On a GitHub runner that path becomes /home/runner/work/interpretune/interpretune/..., so git diff is non-empty even when every resolved pin is identical.

This is a regression introduced by the pyproject.toml migration — the older requirements.in-based invocation used relative paths, which is why 5ad0f82 was clean at the time.

Fix: generate_lockfile() already does pushd "${REPO_ROOT}" at :152, so the path can simply be made repo-relative before building the command:

local rel_output_file="${output_file#${REPO_ROOT}/}"
# ... then use "${rel_output_file}" in the compile_cmd array instead of "${output_file}"

Keep using the absolute ${output_file} for prune_torch_only_deps and the echo messages.

Cause 2 — trailing-newline mismatch

uv pip compile emits the lockfile with no trailing newline. The end-of-file-fixer hook (.pre-commit-config.yaml:28) adds one on commit. The committed requirements/ci/requirements.txt does end with 0a, so every regeneration produces a one-byte diff against it.

Fix: normalize at the end of generate_lockfile():

if [[ -s "${output_file}" && -n "$(tail -c 1 "${output_file}")" ]]; then
    printf '\n' >> "${output_file}"
fi

Verification

After both fixes, regenerating against the committed state should produce no diff:

./requirements/utils/lock_ci_requirements.sh
git --no-pager diff --exit-code requirements/ci/requirements.txt   # expect exit 0

Worth also running it twice consecutively to confirm idempotency.

Note on diff scope

Separately: regen-ci-req-report.yml:65 and regen-ci-req-check.yml:24 set compare_paths to requirements/ci/requirements.txt only, while the lock script also rewrites requirements-oldest.txt and torch-override.txt. Drift in those goes undetected on the report path — but create-pull-request in the scheduled workflow will still commit them. Widening compare_paths (and/or adding add-paths to the create-pull-request step) would make the two consistent.

Environment

• interpretune @ eb54af1
• Affects: .github/workflows/regen-ci-req-report.yml, .github/workflows/regen-ci-req-check.yml, requirements/utils/lock_ci_requirements.sh

[speediedan]

Fixed in f9025b5 (on streamlined-streamable-dashboard-generation-phase-1; auto-closes on merge to main).

Cause 1 — absolute --output-file: generate_lockfile() now derives rel_output_file="${output_file#"${REPO_ROOT}"/}" and passes that to uv pip compile, keeping the absolute ${output_file} for prune_torch_only_deps and the echo messages, exactly as proposed. Header is now:

#    uv pip compile pyproject.toml --extra examples ... --output-file requirements/ci/requirements.txt ...

Cause 2 — trailing newline: normalized at the end of generate_lockfile() with the tail -c 1 guard.

Diff scope: compare_paths widened from requirements/ci/requirements.txt to requirements/ci, so torch-override.txt drift is no longer invisible on the report path. The regen/pinned-requirements PR step gained matching add-paths (requirements.txt + torch-override.txt; requirements-oldest.txt deliberately omitted since generation of that file is best-effort and it is not currently committed — git add on a missing pathspec would fail the step).

Verification

$ ./requirements/utils/lock_ci_requirements.sh --no-upgrade
$ git --no-pager diff --exit-code requirements/ci   # exit 0

Also confirmed idempotent in both modes: two consecutive runs produce byte-identical requirements.txt and torch-override.txt.

Two related fixes made while here

1. torch-override.txt was a second permanent-drift source. generate_torch_override() derived its pin from latest stable torch on PyPI, while overrides.txt — the file the integrated from-source env actually installs from, and the one carrying the paired triton/torchvision/torchaudio pins — is hand-maintained. The two silently disagreed the moment upstream shipped a new minor (here: 2.13.0 on PyPI vs 2.10.0 pinned), which both regenerated this file on every run and risked a mismatched-triton from-source install. overrides.txt is now the source of truth, with the PyPI lookup retained as a fallback when no torch== pin is maintained. Bumping torch is now a single deliberate edit.

2. --no-upgrade mode. generate_lockfile() always passed --upgrade, so any dependency edit that needed to reach the lockfile also absorbed all accumulated pin drift. --no-upgrade re-resolves only what the pyproject.toml change requires, keeping routine pin currency a separate, separately reviewable change. The scheduled workflow still runs the default upgrading form.

PR proliferation

The scheduled workflow used branch: regen/pinned-requirements-${{ github.run_id }}, so every fortnightly run opened a new PR and left an abandoned branch behind (22 such branches had accumulated, all with closed PRs). It now targets a stable regen/pinned-requirements branch with delete-branch: true — create-pull-request force-pushes and updates the existing open PR instead of opening another. The accumulated branches have been deleted.

Note that until the pins are refreshed (the lockfile had not been regenerated in ~6 weeks), the default regen form still reports legitimate drift — notably datasets 4.7.0 → 5.0.0. That is real drift, not the reproducibility defect, and is left for the scheduled workflow's PR so it gets its own CI validation.