Skip to content

fix(deploy): Docker build has failed since 2026-06-15 (pip cannot upgrade itself on Ubuntu 24.04) - #76

Merged
Thomasbehan merged 1 commit into
mainfrom
fix/docker-pip-pep668
Jul 27, 2026
Merged

fix(deploy): Docker build has failed since 2026-06-15 (pip cannot upgrade itself on Ubuntu 24.04)#76
Thomasbehan merged 1 commit into
mainfrom
fix/docker-pip-pep668

Conversation

@Thomasbehan

Copy link
Copy Markdown
Owner

Every Render deploy has failed since 15 June — before any of the recent model work. Pulled from the actual build log:

#12 [ 6/11] RUN python3.12 -m pip install --break-system-packages --upgrade pip
    Attempting uninstall: pip
      Found existing installation: pip 24.0
ERROR: Cannot uninstall pip 24.0, RECORD file not found.
       Hint: The package was installed by debian.
error: failed to solve: ... exit code: 1

Dockerfile line 13, ~20 seconds into the build.

Cause

Ubuntu 24.04 ships an externally-managed Python (PEP 668) whose pip is installed from a .deb and therefore has no RECORD file. pip cannot uninstall a package it has no manifest for, so upgrading pip-over-pip fails.

--break-system-packages doesn't rescue it — that flag waives the PEP 668 "externally managed environment" guard, not the missing-RECORD uninstall. That's why the flag was already present and the build still died.

Fix

Build inside a venv, which owns its own pip, so the upgrade is an ordinary install and the entire class of problem disappears. Also:

  • apt collapsed into one layer with rm -rf /var/lib/apt/lists/* in the same layer (the old version ran apt-get clean in a later layer, so the cache was still baked into the image)
  • adds python3.12-venv, which the venv needs on Ubuntu
  • constructs the served predictor at build time — a missing dependency or bad artifact now fails the build loudly rather than 500ing on a user's first request, which is exactly how the onnxruntime omission would have escaped

Note on my earlier guess

I previously suspected TensorFlow's size was causing a build OOM/timeout. That was wrong — the builds die in ~20s, long before any pip install. Worth having pulled the real log rather than acting on the theory.

Test plan

  • Root cause confirmed from the Render build log, not inferred
  • Verified by the resulting Render deploy (no local Docker available) — will confirm after merge

… 2026-06-15

Every Render deploy since 15 June failed ~20s in, at Dockerfile line 13:

    ERROR: Cannot uninstall pip 24.0, RECORD file not found.
           Hint: The package was installed by debian.

Ubuntu 24.04 ships an externally-managed Python (PEP 668) whose pip comes from
a .deb with no RECORD file, so pip cannot uninstall itself to upgrade.
--break-system-packages does not help — it waives the PEP 668 guard, not the
missing-RECORD uninstall.

Build inside a venv, which owns its own pip, so the upgrade is an ordinary
install. Also collapses the apt layers and adds a build-time construction of
the served predictor, so a missing dependency or bad artifact fails the build
loudly instead of 500ing on the first user request.
@render
render Bot temporarily deployed to fix/docker-pip-pep668 - LesNet PR #76 July 27, 2026 00:55 Destroyed
@Thomasbehan
Thomasbehan merged commit d11734b into main Jul 27, 2026
6 checks passed
@Thomasbehan
Thomasbehan deleted the fix/docker-pip-pep668 branch July 27, 2026 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant