fix(deploy): Docker build has failed since 2026-06-15 (pip cannot upgrade itself on Ubuntu 24.04) - #76
Merged
Merged
Conversation
… 2026-06-15
Every Render deploy since 15 June failed ~20s in, at Dockerfile line 13:
ERROR: Cannot uninstall pip 24.0, RECORD file not found.
Hint: The package was installed by debian.
Ubuntu 24.04 ships an externally-managed Python (PEP 668) whose pip comes from
a .deb with no RECORD file, so pip cannot uninstall itself to upgrade.
--break-system-packages does not help — it waives the PEP 668 guard, not the
missing-RECORD uninstall.
Build inside a venv, which owns its own pip, so the upgrade is an ordinary
install. Also collapses the apt layers and adds a build-time construction of
the served predictor, so a missing dependency or bad artifact fails the build
loudly instead of 500ing on the first user request.
render
Bot
temporarily deployed
to
fix/docker-pip-pep668 - LesNet PR #76
July 27, 2026 00:55
Destroyed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every Render deploy has failed since 15 June — before any of the recent model work. Pulled from the actual build log:
Dockerfile line 13, ~20 seconds into the build.
Cause
Ubuntu 24.04 ships an externally-managed Python (PEP 668) whose pip is installed from a
.deband therefore has noRECORDfile. pip cannot uninstall a package it has no manifest for, so upgrading pip-over-pip fails.--break-system-packagesdoesn't rescue it — that flag waives the PEP 668 "externally managed environment" guard, not the missing-RECORDuninstall. That's why the flag was already present and the build still died.Fix
Build inside a venv, which owns its own pip, so the upgrade is an ordinary install and the entire class of problem disappears. Also:
rm -rf /var/lib/apt/lists/*in the same layer (the old version ranapt-get cleanin a later layer, so the cache was still baked into the image)python3.12-venv, which the venv needs on Ubuntuonnxruntimeomission would have escapedNote on my earlier guess
I previously suspected TensorFlow's size was causing a build OOM/timeout. That was wrong — the builds die in ~20s, long before any pip install. Worth having pulled the real log rather than acting on the theory.
Test plan