Verify packed PEX cache entries before reuse.

[apetti1920]

Problem

PEXBuilder._build_packedapp admits an existing bootstrap-zip or packed-wheel cache entry on atomic_directory(...).is_finalized() alone, which is just os.path.exists of the target dir, and then safe_copys the cached zip into the PEX being built — outside the atomic block, without inspecting it.

If an entry is ever finalized while incomplete, it is reused verbatim by every later build sharing that PEX_ROOT. A single bad write becomes a permanent, deterministic failure for every subsequent build:

RUN PEX_TOOLS=1 python3 /binary-deps.pex venv --scope=deps --compile /bin/app
pex.dist_metadata.MetadataError: Failed to determine project name and version
  for distribution at .../.deps/<wheel>.whl

or, when it is the bootstrap zip that is short, ModuleNotFoundError: No module named 'pex.version'.

We hit this across a large CI fleet. The tell that it is not a bad dependency is that the missing artifact differs every time — we saw botocore, grpcio, rpds_py, ConfigArgParse, aiobotocore, and in one case pex.version itself. The victim is positional, not packaged. Because the cache entry is stable, retries reproduce the failure byte-for-byte; one of our builds was retried 11 times and failed identically each time.

Note the corrupt artifacts are structurally valid zips with correct central directories, so testzip() and namelist() both pass on them. They are simply missing members.

Fix

cache_zip records the digest of the zip inside the atomic_directory work dir, so it lands through the same atomic rename as the zip it describes, and verifies it before reuse. A mismatched entry is discarded and re-created once; a second failure raises rather than shipping a bad PEX.

An entry written before this existed carries no digest and is treated as unverified, so caches already poisoned heal on first contact. That costs one re-pack per stale entry.

Scope

This addresses reuse of a bad entry, which is what turns one bad write into a permanent failure for everyone sharing the cache. It deliberately does not claim to prevent creation of one: the digest is taken from whatever create_zip wrote, so a short write that still yields a structurally valid zip is only caught by the emptiness guard. The creation-side race appears to live in atomic_directory's EEXIST recovery, where safe_mkdir(work_dir, clean=True) cannot distinguish a dead writer from a live one — its own comment says as much. That felt like a maintainer call rather than something to fold in here, so I have left it alone.

Verification

• New test_cache_zip_rejects_incomplete_entry covers cold build, warm reuse without rebuild, a poisoned-but-structurally-valid entry being discarded and re-created, and an entry with no recorded digest being re-created. test_cache_zip_raises_when_no_zip_produced covers the loud-failure path.
• End to end, reproducing the original failure: build a --layout packed PEX, replace the cached packed wheel with a valid zip whose .dist-info is gone, rebuild. Before this change the build exits 0 and silently ships a corrupt PEX whose venv --scope=deps then fails with the MetadataError above; after it, the entry is discarded, re-created, and the PEX is correct.
• tests/test_pex_builder.py and tests/test_atomic_directory.py pass (46 passed). Two test_build_compression cases error with SystemExit: Pex tests must be run via testing/bin/runtests.py both with and without this change, so they are pre-existing to my environment rather than caused here.
• black and isort clean. I could not get uv run dev-cmd format lint typecheck to bootstrap its venv locally (its pip install -U pip step fails in my environment), so please treat CI as the authority on the full gate.

Happy to adjust the approach — including moving the check inside atomic_directory instead, if you would rather it be generic across all cache users.

[jsirois]

@apetti1920 please provide more context on your Pex use. Is it direct or indirect, say via Pants? The current scheme is very deliberate. Pex uses atomic_directory to ensure bad cache dirs can never be written in the 1st place and relies on this for speed of reads. Absent bugs, a bad cache dir can only be created by another entity deleting cache dir contents or feeding Pex bad inputs for the cache dir. Pex considers the former outside the bounds of what it guards against and purposefully trusts its own cache. The latter is a possibility with, for example, --venv-repository (as opposed to using Pip, which is the default resolver) if the venv is malformed and Pex does not currently check for malformed venvs - it trusts you give it a good one. With that understood by you I'd like to understand if your bad cache directories are from a Pex bug - in which case I'd like to fix that root bug instead - or from an external party.

For example of (likely) external bad, see: pantsbuild/pants#23657

Here Pex is being fed corrupt uv venvs as the input source of pre-installed wheels.

[jsirois]

@apetti1920 your profile links klaviyo via Linked In; so I assume this is Pants. If that's true, are you using the uv resolver and what Pants version is this?

[goodwin-klaviyo]

We're on pants 2.33.0 Checking on UV

[jsirois]

It would be this option in pants.toml: https://www.pantsbuild.org/dev/reference/subsystems/python#resolver:

[python]
resolver = "uv"

[goodwin-klaviyo]

Ok yep we're not using that.

[apetti1920]

@apetti1920 please provide more context on your Pex use. Is it direct or indirect, say via Pants? The current scheme is very deliberate. Pex uses atomic_directory to ensure bad cache dirs can never be written in the 1st place and relies on this for speed of reads. Absent bugs, a bad cache dir can only be created by another entity deleting cache dir contents or feeding Pex bad inputs for the cache dir. Pex considers the former outside the bounds of what it guards against and purposefully trusts its own cache. The latter is a possibility with, for example, --venv-repository (as opposed to using Pip, which is the default resolver) if the venv is malformed and Pex does not currently check for malformed venvs - it trusts you give it a good one. With that understood by you I'd like to understand if your bad cache directories are from a Pex bug - in which case I'd like to fix that root bug instead - or from an external party.

For example of (likely) external bad, see: pantsbuild/pants#23657

Here Pex is being fed corrupt uv venvs as the input source of pre-installed wheels.

@jsirois Via Pants 2.33.0, PEX version: 2.97.3, pinned by us (Pants 2.33's own floor is 2.97.1).

The UV resolver is unset so we're on the default (Resolver.pex, pip via PEX). We have no --venv-repository anywhere. uv appears in our repo only as a lockfile-generation tool (a uv pip compile alias) and as a Pants tool resolve it never produces the pre-installed wheels PEX consumes. So I don't think pantsbuild/pants#23657 applies to us; that one is specifically PEX reading a venv another uv sync is mid-write and we never hand PEX a venv as an input source.

Our environment is alittle unusual in that PEX_ROOT is a Pants append only named cache which is shared by every concurrently executing pex process on the machine and never pruned or invalidated by Pants. These are long-lived CI agents that serve many jobs in parallel and builds are routinely SIGKILLed mid-flight by the merge-queue. So high concurrency against one PEX_ROOT plus abrupt termination. Nothing external should write to or deletes from PEX_ROOT so it shouldnt be the case that another entity deleted cache dir contents.
which is why I suspect its corruption is in PEX's own caches, not in anything we feed it
- packed_wheels entries: a structurally valid zip, correct central directory but no .dist-info, so we see MetadataError at PEX_TOOLS=1 ... venv --scope=deps.
- One production failure was a short .bootstrap missing pex/version.py → ModuleNotFoundError: No module named 'pex.version' . That's PEX's own code, with no resolver input involved at all.
- The affected distribution differed every time rather than tied to any particular input.

We also see PEX's own warning in the logs immediately before these:

After obtaining an exclusive lock on .../packed_wheels/1/<hash>/.defN.atomic_directory.lck,
failed to establish a work directory at .../packed_wheels/1/<hash>/defN.lck.work
due to: [Errno 17] File exists
Continuing to forcibly re-create the work directory at ...

If you'd rather fix the root cause than the defensive check I'm happy to close it. I wrote it to stop a reuse loop with one bad write becoming a permanent failure for every subsequent build on that machine but it does not prevent creation of a bad entry. If the creation path gets fixed most of its value goes away, though it might still be useful for the case of SIGKILLS

[jsirois]

Our environment is alittle unusual in that PEX_ROOT is a Pants append only named cache which is shared by every concurrently executing pex process on the machine and never pruned or invalidated by Pants.

That is actually the typical Pants scenario! On your local machine your named cache directory is never pruned unless you do so manually and a single Pants invocation can run many Pex processes in parallel. As a result, there were many Pex bug fixes in the early days of Pants use of Pex as it exercised parallel Pex invocations.

If the creation path gets fixed most of its value goes away, though it might still be useful for the case of SIGKILLS

The atomic_directory works like so:

1. obtain flock lock on a file associated with the final directory
2. if and only if final directory does not exist, create a work directory next to it
3. populate the work directory and not the final directory
4. if and only if 3 completes without raising, do an atomic rename of the work directory to the final directory
5. unlock the flock lock

So, if the SIGKILL happens at any point before 4, there is never a rename and so partial population of the work dir will never be seen at the final directory rename location.

So all a SIGKILL can do is prevent a rename of a work dir to the final dir, in which case the work dir will be left around for the next try, and you'd see what you report:

After obtaining an exclusive lock on .../packed_wheels/1/<hash>/.defN.atomic_directory.lck,
failed to establish a work directory at .../packed_wheels/1/<hash>/defN.lck.work
due to: [Errno 17] File exists
Continuing to forcibly re-create the work directory at ...

This is as noted in the comment here:

pex/pex/atomic_directory.py

Lines 259 to 285 in e0d238b

	# If there is an error making the work_dir that means that either file-locking guarantees have
	# failed somehow and another process has the lock and has made the work_dir already or else a
	# process holding the lock ended abnormally.
	try:
	os.mkdir(atomic_dir.work_dir)
	except OSError as e:
	ident = "[pid:{pid}, tid:{tid}, cwd:{cwd}]".format(
	pid=os.getpid(), tid=threading.current_thread().ident, cwd=os.getcwd()
	)
	pex_warnings.warn(
	"{ident}: After obtaining an exclusive lock on {lockfile}, failed to establish a work "
	"directory at {workdir} due to: {err}".format(
	ident=ident,
	lockfile=atomic_dir.lockfile,
	workdir=atomic_dir.work_dir,
	err=e,
	),
	)
	if e.errno != errno.EEXIST:
	raise
	pex_warnings.warn(
	"{ident}: Continuing to forcibly re-create the work directory at {workdir}.".format(
	ident=ident,
	workdir=atomic_dir.work_dir,
	)
	)
	safe_mkdir(atomic_dir.work_dir, clean=True)

I.E.: workdir cleanup indicates a forceable kill of a prior attempt to populate the atomic directory (OK), or else a failed flock lock scenario where two processes hold the same lock (not OK!).

Its the latter that I suspect in this scenario because I've reviewed this locking code so many times now over the last many years. The known cases where flock fails are network filesystems like NFS and other exotic storage configuration scenarios. I'm currently focused on your use of docker overlayfs2 for the Pants named-caches directory mount from the host to confirm that style of mount does or does not allow flock to be implemented faithfully.