Make cactus reproducible - #1985
Open
glennhickey wants to merge 6 commits into
Open
Conversation
The same input and command produced a different alignment on every run, at any thread count. Four causes lived here; a fifth was in sonLib, whose submodule pointer this bumps to the two commits that fix it. Names came off one shared counter inside the OpenMP loops in bar and in reference building, so which object got which Name depended on how the threads interleaved -- and Names are written into the .c2h. A loop now reserves one block of names up front and hands each iteration a private sub-interval chosen from the iteration index, so naming depends on a flower's position in the list rather than on scheduling. The intervals are sized from an upper bound on what an iteration can build; overrunning one falls back to the shared counter and says so, rather than failing. st_random drew from one process-global sequence. Both places cactus breaks ties randomly -- the reference ordering's simulated annealing, and ancestral base calling -- do so inside parallel loops, so the draws were a different slice of that sequence each run. With sonLib's generator now thread-local, each iteration seeds from its own flower, and the root flower is seeded before the serial call that follows the loop. Two comparators could not give a stable order: sortByEvent ranked segments by the address of their Event, and several others left ties to qsort. They now order by name, with names as tie-breaks, so each is a total order. getFirstSegmentMatchingEvent moves with sortByEvent, since it binary-searches the list that sort produces. Thread components were attached to the dead-end component in the order an stSortedSet of pointers yielded them, which decided which threads got attached to the root. They are now sorted by their lowest thread name, and each component's threads by name. This changes cactus's alignments, not just their serialisation: different tie-breaks and a different random sequence give slightly different -- not worse -- output than 3.3.x. Stored baselines will need refreshing. What this establishes is same input + same command + same binary. It is not bit-identity across platforms: several comparators still hand ties to qsort_r, whose behaviour on equal elements is unspecified and changed in glibc 2.37. Verified: cactus_consolidated byte-identical across runs at 1, 2, 4, 8, 16 and 32 threads on two jobs; the progressive and pangenome pipelines identical in every cactus-produced output; alignment accuracy against the evolver truth maf unchanged, inside the old build's own run-to-run spread; peak RSS and wall time unchanged on a 266 Mb job, where hash entries account for 1.3% of peak and this adds 0.43% of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017E9w7KrbFEfqNRtkam4u1F
hal wrote uninitialised heap bytes into every file: Hdf5ExternalArray's buffer comes from new char[], and not every byte of it is set before the buffer is flushed. Two cactus runs producing the same alignment therefore wrote .hal files that differed in bytes nothing ever reads, which meant h5diff and checksums could not be used to compare them. This is the last piece of making cactus's output reproducible; the rest is in the previous commit. Only that fix comes in: the merge commit in between changes no files, so 77075929..7b418b56 is exactly the 12 added lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017E9w7KrbFEfqNRtkam4u1F
The build forces paffy's nested sonLib to whatever SHA cactus's own sonLib is at, so the two have to agree or a fresh clone checks out a commit paffy has never fetched. 0d26e841..67ca7364 changes one file: paffy's sonLib pointer, now cbb2285 like ours. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017E9w7KrbFEfqNRtkam4u1F
downloadMafTools builds taffy from a pinned commit and initialises its submodules recursively, so taffy brings its own sonLib. Move the pin so that sonLib is the same one cactus, hal and paffy now use. 0b6e42fd..2f964e8c is a fast-forward whose whole file diff is taffy's sonLib pointer; the merge commit in between changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017E9w7KrbFEfqNRtkam4u1F
This was referenced Aug 25, 2026
Open
Every place the workflow collected files from a directory did so in os.listdir order, which is whatever the filesystem hands back. The one that matters is the lastz chunking. faffy names its chunks 0.fa, 1.fa, ... and cactus took them in listdir order, so chunk i -- and hence the lastz job named <event>_i -- was an arbitrary piece of the genome rather than the i'th piece of it. faffy already prints each chunk's path to stdout as it completes it, in the order the chunks tile the input, so take the list from there. Sorting the names would not have worked: they sort lexicographically, 0, 1, 10, 11, 2. The remaining sites take sorted(). Most concatenate a genome supplied as a directory of fastas, where the order sets sequence order in the merged file. This is not a run-to-run fix. listdir is stable for a given directory on a given filesystem, so a machine's own results were already reproducible; the exposure is across filesystems. The same 53 chunk names list in three different orders on tmpfs and on two ext4 volumes, because ext4 hashes directory entries against a seed chosen at mkfs time -- so the same job run on another disk, or in a container, chunked the genome differently. Behaviour is otherwise unchanged: a 120-lastz-job chunked blast produces a byte-identical paf before and after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017E9w7KrbFEfqNRtkam4u1F
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same input / command / cactus version should give identical output (modulo hdf5 timestamp) on different runs.
this addresses longstanding issue where things would sort different in cactus_conslidated between runs, leading to variability in the output (even single-thread).
Enough changes here to require full-genome run before merging. I was a bit concerned that the additional hash indexing for stable iteration would increase memory but it doesn't seem like much of an issue in practice so far.