Lookup hash fix - #397
Open
wrighton wants to merge 2 commits into
Open
Conversation
added 2 commits
July 9, 2026 13:34
When building the in-memory directory tree from an archive (mksquashfs -zip / -tar), add_zipfile()/add_tarfile() call lookup_name() once per path component of every entry. lookup_name() scanned the directory's entry list (dir->list) linearly, so populating a directory of K entries cost O(K^2). For archives with very many files this dominates the whole run and pins a single CPU while the parallel compressor threads sit idle: a 300k-file zip took ~76s at ~100% CPU, with compression under 2% of profile samples and ~50% in lookup_name()/strcmp(). Add a per-directory hash table keyed by entry name: - struct dir_info gains name_hash[]/name_hash_size; struct dir_ent gains name_hash_next. - The table is built lazily once a directory exceeds DIR_HASH_THRESHOLD entries (small directories keep the cheaper linear scan) and doubled when the load factor exceeds one. - add_dir_entry() inserts into the hash; lookup_name() consults it when present, otherwise falls back to the linear scan. - free_dir_entry() unlinks the entry from its bucket (reached via dir_ent->our_dir) so removals (e.g. prune actions) stay consistent, and free_dir() drops the table up front so full-directory teardown skips the per-entry unlink. Lookups become O(1) amortized and directory-tree construction O(K). The 300k -file zip above now builds in ~3.9s at ~1000% CPU (~20x faster) with identical output: unsquashfs extraction byte-compared, inode/file counts unchanged, zero duplicates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eliminate n^2 runtime (in file count) when building from an archive(s)