perf: accelerate imports and harden large-library recovery

.grype.yaml

@@ -341,6 +341,33 @@ ignore:
       version: 2.41-5+dhi3
       type: deb
 
+  # Grype maps these util-linux source-package findings to libuuid1 in the
+  # current DHI. Their affected code is in privileged mount/nsenter commands,
+  # which are absent from the Pullbox runtime image. Debian 13 marks all four
+  # no-dsa/minor and has not published a stable fix. Keep the High gate active;
+  # retain only these exact package matches. Re-review by 2026-10-04 or at the
+  # next DHI refresh, whichever comes first.
+  - vulnerability: CVE-2026-76642
+    package:
+      name: libuuid1
+      version: 2.41.5-0+deb13u1+dhi2
+      type: deb
+  - vulnerability: CVE-2026-78408
+    package:
+      name: libuuid1
+      version: 2.41.5-0+deb13u1+dhi2
+      type: deb
+  - vulnerability: CVE-2026-78409
+    package:
+      name: libuuid1
+      version: 2.41.5-0+deb13u1+dhi2
+      type: deb
+  - vulnerability: CVE-2026-78410
+    package:
+      name: libuuid1
+      version: 2.41.5-0+deb13u1+dhi2
+      type: deb
+
   - vulnerability: CVE-2025-6141
     package:
       name: libncursesw6
@@ -387,6 +414,23 @@ ignore:
       name: zlib1g
       version: 1:1.3.dfsg+really1.3.1-1+dhi2
       type: deb
+
+  # Debian 13 has no fixed zlib package for CVE-2026-85091. The affected path
+  # requires the non-blocking gzwrite API followed by gzprintf/gzvprintf;
+  # Pullbox does not call that native sequence. The duplicated development
+  # package metadata comes from the DHI Python build/runtime dependency tree.
+  # Keep this exact exception only. Re-review by 2026-10-04 or at the next DHI
+  # refresh, whichever comes first.
+  - vulnerability: CVE-2026-85091
+    package:
+      name: zlib1g
+      version: 1:1.3.dfsg+really1.3.1-1+dhi3
+      type: deb
+  - vulnerability: CVE-2026-85091
+    package:
+      name: zlib1g-dev
+      version: 1:1.3.dfsg+really1.3.1-1+dhi3
+      type: deb
   - vulnerability: CVE-2026-34743
     package:
       name: liblzma5

CHANGELOG.md

@@ -30,6 +30,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- Improved Mylar and folder scan throughput with resource-capped archive
+  inspection, batched review inserts, and fewer unnecessary duplicate checks.
+  Corrected current-item scan progress and unknown time estimates, and bounded
+  pending file-processing work without weakening cancellation or archive safety.
 - Reconciled stale Mylar filenames with uniquely verified same-folder comic
   files, including `#1` versus `001` naming changes, without weakening identity
   or archive safety checks. Added a dry-run-first offline repair for saved

docker/.env.example

@@ -145,9 +145,15 @@ PULLBOX_RUNTIME_GID=65532
 # PULLBOX_NAMING_SERIES_FORMAT="{series} ({year})"
 # PULLBOX_NAMING_ISSUE_FORMAT="{series} ({year}) #{issue:03d}"
 
-# Import debug controls. These are for troubleshooting only and should stay
-# disabled for normal production use.
+# Import concurrency. Scan 0 = auto (up to 4 inspectors, capped by visible
+# CPU/container and available-memory limits). Use 1 for slow/shared disks or
+# explicitly test 2-16; an explicit value never exceeds the resource ceiling.
+# Inspection never changes file-safety policy. Step 4 workers remain separately
+# budgeted because conversions can require substantial temporary disk space.
+# PULLBOX_IMPORT_SCAN_WORKER_COUNT=0
 # PULLBOX_IMPORT_FILE_WORKER_COUNT=2
+
+# Import debug controls. These should stay disabled for normal production use.
 # PULLBOX_IMPORT_DEBUG_SLOW_MODE=false
 # PULLBOX_IMPORT_DEBUG_PHASE_DELAY_SECONDS=1.25
 # PULLBOX_IMPORT_DEBUG_ITEM_DELAY_SECONDS=0.4

docs/development/DATABASE_STANDARDS.md

@@ -533,9 +533,10 @@ Pullbox also includes sidecar recovery logic for stale or corrupt `-wal` and
 ### 5.2 Maintenance Coordination
 
 Database-size health thresholds allow for large collections and retained logs:
-the size sub-check is healthy through 1 GiB (1024 MiB), degraded above 1 GiB,
-and unhealthy above 2 GiB (2048 MiB). These are advisory health thresholds, not
-storage limits or an automatic cleanup policy. Disk-space, integrity, query
+the size sub-check reports the current file size as informational data. Database
+health is determined by integrity, representative latency, bloat, and available
+storage rather than a fixed size threshold. The file-size observation does not
+impose a storage limit or an automatic cleanup policy. Disk-space, integrity, query
 latency, and database-bloat checks remain independent; a smaller database does
 not suppress failures in those checks.
 

docs/development/IMPORT_PERFORMANCE.md

@@ -0,0 +1,74 @@
+# Import Performance
+
+## Runtime Contract
+
+Mylar and folder imports share bounded, read-only archive inspection. A worker
+receives paths and immutable safety settings, never an AsyncSession. Completion
+and diagnostic updates flow through one coordinator. Review file rows are
+inserted in batches of at most 500 in the existing transaction; source pages
+retain their durable checkpoints. A singleton issue group cannot contain a
+duplicate or conflict, so it no longer loads files and commits just to check.
+Archive page-name matching runs off the event loop and uses a bounded local
+cache for repeated page-title parsing. It still counts every page toward
+consensus and does not reuse safety decisions between scans.
+
+`PULLBOX_IMPORT_SCAN_WORKER_COUNT=0` selects automatic inspection concurrency,
+up to four workers. CPU affinity, cgroup v2 CPU quotas and parent limits,
+cgroup memory headroom, and OS available memory cap the budget. Common cgroup
+v1 mounts are supported too. Missing resource information falls back
+conservatively. Explicit values 1-16 are also capped. The budget is reevaluated
+between batches; this is not a throughput-learning autotuner or a guarantee
+against other host workloads consuming resources.
+
+The resource ceiling reserves at least one CPU (25% on larger machines) and
+512 MiB of available memory, with an additional 512 MiB allowance per inspector.
+Automatic mode deliberately does not saturate high-core machines: local ZIP
+header parsing did not improve with more than 2-4 workers. Docker Desktop's
+VM resources are the relevant limits, not the Mac's advertised RAM.
+
+Step 4 keeps `PULLBOX_IMPORT_FILE_WORKER_COUNT=2` and its existing temporary-space
+preflight, target-collision serialization, per-worker sessions, and rollback
+journal. It now bounds submitted tasks as well as active workers, rather than
+creating one waiting task per file. Exiting or canceling either worker pool
+drains active work before the job can transition; no orphan filesystem work
+may continue after cancellation is reported complete.
+
+## Progress and Evidence
+
+Unknown inventory totals are indeterminate. Completed series report 100% for
+the current item, independent of overall phase weights. The browser must not
+invent an ETA when the backend reports an unknown estimate. Matching emits
+lightweight completion updates between durable checkpoints without adding a
+database commit per item.
+
+`import_archive_inspection_batch` reports actual workers, effective CPUs,
+available memory, files checked, and elapsed milliseconds. Mylar and folder
+batch events separately report inspection and persistence durations; existing
+Step 2 timing events retain discovery/matching/total durations. These metrics
+are observations, not substitutes for transaction-wait or storage profiling.
+Inspection batch wall time includes policy reads, reconciliation, and progress
+callbacks. Persistence timing covers row materialization/flush, not the later
+checkpoint commit. Do not interpret either value as exclusive disk I/O time.
+
+## Reproducible Benchmarks
+
+```bash
+.venv/bin/python scripts/benchmark_import_scan.py \
+  --series-count 100 --files-per-series 12 --trusted-comicinfo \
+  --archive-pages 32 --inspection-workers 4
+```
+
+The benchmark creates an isolated temporary source tree and database and makes
+no external provider calls. Repeat with workers 1, 2, 4, 8, and 16; use medians
+and retain the effective worker count, not only the requested count. Compare
+identical matched, blocked, conflict, and missing-file outcomes before speed.
+
+`--inspection-delay-ms 10` adds controlled per-archive latency to evaluate I/O
+overlap. It is a simulation, not a NAS measurement. Archive fixtures exercise
+member indexes and bounded ComicInfo reads, not full image decoding or physical
+multi-gigabyte payloads. Run representative CBR/conversion workloads and real
+storage samples before claiming a user's end-to-end speedup.
+
+Do not add persistent cross-scan safety caches, speculative provider concurrency,
+or process pools without evidence and new invalidation/recovery tests. Existing
+compact archive metadata reuse remains intact; safety is freshly evaluated.

docs/development/IMPORT_REVIEW_RECOVERY.md

@@ -1,19 +1,26 @@
 # Recheck A Saved Import Review
 
 Use this maintenance procedure when a review was generated before the Mylar
-sidecar parser and comic-content checks were corrected. It works for Mylar and
-folder imports. This is not a full rescan, a database restore, or an import.
+sidecar parser and comic-content checks were corrected. Normal completed-job
+recovery does not require this command: the **Retry failed** action revalidates
+retryable source changes inside Pullbox before execution. It works for Mylar
+and folder imports. This is not a full rescan, a database restore, or an import.
 
 ## Safety And Scope
 
 - Stop Pullbox and back up its database before running the command. `--offline`
   is the operator's acknowledgement, not an automatic container stop.
-- The job must be idle at Step 3 (`REVIEW`). Do not run this against a scan,
-  import, rollback, or job with a pending control request.
-- By default only automatically rejected series with
-  `trusted_source_identity_conflict` are examined. Repeat `--series-id` to
-  narrow the operation to specific **import-review series IDs**, not ComicVine
-  or library series IDs.
+- The job must be idle at Step 3 (`REVIEW`) or finished (`COMPLETED`). Do not
+  run this against a scan, import, rollback, or job with a pending control
+  request.
+- For a `REVIEW` job, only automatically rejected series with
+  `trusted_source_identity_conflict` are examined by default. Repeat
+  `--series-id` to narrow the operation to specific **import-review series
+  IDs**, not ComicVine or library series IDs.
+- For a `COMPLETED` job, prefer the in-app **Retry failed** action. It performs
+  the same bounded source revalidation automatically and retries only files
+  that remain inside their approved root, pass current safety checks, and do
+  not conflict with the saved source identity.
 - An entire series is left untouched if it has manual overrides, selected
   files, explicit skips, approved exceptions, or other completed file decisions.
   The report counts these as `skipped_series`; discuss them individually.
@@ -48,9 +55,11 @@ docker compose -f pullbox.yml run --rm --no-deps --entrypoint python pullbox \
   -m pullbox.cli recheck-import --job 1 --source-root /mnt/comics --offline
 ```
 
-The JSON result reports `series_prepared`, `files_checked`, `blocked_files`,
-and `skipped_series`. `applied: false` confirms it was only a preview. Review
-the counts before running the same command with `--apply`:
+For a Step 3 review, the JSON result reports `series_prepared`,
+`files_checked`, `blocked_files`, and `skipped_series`. For a completed import,
+it reports `files_prepared`, `files_checked`, `blocked_files`, and
+`skipped_files`. `applied: false` confirms it was only a preview. Review the
+counts before running the same command with `--apply`:
 
 ```bash
 docker compose -f pullbox.yml run --rm --no-deps --entrypoint python pullbox \
@@ -63,13 +72,19 @@ startup recovery resumes from `MATCHING`, preserving the directory inventory
 and unaffected review decisions, and returns to Step 3. It does not select
 files or start Step 4. Genuine source-identity conflicts remain in review.
 
+For a completed import, restart Pullbox, open that import's results, and choose
+**Retry failed**. Only the rechecked failures are prepared; successful files
+and series remain untouched. A file that is still missing, outside an approved
+root, unreadable, or unsafe stays failed with refreshed diagnostics.
+
 ## Replaced Files
 
 Changed or missing scan signatures normally remain blocked. After deliberately
-replacing a defective file, preview a targeted recheck with both `--series-id`
-and `--accept-replaced-files`. This explicitly accepts new scan evidence only
-after containment and archive checks; it is not an archive-safety override.
-Then repeat the reviewed command with `--apply` if appropriate.
+replacing a defective file, preview a targeted recheck with
+`--accept-replaced-files` and, when useful, one or more `--series-id` filters.
+This explicitly accepts new scan evidence only after containment and archive
+checks; it is not an archive-safety override. Then repeat the reviewed command
+with `--apply` if appropriate.
 
 A renamed file at a different path is not automatically discovered by this
 command. Do not use broad filename guessing to repair ownership.

scripts/benchmark_import_scan.py

@@ -9,6 +9,7 @@
 import argparse
 import asyncio
 import json
+import platform
 import tempfile
 import time
 import zipfile
@@ -20,6 +21,7 @@
 
 from pullbox.core import file_safety
 from pullbox.core.archive import ArchiveReader
+from pullbox.core.import_resources import detect_import_resources
 from pullbox.models import import_job as _import_job_models  # noqa: F401
 from pullbox.models import issue as _issue_models  # noqa: F401
 from pullbox.models import library as _library_models  # noqa: F401
@@ -120,6 +122,7 @@ def _build_tree(
     series_count: int,
     files_per_series: int,
     trusted_comicinfo: bool,
+    archive_pages: int = 2,
 ) -> None:
     for series_idx in range(series_count):
         title = f"Series {series_idx:04d}"
@@ -130,11 +133,8 @@ def _build_tree(
         for file_idx in range(1, files_per_series + 1):
             archive_path = folder / f"{title} #{file_idx:03d}.cbz"
             with zipfile.ZipFile(archive_path, "w", zipfile.ZIP_DEFLATED) as archive:
-                archive.writestr(
-                    f"{title} #{file_idx:03d}.jpg",
-                    b"benchmark-page",
-                )
-                archive.writestr(f"{title} #{file_idx:03d} p002.jpg", b"benchmark-page")
+                for page in range(1, archive_pages + 1):
+                    archive.writestr(f"{title} #{file_idx:03d} p{page:04d}.jpg", b"benchmark-page")
                 if trusted_comicinfo:
                     issue_provider_id = (series_provider_id * 1000) + file_idx
                     archive.writestr(
@@ -159,6 +159,9 @@ async def main() -> None:
     parser.add_argument("--series-count", type=int, default=200)
     parser.add_argument("--files-per-series", type=int, default=12)
     parser.add_argument("--trusted-comicinfo", action="store_true")
+    parser.add_argument("--archive-pages", type=int, default=32)
+    parser.add_argument("--inspection-workers", type=int, default=0, choices=range(17))
+    parser.add_argument("--inspection-delay-ms", type=float, default=0)
     args = parser.parse_args()
 
     provider = FakeMetadataProvider()
@@ -169,6 +172,9 @@ async def main() -> None:
         event_bus=cast("Any", SimpleNamespace()),
     )
     benchmark_service = cast("Any", service)
+    benchmark_service._settings = service._settings.model_copy(
+        update={"import_scan_worker_count": args.inspection_workers}
+    )
     benchmark_service._build_scan_metadata_provider = lambda _session: CachedImportMetadataProvider(
         provider
     )
@@ -203,6 +209,8 @@ def counting_read_zip(
 
     def counting_inspect_zip_archive_safety(*call_args: Any, **call_kwargs: Any) -> Any:
         nonlocal archive_metadata_evidence_count, archive_safety_inspection_count
+        if args.inspection_delay_ms > 0:
+            time.sleep(args.inspection_delay_ms / 1000)
         archive_safety_inspection_count += 1
         report = original_inspect_zip_archive_safety(*call_args, **call_kwargs)
         if report is not None and report.comicinfo is not None:
@@ -238,6 +246,7 @@ def counting_archive_entry_issue_hint_from_names(
                 series_count=args.series_count,
                 files_per_series=args.files_per_series,
                 trusted_comicinfo=args.trusted_comicinfo,
+                archive_pages=args.archive_pages,
             )
 
             db_path = Path(tmp) / "benchmark.db"
@@ -273,6 +282,14 @@ async def counted_commit() -> None:
                     "series_count": args.series_count,
                     "files_per_series": args.files_per_series,
                     "trusted_comicinfo": args.trusted_comicinfo,
+                    "archive_pages": args.archive_pages,
+                    "inspection_workers_requested": args.inspection_workers,
+                    "simulated_inspection_delay_ms": args.inspection_delay_ms,
+                    "inspection_workers_effective": detect_import_resources().inspection_workers(
+                        requested=args.inspection_workers
+                    ),
+                    "platform": platform.platform(),
+                    "python": platform.python_version(),
                     "elapsed_ms": elapsed_ms,
                     "archive_read_count": archive_read_count,
                     "archive_member_payload_read_count": archive_member_payload_read_count,

scripts/generate_import_scale_fixture.py

@@ -20,6 +20,13 @@ def build_parser() -> argparse.ArgumentParser:
     parser.add_argument("--series-count", type=int, default=50_000)
     parser.add_argument("--file-count", type=int, default=200_000)
     parser.add_argument("--seed", type=int, default=1300)
+    parser.add_argument("--archive-pages", type=int, default=32)
+    parser.add_argument(
+        "--single-page-every",
+        type=int,
+        default=0,
+        help="Include an explicit one-page safety exception every N files (0 disables)",
+    )
     parser.add_argument(
         "--profile",
         choices=("balanced", "realistic-skew"),
@@ -55,6 +62,8 @@ def main(argv: list[str] | None = None) -> int:
         series_count=args.series_count,
         file_count=args.file_count,
         seed=args.seed,
+        archive_pages=args.archive_pages,
+        single_page_every=args.single_page_every,
         profile=args.profile,
         max_issues_per_series=args.max_issues_per_series,
         layout_profile=args.layout_profile,