feat(snapshot): namespace spec, persistent sources and scoped read foundations

[Ivanbeethoven]

Summary

Start with the plain-language Chinese design overview: docs/monorepo-versioning-design.md. It walks through a complete V100-to-V101 dependency update, includes the concrete manifest and core data structures, shows how each Mega directory type advances the monorepo version, and keeps storage/codec details in linked implementation specs.

The proposed file-transfer design targets many small source files: exact directory sizes, bounded on-demand tar+zstd packages, reusable hot-directory packages, and independently verified large-file chunks backed by a trusted Mega chunk map. Read the worked example and protocol spec. These are design documents; transfer endpoints, caches and performance gates are not implemented.

Draft review checkpoint for Mega/ScorpioFS version management. Not ready to merge or deploy as a snapshot capability. This PR is part of the full namespace-publication effort, not a replacement for it.

• Specify native/import/aggregate namespace views, writer publication transactions, bounded immutable routing, leases, migration and recovery gates.
• Resolve import commits/qualified refs once; bound annotated-tag peeling and reject missing, foreign, ambiguous or corrupt roots without latest fallback.
• Add a strict source identity contract and independently computed cross-repository golden vectors.
• Add additive source identity/scope proof migrations, CLI-generated Callisto entities, immutable transaction-compatible storage and migration regression tests.
• Add a source catalog that observes scoped native refs, derives child roots from an attested fixed native tree, resolves import sources and checks fixed-path object membership. Recorded source proofs survive ref/registry cleanup and path reuse by a different repository.
• Add a canonical bounded immutable radix index, transaction-borrowing node persistence, independent vectors and million-binding algorithm gate. Add a forward PostgreSQL UTC compatibility repair and focused PostgreSQL/index CI job.
• Add shared strict binding/view manifest codecs and independent .NET vectors consumed by Mega and ScorpioFS. Record the confirmed immutable-release policy.
• Add an additive publication transaction core: reserve operations before lending the ref-writing transaction, insert immutable views, CAS sequence/view/epoch, commit publication/receipt/outbox with refs, and replay committed results without repeating writes. Production writer adapters and policy enforcement are still not integrated.

The catalog trusts existing ingestion commit/tag metadata; it does not claim raw commit/tag re-verification. The catalog is not an authorization grant or retention lease, and is not exposed by a new HTTP endpoint.

Research design

The research plan separates transport format from candidate contributions, records relevant prior systems and REAPI, and defines correctness invariants, a proposed bounded transfer selector, falsifiable hypotheses and controlled baselines/ablations. Package-count examples now explicitly require known or concurrent demand. This is an experimental plan, not evidence of novelty or measured performance.

Validation

Transfer-design documentation: JSON examples, relative links, example arithmetic and whitespace checks passed. No transfer implementation or performance benchmark is claimed by this documentation update.

Local WSL Ubuntu 24.04 checks:

• cargo test -p ceres --lib snapshot --locked -j 1 -- --include-ignored --nocapture with the explicit disposable PostgreSQL URL — 37 tests passed, including shared namespace manifests, real PostgreSQL and the million-binding fixture.
• cargo test -p jupiter --lib snapshot_storage --locked -j 2 — 6 tests passed.
• cargo test -p jupiter --lib namespace_storage --locked -j 2 — 3 tests passed.
• cargo test -p jupiter --lib publication_storage --locked -j 1 -- --include-ignored --nocapture — 6 tests passed across SQLite and PostgreSQL, using the real import_refs and publication tables. Includes duplicate-key races, expected-old competition, rollback after head CAS, dropped transactions, no-op fencing by an independent connection and reconnect receipt lookup. The storage fixture uses opaque view bytes; it does not prove full native/import composition.
• cargo test -p jupiter-migrate --lib snapshot --locked -j 2 -- --include-ignored --nocapture — 2 tests passed: SQLite up/down/up preserves legacy data; PostgreSQL UTC forward/down/up preserves known UTC values under a non-UTC session.
• cargo +nightly fmt --all --check — passed.
• cargo clippy -p ceres -p jupiter -p jupiter-migrate --lib --locked -j 2 -- -D warnings — passed.
• Test-code Clippy also passed with only the baseline clippy::items_after_test_module lint allowed; the unmodified mono_storage.rs triggers that lint under Rust 1.95. The strict test-code command without that allowance is not green.

PostgreSQL 16.15 validation covers source/node persistence and publication storage concurrency, not the full application publisher. The million-binding fixture measured 7 node reads and 7 writes (2235 bytes each) per single update, 44 reads for page 32 and 7 for one prefix; these are logical index calls, not SQL counts or full service performance. See docs/spec/namespace-index-v1.md and docs/spec/namespace-publication-core.md for methodology and evidence limits. At 706051f8, the focused PostgreSQL/index job, fmt and Clippy passed; other CI was still pending when checked. CI for the latest publication-core commit must be reviewed separately. Real HTTP/FUSE integration, process/host crash recovery, retention and production writer gates remain unvalidated.

Remaining full-delivery gates

• D1 confirmed: full atomic native + import namespace delivery; not a source-only reduction.
• D4 confirmed: default-off snapshot APIs; explicit source/scope authorization and retention configuration required before enablement.
• D2 confirmed: explicitly marked release directories are immutable after first publication; ordinary development bindings may evolve.
• Confirm D3: workspace switch semantics.
• Implement and benchmark the documented small-package and large-file chunk transport, including authorization, limits and interrupted-download recovery.
• Persist native scope proofs at every relevant creation/merge path, with compatible rollout and backfill.
• Atomically publish native refs, import refs, registry bindings and namespace head; cover every visible writer and expected-old ref CAS.
• Add bounded immutable binding index and real PostgreSQL/SQLite node persistence.
• Add transactional operation receipts/outbox and publication-head CAS storage core.
• Integrate full atomic native/import composition, production writer read sets, release enforcement, pins and historical replay.
• Implement Git object reachability retention/leases and source/path authorization on every fixed read.
• Expose source-aware APIs and integrate immutable ScorpioFS lower/cache generations.
• Validate two-version mounts, controlled updates and recovery; execute the spec's remaining acceptance gates before ready-for-review.

Related: gitmono-dev/scorpiofs#55 and gitmono-dev/scorpiofs#42. These issues remain open; foundations do not close them.

Companion ScorpioFS draft: gitmono-dev/scorpiofs#56.

Existing original worktrees were preserved. No deployment, destructive data backfill or feature enablement is included.