Serve-from-storage: cold semantic queries without attaching (Phase 5)

.env.example

@@ -77,9 +77,24 @@ RUST_LOG=compass=info
 # past the budget; detached collections re-attach on demand). 0 = unbounded.
 # COMPASS_MAX_ATTACHED=0
 
-# ── Telemetry (anonymous; opt out) ──────────────────────────────────────────
+# ── Serve-from-storage (cold reads) ─────────────────────────────────────────
+# Serve-from-storage: semantic queries on UNATTACHED collections are answered
+# directly from object storage (a few range reads, ~100s of ms) instead of
+# waiting for a full index rebuild. Implies COMPASS_LAZY_ATTACH. Cloud only.
+# COMPASS_COLD_SERVE=true
+# Cold hits on a namespace before a background attach warms it (0 = never).
+# COMPASS_WARM_AFTER=3
+# IVF clusters probed per segment per cold query (recall/latency knob).
+# Default 8 reaches warm-parity recall on clustered embedding spaces; raise
+# it for unstructured vector data (see docs/search-quality.md).
+# COMPASS_COLD_NPROBE=8
+
 # Global in-flight request cap (backpressure). Unset = effectively unlimited.
 # COMPASS_MAX_CONCURRENCY=1024
 
-# COMPASS_TELEMETRY=off
-# DO_NOT_TRACK=1
+# ── Telemetry (anonymous; OPT-IN, off by default) ───────────────────────────
+# Compass never phones home unless you set this. When on, it sends a startup
+# event + daily heartbeat (random instance id, version, OS/arch, collection
+# and vector counts — never document content or queries). DO_NOT_TRACK=1 is
+# honored even when opted in.
+# COMPASS_TELEMETRY=on

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +1,5 @@
 blank_issues_enabled: true
 contact_links:
   - name: Security vulnerability
-    url: mailto:security@runcaptain.com
+    url: mailto:support@runcaptain.com
     about: Report security issues privately via email — do not open a public issue.

.github/workflows/release.yml

@@ -27,7 +27,9 @@ jobs:
 
       - name: Verify tag matches Cargo.toml
         run: |
-          CARGO_VERSION=$(grep "^version" crates/compass/Cargo.toml | head -1 | sed 's/.*"\([^"]*\)".*/\1/')
+          # The version lives in [workspace.package] in the ROOT manifest;
+          # crate manifests say `version.workspace = true`.
+          CARGO_VERSION=$(grep "^version" Cargo.toml | head -1 | sed 's/.*"\([^"]*\)".*/\1/')
           if [ "${{ steps.version.outputs.version }}" != "$CARGO_VERSION" ]; then
             echo "Tag version ${{ steps.version.outputs.version }} does not match Cargo.toml version $CARGO_VERSION"
             exit 1

ARCHITECTURE.md

@@ -43,23 +43,27 @@ crates/compass/src/
     mod.rs           SearchMode enum + re-exports.
     backend.rs       VectorIndex trait shim. UsearchHnswIndex (CPU) lives here.
     vector.rs        USearch HNSW build + search + persistence (CPU primitives).
-    tantivy_fts.rs   Full-text search via Tantivy (BM25).
+    tantivy_fts.rs   Full-text search via Tantivy (BM25) + facet treemaps.
     hybrid.rs        Reciprocal Rank Fusion (RRF, k=60) over FTS + semantic.
+    ivf.rs           IVF clustering built at compaction (cold-read layout).
+    cold.rs          Serve-from-storage query path (range reads, no attach).
+    filter_index.rs  Roaring-treemap metadata filter index (warm pushdown).
+    chunk_store.rs / chunk_cache.rs   redb chunk store + bounded LRU cache.
+  collections/
+    partitions.rs    Tenant-partition routing helpers.
+    cloud.rs         Segment codec (CSEG0003), materialize, bucket config.
+  storage/           Storage trait + local disk + object-store backends + LSM.
+  metrics.rs         /metrics counters. telemetry.rs: opt-in usage pings.
 ```
 
-## Vector backend abstraction
+## Vector backends
 
-All vector backends implement `compass_index_api::VectorIndex`. The default backend is `UsearchHnswIndex` (CPU, mmap-backed, disk-persistent). The opt-in GPU backend is `compass_vector_gpu::CuvsHnswIndex` (CAGRA build on GPU, HNSW search on CPU).
-
-Selection happens at startup in `search::backend::build_backend`, driven by the `COMPASS_BACKEND` environment variable:
-
-| Value | Behavior |
-|-------|----------|
-| `cpu` (default) | USearch on CPU. Always available. |
-| `gpu` | cuVS on GPU. Requires the `gpu` feature and a CUDA-capable device. Falls back to CPU with a warning if either is missing. |
-| `auto` | Probe for GPU, fall back to CPU silently if unavailable. |
-
-The trait is intentionally narrow: `build`, `add`, `search`, `len`, `dims`, `save`, `backend_name`. New backends should fit through this surface or extend it via a follow-up trait, not by branching on a concrete type.
+The engine uses USearch HNSW directly (CPU, mmap-backed, disk-persistent)
+for warm serving, plus an IVF layout inside segments (`search/ivf.rs`) for
+serve-from-storage cold reads. `compass-index-api` (a narrow `VectorIndex`
+trait) and `compass-vector-gpu` (cuVS) exist as standalone crates for a
+future GPU integration but are NOT wired into the engine — there is no
+`COMPASS_BACKEND` knob and no `gpu` feature on the `compass` crate today.
 
 ## Storage layout
 
@@ -80,8 +84,8 @@ data/<collection>/
 
 In cloud mode the object-storage bucket additionally holds, per collection:
 `collection.json` (bucket config), `manifest` (LSM manifest, CAS-committed),
-`wal/{uuid}.frag` (WAL fragments), `segments/{uuid}` (CSEG0002 sectioned
-segments), and `id-alloc` (CAS-leased chunk-id blocks).
+`wal/{uuid}.frag` (WAL fragments), `segments/{uuid}` (CSEG0003 sectioned
+segments: row-addressable metadata + IVF-clustered vectors; v2 readable), and `id-alloc` (CAS-leased chunk-id blocks).
 
 The disk format is the contract. Bumping it requires a migration path documented in CHANGELOG.md.
 
@@ -118,7 +122,8 @@ cuVS CAGRA build on an A10G runs ~12x faster than USearch CPU build at the same
 1. Create a new crate `crates/compass-vector-<name>/`.
 2. Depend on `compass-index-api` (workspace dep) and your backend library.
 3. Implement `VectorIndex` (and `LoadableIndex` if loading from disk makes sense).
-4. Add a `#[cfg(feature = "<name>")]`-gated branch in `search::backend::build_backend`.
+4. Wire it into the engine (there is currently no runtime backend selector —
+   proposing that wiring is part of such a PR; open an issue first).
 5. Document the build prerequisites in `ARCHITECTURE.md` (this file).
 6. Add a smoke binary under `src/bin/` that builds, queries, and prints latency.
 

CHANGELOG.md

@@ -6,7 +6,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## [Unreleased]
 
-### Added — tenant-partitioned collections (Phase 6)
+### Added — serve-from-storage ("true serverless")
+
+- **`COMPASS_COLD_SERVE=true`**: semantic queries on collections (and tenant partitions) that are NOT attached are answered directly from object storage — a manifest read, cached centroid/TOC artifacts, and a handful of range-GETs — instead of triggering a full index rebuild. Compaction now writes IVF-clustered vector sections (`cent:`/`clu:`, k-means, unit-normalized) plus a row-addressable metadata index (`meta2`/`metaidx`) into segments (format CSEG0003; v2 segments remain readable, pre-v0.4 readers fail loudly on v3). Cold reads see the full committed state including the WAL tail and tombstones, so read-your-writes holds by construction; metadata filters apply; FTS on a cold namespace returns a clear error (inverted indexes still need an attach). Repeated cold hits (`COMPASS_WARM_AFTER`, default 3) promote a background attach so hot namespaces migrate to the fast path on their own. RAM per cold namespace is megabytes (centroids + directories), independent of collection size.
+
+### Added — tenant-partitioned collections
 
 - **`config.partition_by`**: create a collection partitioned by a metadata field (e.g. `tenant_id`) and every chunk routes to an internal per-tenant partition — a full engine namespace (own LSM, indexes, attach/evict lifecycle) behind one collection API. Searches and deletes filter by the partition field (exact → one partition; `{"in": [...]}` fans out up to 16, merged by score); chunk ids are collection-unique via the parent's CAS id allocator; partitions auto-create on first ingest (writer role included), attach on demand, are hidden from listings, and cascade-delete with the parent. This moves the scale envelope from per-collection to per-tenant: RAM and refresh cost track the HOT tenant set, so one collection can hold billions of vectors across tenants while serving on bounded memory. Not yet routed on partitioned collections (clear errors): relations, facets, TAMS lookup, vector-space CRUD.
 
@@ -31,9 +35,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 - Pork audit (three independent review passes): −1,200 lines of dead weight removed — the unwired VectorIndex/GPU backend plumbing (`COMPASS_BACKEND` did nothing), a third never-called filter evaluator, never-wired filter-index persistence codecs, the legacy vector writer, the `rayon` dependency, and assorted dead fields/params. `delete_by_filter` now resolves ids through the same roaring filter-index pushdown as search (one filter semantics, not three). The `dead_code` lint is enabled again crate-wide. `collections/mod.rs` shrank from 7,100 to 3,700 lines (test modules extracted to files).
 
+### Changed (behavior)
+
+- **Telemetry is now opt-in** (`COMPASS_TELEMETRY=on`); previously it defaulted on. An engine whose promise is "data never leaves your machine" should not phone home by default.
+
 ### Scope & limitations (honest)
 
-- Warm, not cold: attach cost is proportional to collection size until the sectioned segment format + serve-from-storage indexes land (roadmap Phases 5–6). Cross-node convergence is periodic (refresh interval), not synchronous — use `min_seq` when you need read-your-writes.
+- Cold serving is semantic-only: full-text (and hybrid-with-text) queries on a cold namespace return a clear error until it warms — BM25 still needs local indexes. Cold recall depends on embedding-space structure (see docs/search-quality.md); the default nprobe reaches warm parity on clustered embeddings. Cross-node convergence is periodic (refresh interval), not synchronous — use `min_seq` when you need read-your-writes (cold reads have it by construction).
 
 ## [0.3.0] - 2026-07-03
 

CONTRIBUTING.md

@@ -15,15 +15,24 @@ cargo run --release             # serves on http://localhost:4001
 
 Compass is a Cargo workspace. Useful invocations:
 
+Prerequisites on Linux: `cmake`, `pkg-config`, `libssl-dev` (what CI
+installs). On Windows there is a known linker clash between `esaxx-rs` and
+`cxx` — use `cargo check` locally and run builds/tests in Docker or WSL.
+
 ```bash
-cargo build                              # builds the default member (`compass`)
-cargo build -p compass-index-api         # builds just the trait crate
-cargo build --features gpu               # adds the GPU backend (Linux + CUDA only)
-cargo test --workspace                   # runs all tests in all crates
-cargo clippy --workspace -- -D warnings  # lint check (CI requires zero warnings)
-cargo fmt --all --check                  # format check (CI requires clean diff)
+cargo build                                                # default member (`compass`)
+cargo test --workspace --exclude compass-vector-gpu       # all tests CI runs
+cargo test -p compass --features object-storage           # + S3/GCS backend tests
+cargo clippy --workspace --exclude compass-vector-gpu --all-targets -- -D warnings
+cargo fmt --all --check                                    # CI requires a clean diff
 ```
 
+`compass-vector-gpu` is a standalone experimental crate (cuVS; needs CUDA
+12+, CMake, a long first build) that is NOT wired into the engine yet —
+every CI job excludes it, and so should you unless you're working on it.
+Tests against real object storage skip cleanly unless `COMPASS_TEST_S3_BUCKET`
+is set (CI runs them against MinIO).
+
 See [`ARCHITECTURE.md`](ARCHITECTURE.md) for the module map and where to put new code.
 
 ## What we accept
@@ -35,17 +44,16 @@ See [`ARCHITECTURE.md`](ARCHITECTURE.md) for the module map and where to put new
 - Documentation improvements, especially examples.
 
 **Out of scope (for now):**
-- Storage backends other than the local filesystem.
 - New embedding model integrations (we plug into HuggingFace TEI / vLLM via the `embed_endpoint` config; pull requests adding new in-process embedders need a strong motivation).
-- Cluster / replication features (Compass is single-node by design; horizontal scaling is via sharding behind a load balancer).
+- Consensus/quorum replication. Compass scales out via object storage as the source of truth (stateless writers + serving nodes + serve-from-storage cold reads); PRs should build on that model, not introduce node-to-node coordination.
 
 If you're not sure, open an issue first and ask.
 
 ## Pull request checklist
 
 - [ ] `cargo fmt --all` clean.
 - [ ] `cargo clippy --workspace -- -D warnings` clean.
-- [ ] `cargo test --workspace` green.
+- [ ] `cargo test --workspace --exclude compass-vector-gpu` green.
 - [ ] `CHANGELOG.md` updated under the `[Unreleased]` section.
 - [ ] Public API changes have rustdoc comments.
 - [ ] Behavior changes have a test that would have caught the regression.
@@ -75,7 +83,7 @@ Open an issue with:
 
 ## Reporting security issues
 
-Don't open a public issue. Email `founders@runcaptain.com` with the details. We'll acknowledge within two business days.
+Don't open a public issue. Email `support@runcaptain.com` with the details. We'll acknowledge within two business days.
 
 ## Code of conduct
 

Cargo.toml

@@ -10,7 +10,8 @@ default-members = ["crates/compass"]
 [workspace.package]
 version = "0.3.0"
 edition = "2021"
-authors = ["Captain Technologies <founders@runcaptain.com>"]
+authors = ["Captain Technologies <support@runcaptain.com>"]
+license = "Apache-2.0"
 repository = "https://github.com/runcaptain/compass"
 homepage = "https://runcaptain.com"
 rust-version = "1.88"

README.md

@@ -437,7 +437,18 @@ Supported: `s3://bucket[/prefix]` (AWS S3, MinIO, Cloudflare R2 — set `COMPASS
 
 In this mode the bucket is the **source of truth**: every write lands durably in object storage first (an LSM of immutable WAL fragments + a CAS-committed manifest), and a node that boots with an empty disk discovers its collections from the bucket and rebuilds all local indexes — chunks, hierarchy, and typed relations included. Compaction (automatic past a WAL threshold, or via `POST /compact`) folds fragments into segments and physically reclaims deleted data.
 
-Scope, honestly: reads are served from the locally rebuilt indexes (durable-via-cloud, fast-via-local) — this is not stateless multi-node serving, and cold-start recovery materializes the live set in RAM. See [CHANGELOG](CHANGELOG.md) for details.
+### Serverless topologies
+
+With the bucket as the source of truth, nodes become disposable roles you mix per workload (full reference: [docs/deployment.md](docs/deployment.md)):
+
+- **Serving node** (default): full local indexes, fast reads; converges on other nodes' writes via a background manifest refresher (`COMPASS_REFRESH_INTERVAL`, default 5s). Writes return a `seq`; pass it back as `min_seq` for read-your-writes.
+- **Writer node** (`COMPASS_ROLE=writer`): stateless, append-only, boots in milliseconds, refuses reads. Durable immediately; searchable on serving nodes within the refresh interval.
+- **Cold serving** (`COMPASS_COLD_SERVE=true`): answers *semantic* queries on collections it has never attached, straight from object-storage range reads — first query in ~tens of ms instead of a minutes-long index rebuild. Repeated hits promote a background attach (`COMPASS_WARM_AFTER`). Full-text on a cold collection returns a clear error until it warms. Recall characteristics: [docs/search-quality.md](docs/search-quality.md).
+- **Lazy attach + LRU** (`COMPASS_LAZY_ATTACH`, `COMPASS_MAX_ATTACHED`): boot registers namespaces without loading them; RAM tracks the hot set.
+
+### Multi-tenant partitions
+
+Create a collection with `"config": {"partition_by": "tenant_id"}` and every chunk routes to an internal per-tenant partition — its own indexes and attach/evict lifecycle behind one collection API. Searches and deletes filter by the partition field (exact match, or `{"in": [...]}` to fan out across up to 16 tenants); chunk ids stay collection-unique; partitions auto-create on first ingest and cascade-delete with the parent. Serving RAM tracks the hot-tenant set, not the tenant count — this also works fully offline in local mode.
 
 For local development against MinIO:
 
@@ -569,16 +580,20 @@ PUT    /collections/:name/default-vector-space         Switch default space
 GET    /collections/:name/segments/at                  Temporal segment lookup (TAMS)
 
 GET    /health                                         Health check
-GET    /metrics                                        Prometheus-text metrics
+GET    /metrics                                        Prometheus-text metrics (unauthenticated by design, like /health — exposes collection names + counts; firewall it if that matters)
 ```
 
 ## Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, PR guidelines, and commit conventions.
 
+## Telemetry
+
+Telemetry is **off by default** — Compass never phones home unless you set `COMPASS_TELEMETRY=on`. When opted in, it sends a startup event and a daily heartbeat to PostHog (random instance id, version, OS/arch, collection and vector counts — never document content, queries, or metadata). `DO_NOT_TRACK=1` is honored even when opted in.
+
 ## Security
 
-To report a vulnerability, email **security@runcaptain.com**. See [SECURITY.md](SECURITY.md) for details.
+To report a vulnerability, email **support@runcaptain.com**. See [SECURITY.md](SECURITY.md) for details.
 
 ## License
 

SECURITY.md

@@ -2,7 +2,7 @@
 
 ## Reporting a Vulnerability
 
-We take the security of Compass seriously. If you discover a security vulnerability, please email founders@runcaptain.com with the following information:
+We take the security of Compass seriously. If you discover a security vulnerability, please email support@runcaptain.com with the following information:
 
 1. **Description** of the vulnerability
 2. **Steps to reproduce** (if applicable)
@@ -28,7 +28,7 @@ We will acknowledge your report within **two business days** and work with you t
 
 ### Model Weights
 
-- Compass downloads model weights on first run (e.g., BGE-small via Hugging Face Hub).
+Compass never downloads anything at runtime. Model weights are fetched only if you run `scripts/download-models.sh` (or `huggingface-cli`) yourself.
 - Verify downloaded files match expected checksums when possible.
 - For air-gapped deployments, pre-download and verify model weights before use.
 

crates/compass-index-api/Cargo.toml

@@ -3,6 +3,7 @@ name = "compass-index-api"
 version.workspace = true
 edition.workspace = true
 authors.workspace = true
+license.workspace = true
 repository.workspace = true
 homepage.workspace = true
 rust-version.workspace = true