From d05faebf7589da9eea46651fb3b64ebada00ad29 Mon Sep 17 00:00:00 2001 From: Luxian Date: Sun, 6 Sep 2026 08:16:26 +0800 Subject: [PATCH 1/8] docs: specify immutable monorepo views and paper roadmap --- docs/spec/monorepo-versioning.md | 347 +++++++++++++++++++++++++++++++ docs/spec/system-paper-spec.md | 160 ++++++++++++++ 2 files changed, 507 insertions(+) create mode 100644 docs/spec/monorepo-versioning.md create mode 100644 docs/spec/system-paper-spec.md diff --git a/docs/spec/monorepo-versioning.md b/docs/spec/monorepo-versioning.md new file mode 100644 index 0000000..6b790ee --- /dev/null +++ b/docs/spec/monorepo-versioning.md @@ -0,0 +1,347 @@ +# Mega 命名空间版本与 Dicfuse 不可变视图 Spec + +状态：Draft v0.2，2026-09-06。三项产品决策待用户确认（§12）；文中的 MUST 是拟议协议要求，不代表现有实现。命名空间协议由 [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) 跟踪，本文细化 [#42](https://github.com/gitmono-dev/scorpiofs/issues/42)，约束 #43、#44、#49、#50、#51、#53。总路线见 [system-paper-spec.md](system-paper-spec.md)。Mega 侧配套实施草案位于该仓库的 `docs/spec/namespace-snapshot-spec.md`，细化 G01–G06 与 MG01–MG17；目前两仓 spec 均未提交。 + +## 1. 问题与事实基线 + +只读权限不提供版本隔离。Dicfuse 必须固定“某个路径由谁提供、提供哪个不可变对象”，否则一次 build 会混读不同时刻的文件。整个挂载树也不一定属于同一个 Git revision。 + +本次静态核对：ScorpioFS `6a2e7f2ddb7d913b497167f76ff6539bfbb983c2`；Mega `c4c79bc195541a13ac1505b94728c81a8ff3d603`。这是上游代码基线，尚未验证用户正在运行的 Mega 服务端版本、数据库或配置。 + +| 已验证的行为 | 源码依据 | 对设计的影响 | +| --- | --- | --- | +| import 根目录由配置指定，默认 `/third-party`；其余默认根目录有 `project/doc/release/model/toolchains` | [Mega config](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/config/config.toml#L50) | 分类使用服务端登记的归属，不能只看目录名字 | +| REST 根据 import 根和已登记仓库路径切换 Mono / Import handler | [api_handler](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/mono/src/api/mod.rs#L82) | 路由表也是快照的一部分 | +| import 仓库按路径组件寻找最长匹配；新注册检查父子仓库嵌套冲突 | [git_db_storage](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/jupiter/src/storage/git_db_storage.rs#L351) | 不把任意目录都当独立 repo，不靠字符串 starts_with 匹配 | +| 原生目录的 refs 表以 `(path, ref_name)` 标识引用；更新链逐层重建父 tree，并可产生不同 scope 的 commit | [mega_refs](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/jupiter/callisto/src/mega_refs.rs#L9)、[tree update](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/ceres/src/application/api_service/mono/logic/tree.rs#L88) | 子目录 commit 与全库 commit 的根路径不同，必须携带 scope | +| Mono `get_root_tree` 可接受 40 位 commit OID 或 tag；空 refs 取 `/` 当前主引用 | [Mono service](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/ceres/src/application/api_service/mono/service.rs#L120) | 不是所有服务端读接口都缺历史能力；缺的是统一、完整的固定版本读路径 | +| Import `get_root_tree` 忽略传入 refs，使用默认 ref | [Import service](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/ceres/src/application/api_service/import_api_service.rs#L117) | 添加 `?refs=A` 不能保证 import 返回 A | +| 二进制 tree API 只接收 path 和可选 oid；内部先查当前 path，oid 仅校验是否匹配 | [TreeQuery](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/ceres/src/model/git.rs#L55)、[tree_ops](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/ceres/src/application/api_service/tree_ops.rs#L40) | 当前 oid 参数不是历史 tree 寻址接口 | +| import receive-pack 通过创建路径/占位 `.gitkeep` 树连接总目录，同时更新 import refs；不是把完整 import 内容树直接接入原生树 | [import attach](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/ceres/src/application/code_edit/post_receive/import.rs#L78) | 主仓 root OID 不足以重建完整 import 内容及其历史挂接 | +| 原生与 import 元数据共享数据库事务入口 | [Storage](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/jupiter/src/storage/mod.rs#L327) | 组合发布可加入同一事务；对象存储可读性仍须单独保证 | +| import 网页编辑独立更新默认 ref；receive-pack 准备阶段可先登记新仓库 | [web edit](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/ceres/src/application/api_service/import_api_service.rs#L374)、[registration](https://github.com/gitmono-dev/mega/blob/c4c79bc195541a13ac1505b94728c81a8ff3d603/ceres/src/transport/protocol/mod.rs#L168) | publisher 不仅覆盖 push；登记表不等于成功发布的 binding | +| Dicfuse metadata fetch 只传 path，实例按 store/path 复用，内容按 inode 持久化 | [store](https://github.com/gitmono-dev/scorpiofs/blob/6a2e7f2ddb7d913b497167f76ff6539bfbb983c2/src/dicfuse/store.rs#L493)、[manager](https://github.com/gitmono-dev/scorpiofs/blob/6a2e7f2ddb7d913b497167f76ff6539bfbb983c2/src/dicfuse/manager.rs#L47)、[content store](https://github.com/gitmono-dev/scorpiofs/blob/6a2e7f2ddb7d913b497167f76ff6539bfbb983c2/src/dicfuse/content_store.rs#L28) | 需要同时改变读接口、命名空间和缓存身份 | + +结论：不能把方案简化成“每个目录有一个版本号”，也不能把一个主仓 commit 直接当作所有 import 仓库的 commit。 + +## 2. 目录分类与版本来源 + +| 类型 | 示例（示意路径） | 应固定的身份 | 更新规则 | +| --- | --- | --- | --- | +| 原生普通目录 | `/project/a/src`、`/doc`、`/toolchains` | 选定原生 root tree 下的 subtree | 随新的原生视图发布；普通目录无需独立 revision | +| 可独立 clone 的原生 scope | `/project/a` | 原生 source + `scope_path` + commit + scope tree | 独立 checkout 使用 scope commit；全库视图默认从同一个全库 root 派生 | +| import 独立仓库 | `/third-party/vendor/lib` | 稳定 repo ID + commit + tree + hash algorithm | branch/tag 只在 resolve 时解析一次；后续按对象读 | +| import 中间/聚合目录 | `/third-party`、`/third-party/rust/crates` | 原生目录树 + 固定挂接目录索引 | 子仓新增、删除、换路径、版本变化必须生成新 namespace view | +| 版本号命名的 import | `/third-party/rust/crates/…/1.3.0` | 同上，并记录发布策略 | 版本字符串不等于内容哈希；是否允许替换由 D2 决定 | +| CL / 候选变更 | `/project/a` 的 `refs/cl/X` | 基础 view + CL scope + base/head OID + delta digest | CL 移动生成新候选视图；未合并 CL 不推进默认全库视图 | + +`release/model/toolchains` 是名字，不天然代表第三种对象存储。只有服务端显式注册为新 source 类型时才启用新 adapter。Git submodule、LFS 指针和外部 artifact 也不能凭目录名称自动展开：M1 返回原始指针/声明 unsupported；将来展开时，materialization policy 及对象 digest 必须进入身份。 + +## 3. 拟议身份模型 + +使用三个层次，避免把 commit、内容身份和工作区生命周期混成同一个 `generation`： + +```text +SourceSnapshot 一个版本域里的 commit/tree，以及它对应的路径 scope +NamespaceView 原生 snapshot + 不可变挂接索引 + 显式覆写规则 +WorkspaceGeneration 工作区使用的 view、delta 序号与切换事务 +``` + +### 3.1 SourceSnapshot + +```rust +// 协议草图；ObjectId/SourceId 均为有验证器的类型，不接受任意字符串。 +struct SourceSnapshot { + source_id: SourceId, // instance UUID + backend kind + stable repo ID + scope_path: RepoPath, // commit.tree 对应的命名空间位置 + commit_oid: ObjectId, + root_tree_oid: ObjectId, // commit 的 tree；scope 映射经服务端验证 + object_format: ObjectFormat, +} +``` + +`/project/a` 的 scope commit，其 tree 根已经是 `a/` 的内容；读取 `src/lib.rs` 时不能再向它附加 `project/a/`。全库 root commit 的同一路径则要从 `/` 遍历。服务端 MUST 返回并验证 scope，不能仅凭一个存在的 commit OID 猜测。 + +Mega 当前 `mega_commit` 没有 scope 字段，因此需要持久化 `(source_id, scope_path, commit_oid) → root_tree_oid + proof`；同一 commit 可有多个有效 scope，不设唯一反向映射。子 scope ref 被清理不能丢失历史证明；存量 commit 没有证明时返回 `SOURCE_SCOPE_UNVERIFIED`，不能默认它属于 `/`。clone 派生 scope commit/证明本身不代表全库可见树变化。 + +revision selector 使用带类型联合：`published_view`、`source_commit`、`source_ref`。branch/tag 解析结果带完整 ref 名和最终 commit；裸 tree 请求必须显式声明 `tree` 类型，不能伪装成 commit。当前 Mega 的 SHA-1 限制作为 capability 返回；协议保留 SHA-256 类型但不得提前宣称支持。 + +### 3.2 NamespaceView + +```rust +struct NamespaceView { + schema_version: u32, + instance_id: InstanceId, + native: SourceSnapshot, + bindings_root: Digest, // 持久化目录 trie/Merkle index 的根 + overrides_root: Option, // 显式候选 scope 的替换记录 + materialization_policy: Digest, +} +struct Binding { + mount_path: RepoPath, + source: SourceSnapshot, + source_subpath: RelativePath, + policy: BindingPolicy, // 固定目标 + 发布约束 +} +``` + +`view_id = sha256(canonical_descriptor_bytes)`。序列化规范固定 schema、字段顺序/编码、路径字节顺序、可选项表达和 object type；不纳入时间戳、租约、可移动 ref 标签。实现先给 golden vectors，再选择一种确定性编码；JSON 示例只作可读表示，不能直接 hash 任意 JSON 文本。 + +为保证两次不同 commit 但相同 tree 能共享数据，另设 `projection_key`：由实际可见内容图、路由、scope、object format、materialization policy 和有效访问域决定；commit provenance 独立保存在 view。两者不可互相替代。inode 分配、stat 的合成属性策略也须稳定，或进入 projection key。 + +发布记录另存 `{publication_seq, view_id, parent_view_id, reason, created_at}`。seq 只比较同一实例的发布顺序；view_id 用于重放，workspace generation 用于条件更新，delta_seq 用于修改增量查询。 + +### 3.3 大规模目录索引 + +不能为每次 mount 枚举全量 import refs。挂接索引是不可变、可分页、按前缀查询的持久化树；改变一个 import 只重写该索引的祖先节点。挂载拉取 view 根及所需路由分支，未访问对象保持惰性。 + +首次导入既有目录表可做一次受控 O(R) 建索引（R 为登记仓库数）。此后更新成本按变动绑定数和索引深度增长。测试必须包含 large-R/small-working-set，不能只用 64 个小 repo 宣称可扩展。 + +节点 fanout/大小也必须有界；若根节点内嵌百万 child，即使“只写祖先节点”仍会 O(R) 重写。Mega 草案采用受限分支的持久化 byte-radix trie，支持 prefix seek 与分页，并计量节点/字节读写放大。canonical descriptor/index 编码及 golden vectors 在 G01 冻结后才能宣布 v1 capability。 + +## 4. 读取规则与边界 + +1. mount 先 resolve 一个 view 并获取保留租约，成功后才公开可访问路径。 +2. lookup 在该 view 的固定索引内定位路径归属，按路径组件匹配；不得再次查询实时 `git_repo` 表来改变归属。 +3. 普通目录从选定 native tree 下行；遇到 import 边界切换到 Binding 中固定的 source tree。边界视图替换占位内容，不能把占位 `.gitkeep` 与真实 repo 根随意合并。 +4. parent readdir 合并原生名字和 view 索引的边界名字；同名冲突必须是声明过的挂接替换，否则 resolve 拒绝。历史 view 不出现之后新登记的仓库。 +5. directory handle 绑定 view/projection，readdir cookie 仅对该 handle 有效。tree/blob 读取只使用 source + immutable OID；临时错误不得回退到 latest。 +6. 完整性验证区分 Git blob OID 和裸内容 digest。Git 对象哈希包含对象类型与长度头；CAS 不通过“内容看起来像 Git 头”自动剥离真实文件前缀。 +7. Git tree 不携带 blob 长度。size 从同 OID 的 metadata/size index 或对象头取得；缺少 length 不能把非空文件报为零。权限 mode、symlink target、可执行位进入 oracle。 +8. 路径以无歧义的组件编码表示，v1 使用 UTF-8 组件，不做大小写折叠/Unicode 归一化，拒绝 NUL、`.`、`..`、重复分隔和越界路径；非 UTF-8 名字在 capability 中明确拒绝。未来 byte-path 支持单独版本化。symlink 解析不能借路径路由突破 workspace 根。 +9. 在不可变 view 内 TTL 只影响缓存回收/重试，不触发读新 branch。`mutable/latest` 兼容模式使用独立 namespace，且不宣称快照一致。 + +同一个 `base_path` 的 view A/B MUST 允许同时存在。一个 workspace 的更新不得修改其他 workspace 正在使用的 `Arc`。 + +## 5. 服务端发布与历史保留 + +### 5.1 推荐：Mega 发布全库组合视图（D1 待确认） + +一次发布包含原生主仓 root 和固定 import 绑定，必要时还包含候选 scope 的显式覆写。发布流程： + +```text +保存并校验新 objects + → 基于 expected published view 构建新 native tree / bindings index + → 验证对象可读并建立保留根 + → 数据库事务 CAS 更新 published_view 指针及关联 ref/映射 + → 提交后发送更新事件 +``` + +published_view 指针是全库读取的线性化点。序列号可以有空洞，不能重复/倒退。对象准备失败不发布；事务失败留下的孤立对象稍后 GC。事件采用事务 outbox 或等价机制，丢失通知只影响及时性；客户端仍通过期望 seq 拉取恢复。 + +原生 merge、import 同步、登记/删除/重命名、默认分支变化、工具链修改等每条可见写路径都必须参与同一 publication contract。当前部分操作已有事务或 CAS 不等于全库协议已完成。跨数据库/异步对象存储场景不能用“顺序读取几个 HEAD”冒充原子快照；先完成对象可达性，再发布唯一指针。 + +当前 Mega 原生/import 元数据可用同一个应用 DB transaction；应在该事务内更新 refs、bindings、view/head、scope proof、operation receipt 与 outbox。除 published head CAS 外，每个被修改 ref 都必须验证 expected-old；现有 `update_ref_in_txn` 没有该参数，不能仅依赖原生 root CAS。网页编辑固定一个 base 生成 tree 与 parent，避免先读旧 tree 后读新默认 ref。响应丢失通过 operation 查询确定结果，事件失败不反向宣称内容事务回滚。 + +新 import 的提前登记是 staged，不是默认视图中的公开 binding；失败 unpack/无有效默认 commit 不发布空目录。当前 transport 已拒绝删除默认分支，需保留并在事务内对所有入口重验。当前分支 report-status 位于 finalize 之后，publisher 必须保持在此成功边界内；tag 提前持久化的行为单独回归，不擅自扩展为整个 push 的原子承诺。 + +默认 import branch 的哪个 tip 对外发布应有固定策略。向非默认 branch 推送不会自动替换默认全库内容。source ref 变更可与正式发布分离，但必须显示 `staged/unpublished`；只有发布成功的 view 可被 `latest` 返回。写事务如果宣称同时更新主仓与依赖，则二者必须在同一个新 view 内生效。 + +这里“不替换内容”不等于现有实现绝不产生新 native commit：当前 import attach 可能额外创建 root commit；如果保留该行为，provenance 变化仍需发布新 view，即使 projection 可复用。未来消除这种额外 root 写入作为独立优化。selected ref 根据整个成功批次之后的状态选择，不取第一条 push command。 + +### 5.2 存量兼容与能力分级 + +| 能力 | 可以承诺什么 | 不能承诺什么 | +| --- | --- | --- | +| `source-snapshot.v1` | 单一 source/scope 的历史 tree/blob 一致 | 一个全库 root 自动覆盖 imports | +| `view-lock.v1` | 持久化的显式组合可重放 | 各 ref 恰好来自同一全局时刻 | +| `namespace-snapshot.v1` | 经服务端发布的完整 namespace view | 未登记在 view 中的外部资源也被固定 | + +迁移初期先获得当前数据库的一致读快照，冻结 native root、repo 路由和已选择的 import commits，保存为初始 view。对旧历史主仓 commit，如果没有当时的挂接索引，MUST 返回 `HISTORICAL_BINDINGS_UNAVAILABLE`；不能拿今天的 import heads 补齐后声称是历史全库版本。 + +首版回填推荐受控维护窗口：暂停相关元数据 writer，验证并建立初始索引、原子 cutover，再放行全部已接入 publisher 的 writer。普通 DB begin/分页扫描不自动等于一致读快照。旧二进制/直接改表脚本未被隔离前不宣布 namespace capability；更大规模在线回填另做 changelog/catch-up 方案。回退保留已分配 view 的读服务与租约，不能将这些 view_id 重新解释为 legacy latest。 + +若只能先做客户端 lock，返回 `consistency=explicit_composition`，记录各 source 的解析结果。它是可复现组合，不是服务端原子发布；仍需固定 routing，禁止混用实时 registry。此降级作为独立模式而非静默 fallback。 + +### 5.3 保留/GC + +view manifest 永久存在不等于历史内容永久可读。Mega 必须按 source snapshot 保留 tree/blob 的可达闭包（以及显式开启时的 LFS/artifact 对象），允许租约续期，并给出历史保留期。pin 不要求预下载全仓库。 + +GC 与 lease 创建/续期须协调，防止有效续租对象被并发删除；按 pack 存储时还要保护包含有效对象的 pack 及 delta 解码依赖，或安全 repack。已存在的 artifact GC 不是 Git snapshot GC 的证明。lease 只保留对象，不绕过当前鉴权；撤权可失败，不能改读另一个版本。 + +ScorpioFS 持有 active workspace、open handle、refresh prepare 的引用；服务端依据 view lease/历史保留策略阻止对应对象回收。租约失效后禁止承诺未缓存数据可用；离线完整重建只在可达闭包已导出并验证时支持。 + +本地 CAS 的活动 read lease 防止 eviction 竞争；显式 offline pin 保护已缓存对象。已固定远端快照但尚未下载的对象不计为本地驻留空间；预算不足明确返回 `CACHE_PIN_BUDGET_EXCEEDED`，不无界增长。 + +## 6. 拟议 API（尚未实现） + +### 6.1 Mega：发现、固定与读取 + +| API | 请求关键字段 | 响应/约束 | +| --- | --- | --- | +| `GET /api/v1/snapshots/capabilities` | 服务端发现 | instance/schema/算法/路径编码、source/namespace readiness、retention 限制 | +| `POST /api/v1/snapshots/resolve` | selector、scope、expected publication（可选） | view/source descriptor、resolved commit/tree、consistency、lease；symbolic ref 只解析一次 | +| `GET /api/v1/snapshots/{view_id}` | 固定 view_id | 同一 ID 内容永远一致 | +| `GET /api/v1/snapshots/{view_id}/bindings` | prefix、cursor | 固定索引分页；cursor 绑定 view/prefix，不能混页 | +| `GET /api/v1/snapshots/{view_id}/tree` | path、cursor | 依据固定路由的 entries，含 source、tree OID；区分不存在与空目录 | +| `GET /api/v1/sources/{source_id}/trees/{oid}` | 同 descriptor 的 source 和 tree OID | 原始 tree 或有验证关系的结构化 entries；不解析当前 refs | +| `GET /api/v1/sources/{source_id}/blobs/{oid}` | object format、必要的授权上下文 | 精确字节与 size；不因其他 repo 命中缓存而跳过授权 | +| `POST /api/v1/snapshots/{view_id}/leases` | client token、期限 | 创建/续期 pin 的不透明租约 | +| `DELETE /api/v1/snapshot-leases/{lease_id}` | lease_id | 幂等释放；不删除仍被其他引用保留的 view | +| `GET /api/v1/snapshot-operations/{operation_id}` | actor-domain 内的 operation_id | 查询已提交结果，恢复未知响应；不允许跨用户枚举 | + +source ID 是稳定、不透明、可 URL 编码的标识，不是供客户端传任意 URL 的位置。resolve 内部需要校验 source/commit/scope 对应关系。按对象读取也必须约束到有权访问的 source/view。 + +可用固定 tree walk 生成的 object ticket 或等价可验证上下文证明 scope/root 到 OID 的可达性，初始 root 由 resolver 验证；不能以客户端给出任意裸 OID 或全局 CAS 命中代替授权。证明机制应惰性展开，不要求 mount 时全树遍历；ticket 不授予超越当前 ACL 的访问权。 + +错误统一 `{code,message,retryable,details}`。400 参数/路径错误；403 未授权（隐藏存在性政策可统一 404）；404 不存在；409 `EXPECTED_VIEW_MISMATCH` / `REF_MOVED` / `SOURCE_SCOPE_MISMATCH` / `SOURCE_SCOPE_UNVERIFIED` / `BINDING_CONFLICT` / `IMMUTABLE_BINDING` / `DEFAULT_REF_REQUIRED`；410 `SNAPSHOT_EXPIRED`；422 `HISTORICAL_BINDINGS_UNAVAILABLE`；501 capability 不支持；503 `OBJECT_UNAVAILABLE` / `PUBLICATION_NOT_READY`。旧接口的“空数组/空 body”不能用于表示这些失败。 + +### 6.2 ScorpioFS：版本化工作区 + +新接口拟放在 `/antares/v2` 下，保留当前 `/antares` v1 的字段/语义。 + +创建 mount 请求（值为示意占位符）： + +```json +{ + "job_id": "build-a-001", + "mount_path": "/", + "base": {"kind": "published_view", "view_id": "sha256:VIEW_A"}, + "mode": "immutable", + "state_owner": "scorpiofs" +} +``` + +成功响应包含 `workspace_id, view_id, projection_key, generation, delta_seq, resolved_sources, mountpoint, readiness, state_owner`。大规模 sources 返回分页引用，不在每个 status 重复完整列表。同 job_id + 相同规范化请求幂等；不同请求返回 409。 + +`readiness` 区分 `identity_resolved`、`mount_accessible`、`prefetch_complete`。lazy mount 对外可访问只要求身份和根已验证；完全预取不应成为固定版本挂载的必要条件。延迟对比必须使用同一种 readiness。 + +更新计划包含 `expected_generation, expected_view, target_view, expected_delta_seq`；返回 changes summary、dirty/busy/owner checks、expiry。plan 本身不是锁，执行时必须在屏障内重验。 + +真正切换请求另有 `operation_id`（幂等键）、owner token、期望状态及 plan digest。相同 operation_id 不同 payload 返回 409。长期操作返回 202 和 operation URL；GET operation 可恢复未知响应结果。API MUST 同时返回 committed state 和是否已恢复 workload，不混淆两者。 + +## 7. 更新语义 + +### 7.1 日常版本前进 + +```text +Mega 发布 V1 → 新工作区 W1 绑定 V1 +Mega 发布 V2 → 新工作区 W2 绑定 V2；W1 继续使用 V1 +W1 显式请求 update → 检查/暂停 → 新 generation 使用 V2 +``` + +原生子目录发生变化时只需新 native tree 和祖先节点；未变 subtree/OID 继续复用。import branch 更新改变其 Binding 和索引根；没有改动的 source snapshot 与 blobs 继续复用。新增/移除 import 也是 view 变更；旧 view 保留旧名字及来源。 + +全库 view 中一般从同一个 native root 派生所有原生子目录。若指定 `/project/a@A` 和 `/project/b@B` 混搭，必须是显式 `overrides_root` 描述的候选组合，不能伪称一个原生 root commit。路径覆写不能穿过另一个 source 边界，冲突在 resolve 时拒绝。 + +下面用符号 OID 展示更新结果，不表示实际部署内容： + +| 已发布视图 | native root | `/project/a` | `/third-party/vendor/lib` | 使用者 | +| --- | --- | --- | --- | --- | +| V100 | M0 | M0 派生的 tree A0 | repo R 的 commit I0 / tree T0 | 旧 build 固定在此 | +| V101（只改 a） | M1 | M1 派生的 tree A1 | 仍是 I0 / T0 | 新 build 可选择 V101 | +| V102（只更新 import） | M1 | 仍是 A1 | I1 / T1 | 显式 update 的任务选择 V102 | + +V100 即使第一次访问 import 发生在 V102 发布后，也必须下载 I0/T0 的对象。若新增 `/third-party/newlib`，只有包含新 binding 的视图能列出它。A1/I0 和 A1/I1 都是有效组合，但只有发布记录决定哪一份是该时刻的默认视图；不靠访问先后顺序决定。 + +### 7.2 第一阶段 refresh：暂停后切换（D3 待确认） + +1. 验证 target view、取得租约并准备一个独立 target lower。读旧 workspace 仍可继续。 +2. 由 state owner 取得工作区 mutation lock 和 workload 停止/暂停屏障；检查普通 FD、directory FD、mmap、cwd/root 引用、活跃写入与 FUSE 请求。无法证明受控访问时返回 `WORKSPACE_BUSY`，采用创建新 mount + 重新启动任务的方式。 +3. 在屏障内重验 expected generation/view/delta_seq。普通 checkout 要求 clean；dirty 返回 `DIRTY_WORKSPACE`。`.libra` 控制数据排除，但不能因此忽略源码和生成文件的修改。 +4. 持久化 PREPARED 日志，包含旧/新 view、operation_id、目标 mount 配方和保留的 upper 路径。fsync 文件及父目录后继续。 +5. 在外部不可访问期间完成 mount 替换/重建并运行树身份 probe。缓存失效必须包括负 dentry、attr、page cache；不能仅替换一个用户态指针就宣布完成。 +6. 验证成功后 fsync COMMITTED 记录；它是恢复选择新 generation 的唯一依据。对调用者恢复访问前，控制面与实际挂载 identity 必须一致。 +7. 解除屏障并返回新状态，异步释放旧 lower/租约。失败保留 upper 和可解释的 recovery 状态。 + +只有 PREPARED：重启时按旧 view 重建；有有效 COMMITTED：按新 view 重建。介于 mount 变更与 durable commit 的窗口对 workload 必须不可访问。日志损坏无法判断时进入 `FailedRecoverable`，不猜测 latest。进程 crash 和整机断电的测试分开，普通未 fsync 应用写入不额外承诺持久性。 + +若选择 kernel OverlayFS，不得改变仍在使用的 lowerdir 内容；不得把旧 upper/workdir 同时挂到两个 overlay。默认 clean refresh 建立新 upper/workdir（控制元数据另管），旧目录保留到事务结束。dirty commit 清理路径需 §7.3 和 delta manifest 配合，不能盲目复用 origin/index/redirect 元数据。 + +透明 live-refresh 延后为单独 capability：旧 handle/mmap 绑定旧 generation，新 path resolution 绑定新 generation，还需处理 cwd、共享 writable upper 与 kernel caches。不能因实验的普通 open/read 通过就宣称所有 POSIX 访问语义成立。 + +### 7.3 Commit、CL 与 delta 清理 + +Libra 负责 refs/HEAD/index、commit、merge/conflict。Mega 负责验证 source/scope 和发布 namespace view。ScorpioFS 负责按已解析 identity 提供文件树。 + +commit 成功不等于全库 main 已更新：未合并 CL 或尚未发布的 source commit 使用候选 descriptor；其父 view、scope、base/head OID 都固定。CL overlay 的 read-only 层必须记录相同 base view，不能在 CL 编辑中悄悄更换基础依赖。 + +清理已提交 delta 必须以 `{path, entry_seq, committed_oid, mode}` 条件匹配。先 prepare candidate base，再验证 commit 对象存在/可提供，执行受控切换，持久化 commit receipt，最后只清理仍匹配的条目；期间新增修改全部保留。删除的 tombstone 同样带版本。Libra 的 HEAD/index 与 ScorpioFS 的 mount 不是一个数据库事务：使用唯一 state owner 的持久化 saga/恢复记录，直到双方可验证一致才允许新 workload。 + +跨不同 source 的 rename/hardlink v1 返回 `EXDEV`；跨原生普通目录的语义由同一 native scope 的 POSIX 实现负责。原生 scope 与 import 之间的跨域原子 commit 不在第一阶段范围。 + +## 8. Delta、存储和请求调度的联动 + +每个 delta entry 至少保存 `path, kind, base_view, source_id, base_oid, source_path, entry_seq`。`content_oid` 在 write 完成/稳定检查点后才可确定，不能在首次 copy-up 时冒充最终内容。rename-directory/opaque whiteout/hardlink 必须有专项语义测试。 + +更新 manifest 与 upper 不是天然原子：先写 mutation intent，再做文件操作/必要 fsync，最后提交 manifest；崩溃后只 reconcile 未完成 intent 涉及的路径，严重不一致才全扫描。通过 FUSE 之外修改 upper 的场景不得宣称 event index 完整；检测到不可信状态就阻止 clean 判定并修复。 + +存储分开： + +```text +objects//// verified bytes +views/ immutable descriptor +bindings/ immutable routing index +metadata// immutable nodes +workspaces//state + journal + delta mutable control state +workspaces//upper private files +``` + +对象 CAS 原子发布使用临时文件→流式校验→fsync→同文件系统 rename→目录 fsync，配额包含 in-flight 临时字节。原有 inode-keyed db 与新 schema 隔离重建，不原地解释为新对象缓存。 + +FetchKey 至少包含访问域、source/对象存储身份、类型、算法、OID。physical CAS 在已授权域内可对相同已验证内容去重；scheduler 不能只以裸 OID 绕过 source 的访问检查。共享 future/文件句柄返回结果，单个 waiter 取消不取消其他 waiter；refresh 取消无消费者预取，保留仍被旧工作区需求读取的请求。 + +## 9. 架构 ADR：先验证，再选择共享挂载方案 + +当前 Antares 是 `libfuse_fs::unionfs::OverlayFs` 用户态实现，不能把它的 hook 能力直接套到 Linux kernel OverlayFS 上。 + +| 方案 | 预期优势 | 必须先证明的门槛 | +| --- | --- | --- | +| A：共享不可变 Dicfuse mount + kernel OverlayFS | 有机会复用 lower 内核缓存 | FUSE lower 兼容性、权限、upper 写事件/manifest 完整性、受控换代与恢复 | +| B：单 FUSE supermount 路由 workspace | 容易集中维护 generation 与 observer | inode/handle 路由隔离、无 sibling 可见性、connection 瓶颈；单 connection 不自动证明 page-cache 共享 | +| C：现有 per-workspace FUSE | 最短路径验证版本正确性；作为 baseline | 用户态共享 lower/CAS，单独计量每 workspace session 成本 | + +优先在 C 上完成 snapshot 读链路和双版本 oracle，同时限时研究 A/B。A 的实现须提供可靠 delta 观测方案；仅用可能丢事件的 watcher 不能满足 #49。主线 ADR 暂不选生产胜出方案。 + +Linux 文档规定挂载中的 overlay 不允许底层内容变化，部分特性还限制离线更换 lower。来源：[OverlayFS — Changes to underlying filesystems](https://docs.kernel.org/filesystems/overlayfs.html#changes-to-underlying-filesystems)（2026-09-06 查阅）。因此 A 的 refresh 设计必须采用独立 generation 和受控重新挂载。 + +## 10. 验证矩阵与交付门槛 + +以下为拟新增测试/实验 ID，当前仓库尚未具备相应命令。 + +| ID | 场景 | 必须满足的断言 | +| --- | --- | --- | +| V01 | 原生 A/B，branch 前进 | A mount 始终返回 A；新 mount 返回 B | +| V02 | 同一路径 scope commit / 全库 commit | 正确处理 scope；不重复附加前缀；不接受错误 scope | +| V03 | import 显式 A，默认 branch 前进到 B | tree、blob、size、readdir 都来自 A；不忽略 refs | +| V04 | 老总目录 view + 新 import 登记 | 老 view 名字集合不变；新 view 可见新 repo | +| V05 | import 原地更新/删除/改默认分支 | 旧 view 仍可寻址旧对象；新 view 依发布策略变化 | +| V06 | 并发原生 merge 与 import 发布 | 只读到一个有效发布组合；无半旧半新路由 | +| V07 | `/rust` vs `/rust_v1`、嵌套 repo | 组件匹配正确；拒绝冲突；固定分页不串 view | +| V08 | 双 commit 同 tree、双 scope 同 blob | provenance 不丢；允许 metadata/CAS 复用；inode 不串内容 | +| V09 | tree API 旧 oid、空目录、缺失目录 | 明确区分 immutable hit / empty / not found / unsupported | +| V10 | refresh 遇 dirty、FD、mmap、cwd | v1 明确拒绝或使用受控新 workspace；未静默切换 | +| V11 | 每个 refresh phase 注入 crash | 恢复旧或新 generation；COMMITTED 决定恢复选择 | +| V12 | commit 后同 path 再编辑 | selective cleanup 保留新 entry_seq 和新内容 | +| V13 | CL 更新与 main merge | 未合并 CL 不改变默认 view；候选 base/head 可重放 | +| V14 | CAS 损坏、GC/read 竞争、lease 过期 | 无错字节；有效 pin 不回收；过期明确错误 | +| V15 | million bindings，更新一个 import | mount 不枚举全 registry；写放大受索引深度约束 | +| V16 | 模式/链接/目录替换、opaque whiteout | oracle 比较 path/type/mode/content；与基线语义一致 | +| V17 | 旧 commit 无历史 catalog | 拒绝伪历史 snapshot；显式 lock 模式准确标注 | +| V18 | 32/64 workspace 与两种 source 版本 | 修改隔离、共享请求可计量、取消与回收不影响邻居 | + +oracle 用公开 fixture 的 Git object tree + 独立实现的 binding composition 物化期望树；不能调用被测 Dicfuse resolver 自证。每轮冻结 source commits、namespace descriptor、seed、配置、内核、架构模式和实际失败率。 + +## 11. 实施顺序和跨仓依赖 + +| 工作包 | 仓库/现有 issue | 交付物 | 前置与退出条件 | +| --- | --- | --- | --- | +| V0 | ScorpioFS / #55，#42 前置协议追踪 | 目录分类、capabilities、descriptor、公开 fixture | D1 确认；V02/V07/V17 的预期结果冻结 | +| V1 | Mega G01/G02（仅 spec，业务代码未改） | Import 按 commit 读；scope proof、scope-aware tree/blob、共享 fixture | source-snapshot.v1；V01–V03/V09、MG01–MG04/MG13 通过 | +| V2 | Mega G03–G05（拟议） | bounded bindings index、全部 writer 的 publication transaction、leases/迁移 | D1/D2 确认；V04–V07/V14/V15、MG05–MG17 相关门槛通过 | +| V3 | ScorpioFS / #42、#43 | ViewResolver、immutable Dicfuse、CAS/schema 隔离 | 可先接 fake backend；V01–V09 通过后接真实 Mega | +| V4 | ScorpioFS / #44、#49 | refresh journal、upper manifest、受控切换 | D3 确认；V10–V12/V16 通过 | +| V5 | ScorpioFS / #50、#51 | FetchCoordinator，A/B/C bakeoff | correctness 不退化，V18 与资源计量通过 | +| V6 | Libra/Orion 协议对接 / #53、#54 | state-owner saga、candidate view、E2E artifact | V13/V18；源码变更需相应跨仓实施任务 | + +M0 观测/benchmark 与 V0/V1 并行。不要等待全部观测实现才能开始正确性 fixture；也不要在 V0–V3 之前锁定共享 lower 的生产架构。 + +## 12. 待确认决策与明确假设 + +| 决策 | 推荐方案（尚未确认） | 另一选择及成本 | +| --- | --- | --- | +| D1 全库版本边界 | Mega 发布 native root + 固定 import bindings 的组合 view，允许规划服务端协议改造 | 首期只做 source snapshot；全库原子一致性延期，lock 只能标注显式组合 | +| D2 版本号路径策略 | 显式标记为发布版本的目录首次发布后不可变，新内容使用新版本路径；不靠数字目录名推断；普通 import branch 仍可演进 | 可原地替换，但每次发布产生新绑定，旧对象必须按历史策略保留 | +| D3 工作区更新体验 | 运行任务固定旧 view；新任务用新 view；现有 mount 暂停/检查后显式切换 | 要求透明运行中切换，必须扩展 handle/mmap/cwd/upper generation 协议 | + +其他暂定项：M1 支持现有 Mega SHA-1 并显式拒绝不支持的格式；不自动 hydrate LFS/submodule；GC 保留期、部署内核和 import 实际拓扑需在实施前填入环境 manifest。本文不把 A/B 架构选择当成已获确认的决策。 diff --git a/docs/spec/system-paper-spec.md b/docs/spec/system-paper-spec.md new file mode 100644 index 0000000..8d683a7 --- /dev/null +++ b/docs/spec/system-paper-spec.md @@ -0,0 +1,160 @@ +# ScorpioFS 系统论文实施 Spec + +状态：Draft v0.2，2026-09-06。代码基线 `6a2e7f2ddb7d913b497167f76ff6539bfbb983c2`。用户附件是需求草案；本 spec 根据源码核对修正接口假设。文中的模块名、API 和测试 ID 是拟议交付物。 + +总 Epic：[#39](https://github.com/gitmono-dev/scorpiofs/issues/39)。已有路线 issue 全部保持原编号；#45–#48 是已关闭的重复项，不能作为功能已完成的证据。本次没有实施 Rust 代码或运行论文实验。 + +## 1. 目标、范围与核心约束 + +目标是在多个 build/agent workspace 中共享不可变文件树、内容与远端请求，并使每个 workspace 的写入保持独立。真实收益通过实验判断，不能把架构复杂度当作论文贡献。 + +版本协议是第一前置：Mega 原生目录、scope clone、import 仓库和聚合目录需要共同组成可重放视图。详见 [monorepo-versioning.md](monorepo-versioning.md)，该文件规定 source/view/generation、服务端接口、更新及错误语义。 + +Mega 已在独立 WSL checkout 核对，基线 `c4c79bc195541a13ac1505b94728c81a8ff3d603`；配套服务端草案为 Mega `docs/spec/namespace-snapshot-spec.md`。新增 G01–G06 实施包、MG01–MG17 验收组，覆盖 scope proof、同库 publication txn、push/网页编辑写入口、初始回填与保留。两仓目前只有 spec 变更，不宣称协议或 MG 测试已经实现。 + +约束： + +- 所有 immutable read 可追溯到固定 view/source/OID；旧 workspace 不随 latest 改变。 +- 用户态 lower 共享、物理 CAS 去重、remote single-flight、kernel cache/session 共享分别计量，不相互代指。 +- view_id、projection_key、workspace generation、delta_seq、publication_seq 分工明确。 +- Libra 负责 VCS，Mega 负责 namespace 发布，ScorpioFS 负责投影和 upper；嵌入模式的 desired state 只有一个 owner。 +- 不支持的能力明确拒绝。兼容 mutable mode、CL build 和 interactive worktree 使用独立语义标识。 +- 普通进程 crash、daemon crash、机器断电与数据损坏是不同实验；持久化保证以规定的 fsync 边界为准。 + +## 2. Issue 与工作包映射 + +| 草案 | GitHub | 本 spec 交付边界 | +| --- | --- | --- | +| 0 Epic | [#39](https://github.com/gitmono-dev/scorpiofs/issues/39) | 研究假设、依赖、实验 claim 映射 | +| 1 Telemetry | [#40](https://github.com/gitmono-dev/scorpiofs/issues/40) | 版本化 events、可控采样、计量开销 | +| 2 Harness | [#41](https://github.com/gitmono-dev/scorpiofs/issues/41) | fixtures、同等 workload、cache state、raw results | +| 新增前置 Namespace contract | [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) | 原生/import/聚合目录的固定路由与发布契约 | +| 3 Snapshot | [#42](https://github.com/gitmono-dev/scorpiofs/issues/42) | SourceSnapshot + NamespaceView，不再仅单一 base_revision | +| 4 CAS | [#43](https://github.com/gitmono-dev/scorpiofs/issues/43) | verified objects、metadata namespace、GC/pins | +| 5 Refresh | [#44](https://github.com/gitmono-dev/scorpiofs/issues/44) | 受控 generation 切换、journal、恢复 | +| 6 Delta | [#49](https://github.com/gitmono-dev/scorpiofs/issues/49) | provenance、增量变化、条件清理、repair | +| 7 Fetch | [#50](https://github.com/gitmono-dev/scorpiofs/issues/50) | single-flight、优先级、公平、有界资源 | +| 8 Shared lower | [#51](https://github.com/gitmono-dev/scorpiofs/issues/51) | A/B/C 验证后选型；依赖版本/观测正确性 | +| 9 Prefetch | [#52](https://github.com/gitmono-dev/scorpiofs/issues/52) | 可选、严格预算、邻居干扰门槛 | +| 10 Integration | [#53](https://github.com/gitmono-dev/scorpiofs/issues/53) | 唯一 state owner、Libra/Orion E2E、独立 oracle | +| 11 Evaluation | [#54](https://github.com/gitmono-dev/scorpiofs/issues/54) | 全矩阵、ablation、可公开 artifact | + +关键路径：namespace contract → immutable source/view → CAS → refresh/delta → shared architecture → E2E。M0 与 namespace contract 并行；FetchCoordinator 可在 CAS 后与 refresh/delta 并行；prefetch 不阻塞主线。 + +## 3. M0：测量基础 + +### 3.1 Telemetry contract + +拟新增 `src/telemetry/{event,context,sink,metrics}.rs`，只通过轻量 context/sink 埋点；禁用时不创建写盘线程。事件字段：`schema_version,event_id,trace_id,span_id,workspace_id,view_id,projection_key,generation,delta_seq,operation,result,data_source,queue_us,latency_us,bytes,object_type,path_id`。 + +耗时用 monotonic clock，墙上时间只用于关联。并发事件不承诺全局顺序，使用 request/span 关系重建因果。一个共享 fetch 只有一次 physical fetch 计数，所有 workspace 的 logical requests 分开；共享 bytes 不向每个 waiter重复计为物理网络字节。 + +HMAC path key 在一次实验内稳定，跨实验默认轮换；raw path 显式开启。输出有界队列、采样、rotation、丢弃计数；观测不能阻塞 demand read。指标标签限制基数，路径/OID 不作为无界 histogram 标签。schema 区分 `unknown` 与 0，并声明单位。 + +验收 T01：trace 关联 create→ready→read/write→refresh→delete；T02：同 blob 64 logical reads 的 physical 计数正确；T03：disabled 无后台写盘；T04：固定 replay 的开启/关闭 overhead 报告，目标中位吞吐损失 ≤5%；T05：旧 schema fixture 兼容。 + +### 3.2 Harness contract + +拟新增 `bench/`，入口及布局在首个实现 PR 固定；建议 CLI 为 `python3 bench/run.py --manifest bench/manifests/smoke.json`。该命令目前未实现。 + +manifest 必填：schema、fixture seed/commit、view descriptor、workload、baseline、workspace count、unique views/sources、cache state、network profile、repeats、timeout、hardware/kernel/config。产物 `manifest.json, events.jsonl, samples.csv, summary.json`；每次运行单独目录，失败不覆盖原始日志。 + +baseline adapter 负责 prepare/cache-precondition/run/collect/cleanup；校验所有 baseline 的可见树 digest 相同，partial/sparse baseline 提供 workload 必需路径和依赖。Git worktree baseline 的共享 object store 占用只计一次，per-workspace 文件占用单列。 + +| 状态 | 前提检查 | +| --- | --- | +| C0 | 新 daemon + 空专用 disk cache，记录 kernel cache 处理策略 | +| C1 | daemon 已热但目标 snapshot 未访问，CAS 状态明确记录 | +| C2 | 目标 shared snapshot 已热，新 workspace 未访问 | +| C3 | 目标 workspace 重复执行 | +| C4 | content cache 保留，kernel/FUSE 状态通过隔离环境重置并验证 | + +不能只通过重启 daemon 宣称 kernel cold。可能影响整机的 drop_caches 只在专用实验 VM 中执行。precondition 无法验证则标记 invalid，不与通过验证的 cache 状态混合。 + +M0 smoke：两版本公开 fixture，native/import/placeholder/conflict，1/4 workspace，metadata/read/write；full 扩至 1/4/16/64。至少 10 次独立重复用于主要性能点；不要拿 10 次样本的一个最大值当可靠 P99，尾延迟另收集足够操作样本并说明嵌套相关性与 CI 方法。WSL 可用于功能 smoke，论文主结果的内核/宿主资源噪声需在专用 Linux 环境控制。 + +## 4. M1：版本、存储、切换与 delta + +规范细节以 [版本 spec §3–8](monorepo-versioning.md#3-拟议身份模型) 为准，模块边界如下： + +| 拟新增模块 | 输入/输出 | 禁止承担的职责 | +| --- | --- | --- | +| `src/snapshot/identity.rs` | typed source/view IDs、canonical vectors | 解析用户凭据、隐式 latest | +| `src/snapshot/backend.rs` | capabilities、resolve、immutable tree/blob | 假定所有 scope 同一 commit | +| `src/snapshot/resolver.rs` | view + path → fixed source/object | 在读路径查实时 ref/registry | +| `src/cache/` | verified CAS、metadata、pins、配额 | 把 inode 当内容 ID | +| `src/workspace/transaction.rs` | generation/operation journal | 双写 Libra desired state | +| `src/workspace/delta.rs` | mutation intent、entry seq、changes cursor | 以路径集合 hash 代表内容版本 | + +第一 PR 先引入 backend trait 和 fake backend，允许旧 Dicfuse adapter 并存。错误类型和 descriptor 确定后再迁移实际 fetch，不一次性改动所有 daemon 逻辑。新版数据写到独立 schema 根，旧 state 显式标注 legacy。 + +refresh 先支持清洁、受控、可暂停 workspace；dirty commit 通过 owner 协议和 selective cleanup 单独交付。协议 v2 暴露 `generation` 与 `delta_seq`；旧 `/changes` 的 path fingerprint 字段保持旧语义。 + +M1 退出：版本 spec V01–V14、V16–V17 correctness 测试通过；真实 Mega adapter 至少完成 native A/B + import A/B + old namespace routing；未支持 namespace 发布时只能交付 source-snapshot capability。 + +## 5. M2：共享 fetch、架构对照与可选预取 + +### 5.1 FetchCoordinator + +队列按 workspace 做有界 admission，并给 demand metadata/read 高于 speculative prefetch 的优先级。每类队列采用带权公平调度；后台任务也有受控最低进展/aging，避免严格优先级的永久饥饿。跨域请求不错误合并。 + +single-flight state：Absent→Queued→InFlight→Published/Failed；每个 waiter 独立 deadline/cancel。leader 的执行寿命与某个首发 workspace 解耦。对象大小未知时按块预留带宽与 in-flight bytes，超过单对象预算用磁盘流或明确拒绝，不能突破内存上限。 + +相同对象的 prefetch 后来遇 demand 时升级队列优先级；已开始的 HTTP 不宣称可以无成本抢占，只通过 demand 预留并发位保证进展。全局和 per-workspace 限额同时生效。永久错误的短负缓存限定 source/OID/权限语境；鉴权失败不变为全局 missing。 + +验收 F01：64 同对象冷读一次后端请求；F02：取消一个 waiter 不影响其余；F03：retry 和 publish 幂等；F04：队列和内存硬上限；F05：noisy neighbor 下有进展及 demand 延迟报告。用可确定性 fake backend 控制完成顺序与失败点。 + +### 5.2 A/B/C 架构 bakeoff + +三个方案定义见 [版本 spec §9](monorepo-versioning.md#9-架构-adr先验证再选择共享挂载方案)。先在 C 上提供正确性基线；A/B 的首轮原型时间箱暂定各 3–5 人日（工程估算，不是排期承诺）。到期产出 ADR：可以进入生产、继续调查或淘汰；没有功能/隔离/恢复证据不扩展为生产重构。 + +分别记录 unique SourceSnapshot 数、unique NamespaceView/projection 数、FUSE connections、mounts、upper、RSS、CPU、kernel slab/page cache。组合 view 的数量可能大于 source snapshot 数；不能把 O(U) 结论里的 U 随实验更换定义。 + +功能门槛：shared lower 不变、私有 upper 隔离、unmount 一个不伤其他、refresh 不修改活动 lower、delta index 完整且 crash 可修复。性能保留门槛沿用 #51：64 workspace 的 P95 provision ≥5×改善，或预注册的 CPU/RSS/session 成本指标 ≥3×改善，或真实并发 build throughput ≥25% 改善；全部报告，不能事后换指标挑赢家。 + +### 5.3 预取 + +统一 planner 输入 context/hints/budget，输出有序对象候选和可解释 score；budget 同时限制网络 bytes、objects、time、in-flight。已命中 CAS 不计远端预算。准确率以实际后续 demand 事件判断，定义 unused/late/evicted-before-use，取消后只保留仍有需求的 fetch。 + +先 explicit paths 和 trace hotset，再有证据时做 build hints。#52 为可选：两个真实 workload 的 P95 first-successful-action ≥20%改善、waste ≤ demand bytes 30%、邻居 P95 退化 <5% 才入主论文。静态模式、no-prefetch 和预算变化保留作为对照。 + +## 6. M3：集成、实验与 artifact + +Libra attach 传入 scope commit 或 published view；Mega 提供验证后的 immutable descriptor；ScorpioFS resolve/mount；readiness 后 Libra 才放行进程。Orion 每个 task 记录 view_id 和 CL base/head，重试使用原身份，不能重新读取 latest。 + +唯一 owner 保存 desired state 和跨步骤 receipt；ScorpioFS 保存 runtime handles 及必要执行 journal。嵌入 External 模式不得另写独立 desired-state 文件；事务 journal 的持有者/持久化 callback 在集成 PR 明确。HEAD/index 与 mount 可能短暂处于恢复中，但恢复屏障完成前不允许任务访问。 + +公开 correctness oracle 单独物化期望树，验证 path/type/mode/content（额外元数据的合成规则固定）；每次 attach/refresh/commit 都比对。LFS/submodule 未支持时使用显式 fixture 或 fail，不把错误忽略计为成功。 + +实验矩阵：B0 full checkout、B1 warm bare+worktree、B2 partial+sparse、B3 materialized+kernel OverlayFS、B4 固定现有 ScorpioFS、B5 snapshot+CAS、B6 +fetch、B7 选定 shared architecture、B8 可选 prefetch。组合命名空间 baseline 必须物化同一 bindings，不能把 B4 的浮动 latest 当作正确性对照。 + +workload 至少覆盖 Buck2、Bazel、Cargo、CMake/Ninja、百万文件 sparse 合成、scripted agent、parallel patch。每个真实仓库先 smoke，再固定公开 commit/target/toolchain。M0 用小 fixture 启动，完整矩阵在机制通过正确性后展开。 + +每个 claim 对应 experiment ID、原始文件、聚合脚本和图表；预注册 outlier/失败处理，报告绝对值、相对值、median/tail/CI 和失败率。agent 同时报告任务成功率。存储报告 resident/cached/upper/object logical/physical，root-trie 创建成本和服务器租约/GC 成本单列。 + +artifact 包含一键 smoke、fake backend、固定 namespace fixtures、环境 manifest、故障矩阵、raw JSONL/CSV 和画图脚本。需要 FUSE/mount namespace 的测试在具备条件的 runner 中执行，普通 CI 运行无 FUSE 的协议/schema/调度/恢复模型测试。 + +## 7. 第一轮可执行拆分与验证 + +| PR 工作包 | 改动范围 | 验收 | 依赖 | +| --- | --- | --- | --- | +| P01 Contract + fixtures | `src/snapshot/` 类型草图、fixtures、API/schema 文档 | canonical vectors、scope mismatch、old import refs 回归预期 | D1；Mega 接口审阅 | +| P02 M0 event core | `src/telemetry/`、最小 create/fetch 埋点 | T01/T02/T03/T05，先得到 physical/logical 计数 | 可与 P01 并行 | +| P03 Smoke harness | `bench/`、baseline adapter、oracle | 相同 native/import A/B 树、失败返回非零、manifest 完整 | fixtures，可与 P02 并行 | +| P04 Immutable read slice | Dicfuse manager/store + fake/Mega adapter | V01/V02/V03/V09，两个版本并存且旧版不漂移 | P01、服务端 source-snapshot | +| P05 CAS + metadata namespace | `src/cache/`、迁移 marker | V08/V14、损坏恢复、配额、旧 schema 不误读 | P04 | +| P06 Namespace composition | bindings resolver + publication adapter | V04–V07/V15/V17 | Mega namespace contract、P05 | +| P07 Refresh + delta | journal/manifest/observer、API v2 | V10–V12/V16、每阶段 crash | D3、P05，namespace 模式还依赖 P06 | +| P08 Scheduler + architecture ADR | fetch coordinator，A/B 最小原型 | F01–F05、#51 功能/性能 gate | M0、P05/P07 | + +跨仓衔接：Mega G01 与 P01 共用 canonical/目录 fixture；G02 对接 P04；G03/G04 提供 P06 的固定索引和发布读链路；G05 完成真实部署保留/迁移后，G06 与 P06/P07 联调。G02 可先交付单 source，但 P06 的全 namespace 退出条件不能据此勾选完成。Mega writer 覆盖和 rollout gate 不属于 ScorpioFS 单仓可代办的事项。 + +后续集成/全量实验在 P04–P08 的实测风险明确后估算；目前没有人力和截止日期，不提供虚假的日历交付时间。 + +文档变更本次检查：本地相对链接、Issue 编号映射、schema 示例、diff whitespace。实现 PR 应运行相应 unit/integration/FUSE tests；拟议 tests/CLI 尚不存在，不能把它们列为已经通过。 + +## 8. 决策记录 + +D1/D2/D3 见 [版本 spec §12](monorepo-versioning.md#12-待确认决策与明确假设)。在收到确认之前，方案按推荐方向写成可审阅草案，不锁定服务端发布政策或 live-refresh 承诺。 + +上一轮建议的“Option A 直接定为生产首选”撤回为待 bakeoff 的候选。原因是完整 namespace 版本、kernel lower 换代和 delta observer 是联合约束；现有 per-job FUSE 继续承担第一阶段正确性基线。 From e4f8095a728579ce56a6991ccb6251a2ed2b97b4 Mon Sep 17 00:00:00 2001 From: Luxian Date: Sun, 6 Sep 2026 09:15:27 +0800 Subject: [PATCH 2/8] feat(snapshot): add typed identities and fixed-source object reader --- docs/spec/monorepo-versioning.md | 8 +- docs/spec/source-snapshot-v1.md | 66 +++++ src/lib.rs | 1 + src/snapshot/backend.rs | 334 +++++++++++++++++++++ src/snapshot/backend/tests.rs | 390 +++++++++++++++++++++++++ src/snapshot/identity.rs | 362 +++++++++++++++++++++++ src/snapshot/mod.rs | 5 + tests/fixtures/snapshot/source-v1.json | 24 ++ 8 files changed, 1188 insertions(+), 2 deletions(-) create mode 100644 docs/spec/source-snapshot-v1.md create mode 100644 src/snapshot/backend.rs create mode 100644 src/snapshot/backend/tests.rs create mode 100644 src/snapshot/identity.rs create mode 100644 src/snapshot/mod.rs create mode 100644 tests/fixtures/snapshot/source-v1.json diff --git a/docs/spec/monorepo-versioning.md b/docs/spec/monorepo-versioning.md index 6b790ee..c3b4eb0 100644 --- a/docs/spec/monorepo-versioning.md +++ b/docs/spec/monorepo-versioning.md @@ -1,6 +1,8 @@ # Mega 命名空间版本与 Dicfuse 不可变视图 Spec -状态：Draft v0.2，2026-09-06。三项产品决策待用户确认（§12）；文中的 MUST 是拟议协议要求，不代表现有实现。命名空间协议由 [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) 跟踪，本文细化 [#42](https://github.com/gitmono-dev/scorpiofs/issues/42)，约束 #43、#44、#49、#50、#51、#53。总路线见 [system-paper-spec.md](system-paper-spec.md)。Mega 侧配套实施草案位于该仓库的 `docs/spec/namespace-snapshot-spec.md`，细化 G01–G06 与 MG01–MG17；目前两仓 spec 均未提交。 +状态：Draft v0.3，2026-09-06。三项产品决策待用户确认（§12）；文中的 MUST 是目标协议要求，不代表现有实现。命名空间协议由 [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) 跟踪，本文细化 [#42](https://github.com/gitmono-dev/scorpiofs/issues/42)，约束 #43、#44、#49、#50、#51、#53。总路线见 [system-paper-spec.md](system-paper-spec.md)。Mega 侧配套实施草案位于该仓库的 `docs/spec/namespace-snapshot-spec.md`，细化 G01–G06 与 MG01–MG17；两仓 spec 已提交到工作分支。 + +当前实现进度：已增加严格 source identity 与不可变 SourceReader 库层，跨仓黄金向量和固定对象读取测试见 [source-snapshot-v1.md](source-snapshot-v1.md)。尚未接入实际 Dicfuse/Antares 挂载、HTTP 对象后端、lease/CAS 或工作区切换；现有挂载因此仍不具备本文承诺的版本隔离。完整 namespace 发布及所有写入者覆盖同样尚未完成。 ## 1. 问题与事实基线 @@ -52,7 +54,7 @@ WorkspaceGeneration 工作区使用的 view、delta 序号与切换事务 ```rust // 协议草图；ObjectId/SourceId 均为有验证器的类型，不接受任意字符串。 struct SourceSnapshot { - source_id: SourceId, // instance UUID + backend kind + stable repo ID + source_id: SourceId, // persistent UUID mapped to instance/backend/repo scope_path: RepoPath, // commit.tree 对应的命名空间位置 commit_oid: ObjectId, root_tree_oid: ObjectId, // commit 的 tree；scope 映射经服务端验证 @@ -66,6 +68,8 @@ Mega 当前 `mega_commit` 没有 scope 字段，因此需要持久化 `(source_i revision selector 使用带类型联合：`published_view`、`source_commit`、`source_ref`。branch/tag 解析结果带完整 ref 名和最终 commit；裸 tree 请求必须显式声明 `tree` 类型，不能伪装成 commit。当前 Mega 的 SHA-1 限制作为 capability 返回；协议保留 SHA-256 类型但不得提前宣称支持。 +单 source 的 v1 校验器和 canonical 编码现已落地，详见 [source-snapshot-v1.md](source-snapshot-v1.md)。该基础实现不等于 namespace 发布、leases 或 FUSE 集成已完成；view/index 的编码仍在各自实施闸门内。 + ### 3.2 NamespaceView ```rust diff --git a/docs/spec/source-snapshot-v1.md b/docs/spec/source-snapshot-v1.md new file mode 100644 index 0000000..e4736e7 --- /dev/null +++ b/docs/spec/source-snapshot-v1.md @@ -0,0 +1,66 @@ +# Source snapshot v1 contract + +Status: implemented identity/read foundations, 2026-09-06. This contract does not advertise a deployed snapshot capability. Namespace publication, leases, scope attestation coverage and FUSE integration remain separate gates in the versioning specs. + +## JSON and validation + +The source descriptor has exactly these five fields: + +```json +{ + "source_id": "11111111-1111-4111-8111-111111111111", + "scope_path": "/project/a", + "object_format": "sha1", + "commit_oid": "1111111111111111111111111111111111111111", + "root_tree_oid": "4b825dc642cb6eb9a060e54bf8d69288fbee4904" +} + +``` + +The hashes above illustrate structure, not a deployed source or a claimed commit/tree relationship. + +- source_id is a non-nil, lowercase, hyphenated UUID persisted by Mega. Its server-side mapping includes instance, backend kind and repo ID. Paths are not source IDs; recreating a different logical source must not reuse an ID. +- scope_path is canonical absolute UTF-8. / is the root; other paths cannot end in / or contain empty, dot or parent components or NUL. Limit: 4096 UTF-8 bytes per protocol path, 255 per component. This protocol limit is not a guarantee that any host-local mountpoint prefix fits an OS path limit. +- Names retain case, Unicode composition, plus signs and literal backslashes. No Windows path normalization is applied. Non-UTF-8 names are unsupported in v1. +- object_format is sha1. Object IDs contain exactly 40 lowercase hexadecimal digits. Future algorithms require negotiation, not automatic acceptance. +- Unknown fields, invalid IDs and unknown algorithm tags fail deserialization. Structural validation does not prove scope, object membership or commit.tree: Mega must attest those relationships. + +The source_ref selector requires a fully qualified refs/heads/... or refs/tags/... name. The source_commit selector accepts only a commit OID, never an arbitrary tree/tag OID. The compatibility parser in existing Mega browsing APIs still accepts an unqualified tag name; the new typed contract does not. + +## Canonical source identity + +Do not hash a JSON serialization. Canonical source bytes consist of: + +1. The ASCII domain mega.source-snapshot.v1 followed by one NUL byte. +2. The five fields in this order: source_id, scope_path, object_format, commit_oid, root_tree_oid. +3. Each field is encoded as its unsigned 32-bit big-endian UTF-8 byte length followed immediately by those UTF-8 bytes. OIDs are lowercase hex text here, not raw 20-byte values. + +source identity = sha256: followed by the lowercase SHA-256 hex digest of those bytes. + +The same shared fixture is tested by both implementations. It includes an ASCII scope and a Unicode/plus-sign scope. The first vector's identity is sha256:6e3f8a7e41d3a9759bc05cbc1dab153ad27ba0e0ff494f7692392dbfd5a95451. Fixture bytes/digests were independently computed with .NET; Ceres uses RustCrypto SHA-256 and ScorpioFS uses ring. + +This identity includes commit provenance. It is not namespace view_id, publication_seq, a lease, or a projection_key. A same-tree/different-commit pair has different source identities but may still share verified physical objects within an authorized domain. + +## Immutable object boundary + +ScorpioFS SourceReader owns a fixed SourceSnapshot and only asks ObjectBackend for (source, object kind, OID, root-relative source_path, byte limit). It exposes no mutable ref selector. A different version requires another reader; a caller cannot mutate the descriptor held by an existing reader. + +source_path is a membership/authorization context, not a lookup through current routing: the server must walk from the descriptor's fixed root and verify the resulting kind/OID before returning an object. Root uses the empty relative path. This avoids a whole-repository reachability scan per request. Signed object tickets may optimize the same check later; arbitrary caller-supplied OIDs are never sufficient proof. + +Backends must check current authorization and retention even on a global CAS hit, enforce limits during download, and return raw object payloads. The client verifies SHA-1 over Git's type + space + decimal length + NUL + payload. A file beginning with Git-like header bytes retains those bytes. + +Tree traversal is relative to the source root. Prefix neighbors such as /project/ab do not match scope /project/a. A scope commit's tree is already rooted at the scope; the prefix is never applied twice. Tree names, entry modes, symlink targets, missing paths and failed object fetches remain distinct. + +The initial reader is a bounded whole-object implementation: default limits are 16 MiB/tree and 64 MiB/blob. It returns an explicit size-limit error, never empty bytes, on oversized objects. This is not the final streaming/CAS/FUSE adapter or a claim that stat is metadata-only. Namespace routing, chunked large-object reads and controlled workspace generation changes are not completed by these tests. + +## Verification + +Run the relevant repository command: + +```sh +cargo test -p ceres --lib snapshot --locked +cargo test --lib snapshot --locked + +``` + +The ScorpioFS reader fixtures obtain their OIDs from git hash-object --stdin (without -w). Git must be installed, but these tests need no network, FUSE mount, mutable global Git configuration or existing repository objects. diff --git a/src/lib.rs b/src/lib.rs index 7714cf0..d42f6e8 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -188,6 +188,7 @@ pub mod doctor; pub mod fuse; pub mod manager; pub mod server; +pub mod snapshot; pub mod util; /// Commonly used types and traits for working with Antares. diff --git a/src/snapshot/backend.rs b/src/snapshot/backend.rs new file mode 100644 index 0000000..bdec46a --- /dev/null +++ b/src/snapshot/backend.rs @@ -0,0 +1,334 @@ +//! A fixed-source read boundary. Backend adapters must authorize every request +//! against this descriptor. Paths are root-relative membership proofs, never +//! lookups through a moving ref or the current namespace registry. + +use std::{collections::HashSet, sync::Arc}; + +use async_trait::async_trait; +use bytes::Bytes; +use thiserror::Error; + +use super::identity::{ObjectId, RelativePath, RepoPath, SourceSnapshot, MAX_COMPONENT_BYTES}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub enum ObjectKind { + Tree, + Blob, +} + +impl ObjectKind { + fn as_str(self) -> &'static str { + match self { + Self::Tree => "tree", + Self::Blob => "blob", + } + } +} + +#[derive(Debug, Error)] +pub enum SnapshotReadError { + #[error("path is outside the resolved source scope")] + OutsideScope, + #[error("path does not exist in the fixed source")] + PathNotFound, + #[error("expected a directory")] + NotDirectory, + #[error("expected a regular file")] + NotFile, + #[error("expected a symlink")] + NotSymlink, + #[error("source access denied")] + Forbidden, + #[error("snapshot retention lease expired")] + Expired, + #[error("object is unavailable: {0}")] + Unavailable(String), + #[error("Git {kind:?} hash mismatch for {oid}")] + Integrity { kind: ObjectKind, oid: ObjectId }, + #[error("malformed Git tree: {0}")] + MalformedTree(&'static str), + #[error("unsupported snapshot entry: {0}")] + Unsupported(&'static str), + #[error("object exceeds configured byte limit {limit}")] + ObjectTooLarge { limit: usize }, + #[error("object limits must be nonzero")] + InvalidLimits, +} + +#[async_trait] +pub trait ObjectBackend: Send + Sync { + /// Return raw object payload (no Git header). An implementation must enforce + /// max_bytes while receiving/allocating bytes, not merely after download. + /// source also identifies authorization/scope/retention context: a global + /// physical CAS hit must not bypass that check. No latest fallback is allowed. + /// source_path must resolve to kind/OID beneath source.root_tree_oid. This + /// permits a bounded-depth server membership/ACL check without enumerating + /// the entire reachable object graph or trusting an arbitrary bare OID. + async fn fetch( + &self, + source: &SourceSnapshot, + kind: ObjectKind, + oid: &ObjectId, + source_path: &RelativePath, + max_bytes: usize, + ) -> Result; +} + +#[derive(Debug, Clone, Copy)] +pub struct ReadLimits { + pub tree_bytes: usize, + pub blob_bytes: usize, +} + +impl Default for ReadLimits { + fn default() -> Self { + Self { + tree_bytes: 16 * 1024 * 1024, + blob_bytes: 64 * 1024 * 1024, + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum EntryKind { + Directory, + File, + Executable, + Symlink, + Gitlink, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TreeEntry { + pub name: String, + pub kind: EntryKind, + pub oid: ObjectId, +} + +/// One immutable descriptor for this reader's entire lifetime. A caller must +/// resolve/attest/pin it before exposing a mount; this type doesn't publish or +/// acquire leases on its own. Updates construct another reader. +pub struct SourceReader { + source: SourceSnapshot, + backend: Arc, + limits: ReadLimits, +} + +impl SourceReader { + pub fn new( + source: SourceSnapshot, + backend: Arc, + limits: ReadLimits, + ) -> Result { + if limits.tree_bytes == 0 || limits.blob_bytes == 0 { + return Err(SnapshotReadError::InvalidLimits); + } + Ok(Self { + source, + backend, + limits, + }) + } + + pub fn source(&self) -> &SourceSnapshot { + &self.source + } + + async fn object( + &self, + kind: ObjectKind, + oid: &ObjectId, + source_path: &RelativePath, + ) -> Result { + let limit = match kind { + ObjectKind::Tree => self.limits.tree_bytes, + ObjectKind::Blob => self.limits.blob_bytes, + }; + let bytes = self + .backend + .fetch(&self.source, kind, oid, source_path, limit) + .await?; + if bytes.len() > limit { + return Err(SnapshotReadError::ObjectTooLarge { limit }); + } + verify_object(kind, oid, &bytes)?; + Ok(bytes) + } + + async fn tree( + &self, + oid: &ObjectId, + source_path: &RelativePath, + ) -> Result, SnapshotReadError> { + let bytes = self.object(ObjectKind::Tree, oid, source_path).await?; + decode_tree(&bytes) + } + + pub async fn verify_root(&self) -> Result<(), SnapshotReadError> { + self.tree( + &self.source.root_tree_oid, + &RelativePath::new("").expect("valid root path"), + ) + .await + .map(|_| ()) + } + + pub async fn lookup(&self, path: &RepoPath) -> Result { + let relative = path + .relative_to(&self.source.scope_path) + .ok_or(SnapshotReadError::OutsideScope)?; + self.lookup_relative(&relative).await + } + + pub async fn lookup_relative( + &self, + path: &RelativePath, + ) -> Result { + let mut entry = TreeEntry { + name: String::new(), + kind: EntryKind::Directory, + oid: self.source.root_tree_oid.clone(), + }; + if path.as_str().is_empty() { + return Ok(entry); + } + let mut walked = String::new(); + for name in path.as_str().split('/') { + if entry.kind != EntryKind::Directory { + return Err(if entry.kind == EntryKind::Gitlink { + SnapshotReadError::Unsupported("submodule traversal") + } else { + SnapshotReadError::NotDirectory + }); + } + entry = self + .tree( + &entry.oid, + &RelativePath::new(&walked).expect("prefix of a validated path"), + ) + .await? + .into_iter() + .find(|entry| entry.name == name) + .ok_or(SnapshotReadError::PathNotFound)?; + if !walked.is_empty() { + walked.push('/'); + } + walked.push_str(name); + } + Ok(entry) + } + + pub async fn list_dir(&self, path: &RepoPath) -> Result, SnapshotReadError> { + let entry = self.lookup(path).await?; + if entry.kind != EntryKind::Directory { + return Err(SnapshotReadError::NotDirectory); + } + let relative = path + .relative_to(&self.source.scope_path) + .ok_or(SnapshotReadError::OutsideScope)?; + self.tree(&entry.oid, &relative).await + } + + /// This initial, bounded whole-object reader provides exact length through + /// returned bytes. It never reports unknown stat size as zero; large-object + /// streaming and an authenticated size-index adapter are separate work. + pub async fn read_file(&self, path: &RepoPath) -> Result { + let entry = self.lookup(path).await?; + match entry.kind { + EntryKind::File | EntryKind::Executable => { + let relative = path + .relative_to(&self.source.scope_path) + .ok_or(SnapshotReadError::OutsideScope)?; + self.object(ObjectKind::Blob, &entry.oid, &relative).await + } + EntryKind::Gitlink => Err(SnapshotReadError::Unsupported("submodule hydration")), + _ => Err(SnapshotReadError::NotFile), + } + } + + /// Return the raw symlink target without following it through live routing. + pub async fn read_link(&self, path: &RepoPath) -> Result { + let entry = self.lookup(path).await?; + if entry.kind != EntryKind::Symlink { + return Err(SnapshotReadError::NotSymlink); + } + let relative = path + .relative_to(&self.source.scope_path) + .ok_or(SnapshotReadError::OutsideScope)?; + self.object(ObjectKind::Blob, &entry.oid, &relative).await + } +} + +pub fn verify_object( + kind: ObjectKind, + oid: &ObjectId, + bytes: &[u8], +) -> Result<(), SnapshotReadError> { + // SHA-1 is for Git compatibility, never for authentication. Select the + // algorithm explicitly rather than using git-internal's thread-local mode. + let mut digest = ring::digest::Context::new(&ring::digest::SHA1_FOR_LEGACY_USE_ONLY); + digest.update(format!("{} {}\0", kind.as_str(), bytes.len()).as_bytes()); + digest.update(bytes); + if hex::encode(digest.finish().as_ref()) != oid.as_str() { + return Err(SnapshotReadError::Integrity { + kind, + oid: oid.clone(), + }); + } + Ok(()) +} + +fn decode_tree(mut bytes: &[u8]) -> Result, SnapshotReadError> { + let mut entries = Vec::new(); + let mut names = HashSet::new(); + while !bytes.is_empty() { + let space = bytes + .iter() + .position(|&b| b == b' ') + .ok_or(SnapshotReadError::MalformedTree("missing mode separator"))?; + let kind = match &bytes[..space] { + b"40000" | b"040000" => EntryKind::Directory, + b"100644" => EntryKind::File, + b"100755" => EntryKind::Executable, + b"120000" => EntryKind::Symlink, + b"160000" => EntryKind::Gitlink, + _ => return Err(SnapshotReadError::Unsupported("Git tree mode")), + }; + bytes = &bytes[space + 1..]; + let nul = bytes + .iter() + .position(|&b| b == 0) + .ok_or(SnapshotReadError::MalformedTree("missing name terminator"))?; + let name = std::str::from_utf8(&bytes[..nul]) + .map_err(|_| SnapshotReadError::Unsupported("non-UTF-8 filename"))?; + if name.is_empty() + || name == "." + || name == ".." + || name.contains('/') + || name.len() > MAX_COMPONENT_BYTES + { + return Err(SnapshotReadError::MalformedTree("invalid filename")); + } + if !names.insert(name.to_owned()) { + return Err(SnapshotReadError::MalformedTree("duplicate filename")); + } + bytes = &bytes[nul + 1..]; + let raw_oid = bytes + .get(..20) + .ok_or(SnapshotReadError::MalformedTree("truncated SHA-1"))?; + let oid = ObjectId::new(hex::encode(raw_oid)).expect("20 bytes produce a valid SHA-1 ID"); + entries.push(TreeEntry { + name: name.to_owned(), + kind, + oid, + }); + bytes = &bytes[20..]; + } + // Directory iteration order is stable even if the source tree uses Git's + // directory-slash ordering rather than bytewise filename ordering. + entries.sort_by(|a, b| a.name.as_bytes().cmp(b.name.as_bytes())); + Ok(entries) +} + +#[cfg(test)] +mod tests; diff --git a/src/snapshot/backend/tests.rs b/src/snapshot/backend/tests.rs new file mode 100644 index 0000000..da22bbd --- /dev/null +++ b/src/snapshot/backend/tests.rs @@ -0,0 +1,390 @@ +use std::{ + collections::HashMap, + io::Write, + process::{Command, Stdio}, + sync::{ + atomic::{AtomicU8, Ordering}, + Mutex, + }, +}; + +use super::*; +use crate::snapshot::identity::{ObjectFormat, SourceId}; + +type Key = (SourceId, ObjectKind, ObjectId); + +#[derive(Default)] +struct FakeBackend { + objects: Mutex>, + calls: Mutex>, + failure: AtomicU8, +} + +#[async_trait] +impl ObjectBackend for FakeBackend { + async fn fetch( + &self, + source: &SourceSnapshot, + kind: ObjectKind, + oid: &ObjectId, + source_path: &RelativePath, + max_bytes: usize, + ) -> Result { + self.calls + .lock() + .unwrap() + .push((source.clone(), kind, oid.clone(), source_path.clone())); + match self.failure.load(Ordering::SeqCst) { + 1 => return Err(SnapshotReadError::Forbidden), + 2 => return Err(SnapshotReadError::Expired), + 3 => { + return Err(SnapshotReadError::Unavailable( + "injected network failure".into(), + )) + } + _ => {} + } + let bytes = self + .objects + .lock() + .unwrap() + .get(&(source.source_id.clone(), kind, oid.clone())) + .cloned() + .ok_or_else(|| SnapshotReadError::Unavailable("missing retained object".into()))?; + if bytes.len() > max_bytes { + return Err(SnapshotReadError::ObjectTooLarge { limit: max_bytes }); + } + Ok(bytes) + } +} + +/// Git itself computes fixture identities. The reader/hash verifier under test +/// does not generate the oracle's OIDs, and no objects are written to a repo. +fn git_oid(kind: &str, payload: &[u8]) -> ObjectId { + let mut child = Command::new("git") + .args(["hash-object", "--stdin", "-t", kind]) + .env("GIT_DEFAULT_HASH", "sha1") + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("snapshot fixtures require Git"); + child.stdin.take().unwrap().write_all(payload).unwrap(); + let output = child.wait_with_output().unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + ObjectId::new(String::from_utf8(output.stdout).unwrap().trim()).unwrap() +} + +fn store(backend: &FakeBackend, source: &SourceId, kind: ObjectKind, payload: &[u8]) -> ObjectId { + let oid = git_oid(kind.as_str(), payload); + backend.objects.lock().unwrap().insert( + (source.clone(), kind, oid.clone()), + Bytes::copy_from_slice(payload), + ); + oid +} + +fn raw_entry(mode: &str, name: &[u8], oid: &ObjectId) -> Vec { + let mut result = format!("{mode} ").into_bytes(); + result.extend(name); + result.push(0); + result.extend(hex::decode(oid.as_str()).unwrap()); + result +} + +fn fixture(backend: &FakeBackend, content: &[u8]) -> SourceSnapshot { + let id = SourceId::new("11111111-1111-4111-8111-111111111111").unwrap(); + let blob = store(backend, &id, ObjectKind::Blob, content); + let subtree = store( + backend, + &id, + ObjectKind::Tree, + &raw_entry("100644", b"file.rs", &blob), + ); + let link = store(backend, &id, ObjectKind::Blob, b"src/file.rs"); + let mut tree = raw_entry("120000", b"link", &link); + tree.extend(raw_entry("100755", b"run.sh", &blob)); + tree.extend(raw_entry("40000", b"src", &subtree)); + let root = store(backend, &id, ObjectKind::Tree, &tree); + let commit = format!("tree {root}\nauthor Fixture 0 +0000\ncommitter Fixture 0 +0000\n\nfixture\n"); + SourceSnapshot { + source_id: id, + scope_path: RepoPath::new("/project/a").unwrap(), + object_format: ObjectFormat::Sha1, + commit_oid: git_oid("commit", commit.as_bytes()), + root_tree_oid: root, + } +} + +fn reader(source: &SourceSnapshot, backend: &Arc) -> SourceReader { + SourceReader::new(source.clone(), backend.clone(), ReadLimits::default()).unwrap() +} + +fn path(relative: &str) -> RepoPath { + RepoPath::new(format!("/project/a/{relative}")).unwrap() +} + +#[tokio::test] +async fn first_lazy_read_after_new_revision_still_reads_old_objects() { + let backend = Arc::new(FakeBackend::default()); + let old = fixture(&backend, b"old"); + let old_reader = reader(&old, &backend); + let new = fixture(&backend, b"new and longer"); + assert_ne!(old.commit_oid, new.commit_oid); + assert!(backend.calls.lock().unwrap().is_empty()); + assert_eq!( + old_reader + .read_file(&path("src/file.rs")) + .await + .unwrap() + .as_ref(), + b"old" + ); + assert!(backend + .calls + .lock() + .unwrap() + .iter() + .all(|(source, _, _, _)| source == &old)); + assert_eq!( + backend + .calls + .lock() + .unwrap() + .iter() + .map(|(_, _, _, path)| path.as_str()) + .collect::>(), + ["", "src", "src/file.rs"] + ); + assert_eq!( + reader(&new, &backend) + .read_file(&path("src/file.rs")) + .await + .unwrap() + .as_ref(), + b"new and longer" + ); + assert_eq!( + old_reader + .read_file(&path("src/file.rs")) + .await + .unwrap() + .len(), + 3 + ); +} + +#[tokio::test] +async fn native_scope_is_not_applied_twice_and_prefix_neighbors_are_outside() { + let backend = Arc::new(FakeBackend::default()); + let source = fixture(&backend, b"hello"); + let reader = reader(&source, &backend); + let absolute = reader.lookup(&path("src/file.rs")).await.unwrap(); + let relative = reader + .lookup_relative(&RelativePath::new("src/file.rs").unwrap()) + .await + .unwrap(); + assert_eq!(absolute, relative); + assert!(matches!( + reader + .lookup(&RepoPath::new("/project/ab/src/file.rs").unwrap()) + .await, + Err(SnapshotReadError::OutsideScope) + )); + assert!(matches!( + reader + .lookup_relative(&RelativePath::new("project/a/src/file.rs").unwrap()) + .await, + Err(SnapshotReadError::PathNotFound) + )); +} + +#[tokio::test] +async fn modes_and_symlink_target_are_preserved_without_implicit_traversal() { + let backend = Arc::new(FakeBackend::default()); + let source = fixture(&backend, b"#!/bin/sh\n"); + let reader = reader(&source, &backend); + reader.verify_root().await.unwrap(); + let entries = reader.list_dir(&source.scope_path).await.unwrap(); + assert_eq!( + entries + .iter() + .map(|entry| (entry.name.as_str(), entry.kind)) + .collect::>(), + [ + ("link", EntryKind::Symlink), + ("run.sh", EntryKind::Executable), + ("src", EntryKind::Directory) + ] + ); + assert_eq!( + reader.read_link(&path("link")).await.unwrap().as_ref(), + b"src/file.rs" + ); + assert!(matches!( + reader.read_file(&path("link")).await, + Err(SnapshotReadError::NotFile) + )); + assert!(matches!( + reader.lookup(&path("link/child")).await, + Err(SnapshotReadError::NotDirectory) + )); +} + +#[tokio::test] +async fn failures_do_not_become_missing_paths_or_empty_successes() { + let backend = Arc::new(FakeBackend::default()); + let source = fixture(&backend, b"hello"); + let reader = reader(&source, &backend); + assert!(matches!( + reader.lookup(&path("absent")).await, + Err(SnapshotReadError::PathNotFound) + )); + backend.failure.store(1, Ordering::SeqCst); + assert!(matches!( + reader.read_file(&path("src/file.rs")).await, + Err(SnapshotReadError::Forbidden) + )); + backend.failure.store(2, Ordering::SeqCst); + assert!(matches!( + reader.read_file(&path("src/file.rs")).await, + Err(SnapshotReadError::Expired) + )); + backend.failure.store(3, Ordering::SeqCst); + assert!(matches!( + reader.read_file(&path("src/file.rs")).await, + Err(SnapshotReadError::Unavailable(_)) + )); + backend.failure.store(0, Ordering::SeqCst); + backend.objects.lock().unwrap().remove(&( + source.source_id.clone(), + ObjectKind::Tree, + source.root_tree_oid.clone(), + )); + assert!(matches!( + reader.list_dir(&source.scope_path).await, + Err(SnapshotReadError::Unavailable(_)) + )); +} + +#[tokio::test] +async fn bad_bytes_are_rejected_and_git_like_file_prefix_is_preserved() { + let backend = Arc::new(FakeBackend::default()); + let content = b"blob 3\0abc"; + let source = fixture(&backend, content); + let reader = reader(&source, &backend); + assert_eq!( + reader + .read_file(&path("src/file.rs")) + .await + .unwrap() + .as_ref(), + content + ); + let entry = reader.lookup(&path("src/file.rs")).await.unwrap(); + assert!(matches!( + verify_object(ObjectKind::Tree, &entry.oid, content), + Err(SnapshotReadError::Integrity { .. }) + )); + backend.objects.lock().unwrap().insert( + (source.source_id.clone(), ObjectKind::Blob, entry.oid), + Bytes::from_static(b"wrong"), + ); + assert!(matches!( + reader.read_file(&path("src/file.rs")).await, + Err(SnapshotReadError::Integrity { + kind: ObjectKind::Blob, + .. + }) + )); +} + +#[tokio::test] +async fn zero_length_and_byte_limits_are_explicit() { + let backend = Arc::new(FakeBackend::default()); + let empty = fixture(&backend, b""); + assert!(reader(&empty, &backend) + .read_file(&path("src/file.rs")) + .await + .unwrap() + .is_empty()); + let source = fixture(&backend, b"more than four bytes"); + let limited = SourceReader::new( + source, + backend, + ReadLimits { + tree_bytes: 4096, + blob_bytes: 4, + }, + ) + .unwrap(); + assert!(matches!( + limited.read_file(&path("src/file.rs")).await, + Err(SnapshotReadError::ObjectTooLarge { limit: 4 }) + )); +} + +#[tokio::test] +async fn same_oid_in_another_source_does_not_bypass_source_membership() { + let backend = Arc::new(FakeBackend::default()); + let source = fixture(&backend, b"private"); + let mut other = source.clone(); + other.source_id = SourceId::new("22222222-2222-4222-8222-222222222222").unwrap(); + // A second source receives the trees, but has no authorized copy of the blob. + let trees = backend + .objects + .lock() + .unwrap() + .iter() + .filter(|((_, kind, _), _)| *kind == ObjectKind::Tree) + .map(|((_, kind, oid), bytes)| { + ((other.source_id.clone(), *kind, oid.clone()), bytes.clone()) + }) + .collect::>(); + backend.objects.lock().unwrap().extend(trees); + assert!(matches!( + reader(&other, &backend) + .read_file(&path("src/file.rs")) + .await, + Err(SnapshotReadError::Unavailable(_)) + )); + assert_eq!( + reader(&source, &backend) + .read_file(&path("src/file.rs")) + .await + .unwrap() + .as_ref(), + b"private" + ); +} + +#[test] +fn malformed_tree_names_and_truncated_objects_are_rejected() { + let oid = ObjectId::new("1".repeat(40)).unwrap(); + for name in [b"".as_slice(), b".", b"..", b"a/b"] { + assert!(matches!( + decode_tree(&raw_entry("100644", name, &oid)), + Err(SnapshotReadError::MalformedTree(_)) + )); + } + assert!(matches!( + decode_tree(&raw_entry("100644", &[255], &oid)), + Err(SnapshotReadError::Unsupported(_)) + )); + let mut duplicate = raw_entry("100644", b"same", &oid); + duplicate.extend(raw_entry("40000", b"same", &oid)); + assert!(matches!( + decode_tree(&duplicate), + Err(SnapshotReadError::MalformedTree("duplicate filename")) + )); + let mut truncated = raw_entry("100644", b"file", &oid); + truncated.pop(); + assert!(matches!( + decode_tree(&truncated), + Err(SnapshotReadError::MalformedTree(_)) + )); +} diff --git a/src/snapshot/identity.rs b/src/snapshot/identity.rs new file mode 100644 index 0000000..2acdf66 --- /dev/null +++ b/src/snapshot/identity.rs @@ -0,0 +1,362 @@ +//! Version-one source identities shared with the Mega snapshot contract. +//! +//! The JSON form is a wire representation, not the bytes to hash. IDs use the +//! explicitly framed encoding below; moving refs and leases are not identity. + +use std::{fmt, str::FromStr}; + +use serde::{Deserialize, Serialize}; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct IdentityError(pub &'static str); + +impl fmt::Display for IdentityError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.0) + } +} +impl std::error::Error for IdentityError {} + +macro_rules! validated_string { + ($name:ident, $validator:ident, $message:literal) => { + #[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] + #[serde(try_from = "String", into = "String")] + pub struct $name(String); + + impl $name { + pub fn new(value: impl Into) -> Result { + let value = value.into(); + if !$validator(&value) { + return Err(IdentityError($message)); + } + Ok(Self(value)) + } + pub fn as_str(&self) -> &str { + &self.0 + } + } + impl TryFrom for $name { + type Error = IdentityError; + fn try_from(value: String) -> Result { + Self::new(value) + } + } + impl From<$name> for String { + fn from(value: $name) -> Self { + value.0 + } + } + impl FromStr for $name { + type Err = IdentityError; + fn from_str(value: &str) -> Result { + Self::new(value) + } + } + impl fmt::Display for $name { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(&self.0) + } + } + }; +} + +fn valid_source(value: &str) -> bool { + uuid::Uuid::parse_str(value) + .map(|id| !id.is_nil() && id.to_string() == value) + .unwrap_or(false) +} + +fn lowercase_hex(value: &str, len: usize) -> bool { + value.len() == len + && value + .bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) +} + +fn valid_oid(value: &str) -> bool { + lowercase_hex(value, 40) +} + +fn valid_digest(value: &str) -> bool { + value + .strip_prefix("sha256:") + .is_some_and(|hex| lowercase_hex(hex, 64)) +} + +/// Limits are byte lengths, matching the v1 Linux projection contract. +pub const MAX_PATH_BYTES: usize = 4096; +pub const MAX_COMPONENT_BYTES: usize = 255; + +fn valid_components(value: &str) -> bool { + value.split('/').all(|part| { + !part.is_empty() + && part != "." + && part != ".." + && part.len() <= MAX_COMPONENT_BYTES + && !part.contains('\0') + }) +} + +fn valid_absolute_path(value: &str) -> bool { + value == "/" + || (value.len() <= MAX_PATH_BYTES && value.strip_prefix('/').is_some_and(valid_components)) +} + +fn valid_relative_path(value: &str) -> bool { + value.is_empty() || (value.len() <= MAX_PATH_BYTES && valid_components(value)) +} + +fn valid_ref(value: &str) -> bool { + (value.starts_with("refs/heads/") || value.starts_with("refs/tags/")) + && value.len() <= 1024 + && !value.ends_with('.') + && !value.contains("..") + && !value.contains("@{") + && !value + .bytes() + .any(|b| b <= b' ' || b == 127 || b"~^:?*[\\".contains(&b)) + && value + .split('/') + .all(|part| !part.is_empty() && !part.starts_with('.') && !part.ends_with(".lock")) +} + +validated_string!( + SourceId, + valid_source, + "source ID must be a non-nil canonical UUID" +); +validated_string!( + ObjectId, + valid_oid, + "v1 Git object ID must be 40 lowercase hexadecimal digits" +); +validated_string!( + ManifestDigest, + valid_digest, + "digest must be sha256 followed by 64 lowercase hexadecimal digits" +); +validated_string!( + RepoPath, + valid_absolute_path, + "path must be absolute, canonical UTF-8 and within v1 byte limits" +); +validated_string!( + RelativePath, + valid_relative_path, + "relative path must be canonical UTF-8 and within v1 byte limits" +); +validated_string!( + RefName, + valid_ref, + "ref must be a canonical fully qualified branch or tag" +); + +impl RepoPath { + /// Component-aware containment, not a raw starts_with check. + pub fn relative_to(&self, scope: &RepoPath) -> Option { + let relative = if scope.as_str() == "/" { + self.as_str().strip_prefix('/')? + } else if self == scope { + "" + } else { + self.as_str() + .strip_prefix(scope.as_str())? + .strip_prefix('/')? + }; + RelativePath::new(relative).ok() + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ObjectFormat { + Sha1, +} + +impl ObjectFormat { + pub fn as_str(self) -> &'static str { + match self { + Self::Sha1 => "sha1", + } + } +} + +/// Structural validity is not a server attestation: the resolver must also +/// prove the commit/tree/scope relationship and enforce authorization. +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct SourceSnapshot { + pub source_id: SourceId, + pub scope_path: RepoPath, + pub object_format: ObjectFormat, + pub commit_oid: ObjectId, + pub root_tree_oid: ObjectId, +} + +impl SourceSnapshot { + /// Domain bytes, then five UTF-8 fields with unsigned big-endian u32 lengths. + /// Field order is part of v1. There are no optional or floating-point fields. + pub fn canonical_bytes(&self) -> Vec { + let mut bytes = b"mega.source-snapshot.v1\0".to_vec(); + for field in [ + self.source_id.as_str(), + self.scope_path.as_str(), + self.object_format.as_str(), + self.commit_oid.as_str(), + self.root_tree_oid.as_str(), + ] { + // All fields are validated and bounded well below u32::MAX. + bytes.extend_from_slice(&(field.len() as u32).to_be_bytes()); + bytes.extend_from_slice(field.as_bytes()); + } + bytes + } + + /// Provenance identity. This is NOT a namespace view or a projection key: + /// two commits with the same tree may still share a verified object cache. + pub fn id(&self) -> ManifestDigest { + let bytes = self.canonical_bytes(); + let digest = hex::encode(ring::digest::digest(&ring::digest::SHA256, &bytes).as_ref()); + ManifestDigest(format!("sha256:{digest}")) + } +} + +/// Only used before resolving. Immutable readers never retain this selector. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)] +pub enum SourceSelector { + SourceCommit { + source_id: SourceId, + scope_path: RepoPath, + commit_oid: ObjectId, + }, + SourceRef { + source_id: SourceId, + scope_path: RepoPath, + ref_name: RefName, + }, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[derive(Deserialize)] + struct Vector { + source: SourceSnapshot, + canonical_hex: String, + source_id_digest: ManifestDigest, + } + + #[test] + fn source_golden_vectors_match_independent_encoding() { + let vectors: Vec = + serde_json::from_str(include_str!("../../tests/fixtures/snapshot/source-v1.json")) + .unwrap(); + for vector in vectors { + assert_eq!( + hex::encode(vector.source.canonical_bytes()), + vector.canonical_hex + ); + assert_eq!(vector.source.id(), vector.source_id_digest); + let json = serde_json::to_string(&vector.source).unwrap(); + assert_eq!( + serde_json::from_str::(&json).unwrap(), + vector.source + ); + } + } + + #[test] + fn paths_preserve_names_and_enforce_component_boundaries() { + let scope = RepoPath::new("/project/a").unwrap(); + assert_eq!( + RepoPath::new("/project/a/src/a+b.rs") + .unwrap() + .relative_to(&scope) + .unwrap() + .as_str(), + "src/a+b.rs" + ); + assert!(RepoPath::new("/project/ab") + .unwrap() + .relative_to(&scope) + .is_none()); + assert_eq!(scope.relative_to(&scope).unwrap().as_str(), ""); + let unicode = RepoPath::new("/第三方/e\u{301}").unwrap(); + assert_eq!(unicode.as_str(), "/第三方/e\u{301}"); + for invalid in [ + "", + "project/a", + "//a", + "/a/", + "/a//b", + "/a/./b", + "/a/../b", + "/a\0b", + ] { + assert!(RepoPath::new(invalid).is_err(), "{invalid:?}"); + } + assert!(RepoPath::new(format!("/{}", "a".repeat(256))).is_err()); + assert!(RelativePath::new("/absolute").is_err()); + assert!(RelativePath::new("../outside").is_err()); + } + + #[test] + fn deserialization_cannot_bypass_identity_validation() { + for invalid in [ + "\"BAD\"", + "\"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\"", + "\"0000000000000000000000000000000000000000000000000000000000000000\"", + ] { + assert!(serde_json::from_str::(invalid).is_err()); + } + assert!(SourceId::new("00000000-0000-0000-0000-000000000000").is_err()); + assert!(SourceId::new("https://example.test/repo").is_err()); + assert!(serde_json::from_str::("\"sha256\"").is_err()); + assert!(ManifestDigest::new(format!("sha256:{}", "A".repeat(64))).is_err()); + } + + #[test] + fn symbolic_refs_are_typed_and_fully_qualified() { + for valid in [ + "refs/heads/main", + "refs/tags/v1.2.3+build", + "refs/heads/团队/分支", + ] { + assert!(RefName::new(valid).is_ok(), "{valid}"); + } + for invalid in [ + "main", + "HEAD", + "refs/cl/123", + "refs/heads/", + "refs/heads/a..b", + "refs/tags/.x", + "refs/tags/x.lock", + "refs/heads/a@{b", + "refs/heads/a//b", + "refs/heads/a?b", + "refs/heads/a\\b", + ] { + assert!(RefName::new(invalid).is_err(), "{invalid}"); + } + } + + #[test] + fn provenance_includes_scope_source_and_commit_even_when_tree_is_equal() { + let vectors: Vec = + serde_json::from_str(include_str!("../../tests/fixtures/snapshot/source-v1.json")) + .unwrap(); + let source = vectors.into_iter().next().unwrap().source; + let mut other = source.clone(); + other.commit_oid = ObjectId::new("2".repeat(40)).unwrap(); + assert_ne!(source.id(), other.id()); + other = source.clone(); + other.scope_path = RepoPath::new("/different").unwrap(); + assert_ne!(source.id(), other.id()); + other = source.clone(); + other.source_id = SourceId::new("22222222-2222-4222-8222-222222222222").unwrap(); + assert_ne!(source.id(), other.id()); + } +} diff --git a/src/snapshot/mod.rs b/src/snapshot/mod.rs new file mode 100644 index 0000000..9df30b7 --- /dev/null +++ b/src/snapshot/mod.rs @@ -0,0 +1,5 @@ +//! Immutable source identities and reads. This module does not follow live refs +//! or claim a server-published namespace until the backend advertises it. + +pub mod backend; +pub mod identity; diff --git a/tests/fixtures/snapshot/source-v1.json b/tests/fixtures/snapshot/source-v1.json new file mode 100644 index 0000000..e3cda42 --- /dev/null +++ b/tests/fixtures/snapshot/source-v1.json @@ -0,0 +1,24 @@ +[ + { + "source": { + "source_id": "11111111-1111-4111-8111-111111111111", + "scope_path": "/project/a", + "object_format": "sha1", + "commit_oid": "1111111111111111111111111111111111111111", + "root_tree_oid": "4b825dc642cb6eb9a060e54bf8d69288fbee4904" + }, + "canonical_hex": "6d6567612e736f757263652d736e617073686f742e7631000000002431313131313131312d313131312d343131312d383131312d3131313131313131313131310000000a2f70726f6a6563742f61000000047368613100000028313131313131313131313131313131313131313131313131313131313131313131313131313131310000002834623832356463363432636236656239613036306535346266386436393238386662656534393034", + "source_id_digest": "sha256:6e3f8a7e41d3a9759bc05cbc1dab153ad27ba0e0ff494f7692392dbfd5a95451" + }, + { + "source": { + "source_id": "33333333-3333-4333-8333-333333333333", + "scope_path": "/third-party/库+1", + "object_format": "sha1", + "commit_oid": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "root_tree_oid": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "canonical_hex": "6d6567612e736f757263652d736e617073686f742e7631000000002433333333333333332d333333332d343333332d383333332d333333333333333333333333000000122f74686972642d70617274792fe5ba932b31000000047368613100000028616161616161616161616161616161616161616161616161616161616161616161616161616161610000002862626262626262626262626262626262626262626262626262626262626262626262626262626262", + "source_id_digest": "sha256:4e5857227f3b44e979c35aadcdc92835646a1eadae013d3e01ec07a08017dd74" + } +] From f37e34f26502d85db29b0e7a45bfc54d9c52a855 Mon Sep 17 00:00:00 2001 From: Luxian Date: Sun, 6 Sep 2026 09:26:42 +0800 Subject: [PATCH 3/8] docs(snapshot): clarify server observations and remaining read gates --- docs/spec/source-snapshot-v1.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/spec/source-snapshot-v1.md b/docs/spec/source-snapshot-v1.md index e4736e7..f621fc7 100644 --- a/docs/spec/source-snapshot-v1.md +++ b/docs/spec/source-snapshot-v1.md @@ -53,6 +53,14 @@ Tree traversal is relative to the source root. Prefix neighbors such as /project The initial reader is a bounded whole-object implementation: default limits are 16 MiB/tree and 64 MiB/blob. It returns an explicit size-limit error, never empty bytes, on oversized objects. This is not the final streaming/CAS/FUSE adapter or a claim that stat is metadata-only. Namespace routing, chunked large-object reads and controlled workspace generation changes are not completed by these tests. +## Mega source observations and scope proofs + +SourceCatalog registers stable backend IDs and resolves typed selectors. A new import observation uses its registered root and a repo-scoped commit/tag resolution. A new native observation requires an exact scoped ref whose stored root agrees with commit.tree; it records native_ref_observed, not a claim that an older writer emitted a creation proof. Native projection derives a child by walking an already attested fixed root and preserves the base commit provenance. + +Explicit native commits without a proof for the requested scope return SCOPE_UNKNOWN. A recorded descriptor can be resolved after refs or registry entries are removed; a reused path assigned to a different repo ID receives a different source ID. No current registry lookup is used to read an already attested source. + +The catalog checks descriptor attestation and walks root-relative paths to bind object kind/OID to source membership. It strictly decodes UTF-8 trees and checks SHA-1 independent of git-internal's thread-local algorithm. It is an internal metadata service, not an authorization or retention grant. Public HTTP reads must add those checks, and no snapshot endpoint or capability is enabled by the catalog alone. Observing individual sources is not an atomic multi-source namespace publication. Existing commit metadata is trusted ingestion state; this foundation does not claim raw commit/tag payload re-verification or complete proof capture by every writer. + ## Verification Run the relevant repository command: From d0175b17206f658b962c394020c1c9a06f77d9e8 Mon Sep 17 00:00:00 2001 From: Luxian Date: Sun, 6 Sep 2026 09:40:08 +0800 Subject: [PATCH 4/8] feat(snapshot): add bounded source-aware HTTP object transport --- docs/spec/monorepo-versioning.md | 18 +- docs/spec/source-snapshot-v1.md | 10 + src/snapshot/http.rs | 222 ++++++++++++++++++ src/snapshot/http/tests.rs | 390 +++++++++++++++++++++++++++++++ src/snapshot/mod.rs | 1 + 5 files changed, 635 insertions(+), 6 deletions(-) create mode 100644 src/snapshot/http.rs create mode 100644 src/snapshot/http/tests.rs diff --git a/docs/spec/monorepo-versioning.md b/docs/spec/monorepo-versioning.md index c3b4eb0..4355c22 100644 --- a/docs/spec/monorepo-versioning.md +++ b/docs/spec/monorepo-versioning.md @@ -1,8 +1,8 @@ # Mega 命名空间版本与 Dicfuse 不可变视图 Spec -状态：Draft v0.3，2026-09-06。三项产品决策待用户确认（§12）；文中的 MUST 是目标协议要求，不代表现有实现。命名空间协议由 [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) 跟踪，本文细化 [#42](https://github.com/gitmono-dev/scorpiofs/issues/42)，约束 #43、#44、#49、#50、#51、#53。总路线见 [system-paper-spec.md](system-paper-spec.md)。Mega 侧配套实施草案位于该仓库的 `docs/spec/namespace-snapshot-spec.md`，细化 G01–G06 与 MG01–MG17；两仓 spec 已提交到工作分支。 +状态：Draft v0.3，2026-09-06。D1（完整 native + import 原子组合视图）与 D4（安全启用门槛）已获用户确认；D2/D3 待确认（§12）。文中的 MUST 是目标协议要求，不代表现有实现。命名空间协议由 [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) 跟踪，本文细化 [#42](https://github.com/gitmono-dev/scorpiofs/issues/42)，约束 #43、#44、#49、#50、#51、#53。总路线见 [system-paper-spec.md](system-paper-spec.md)。Mega 侧配套实施草案位于该仓库的 `docs/spec/namespace-snapshot-spec.md`，细化 G01–G06 与 MG01–MG17；两仓 spec 已提交到工作分支。 -当前实现进度：已增加严格 source identity 与不可变 SourceReader 库层，跨仓黄金向量和固定对象读取测试见 [source-snapshot-v1.md](source-snapshot-v1.md)。尚未接入实际 Dicfuse/Antares 挂载、HTTP 对象后端、lease/CAS 或工作区切换；现有挂载因此仍不具备本文承诺的版本隔离。完整 namespace 发布及所有写入者覆盖同样尚未完成。 +当前实现进度：已增加严格 source identity、不可变 SourceReader 库层和 source-aware HTTP 客户端适配器，跨仓黄金向量、固定对象读取及本机 HTTP 测试见 [source-snapshot-v1.md](source-snapshot-v1.md)。尚未接入实际 Mega snapshot HTTP 服务、Dicfuse/Antares 挂载、lease/CAS 或工作区切换；现有挂载因此仍不具备本文承诺的版本隔离。完整 namespace 发布及所有写入者覆盖同样尚未完成。 ## 1. 问题与事实基线 @@ -55,7 +55,7 @@ WorkspaceGeneration 工作区使用的 view、delta 序号与切换事务 // 协议草图；ObjectId/SourceId 均为有验证器的类型，不接受任意字符串。 struct SourceSnapshot { source_id: SourceId, // persistent UUID mapped to instance/backend/repo - scope_path: RepoPath, // commit.tree 对应的命名空间位置 + scope_path: RepoPath, // 已验证的投影根对应的命名空间位置 commit_oid: ObjectId, root_tree_oid: ObjectId, // commit 的 tree；scope 映射经服务端验证 object_format: ObjectFormat, @@ -64,6 +64,8 @@ struct SourceSnapshot { `/project/a` 的 scope commit，其 tree 根已经是 `a/` 的内容；读取 `src/lib.rs` 时不能再向它附加 `project/a/`。全库 root commit 的同一路径则要从 `/` 遍历。服务端 MUST 返回并验证 scope，不能仅凭一个存在的 commit OID 猜测。 +若服务端从已证明的 native source 派生子目录 descriptor，commit_oid 保留 base commit provenance，root_tree_oid 是从固定 base tree 验证得到的 subtree，不要求再次等于 commit.tree。它与直接 scope commit 的证明类型不同；两者即使投影内容相同，provenance identity 也可能不同。 + Mega 当前 `mega_commit` 没有 scope 字段，因此需要持久化 `(source_id, scope_path, commit_oid) → root_tree_oid + proof`；同一 commit 可有多个有效 scope，不设唯一反向映射。子 scope ref 被清理不能丢失历史证明；存量 commit 没有证明时返回 `SOURCE_SCOPE_UNVERIFIED`，不能默认它属于 `/`。clone 派生 scope commit/证明本身不代表全库可见树变化。 revision selector 使用带类型联合：`published_view`、`source_commit`、`source_ref`。branch/tag 解析结果带完整 ref 名和最终 commit；裸 tree 请求必须显式声明 `tree` 类型，不能伪装成 commit。当前 Mega 的 SHA-1 限制作为 capability 返回；协议保留 SHA-256 类型但不得提前宣称支持。 @@ -331,7 +333,7 @@ oracle 用公开 fixture 的 Git object tree + 独立实现的 binding compositi | 工作包 | 仓库/现有 issue | 交付物 | 前置与退出条件 | | --- | --- | --- | --- | | V0 | ScorpioFS / #55，#42 前置协议追踪 | 目录分类、capabilities、descriptor、公开 fixture | D1 确认；V02/V07/V17 的预期结果冻结 | -| V1 | Mega G01/G02（仅 spec，业务代码未改） | Import 按 commit 读；scope proof、scope-aware tree/blob、共享 fixture | source-snapshot.v1；V01–V03/V09、MG01–MG04/MG13 通过 | +| V1 | Mega G01/G02（基础已实现，读服务未接通） | Import 按 commit 读；scope proof、scope-aware tree/blob、共享 fixture | source-snapshot.v1；V01–V03/V09、MG01–MG04/MG13 通过 | | V2 | Mega G03–G05（拟议） | bounded bindings index、全部 writer 的 publication transaction、leases/迁移 | D1/D2 确认；V04–V07/V14/V15、MG05–MG17 相关门槛通过 | | V3 | ScorpioFS / #42、#43 | ViewResolver、immutable Dicfuse、CAS/schema 隔离 | 可先接 fake backend；V01–V09 通过后接真实 Mega | | V4 | ScorpioFS / #44、#49 | refresh journal、upper manifest、受控切换 | D3 确认；V10–V12/V16 通过 | @@ -342,10 +344,14 @@ M0 观测/benchmark 与 V0/V1 并行。不要等待全部观测实现才能开 ## 12. 待确认决策与明确假设 -| 决策 | 推荐方案（尚未确认） | 另一选择及成本 | +持续审阅：[Mega Draft PR #2181](https://github.com/gitmono-dev/mega/pull/2181)、[ScorpioFS Draft PR #56](https://github.com/gitmono-dev/scorpiofs/pull/56)。两者仍是基础实现检查点，不表示下列完整目标已经交付。 + +| 决策 | 方案及确认状态 | 另一选择及成本 | | --- | --- | --- | -| D1 全库版本边界 | Mega 发布 native root + 固定 import bindings 的组合 view，允许规划服务端协议改造 | 首期只做 source snapshot；全库原子一致性延期，lock 只能标注显式组合 | +| D1 全库版本边界 | **已确认（2026-09-06）**：Mega 原子发布 native root + 固定 import bindings 的组合 view，ScorpioFS 固定该 view 读取 | 未选：首期只做 source snapshot 并延期全库原子一致性；单 source 仅作中间工作包 | | D2 版本号路径策略 | 显式标记为发布版本的目录首次发布后不可变，新内容使用新版本路径；不靠数字目录名推断；普通 import branch 仍可演进 | 可原地替换，但每次发布产生新绑定，旧对象必须按历史策略保留 | | D3 工作区更新体验 | 运行任务固定旧 view；新任务用新 view；现有 mount 暂停/检查后显式切换 | 要求透明运行中切换，必须扩展 handle/mmap/cwd/upper generation 协议 | +**D4 已于 2026-09-06 获用户确认**：Mega snapshot 读 API 默认关闭，仅在显式配置 source/scope 读授权与对象保留策略后启用。配置缺失/无效或后端门槛未满足时，ScorpioFS 不得静默转用 legacy latest。Mega 当前开发期通用 guard 不能替代 source/path 授权；当前基础代码尚未开放新 HTTP 路由。 + 其他暂定项：M1 支持现有 Mega SHA-1 并显式拒绝不支持的格式；不自动 hydrate LFS/submodule；GC 保留期、部署内核和 import 实际拓扑需在实施前填入环境 manifest。本文不把 A/B 架构选择当成已获确认的决策。 diff --git a/docs/spec/source-snapshot-v1.md b/docs/spec/source-snapshot-v1.md index f621fc7..18106a4 100644 --- a/docs/spec/source-snapshot-v1.md +++ b/docs/spec/source-snapshot-v1.md @@ -61,6 +61,16 @@ Explicit native commits without a proof for the requested scope return SCOPE_UNK The catalog checks descriptor attestation and walks root-relative paths to bind object kind/OID to source membership. It strictly decodes UTF-8 trees and checks SHA-1 independent of git-internal's thread-local algorithm. It is an internal metadata service, not an authorization or retention grant. Public HTTP reads must add those checks, and no snapshot endpoint or capability is enabled by the catalog alone. Observing individual sources is not an atomic multi-source namespace publication. Existing commit metadata is trusted ingestion state; this foundation does not claim raw commit/tag payload re-verification or complete proof capture by every writer. +## Source object HTTP binding (client adapter implemented) + +GET api/v1/sources/{source_id}/trees/{oid} or blobs/{oid}, relative to the configured server base URL, carries exactly five percent-encoded query fields: object_format, scope_path, commit_oid, root_tree_oid and source_path. The path supplies source_id and the expected object kind/OID; together these reconstruct the attested descriptor and fixed-root membership request. There is no ref/latest query. + +Authorization: Bearer carries a current Mono access token; X-Mega-Snapshot-Lease carries the retention lease identifier. Neither is a query parameter or Debug field. The server must validate both and must not let the lease substitute for source/scope authorization. HTTP access logging should redact these headers and avoid logging private query paths. + +A successful full object response is HTTP 200, Content-Type application/octet-stream, with raw bytes. The client does not follow redirects, accept partial/204 responses as full objects, or treat JSON/HTML login/error pages as objects. SourceReader verifies the returned Git hash. The adapter checks Content-Length and also bounds collection of streamed chunks when length is absent. 401/403 become Forbidden, 410 becomes Expired; other failures, including object/source 404, remain Unavailable rather than being misreported as an absent directory entry. A missing entry discovered in a verified tree is a separate PathNotFound result. + +The adapter requires HTTPS except for loopback HTTP test/development servers, rejects base URLs containing userinfo/query/fragment, sets connect/request deadlines, and retains no reqwest URL-bearing error text. It does not acquire or renew leases, negotiate capabilities, authorize requests on behalf of Mega, or connect existing mounts automatically. Local Axum transport tests are not a deployed Mega end-to-end test. The server routes remain an implementation gate under the confirmed default-off policy. + ## Verification Run the relevant repository command: diff --git a/src/snapshot/http.rs b/src/snapshot/http.rs new file mode 100644 index 0000000..aaa7780 --- /dev/null +++ b/src/snapshot/http.rs @@ -0,0 +1,222 @@ +//! Source-aware HTTP transport. Not connected to legacy mounts: callers must +//! negotiate the source capability and acquire an attested descriptor + lease. + +use std::{fmt, time::Duration}; + +use async_trait::async_trait; +use bytes::Bytes; +use reqwest::{ + header::{HeaderValue, ACCEPT, ACCEPT_ENCODING, AUTHORIZATION, CONTENT_TYPE}, + Client, StatusCode, +}; +use thiserror::Error; +use url::{Host, Url}; + +use super::{ + backend::{ObjectBackend, ObjectKind, SnapshotReadError}, + identity::{ObjectId, RelativePath, SourceSnapshot}, +}; + +const LEASE_HEADER: &str = "x-mega-snapshot-lease"; + +#[derive(Debug, Error)] +pub enum HttpBackendConfigError { + #[error("snapshot base URL must use HTTPS (HTTP is allowed only on loopback) and have no userinfo, query or fragment")] + InvalidBaseUrl, + #[error("snapshot credentials must be nonempty, bounded HTTP header values")] + InvalidCredentials, + #[error("snapshot HTTP timeouts must be nonzero")] + InvalidTimeout, + #[error("failed to initialize snapshot HTTP client")] + ClientInitialization, +} + +#[derive(Debug, Clone, Copy)] +pub struct HttpTimeouts { + pub connect: Duration, + pub request: Duration, +} + +impl Default for HttpTimeouts { + fn default() -> Self { + Self { + connect: Duration::from_secs(5), + request: Duration::from_secs(30), + } + } +} + +/// Tokens are sensitive headers, never URL parameters or Debug fields. A lease +/// retains a snapshot but grants no read permission; the server checks both. +pub struct HttpObjectBackend { + base: Url, + client: Client, + authorization: HeaderValue, + lease: HeaderValue, +} + +impl fmt::Debug for HttpObjectBackend { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("HttpObjectBackend") + .field("origin", &self.base.origin().ascii_serialization()) + .finish_non_exhaustive() + } +} + +impl HttpObjectBackend { + pub fn new( + mut base: Url, + bearer_token: &str, + lease: &str, + timeouts: HttpTimeouts, + ) -> Result { + let loopback = match base.host() { + Some(Host::Domain(domain)) => domain == "localhost", + Some(Host::Ipv4(address)) => address.is_loopback(), + Some(Host::Ipv6(address)) => address.is_loopback(), + None => false, + }; + if base.cannot_be_a_base() + || base.host().is_none() + || !(base.scheme() == "https" || (base.scheme() == "http" && loopback)) + || !base.username().is_empty() + || base.password().is_some() + || base.query().is_some() + || base.fragment().is_some() + { + return Err(HttpBackendConfigError::InvalidBaseUrl); + } + if timeouts.connect.is_zero() || timeouts.request.is_zero() { + return Err(HttpBackendConfigError::InvalidTimeout); + } + if bearer_token.is_empty() + || bearer_token.len() > 8192 + || bearer_token.bytes().any(|b| b.is_ascii_whitespace()) + || lease.is_empty() + || lease.len() > 512 + || lease.bytes().any(|b| b.is_ascii_whitespace()) + { + return Err(HttpBackendConfigError::InvalidCredentials); + } + let mut authorization = HeaderValue::from_str(&format!("Bearer {bearer_token}")) + .map_err(|_| HttpBackendConfigError::InvalidCredentials)?; + let mut lease = + HeaderValue::from_str(lease).map_err(|_| HttpBackendConfigError::InvalidCredentials)?; + authorization.set_sensitive(true); + lease.set_sensitive(true); + // Preserve an explicitly configured reverse-proxy prefix. + if !base.path().ends_with('/') { + base.path_segments_mut() + .map_err(|_| HttpBackendConfigError::InvalidBaseUrl)? + .push(""); + } + let client = Client::builder() + .redirect(reqwest::redirect::Policy::none()) + .connect_timeout(timeouts.connect) + .timeout(timeouts.request) + .build() + .map_err(|_| HttpBackendConfigError::ClientInitialization)?; + Ok(Self { + base, + client, + authorization, + lease, + }) + } +} + +#[async_trait] +impl ObjectBackend for HttpObjectBackend { + async fn fetch( + &self, + source: &SourceSnapshot, + kind: ObjectKind, + oid: &ObjectId, + source_path: &RelativePath, + max_bytes: usize, + ) -> Result { + if max_bytes == 0 { + return Err(SnapshotReadError::InvalidLimits); + } + let collection = match kind { + ObjectKind::Tree => "trees", + ObjectKind::Blob => "blobs", + }; + let mut url = self + .base + .join(&format!( + "api/v1/sources/{}/{collection}/{oid}", + source.source_id + )) + .map_err(|_| SnapshotReadError::Unavailable("invalid snapshot endpoint".into()))?; + url.query_pairs_mut().extend_pairs([ + ("object_format", source.object_format.as_str()), + ("scope_path", source.scope_path.as_str()), + ("commit_oid", source.commit_oid.as_str()), + ("root_tree_oid", source.root_tree_oid.as_str()), + ("source_path", source_path.as_str()), + ]); + let mut response = self + .client + .get(url) + .header(AUTHORIZATION, self.authorization.clone()) + .header(LEASE_HEADER, self.lease.clone()) + .header(ACCEPT, "application/octet-stream") + .header(ACCEPT_ENCODING, "identity") + .send() + .await + .map_err(transport_error)?; + match response.status() { + StatusCode::OK => {} + StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => { + return Err(SnapshotReadError::Forbidden) + } + StatusCode::GONE => return Err(SnapshotReadError::Expired), + status => { + return Err(SnapshotReadError::Unavailable(format!( + "snapshot object HTTP {}", + status.as_u16() + ))) + } + } + // Reject successful HTML/login/JSON error pages, partial content and + // redirects. Only a 200 full raw object is the v1 representation. + let content_type = response + .headers() + .get(CONTENT_TYPE) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.split(';').next()); + if content_type != Some("application/octet-stream") { + return Err(SnapshotReadError::Unavailable( + "snapshot response is not a raw object".into(), + )); + } + if response + .content_length() + .is_some_and(|size| size > max_bytes as u64) + { + return Err(SnapshotReadError::ObjectTooLarge { limit: max_bytes }); + } + let mut bytes = Vec::new(); + while let Some(chunk) = response.chunk().await.map_err(transport_error)? { + if chunk.len() > max_bytes - bytes.len() { + return Err(SnapshotReadError::ObjectTooLarge { limit: max_bytes }); + } + bytes.extend_from_slice(&chunk); + } + Ok(Bytes::from(bytes)) + } +} + +fn transport_error(error: reqwest::Error) -> SnapshotReadError { + // reqwest's Display may include private source paths in the URL. Do not + // retain it (or headers/tokens) in user-visible errors or diagnostics. + SnapshotReadError::Unavailable(if error.is_timeout() { + "snapshot transport timed out".into() + } else { + "snapshot transport failed".into() + }) +} + +#[cfg(test)] +mod tests; diff --git a/src/snapshot/http/tests.rs b/src/snapshot/http/tests.rs new file mode 100644 index 0000000..5d4b557 --- /dev/null +++ b/src/snapshot/http/tests.rs @@ -0,0 +1,390 @@ +use std::{ + collections::HashMap, + sync::{Arc, Mutex}, +}; + +use axum::{ + body::Body, + extract::{Request, State}, + response::Response, + Router, +}; +use tokio::{net::TcpListener, task::JoinHandle}; + +use super::*; +use crate::snapshot::{ + backend::{ReadLimits, SourceReader}, + identity::{ObjectFormat, RepoPath, SourceId}, +}; + +#[derive(Clone)] +enum Reply { + Full(StatusCode, &'static str, Bytes), + Chunked, + Redirect, + Wait, + Objects(HashMap), +} + +#[derive(Clone, Debug)] +struct Observed { + path: String, + query: HashMap, + authorized: bool, + leased: bool, + method: String, +} + +#[derive(Clone)] +struct ServerState { + reply: Reply, + seen: Arc>>, +} + +struct TestServer { + base: Url, + seen: Arc>>, + task: JoinHandle<()>, +} + +impl Drop for TestServer { + fn drop(&mut self) { + self.task.abort(); + } +} + +impl TestServer { + async fn start(reply: Reply) -> Self { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base = Url::parse(&format!("http://{}/proxy", listener.local_addr().unwrap())).unwrap(); + let seen = Arc::new(Mutex::new(Vec::new())); + let state = ServerState { + reply, + seen: seen.clone(), + }; + let router = Router::new().fallback(handle).with_state(state); + let task = tokio::spawn(async move { + axum::serve(listener, router).await.unwrap(); + }); + Self { base, seen, task } + } + + fn backend(&self) -> HttpObjectBackend { + HttpObjectBackend::new( + self.base.clone(), + "fixture-access-token", + "fixture-lease", + HttpTimeouts::default(), + ) + .unwrap() + } +} + +async fn handle(State(state): State, request: Request) -> Response { + let path = request.uri().path().to_owned(); + state.seen.lock().unwrap().push(Observed { + path: path.clone(), + query: url::form_urlencoded::parse(request.uri().query().unwrap_or_default().as_bytes()) + .into_owned() + .collect(), + authorized: request + .headers() + .get(AUTHORIZATION) + .is_some_and(|v| v == "Bearer fixture-access-token"), + leased: request + .headers() + .get(LEASE_HEADER) + .is_some_and(|v| v == "fixture-lease"), + method: request.method().to_string(), + }); + match &state.reply { + Reply::Full(status, content_type, bytes) => Response::builder() + .status(*status) + .header(CONTENT_TYPE, *content_type) + .body(Body::from(bytes.clone())) + .unwrap(), + Reply::Chunked => { + let chunks = futures::stream::iter([ + Ok::<_, std::io::Error>(Bytes::from_static(b"abc")), + Ok(Bytes::from_static(b"def")), + ]); + Response::builder() + .header(CONTENT_TYPE, "application/octet-stream") + .body(Body::from_stream(chunks)) + .unwrap() + } + Reply::Redirect => Response::builder() + .status(302) + .header("location", "/redirected") + .body(Body::empty()) + .unwrap(), + Reply::Wait => std::future::pending().await, + Reply::Objects(objects) => { + let oid = path.rsplit('/').next().unwrap(); + match objects.get(oid) { + Some(bytes) => Response::builder() + .header(CONTENT_TYPE, "application/octet-stream") + .body(Body::from(bytes.clone())) + .unwrap(), + None => Response::builder().status(404).body(Body::empty()).unwrap(), + } + } + } +} + +fn source() -> SourceSnapshot { + SourceSnapshot { + source_id: SourceId::new("11111111-1111-4111-8111-111111111111").unwrap(), + scope_path: RepoPath::new("/third-party/库+1\\literal").unwrap(), + object_format: ObjectFormat::Sha1, + commit_oid: ObjectId::new("1".repeat(40)).unwrap(), + root_tree_oid: ObjectId::new("2".repeat(40)).unwrap(), + } +} + +async fn fetch(server: &TestServer, limit: usize) -> Result { + let source = source(); + server + .backend() + .fetch( + &source, + ObjectKind::Blob, + &source.commit_oid, + &RelativePath::new("src/a+b.rs").unwrap(), + limit, + ) + .await +} + +#[tokio::test] +async fn request_binds_fixed_descriptor_path_and_sensitive_headers() { + let server = TestServer::start(Reply::Full( + StatusCode::OK, + "application/octet-stream", + Bytes::from_static(b"blob 4\0data"), + )) + .await; + let result = fetch(&server, 1024).await.unwrap(); + assert_eq!(result.as_ref(), b"blob 4\0data"); + let source = source(); + let seen = server.seen.lock().unwrap(); + assert_eq!(seen.len(), 1); + assert_eq!( + seen[0].path, + format!( + "/proxy/api/v1/sources/{}/blobs/{}", + source.source_id, source.commit_oid + ) + ); + assert_eq!(seen[0].query["scope_path"], source.scope_path.as_str()); + assert_eq!(seen[0].query["source_path"], "src/a+b.rs"); + assert_eq!( + seen[0].query["root_tree_oid"], + source.root_tree_oid.as_str() + ); + assert_eq!(seen[0].query["commit_oid"], source.commit_oid.as_str()); + assert_eq!(seen[0].query["object_format"], "sha1"); + assert_eq!(seen[0].query.len(), 5); + assert!(seen[0].authorized && seen[0].leased); + assert_eq!(seen[0].method, "GET"); + let backend = server.backend(); + assert!(backend.authorization.is_sensitive() && backend.lease.is_sensitive()); + assert!(!format!("{backend:?}").contains("fixture-access-token")); + assert!(!format!("{backend:?}").contains("fixture-lease")); +} + +#[tokio::test] +async fn error_statuses_do_not_become_empty_successes_or_latest_fallbacks() { + for status in [401, 403, 404, 410, 429, 500, 503, 204, 206] { + let server = TestServer::start(Reply::Full( + StatusCode::from_u16(status).unwrap(), + "application/octet-stream", + Bytes::new(), + )) + .await; + let result = fetch(&server, 1024).await; + match status { + 401 | 403 => assert!(matches!(result, Err(SnapshotReadError::Forbidden))), + 410 => assert!(matches!(result, Err(SnapshotReadError::Expired))), + _ => assert!(matches!(result, Err(SnapshotReadError::Unavailable(_)))), + } + assert_eq!(server.seen.lock().unwrap().len(), 1); + } + let redirect = TestServer::start(Reply::Redirect).await; + assert!(matches!( + fetch(&redirect, 1024).await, + Err(SnapshotReadError::Unavailable(_)) + )); + assert_eq!(redirect.seen.lock().unwrap().len(), 1); +} + +#[tokio::test] +async fn full_and_chunked_body_limits_and_content_type_are_checked() { + for reply in [ + Reply::Full( + StatusCode::OK, + "application/octet-stream", + Bytes::from_static(b"abcdef"), + ), + Reply::Chunked, + ] { + let server = TestServer::start(reply).await; + assert!(matches!( + fetch(&server, 3).await, + Err(SnapshotReadError::ObjectTooLarge { limit: 3 }) + )); + } + let server = TestServer::start(Reply::Full( + StatusCode::OK, + "application/json", + Bytes::from_static(b"{}"), + )) + .await; + assert!(matches!( + fetch(&server, 1024).await, + Err(SnapshotReadError::Unavailable(_)) + )); + let empty = TestServer::start(Reply::Full( + StatusCode::OK, + "application/octet-stream", + Bytes::new(), + )) + .await; + assert!(fetch(&empty, 1024).await.unwrap().is_empty()); + assert!(matches!( + fetch(&empty, 0).await, + Err(SnapshotReadError::InvalidLimits) + )); + assert_eq!(empty.seen.lock().unwrap().len(), 1); +} + +#[tokio::test] +async fn transport_deadline_has_a_redacted_error() { + let server = TestServer::start(Reply::Wait).await; + let backend = HttpObjectBackend::new( + server.base.clone(), + "fixture-access-token", + "fixture-lease", + HttpTimeouts { + connect: Duration::from_secs(1), + request: Duration::from_millis(100), + }, + ) + .unwrap(); + let source = source(); + let error = backend + .fetch( + &source, + ObjectKind::Tree, + &source.root_tree_oid, + &RelativePath::new("").unwrap(), + 1024, + ) + .await + .unwrap_err(); + assert_eq!( + error.to_string(), + "object is unavailable: snapshot transport timed out" + ); +} + +#[test] +fn insecure_origins_ambiguous_urls_credentials_and_zero_timeouts_are_rejected() { + for url in [ + "http://example.com", + "https://user:secret@example.com", + "https://example.com?token=secret", + "https://example.com/#fragment", + "file:///tmp/objects", + ] { + assert!(HttpObjectBackend::new( + Url::parse(url).unwrap(), + "token", + "lease", + HttpTimeouts::default() + ) + .is_err()); + } + for (token, lease) in [ + ("", "lease"), + ("token", ""), + ("token\r\ninjected", "lease"), + ("token", "lease with space"), + ] { + assert!(HttpObjectBackend::new( + Url::parse("https://example.com").unwrap(), + token, + lease, + HttpTimeouts::default() + ) + .is_err()); + } + assert!(HttpObjectBackend::new( + Url::parse("https://example.com").unwrap(), + "token", + "lease", + HttpTimeouts { + connect: Duration::ZERO, + request: Duration::from_secs(1) + } + ) + .is_err()); +} + +#[tokio::test] +async fn reader_verifies_real_http_bytes_against_git_object_ids() { + use std::{ + io::Write, + process::{Command, Stdio}, + }; + fn git_oid(kind: &str, bytes: &[u8]) -> ObjectId { + let mut child = Command::new("git") + .args(["hash-object", "--stdin", "-t", kind]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .spawn() + .unwrap(); + child.stdin.take().unwrap().write_all(bytes).unwrap(); + let output = child.wait_with_output().unwrap(); + assert!(output.status.success()); + ObjectId::new(String::from_utf8(output.stdout).unwrap().trim()).unwrap() + } + let payload = Bytes::from_static(b"old fixed content"); + let blob = git_oid("blob", &payload); + let tree_payload = Bytes::from( + [ + b"100644 file.txt\0".as_slice(), + &hex::decode(blob.as_str()).unwrap(), + ] + .concat(), + ); + let root = git_oid("tree", &tree_payload); + let server = TestServer::start(Reply::Objects(HashMap::from([ + (blob.to_string(), payload.clone()), + (root.to_string(), tree_payload), + ]))) + .await; + let source = SourceSnapshot { + root_tree_oid: root, + ..source() + }; + let reader = SourceReader::new( + source.clone(), + Arc::new(server.backend()), + ReadLimits::default(), + ) + .unwrap(); + assert_eq!( + reader + .read_file(&RepoPath::new(format!("{}/file.txt", source.scope_path)).unwrap()) + .await + .unwrap(), + payload + ); + let seen = server.seen.lock().unwrap(); + assert!(seen + .iter() + .all(|r| r.query["commit_oid"] == source.commit_oid.as_str())); + assert!(seen.iter().any(|r| r.query["source_path"].is_empty())); + assert!(seen.iter().any(|r| r.query["source_path"] == "file.txt")); +} diff --git a/src/snapshot/mod.rs b/src/snapshot/mod.rs index 9df30b7..e46a709 100644 --- a/src/snapshot/mod.rs +++ b/src/snapshot/mod.rs @@ -2,4 +2,5 @@ //! or claim a server-published namespace until the backend advertises it. pub mod backend; +pub mod http; pub mod identity; From bea13d3c12902d8cf491646fff16662568427ffa Mon Sep 17 00:00:00 2001 From: Luxian Date: Sun, 6 Sep 2026 11:46:22 +0800 Subject: [PATCH 5/8] feat(snapshot): verify shared namespace view and binding identities --- docs/spec/monorepo-versioning.md | 4 +- docs/spec/namespace-manifest-v1.md | 81 +++++ src/snapshot/mod.rs | 1 + src/snapshot/namespace.rs | 322 ++++++++++++++++++ src/snapshot/namespace/tests.rs | 210 ++++++++++++ .../snapshot/namespace-v1-vectors.ps1 | 69 ++++ tests/fixtures/snapshot/namespace-v1.json | 74 ++++ 7 files changed, 759 insertions(+), 2 deletions(-) create mode 100644 docs/spec/namespace-manifest-v1.md create mode 100644 src/snapshot/namespace.rs create mode 100644 src/snapshot/namespace/tests.rs create mode 100644 tests/fixtures/snapshot/namespace-v1-vectors.ps1 create mode 100644 tests/fixtures/snapshot/namespace-v1.json diff --git a/docs/spec/monorepo-versioning.md b/docs/spec/monorepo-versioning.md index 4355c22..8090573 100644 --- a/docs/spec/monorepo-versioning.md +++ b/docs/spec/monorepo-versioning.md @@ -1,6 +1,6 @@ # Mega 命名空间版本与 Dicfuse 不可变视图 Spec -状态：Draft v0.3，2026-09-06。D1（完整 native + import 原子组合视图）与 D4（安全启用门槛）已获用户确认；D2/D3 待确认（§12）。文中的 MUST 是目标协议要求，不代表现有实现。命名空间协议由 [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) 跟踪，本文细化 [#42](https://github.com/gitmono-dev/scorpiofs/issues/42)，约束 #43、#44、#49、#50、#51、#53。总路线见 [system-paper-spec.md](system-paper-spec.md)。Mega 侧配套实施草案位于该仓库的 `docs/spec/namespace-snapshot-spec.md`，细化 G01–G06 与 MG01–MG17；两仓 spec 已提交到工作分支。 +状态：Draft v0.4，2026-09-06。D1（完整 native + import 原子组合视图）、D2（显式 release 目录发布后不可变）与 D4（安全启用门槛）已获用户确认；D3 待确认（§12）。文中的 MUST 是目标协议要求，不代表现有实现。命名空间协议由 [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) 跟踪，本文细化 [#42](https://github.com/gitmono-dev/scorpiofs/issues/42)，约束 #43、#44、#49、#50、#51、#53。总路线见 [system-paper-spec.md](system-paper-spec.md)。Mega 侧配套实施草案位于该仓库的 `docs/spec/namespace-snapshot-spec.md`，细化 G01–G06 与 MG01–MG17；两仓共享且已验证的内容身份编码见 [namespace-manifest-v1](namespace-manifest-v1.md)，它不等于已实现实际挂载或原子发布。 当前实现进度：已增加严格 source identity、不可变 SourceReader 库层和 source-aware HTTP 客户端适配器，跨仓黄金向量、固定对象读取及本机 HTTP 测试见 [source-snapshot-v1.md](source-snapshot-v1.md)。尚未接入实际 Mega snapshot HTTP 服务、Dicfuse/Antares 挂载、lease/CAS 或工作区切换；现有挂载因此仍不具备本文承诺的版本隔离。完整 namespace 发布及所有写入者覆盖同样尚未完成。 @@ -349,7 +349,7 @@ M0 观测/benchmark 与 V0/V1 并行。不要等待全部观测实现才能开 | 决策 | 方案及确认状态 | 另一选择及成本 | | --- | --- | --- | | D1 全库版本边界 | **已确认（2026-09-06）**：Mega 原子发布 native root + 固定 import bindings 的组合 view，ScorpioFS 固定该 view 读取 | 未选：首期只做 source snapshot 并延期全库原子一致性；单 source 仅作中间工作包 | -| D2 版本号路径策略 | 显式标记为发布版本的目录首次发布后不可变，新内容使用新版本路径；不靠数字目录名推断；普通 import branch 仍可演进 | 可原地替换，但每次发布产生新绑定，旧对象必须按历史策略保留 | +| D2 版本号路径策略 | **已确认（2026-09-06）**：显式标记为 release 的目录首次发布后不可变，新内容使用新版本路径；不靠数字目录名推断；普通 import 开发 branch 仍可演进 | 未选：release 原地替换；所有写入口必须拒绝内容变更与 policy 降级绕过 | | D3 工作区更新体验 | 运行任务固定旧 view；新任务用新 view；现有 mount 暂停/检查后显式切换 | 要求透明运行中切换，必须扩展 handle/mmap/cwd/upper generation 协议 | **D4 已于 2026-09-06 获用户确认**：Mega snapshot 读 API 默认关闭，仅在显式配置 source/scope 读授权与对象保留策略后启用。配置缺失/无效或后端门槛未满足时，ScorpioFS 不得静默转用 legacy latest。Mega 当前开发期通用 guard 不能替代 source/path 授权；当前基础代码尚未开放新 HTTP 路由。 diff --git a/docs/spec/namespace-manifest-v1.md b/docs/spec/namespace-manifest-v1.md new file mode 100644 index 0000000..9aaea4a --- /dev/null +++ b/docs/spec/namespace-manifest-v1.md @@ -0,0 +1,81 @@ +# Namespace manifest identity v1 + +Status: shared codec implemented and tested, 2026-09-06. This is a content +identity contract, not a claim of publication, authorization, leases or FUSE +integration. Both repositories consume the same `namespace-v1.json` fixture; +the committed PowerShell 7 generator independently frames bytes and hashes with +.NET, without calling the Rust implementation. + +## Binding + +JSON has exactly `mount_path`, `source_snapshot`, `source_subpath` and `policy`. +The first is a canonical RepoPath; source_subpath is a canonical RelativePath +relative to the attested source scope. Their source-side composition must still +fit the 4096-byte absolute-path limit. Tree existence, source membership, +ancestor/descendant binding conflicts and release enforcement are publisher +checks, not proofs supplied by this structural codec. + +Canonical bytes are ASCII `mega.namespace-binding.v1` plus NUL, followed by: + +1. Mount path: u32 big-endian byte length, then UTF-8 bytes. +2. Full canonical SourceSnapshot bytes from source-snapshot-v1, framed by u32 + big-endian byte length. This is not JSON and not merely the source UUID. +3. Source subpath: u32 big-endian byte length, then UTF-8 bytes. +4. Policy u8: 1 = `mutable`, 2 = `immutable_release`. + +Policy is explicit and part of identity; it is never guessed from a numeric +directory name. **D2 is confirmed:** an explicitly marked release directory +cannot change content after its first publication; ordinary development +bindings may evolve. A codec that can encode both values does not itself enforce +this rule on writers. + +## View + +JSON has exactly `schema_version`, `instance_id`, `native`, `bindings_root`, +`overrides_root` and `materialization_policy`. Schema version must be integer 1. +Instance ID is a distinct non-nil canonical UUID type, not a source ID. +The native SourceSnapshot must have root scope `/`; the server must separately +attest that its source is the instance's native backend. + +Canonical bytes are ASCII `mega.namespace-view.v1` plus NUL, followed by: + +1. Schema version: u16 big-endian, exactly 1. +2. Instance UUID: u32 big-endian byte length, then canonical lowercase UUID text. +3. Full canonical native SourceSnapshot bytes, framed by u32 big-endian length. +4. Bindings root: 32 raw SHA-256 bytes, with no textual prefix. +5. Overrides presence: u8 0 for absent, or u8 1 followed by 32 raw digest bytes. +6. Materialization policy u8: 1 = `git_raw_v1`. + +`git_raw_v1` identifies raw Git projection without implicit LFS hydration or +submodule expansion. It is not permission to traverse arbitrary external +symlinks. Overrides are representable in the codec; the reader must explicitly +reject that capability until the override route semantics are implemented. +The absent root is distinct from a present empty-index root. + +No timestamp, actor, operation ID, publication sequence, parent view, floating +ref, lease or client generation is hashed into a view. A different commit with +the same tree changes provenance and therefore changes view_id. Re-publishing +identical content can reuse view_id while publication metadata remains separate. + +## Strict decoding and cross-repository evidence + +Every complete manifest is limited to 16384 bytes. Hash identity is `sha256:` +plus lowercase hex SHA-256 of the entire domain-separated canonical byte +sequence. Binary decoding rejects truncation, length overflow, trailing bytes, +unknown schema/policy/optional tags, invalid UTF-8 or paths and mismatched +domains. JSON rejects unknown fields and passes through the same structural +validation as constructors. Decoding bytes is not digest verification against a +requested ID; the content-store/read boundary must do that separately. + +Golden view without overrides: +`sha256:3c8632afb308bf562973b3af517ae5d0a27c05651f3f7511f91e16d7ad8f1231`. + +Golden mutable binding: +`sha256:adebe124b05761074c9460ed20426acf3023645e2bfa7e46b12239da68b14a88`. + +Both repositories pass five codec tests: independent binary/JSON vectors, +identity changes with provenance/routing/instance/policy, JSON rejection, +all-prefix truncation and malformed tags/lengths, and maximum-length paths. +Mega uses SHA-2 and ScorpioFS uses ring, while the oracle uses .NET SHA-256. +This closes the shared manifest-identity subtask, not the full G01–G06/V01–V18 +acceptance suite. diff --git a/src/snapshot/mod.rs b/src/snapshot/mod.rs index e46a709..9259646 100644 --- a/src/snapshot/mod.rs +++ b/src/snapshot/mod.rs @@ -4,3 +4,4 @@ pub mod backend; pub mod http; pub mod identity; +pub mod namespace; diff --git a/src/snapshot/namespace.rs b/src/snapshot/namespace.rs new file mode 100644 index 0000000..67162d4 --- /dev/null +++ b/src/snapshot/namespace.rs @@ -0,0 +1,322 @@ +//! Immutable namespace identity codec. Structural validity is not publication, +//! source authorization, scope attestation or an object-retention lease. + +use serde::{Deserialize, Serialize}; + +use super::identity::{ + IdentityError, ManifestDigest, ObjectFormat, ObjectId, RelativePath, RepoPath, SourceId, + SourceSnapshot, MAX_PATH_BYTES, +}; + +pub const MAX_MANIFEST_BYTES: usize = 16 * 1024; +const BINDING_DOMAIN: &[u8] = b"mega.namespace-binding.v1\0"; +const VIEW_DOMAIN: &[u8] = b"mega.namespace-view.v1\0"; + +/// Distinct from a source UUID even though both use the same UUID syntax. +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(transparent)] +pub struct InstanceId(SourceId); + +impl InstanceId { + pub fn new(value: impl Into) -> Result { + SourceId::new(value).map(Self) + } + pub fn as_str(&self) -> &str { + self.0.as_str() + } +} + +/// Explicit data, never inferred from a numeric directory name. Encoding both +/// policies does not choose the deployment's release-directory policy. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum BindingPolicy { + Mutable, + ImmutableRelease, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum MaterializationPolicy { + GitRawV1, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(try_from = "BindingFields", into = "BindingFields")] +pub struct NamespaceBinding(BindingFields); + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct BindingFields { + mount_path: RepoPath, + source_snapshot: SourceSnapshot, + source_subpath: RelativePath, + policy: BindingPolicy, +} + +impl TryFrom for NamespaceBinding { + type Error = IdentityError; + fn try_from(fields: BindingFields) -> Result { + // A binding may expose a subtree of an attested source. It must still + // have a representable absolute source path for membership proofs. + let scope = fields.source_snapshot.scope_path.as_str(); + let subpath = fields.source_subpath.as_str(); + let length = scope.len() + usize::from(scope != "/" && !subpath.is_empty()) + subpath.len(); + if length > MAX_PATH_BYTES { + return Err(IdentityError("binding source path exceeds v1 limit")); + } + Ok(Self(fields)) + } +} +impl From for BindingFields { + fn from(value: NamespaceBinding) -> Self { + value.0 + } +} + +impl NamespaceBinding { + pub fn new( + mount_path: RepoPath, + source_snapshot: SourceSnapshot, + source_subpath: RelativePath, + policy: BindingPolicy, + ) -> Result { + BindingFields { + mount_path, + source_snapshot, + source_subpath, + policy, + } + .try_into() + } + pub fn mount_path(&self) -> &RepoPath { + &self.0.mount_path + } + pub fn source_snapshot(&self) -> &SourceSnapshot { + &self.0.source_snapshot + } + pub fn source_subpath(&self) -> &RelativePath { + &self.0.source_subpath + } + pub fn policy(&self) -> BindingPolicy { + self.0.policy + } + + pub fn canonical_bytes(&self) -> Vec { + let mut out = BINDING_DOMAIN.to_vec(); + frame(&mut out, self.mount_path().as_str().as_bytes()); + frame(&mut out, &self.source_snapshot().canonical_bytes()); + frame(&mut out, self.source_subpath().as_str().as_bytes()); + out.push(match self.policy() { + BindingPolicy::Mutable => 1, + BindingPolicy::ImmutableRelease => 2, + }); + out + } + + pub fn from_canonical_bytes(bytes: &[u8]) -> Result { + let mut r = Reader::new(bytes, BINDING_DOMAIN)?; + let mount = RepoPath::new(r.text()?)?; + let source = read_source(r.field()?)?; + let subpath = RelativePath::new(r.text()?)?; + let policy = match r.byte()? { + 1 => BindingPolicy::Mutable, + 2 => BindingPolicy::ImmutableRelease, + _ => return Err(IdentityError("unknown binding policy")), + }; + r.finish()?; + Self::new(mount, source, subpath, policy) + } + + pub fn id(&self) -> ManifestDigest { + hash_bytes(&self.canonical_bytes()) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(try_from = "ViewFields", into = "ViewFields")] +pub struct NamespaceView(ViewFields); + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct ViewFields { + schema_version: u16, + instance_id: InstanceId, + native: SourceSnapshot, + bindings_root: ManifestDigest, + overrides_root: Option, + materialization_policy: MaterializationPolicy, +} + +impl TryFrom for NamespaceView { + type Error = IdentityError; + fn try_from(fields: ViewFields) -> Result { + if fields.schema_version != 1 { + return Err(IdentityError("unknown namespace view schema")); + } + if fields.native.scope_path.as_str() != "/" { + return Err(IdentityError( + "namespace native snapshot must cover root scope", + )); + } + Ok(Self(fields)) + } +} +impl From for ViewFields { + fn from(value: NamespaceView) -> Self { + value.0 + } +} + +impl NamespaceView { + pub fn new( + instance_id: InstanceId, + native: SourceSnapshot, + bindings_root: ManifestDigest, + overrides_root: Option, + materialization_policy: MaterializationPolicy, + ) -> Result { + ViewFields { + schema_version: 1, + instance_id, + native, + bindings_root, + overrides_root, + materialization_policy, + } + .try_into() + } + pub fn instance_id(&self) -> &InstanceId { + &self.0.instance_id + } + pub fn native(&self) -> &SourceSnapshot { + &self.0.native + } + pub fn bindings_root(&self) -> &ManifestDigest { + &self.0.bindings_root + } + pub fn overrides_root(&self) -> Option<&ManifestDigest> { + self.0.overrides_root.as_ref() + } + pub fn materialization_policy(&self) -> MaterializationPolicy { + self.0.materialization_policy + } + + pub fn canonical_bytes(&self) -> Vec { + let mut out = VIEW_DOMAIN.to_vec(); + out.extend_from_slice(&1u16.to_be_bytes()); + frame(&mut out, self.instance_id().as_str().as_bytes()); + frame(&mut out, &self.native().canonical_bytes()); + out.extend_from_slice(&raw_digest(self.bindings_root())); + match self.overrides_root() { + None => out.push(0), + Some(root) => { + out.push(1); + out.extend_from_slice(&raw_digest(root)); + } + } + out.push(match self.materialization_policy() { + MaterializationPolicy::GitRawV1 => 1, + }); + out + } + + pub fn from_canonical_bytes(bytes: &[u8]) -> Result { + let mut r = Reader::new(bytes, VIEW_DOMAIN)?; + if r.take(2)? != [0, 1] { + return Err(IdentityError("unknown namespace view schema")); + } + let instance = InstanceId::new(r.text()?)?; + let native = read_source(r.field()?)?; + let bindings = r.digest()?; + let overrides = match r.byte()? { + 0 => None, + 1 => Some(r.digest()?), + _ => return Err(IdentityError("invalid overrides presence tag")), + }; + let policy = match r.byte()? { + 1 => MaterializationPolicy::GitRawV1, + _ => return Err(IdentityError("unknown materialization policy")), + }; + r.finish()?; + Self::new(instance, native, bindings, overrides, policy) + } + + pub fn id(&self) -> ManifestDigest { + hash_bytes(&self.canonical_bytes()) + } +} + +fn frame(out: &mut Vec, value: &[u8]) { + out.extend_from_slice(&(value.len() as u32).to_be_bytes()); + out.extend_from_slice(value); +} +fn raw_digest(digest: &ManifestDigest) -> Vec { + hex::decode(&digest.as_str()[7..]).expect("validated digest") +} +fn hash_bytes(bytes: &[u8]) -> ManifestDigest { + let hex_digest = hex::encode(ring::digest::digest(&ring::digest::SHA256, bytes).as_ref()); + ManifestDigest::new(format!("sha256:{hex_digest}")).expect("SHA-256 digest") +} +struct Reader<'a>(&'a [u8]); +impl<'a> Reader<'a> { + fn new(bytes: &'a [u8], domain: &[u8]) -> Result { + if bytes.len() > MAX_MANIFEST_BYTES { + return Err(IdentityError("namespace manifest exceeds byte limit")); + } + bytes + .strip_prefix(domain) + .map(Self) + .ok_or(IdentityError("invalid manifest domain")) + } + fn take(&mut self, len: usize) -> Result<&'a [u8], IdentityError> { + if len > self.0.len() { + return Err(IdentityError("truncated manifest")); + } + let (head, tail) = self.0.split_at(len); + self.0 = tail; + Ok(head) + } + fn byte(&mut self) -> Result { + Ok(self.take(1)?[0]) + } + fn field(&mut self) -> Result<&'a [u8], IdentityError> { + let size = u32::from_be_bytes(self.take(4)?.try_into().expect("four bytes")) as usize; + self.take(size) + } + fn text(&mut self) -> Result<&'a str, IdentityError> { + std::str::from_utf8(self.field()?).map_err(|_| IdentityError("non-UTF-8 manifest field")) + } + fn digest(&mut self) -> Result { + ManifestDigest::new(format!("sha256:{}", hex::encode(self.take(32)?))) + } + fn finish(self) -> Result<(), IdentityError> { + if self.0.is_empty() { + Ok(()) + } else { + Err(IdentityError("trailing manifest bytes")) + } + } +} +fn read_source(bytes: &[u8]) -> Result { + let mut r = Reader::new(bytes, b"mega.source-snapshot.v1\0")?; + let source_id = SourceId::new(r.text()?)?; + let scope_path = RepoPath::new(r.text()?)?; + let object_format = match r.text()? { + "sha1" => ObjectFormat::Sha1, + _ => return Err(IdentityError("unknown source object format")), + }; + let commit_oid = ObjectId::new(r.text()?)?; + let root_tree_oid = ObjectId::new(r.text()?)?; + r.finish()?; + Ok(SourceSnapshot { + source_id, + scope_path, + object_format, + commit_oid, + root_tree_oid, + }) +} + +#[cfg(test)] +mod tests; diff --git a/src/snapshot/namespace/tests.rs b/src/snapshot/namespace/tests.rs new file mode 100644 index 0000000..f17a016 --- /dev/null +++ b/src/snapshot/namespace/tests.rs @@ -0,0 +1,210 @@ +use super::*; + +fn vectors() -> serde_json::Value { + serde_json::from_str(include_str!( + "../../../tests/fixtures/snapshot/namespace-v1.json" + )) + .unwrap() +} +fn binding() -> NamespaceBinding { + serde_json::from_value(vectors()["bindings"][0]["binding"].clone()).unwrap() +} +fn view() -> NamespaceView { + serde_json::from_value(vectors()["views"][0]["view"].clone()).unwrap() +} + +#[test] +fn snapshot_namespace_vectors_match_independent_dotnet_and_json_roundtrip() { + for vector in vectors()["bindings"].as_array().unwrap() { + let value: NamespaceBinding = serde_json::from_value(vector["binding"].clone()).unwrap(); + let bytes = hex::decode(vector["canonical_hex"].as_str().unwrap()).unwrap(); + assert_eq!(value.canonical_bytes(), bytes); + assert_eq!(value.id().as_str(), vector["digest"].as_str().unwrap()); + assert_eq!( + NamespaceBinding::from_canonical_bytes(&bytes).unwrap(), + value + ); + assert_eq!(serde_json::to_value(value).unwrap(), vector["binding"]); + } + for vector in vectors()["views"].as_array().unwrap() { + let value: NamespaceView = serde_json::from_value(vector["view"].clone()).unwrap(); + let bytes = hex::decode(vector["canonical_hex"].as_str().unwrap()).unwrap(); + assert_eq!(value.canonical_bytes(), bytes); + assert_eq!(value.id().as_str(), vector["digest"].as_str().unwrap()); + assert_eq!(NamespaceView::from_canonical_bytes(&bytes).unwrap(), value); + assert_eq!(serde_json::to_value(value).unwrap(), vector["view"]); + } +} + +#[test] +fn snapshot_namespace_provenance_routing_policy_and_instance_affect_identity() { + let base = view(); + let other_instance = NamespaceView::new( + InstanceId::new("99999999-9999-4999-8999-999999999999").unwrap(), + base.native().clone(), + base.bindings_root().clone(), + None, + base.materialization_policy(), + ) + .unwrap(); + assert_ne!(base.id(), other_instance.id()); + let mut native = base.native().clone(); + native.commit_oid = ObjectId::new("f".repeat(40)).unwrap(); + let other_commit = NamespaceView::new( + base.instance_id().clone(), + native, + base.bindings_root().clone(), + None, + base.materialization_policy(), + ) + .unwrap(); + assert_eq!( + base.native().root_tree_oid, + other_commit.native().root_tree_oid + ); + assert_ne!(base.id(), other_commit.id()); + let other_bindings = NamespaceView::new( + base.instance_id().clone(), + base.native().clone(), + hash_bytes(b"different routing"), + None, + base.materialization_policy(), + ) + .unwrap(); + assert_ne!(base.id(), other_bindings.id()); + let base_binding = binding(); + let other_policy = NamespaceBinding::new( + base_binding.mount_path().clone(), + base_binding.source_snapshot().clone(), + base_binding.source_subpath().clone(), + BindingPolicy::ImmutableRelease, + ) + .unwrap(); + assert_ne!(base_binding.id(), other_policy.id()); + let moved = NamespaceBinding::new( + RepoPath::new("/other").unwrap(), + base_binding.source_snapshot().clone(), + base_binding.source_subpath().clone(), + base_binding.policy(), + ) + .unwrap(); + assert_ne!(base_binding.id(), moved.id()); + let subpath = NamespaceBinding::new( + base_binding.mount_path().clone(), + base_binding.source_snapshot().clone(), + RelativePath::new("other").unwrap(), + base_binding.policy(), + ) + .unwrap(); + assert_ne!(base_binding.id(), subpath.id()); +} + +#[test] +fn snapshot_namespace_json_cannot_bypass_schema_scope_or_unknown_field_checks() { + let raw = vectors()["views"][0]["view"].clone(); + for (field, value) in [ + ("schema_version", serde_json::json!(2)), + ( + "instance_id", + serde_json::json!("00000000-0000-0000-0000-000000000000"), + ), + ( + "materialization_policy", + serde_json::json!("hydrate_everything"), + ), + ("lease", serde_json::json!("not identity")), + ("publication_seq", serde_json::json!(1)), + ] { + let mut changed = raw.clone(); + changed[field] = value; + assert!( + serde_json::from_value::(changed).is_err(), + "{field}" + ); + } + let mut scoped = raw; + scoped["native"]["scope_path"] = serde_json::json!("/child"); + assert!(serde_json::from_value::(scoped).is_err()); + let raw = vectors()["bindings"][0]["binding"].clone(); + for (field, value) in [ + ("mount_path", serde_json::json!("/deps//bad")), + ("source_subpath", serde_json::json!("../escape")), + ("policy", serde_json::json!("guess_from_path")), + ("ref_name", serde_json::json!("refs/heads/main")), + ] { + let mut changed = raw.clone(); + changed[field] = value; + assert!( + serde_json::from_value::(changed).is_err(), + "{field}" + ); + } +} + +#[test] +fn snapshot_namespace_codec_rejects_truncation_oversize_unknown_tags_and_domains() { + let binding = binding().canonical_bytes(); + let view = view().canonical_bytes(); + for end in 0..binding.len() { + assert!(NamespaceBinding::from_canonical_bytes(&binding[..end]).is_err()); + } + for end in 0..view.len() { + assert!(NamespaceView::from_canonical_bytes(&view[..end]).is_err()); + } + assert!(NamespaceBinding::from_canonical_bytes(&view).is_err()); + assert!(NamespaceView::from_canonical_bytes(&binding).is_err()); + for original in [&binding, &view] { + let mut extra = original.clone(); + extra.push(0); + assert!(NamespaceBinding::from_canonical_bytes(&extra).is_err()); + assert!(NamespaceView::from_canonical_bytes(&extra).is_err()); + } + let mut bad = binding.clone(); + *bad.last_mut().unwrap() = 0; + assert!(NamespaceBinding::from_canonical_bytes(&bad).is_err()); + let mut bad = view.clone(); + *bad.last_mut().unwrap() = 99; + assert!(NamespaceView::from_canonical_bytes(&bad).is_err()); + let mut bad = view.clone(); + bad[view.len() - 2] = 3; + assert!(NamespaceView::from_canonical_bytes(&bad).is_err()); + let mut bad = view; + bad[VIEW_DOMAIN.len() + 1] = 2; + assert!(NamespaceView::from_canonical_bytes(&bad).is_err()); + let mut bad = binding; + bad[BINDING_DOMAIN.len()..BINDING_DOMAIN.len() + 4].copy_from_slice(&u32::MAX.to_be_bytes()); + assert!(NamespaceBinding::from_canonical_bytes(&bad).is_err()); + let huge = vec![0; MAX_MANIFEST_BYTES + 1]; + assert!(NamespaceView::from_canonical_bytes(&huge).is_err()); + assert!(NamespaceBinding::from_canonical_bytes(&huge).is_err()); +} + +#[test] +fn snapshot_namespace_maximum_paths_fit_bounded_manifests_and_proofs() { + let path = format!("/{}", vec!["x".repeat(255); 16].join("/")); + assert_eq!(path.len(), MAX_PATH_BYTES); + let mut native = view().native().clone(); + native.scope_path = RepoPath::new(&path).unwrap(); + let maximal = NamespaceBinding::new( + RepoPath::new(&path).unwrap(), + native.clone(), + RelativePath::new("").unwrap(), + BindingPolicy::Mutable, + ) + .unwrap(); + assert!(maximal.canonical_bytes().len() < MAX_MANIFEST_BYTES); + assert_eq!( + NamespaceBinding::from_canonical_bytes(&maximal.canonical_bytes()).unwrap(), + maximal + ); + assert!(NamespaceBinding::new( + RepoPath::new("/deps").unwrap(), + native, + RelativePath::new("x").unwrap(), + BindingPolicy::Mutable, + ) + .is_err()); + let mut raw = serde_json::to_value(maximal).unwrap(); + raw["source_subpath"] = serde_json::json!("x"); + assert!(serde_json::from_value::(raw).is_err()); +} diff --git a/tests/fixtures/snapshot/namespace-v1-vectors.ps1 b/tests/fixtures/snapshot/namespace-v1-vectors.ps1 new file mode 100644 index 0000000..b68feb0 --- /dev/null +++ b/tests/fixtures/snapshot/namespace-v1-vectors.ps1 @@ -0,0 +1,69 @@ +# Independent .NET framing/SHA-256 oracle. Emits JSON only; never calls Rust. +# Run in PowerShell 7. Do not regenerate expected vectors using the codec under test. +$ErrorActionPreference = 'Stop' +function New-Bytes([string]$domain) { + $buffer = [System.Collections.Generic.List[byte]]::new() + $buffer.AddRange([System.Text.Encoding]::UTF8.GetBytes($domain)) + $buffer.Add(0) + return ,$buffer +} +function Add-Field($buffer, [byte[]]$bytes) { + $length = [System.BitConverter]::GetBytes([uint32]$bytes.Length) + if ([System.BitConverter]::IsLittleEndian) { [array]::Reverse($length) } + $buffer.AddRange($length) + $buffer.AddRange($bytes) +} +function Add-Text($buffer, [string]$value) { + Add-Field $buffer ([System.Text.Encoding]::UTF8.GetBytes($value)) +} +function Source-Bytes($source) { + $buffer = New-Bytes 'mega.source-snapshot.v1' + foreach ($field in @('source_id','scope_path','object_format','commit_oid','root_tree_oid')) { + Add-Text $buffer $source[$field] + } + return ,$buffer.ToArray() +} +function Digest([byte[]]$bytes) { + return 'sha256:' + [System.Convert]::ToHexString([System.Security.Cryptography.SHA256]::HashData($bytes)).ToLowerInvariant() +} +function Hex([byte[]]$bytes) { return [System.Convert]::ToHexString($bytes).ToLowerInvariant() } +$native = [ordered]@{ + source_id='11111111-1111-4111-8111-111111111111'; scope_path='/'; object_format='sha1' + commit_oid=('1' * 40); root_tree_oid='4b825dc642cb6eb9a060e54bf8d69288fbee4904' +} +$import = [ordered]@{ + source_id='33333333-3333-4333-8333-333333333333'; scope_path='/third-party/库+1'; object_format='sha1' + commit_oid=('a' * 40); root_tree_oid=('b' * 40) +} +$bindings = @() +foreach ($policy in @('mutable','immutable_release')) { + $binding = [ordered]@{ + mount_path='/deps/库+1'; source_snapshot=$import; source_subpath='src'; policy=$policy + } + $buffer = New-Bytes 'mega.namespace-binding.v1' + Add-Text $buffer $binding.mount_path + Add-Field $buffer (Source-Bytes $import) + Add-Text $buffer $binding.source_subpath + if ($policy -eq 'mutable') { $buffer.Add(1) } else { $buffer.Add(2) } + $bindings += [ordered]@{binding=$binding; canonical_hex=(Hex $buffer.ToArray()); digest=(Digest $buffer.ToArray())} +} +$empty = 'sha256:18946486089198dfa8eeb70fa90e04b137c579dc08ae1e6f8bceafc0d35ef677' +$views = @() +foreach ($overrides in @($null, ('sha256:' + ('c' * 64)))) { + $view = [ordered]@{ + schema_version=1; instance_id='22222222-2222-4222-8222-222222222222'; native=$native + bindings_root=$empty; overrides_root=$overrides; materialization_policy='git_raw_v1' + } + $buffer = New-Bytes 'mega.namespace-view.v1' + $buffer.AddRange([byte[]]@(0,1)) + Add-Text $buffer $view.instance_id + Add-Field $buffer (Source-Bytes $native) + $buffer.AddRange([System.Convert]::FromHexString($empty.Substring(7))) + if ($null -eq $overrides) { $buffer.Add(0) } else { + $buffer.Add(1) + $buffer.AddRange([System.Convert]::FromHexString($overrides.Substring(7))) + } + $buffer.Add(1) + $views += [ordered]@{view=$view; canonical_hex=(Hex $buffer.ToArray()); digest=(Digest $buffer.ToArray())} +} +[ordered]@{bindings=$bindings; views=$views} | ConvertTo-Json -Depth 10 diff --git a/tests/fixtures/snapshot/namespace-v1.json b/tests/fixtures/snapshot/namespace-v1.json new file mode 100644 index 0000000..0d84de5 --- /dev/null +++ b/tests/fixtures/snapshot/namespace-v1.json @@ -0,0 +1,74 @@ +{ + "bindings": [ + { + "binding": { + "mount_path": "/deps/库+1", + "source_snapshot": { + "source_id": "33333333-3333-4333-8333-333333333333", + "scope_path": "/third-party/库+1", + "object_format": "sha1", + "commit_oid": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "root_tree_oid": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "source_subpath": "src", + "policy": "mutable" + }, + "canonical_hex": "6d6567612e6e616d6573706163652d62696e64696e672e7631000000000b2f646570732fe5ba932b31000000b66d6567612e736f757263652d736e617073686f742e7631000000002433333333333333332d333333332d343333332d383333332d333333333333333333333333000000122f74686972642d70617274792fe5ba932b310000000473686131000000286161616161616161616161616161616161616161616161616161616161616161616161616161616100000028626262626262626262626262626262626262626262626262626262626262626262626262626262620000000373726301", + "digest": "sha256:adebe124b05761074c9460ed20426acf3023645e2bfa7e46b12239da68b14a88" + }, + { + "binding": { + "mount_path": "/deps/库+1", + "source_snapshot": { + "source_id": "33333333-3333-4333-8333-333333333333", + "scope_path": "/third-party/库+1", + "object_format": "sha1", + "commit_oid": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "root_tree_oid": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "source_subpath": "src", + "policy": "immutable_release" + }, + "canonical_hex": "6d6567612e6e616d6573706163652d62696e64696e672e7631000000000b2f646570732fe5ba932b31000000b66d6567612e736f757263652d736e617073686f742e7631000000002433333333333333332d333333332d343333332d383333332d333333333333333333333333000000122f74686972642d70617274792fe5ba932b310000000473686131000000286161616161616161616161616161616161616161616161616161616161616161616161616161616100000028626262626262626262626262626262626262626262626262626262626262626262626262626262620000000373726302", + "digest": "sha256:4f4263e0096171458b5bb5915497e20c74af2a39c01270ce591d629af63d5ae0" + } + ], + "views": [ + { + "view": { + "schema_version": 1, + "instance_id": "22222222-2222-4222-8222-222222222222", + "native": { + "source_id": "11111111-1111-4111-8111-111111111111", + "scope_path": "/", + "object_format": "sha1", + "commit_oid": "1111111111111111111111111111111111111111", + "root_tree_oid": "4b825dc642cb6eb9a060e54bf8d69288fbee4904" + }, + "bindings_root": "sha256:18946486089198dfa8eeb70fa90e04b137c579dc08ae1e6f8bceafc0d35ef677", + "overrides_root": null, + "materialization_policy": "git_raw_v1" + }, + "canonical_hex": "6d6567612e6e616d6573706163652d766965772e76310000010000002432323232323232322d323232322d343232322d383232322d323232323232323232323232000000a56d6567612e736f757263652d736e617073686f742e7631000000002431313131313131312d313131312d343131312d383131312d313131313131313131313131000000012f00000004736861310000002831313131313131313131313131313131313131313131313131313131313131313131313131313131000000283462383235646336343263623665623961303630653534626638643639323838666265653439303418946486089198dfa8eeb70fa90e04b137c579dc08ae1e6f8bceafc0d35ef6770001", + "digest": "sha256:3c8632afb308bf562973b3af517ae5d0a27c05651f3f7511f91e16d7ad8f1231" + }, + { + "view": { + "schema_version": 1, + "instance_id": "22222222-2222-4222-8222-222222222222", + "native": { + "source_id": "11111111-1111-4111-8111-111111111111", + "scope_path": "/", + "object_format": "sha1", + "commit_oid": "1111111111111111111111111111111111111111", + "root_tree_oid": "4b825dc642cb6eb9a060e54bf8d69288fbee4904" + }, + "bindings_root": "sha256:18946486089198dfa8eeb70fa90e04b137c579dc08ae1e6f8bceafc0d35ef677", + "overrides_root": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "materialization_policy": "git_raw_v1" + }, + "canonical_hex": "6d6567612e6e616d6573706163652d766965772e76310000010000002432323232323232322d323232322d343232322d383232322d323232323232323232323232000000a56d6567612e736f757263652d736e617073686f742e7631000000002431313131313131312d313131312d343131312d383131312d313131313131313131313131000000012f00000004736861310000002831313131313131313131313131313131313131313131313131313131313131313131313131313131000000283462383235646336343263623665623961303630653534626638643639323838666265653439303418946486089198dfa8eeb70fa90e04b137c579dc08ae1e6f8bceafc0d35ef67701cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc01", + "digest": "sha256:671360699c27ba18ae4de2b1288c90f5849b3e381170c33b83b370c42a660a4e" + } + ] +} From 5e19889e78b04f83a63091c546f49cc5fc5e0caf Mon Sep 17 00:00:00 2001 From: Luxian Date: Sun, 6 Sep 2026 12:45:33 +0800 Subject: [PATCH 6/8] docs(snapshot): add versioned readonly layer design --- docs/spec/monorepo-versioning.md | 2 + docs/versioned-readonly-layer-design.md | 133 ++++++++++++++++++++++++ 2 files changed, 135 insertions(+) create mode 100644 docs/versioned-readonly-layer-design.md diff --git a/docs/spec/monorepo-versioning.md b/docs/spec/monorepo-versioning.md index 8090573..5a75294 100644 --- a/docs/spec/monorepo-versioning.md +++ b/docs/spec/monorepo-versioning.md @@ -1,5 +1,7 @@ # Mega 命名空间版本与 Dicfuse 不可变视图 Spec +> 第一次阅读请从 [ScorpioFS 的版本化只读层设计](../versioned-readonly-layer-design.md) 开始；本文保留实现级协议和验收细节。 + 状态：Draft v0.4，2026-09-06。D1（完整 native + import 原子组合视图）、D2（显式 release 目录发布后不可变）与 D4（安全启用门槛）已获用户确认；D3 待确认（§12）。文中的 MUST 是目标协议要求，不代表现有实现。命名空间协议由 [#55](https://github.com/gitmono-dev/scorpiofs/issues/55) 跟踪，本文细化 [#42](https://github.com/gitmono-dev/scorpiofs/issues/42)，约束 #43、#44、#49、#50、#51、#53。总路线见 [system-paper-spec.md](system-paper-spec.md)。Mega 侧配套实施草案位于该仓库的 `docs/spec/namespace-snapshot-spec.md`，细化 G01–G06 与 MG01–MG17；两仓共享且已验证的内容身份编码见 [namespace-manifest-v1](namespace-manifest-v1.md)，它不等于已实现实际挂载或原子发布。 当前实现进度：已增加严格 source identity、不可变 SourceReader 库层和 source-aware HTTP 客户端适配器，跨仓黄金向量、固定对象读取及本机 HTTP 测试见 [source-snapshot-v1.md](source-snapshot-v1.md)。尚未接入实际 Mega snapshot HTTP 服务、Dicfuse/Antares 挂载、lease/CAS 或工作区切换；现有挂载因此仍不具备本文承诺的版本隔离。完整 namespace 发布及所有写入者覆盖同样尚未完成。 diff --git a/docs/versioned-readonly-layer-design.md b/docs/versioned-readonly-layer-design.md new file mode 100644 index 0000000..d7ec49a --- /dev/null +++ b/docs/versioned-readonly-layer-design.md @@ -0,0 +1,133 @@ +# ScorpioFS 的版本化只读层设计 + +这份文档是设计入口。ScorpioFS 的职责可以概括为： + +> 从 Mega 取得一个完整 monorepo 版本 ID，并让一个 workspace 在整个生命周期内始终读取这个版本。 + +ScorpioFS 不决定各 Git 仓库发布哪个 commit，也不重建 Mega 的历史挂载关系。Libra 继续负责 Git refs、index、commit 和 push；ScorpioFS 负责文件投影、惰性读取、缓存和 writable upper。 + +## 当前问题 + +现有 Dicfuse readonly 层按路径访问 Mega，但路径背后的 import 可能已经更新： + +~~~text +build 开始： + /project/app = 主仓 M1 + /third-party/lib = import I1 + +运行期间 Mega 更新 lib 到 I2 + +如果 readonly 层读取 latest： + 已缓存文件仍来自 I1 + 第一次访问的文件可能来自 I2 +~~~ + +这个 workspace 已经不是任何真实发布过的 monorepo 状态。 + +## 挂载时固定一个 Mega 版本 + +~~~text +workspace +└── Mega 版本 V100 + ├── 主仓 root M1 + ├── /third-party/lib → I1 + └── /toolchains/rust → R7 +~~~ + +此后所有路径查询和对象读取都使用 V100 中固定的来源和 commit。Mega 发布 V101 后，原 workspace 仍读取 V100。 + +V100 和 V101 可以同时挂载： + +~~~text +build-old → workspace A → V100 +build-new → workspace B → V101 +~~~ + +内容相同且授权兼容的缓存对象可以复用，但会改变路径解释的状态不能共享。 + +## 如何读取路径 + +1. 在固定版本的挂载表中判断路径属于主仓还是 import； +2. 取得该来源的固定 commit 和 root tree； +3. 从 root tree 沿相对路径遍历； +4. 向 Mega 请求精确 tree/blob； +5. 本地验证 Git 哈希后再缓存和返回。 + +不能查询当前 branch。Mega 返回错误时不能退回 legacy latest，也不能伪造空文件或空目录。 + +目录列表也固定在同一个版本。打开的 directory handle 和分页 cookie 不能跨版本使用，否则两页结果可能来自不同挂载表。 + +## Dicfuse、Antares 与 Libra + +~~~text +Libra + Git HEAD / index / commit / push + │ + ▼ +ScorpioFS / Antares + workspace 生命周期、readonly lower、writable upper + │ + ▼ +Dicfuse lower + 按固定 Mega 版本惰性读取 +~~~ + +Dicfuse lower 不需要自己实现 Git 版本管理。它只持有 Mega 版本，并保证 lookup、readdir、readlink 和 read 使用同一个版本。 + +Antares 创建 job mount 时固定版本。CL/upper 记录自己的 base 版本，避免未提交修改在不知情时换到另一套依赖。.libra 持久状态仍由 Libra 管理，不放进 ScorpioFS upper。 + +## 缓存隔离 + +对象缓存至少区分：访问域、对象类型、哈希算法和 OID。路径及目录元数据还要绑定版本 ID，因为同一路径在不同版本可能属于不同 source。 + +这样相同 blob 可以安全复用，V100 的目录和负查询不会污染 V101，私有 source 的 CAS 命中也不会造成越权。TTL 到期只触发验证或下载，不会改读新 branch。 + +固定远端版本不等于全部文件已下载。offline pin 只保护已经缓存且验证过的对象；完全离线需要显式导出并验证完整内容。 + +## workspace 更新 + +推荐的第一阶段语义是受控切换，最终决定仍待确认： + +~~~text +运行中的任务继续使用 V100 +新任务直接使用 V101 +已有 workspace 更新时： + 准备 V101 lower + 暂停 workload + 检查 dirty upper、打开句柄、mmap、cwd 和进行中的请求 + 持久化 PREPARED + 切换并验证 mount + 持久化 COMMITTED + 恢复 workload +~~~ + +如果不能证明 workload 已暂停，则返回 busy，让调用方创建新 mount 并重启任务。透明 live switch 还需定义旧 FD、mmap、inode、page cache 和新路径分别使用哪个版本，首版不能在这些语义未完成时宣称支持。 + +崩溃恢复只认持久化记录：只有 PREPARED 时恢复旧版本；已有 COMMITTED 时恢复新版本；日志损坏时进入可恢复失败状态，不猜测 latest。 + +upper 有修改时默认不切换。Libra 负责 commit 或形成 CL；ScorpioFS 只在收到可验证结果后清理仍匹配原记录的 delta。 + +## API 轮廓 + +现有 /antares 保持兼容；版本化 workspace 使用新 v2 请求，核心输入是 job ID、mount path 和 Mega view ID。相同 job ID 加相同请求是幂等重试；相同 job ID 加不同 view 返回冲突。 + +更新分为 plan 和 execute。plan 展示变化及 dirty/busy 检查；execute 带期望旧版本、期望 workspace 代次和幂等操作 ID，并在切换屏障内重新检查。 + +## 安全与失败 + +snapshot 功能默认关闭。只有 Mega 声明服务就绪，且 ScorpioFS 配置授权凭据、租约和允许的服务地址后才能使用。 + +无权限、版本过期、路径不存在、对象不可用、哈希损坏、workspace dirty/busy 和期望版本变化必须分别报告。任何情况都不能转换成 latest、空文件或空目录。凭据和 lease header 不进入日志，HTTP 不跟随到其他 origin 的重定向。 + +## 当前状态 + +已实现并测试：与 Mega 一致的版本清单编码、固定 source reader、tree/blob 哈希验证、路径/字节限制、可执行位和 symlink 语义，以及固定上下文的 HTTP 客户端。当前共有 24 个 snapshot 单元和 HTTP fixture 测试。 + +仍未完成，因此 PR 保持 Draft：获取真实完整版本及挂载表、接入 Dicfuse/Antares lower、按版本和访问域隔离缓存、lease/offline pin、更新日志和恢复、真实双版本 FUSE 挂载及 Libra 联调。 + +## 详细资料 + +- [完整客户端实施 Spec](spec/monorepo-versioning.md) +- [共享版本清单编码](spec/namespace-manifest-v1.md) +- [单 source 快照契约](spec/source-snapshot-v1.md) +- [系统论文路线](spec/system-paper-spec.md) From 4e508e23c2497c875810bd74fc1be396e3eb9d25 Mon Sep 17 00:00:00 2001 From: Luxian Date: Sun, 6 Sep 2026 15:20:59 +0800 Subject: [PATCH 7/8] docs(snapshot): specify batched transfer and chunk-reader integration --- docs/spec/monorepo-versioning.md | 2 ++ docs/versioned-readonly-layer-design.md | 12 ++++++++++++ 2 files changed, 14 insertions(+) diff --git a/docs/spec/monorepo-versioning.md b/docs/spec/monorepo-versioning.md index 5a75294..a3917ee 100644 --- a/docs/spec/monorepo-versioning.md +++ b/docs/spec/monorepo-versioning.md @@ -175,6 +175,8 @@ ScorpioFS 持有 active workspace、open handle、refresh prepare 的引用； ### 6.1 Mega：发现、固定与读取 +新增传输提案见 Mega 的 [文件传输协议 v1](https://github.com/gitmono-dev/mega/blob/codex/namespace-snapshot-spec/docs/spec/scorpiofs-transfer-v1.md)：准确目录元数据、按需 tar + zstd 小包、热点包和大文件分块。ScorpioFS 对接同一协议，在既有 SourceReader 之外增加批量调度及 range reader；完整对象接口不得返回部分内容。分块表的服务端信任方式已确认，协议本身尚未实现。 + | API | 请求关键字段 | 响应/约束 | | --- | --- | --- | | `GET /api/v1/snapshots/capabilities` | 服务端发现 | instance/schema/算法/路径编码、source/namespace readiness、retention 限制 | diff --git a/docs/versioned-readonly-layer-design.md b/docs/versioned-readonly-layer-design.md index d7ec49a..ba23dfa 100644 --- a/docs/versioned-readonly-layer-design.md +++ b/docs/versioned-readonly-layer-design.md @@ -57,6 +57,18 @@ build-new → workspace B → V101 目录列表也固定在同一个版本。打开的 directory handle 和分页 cookie 不能跨版本使用，否则两页结果可能来自不同挂载表。 +## 大量源码文件怎样下载 + +固定版本之后，ScorpioFS 先取得目录条目的准确大小和 blob ID，再检查本地内容缓存。并发读取缺少的小文件时，将它们组成一个请求，让 Mega 返回 tar + zstd 小包；热点目录也可选择服务端已经准备好的包。 + +例如 1,000 个 8 KiB 源码文件，全部缺失时可以按 128 个文件一包请求 8 个包。V101 只改了其中 10 个时，就只请求这 10 个，继续使用已验证的 990 个缓存对象。目录分页另计，包数不是构建加速比。 + +包内逐个对象进行 Git 哈希验证并写入对象缓存，不直接解包到 workspace。缓存以文件为单位，包布局变化不会让已有内容失效。一次只打开一个文件时立即读取；预取和批量聚合均有时间、内存和带宽上限。 + +超过阈值的大文件使用受认证 Mega 生成的分块表，按 read(offset, length) 获取并校验所需块。用户已确认信任 Mega 校验整文件后建立的 blob 与分块表关系；局部块校验与完整 Git blob 校验明确区分。 + +协议提案及两端职责见 Mega 的 [通俗设计](https://github.com/gitmono-dev/mega/blob/codex/namespace-snapshot-spec/docs/scorpiofs-transfer-design.md) 和 [传输 Spec](https://github.com/gitmono-dev/mega/blob/codex/namespace-snapshot-spec/docs/spec/scorpiofs-transfer-v1.md)。两仓实现使用同一契约和 fixtures，ScorpioFS 不维护第二套有分歧的服务端字段定义。本节是待实现设计，不表示当前 lower 已支持批量、分块或受控版本切换。 + ## Dicfuse、Antares 与 Libra ~~~text From 9ec0c6115d762b42256d0d5383bd80b9f35f91dc Mon Sep 17 00:00:00 2001 From: Luxian Date: Sun, 6 Sep 2026 15:50:47 +0800 Subject: [PATCH 8/8] docs(paper): align transfer evaluation with research constraints --- docs/spec/system-paper-spec.md | 8 +++++++- docs/versioned-readonly-layer-design.md | 4 +++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/spec/system-paper-spec.md b/docs/spec/system-paper-spec.md index 8d683a7..dba1b29 100644 --- a/docs/spec/system-paper-spec.md +++ b/docs/spec/system-paper-spec.md @@ -10,7 +10,7 @@ 版本协议是第一前置：Mega 原生目录、scope clone、import 仓库和聚合目录需要共同组成可重放视图。详见 [monorepo-versioning.md](monorepo-versioning.md)，该文件规定 source/view/generation、服务端接口、更新及错误语义。 -Mega 已在独立 WSL checkout 核对，基线 `c4c79bc195541a13ac1505b94728c81a8ff3d603`；配套服务端草案为 Mega `docs/spec/namespace-snapshot-spec.md`。新增 G01–G06 实施包、MG01–MG17 验收组，覆盖 scope proof、同库 publication txn、push/网页编辑写入口、初始回填与保留。两仓目前只有 spec 变更，不宣称协议或 MG 测试已经实现。 +Mega 已在独立 WSL checkout 核对，基线 `c4c79bc195541a13ac1505b94728c81a8ff3d603`；配套服务端草案为 Mega `docs/spec/namespace-snapshot-spec.md`。新增 G01–G06 实施包、MG01–MG17 验收组，覆盖 scope proof、同库 publication txn、push/网页编辑写入口、初始回填与保留。后续两仓已加入固定 source reader、共享 codec、服务端索引及事务存储基础；完整 writer/HTTP/FUSE 链路和论文实验仍未完成，具体证据见两仓 Draft PR。 约束： @@ -104,6 +104,10 @@ single-flight state：Absent→Queued→InFlight→Published/Failed；每个 wai 验收 F01：64 同对象冷读一次后端请求；F02：取消一个 waiter 不影响其余；F03：retry 和 publish 幂等；F04：队列和内存硬上限；F05：noisy neighbor 下有进展及 demand 延迟报告。用可确定性 fake backend 控制完成顺序与失败点。 +传输子课题使用 Mega 的 [研究设计](https://github.com/gitmono-dev/mega/blob/codex/namespace-snapshot-spec/docs/spec/scorpiofs-transfer-research.md) 和 [协议 Spec](https://github.com/gitmono-dev/mega/blob/codex/namespace-snapshot-spec/docs/spec/scorpiofs-transfer-v1.md)。它补充三类动作：并发单对象、精确缺失包、服务端缓存包。先以固定策略建立实测基线，再决定是否实现按缓存和成本选择的算法；不将 tar.zstd、CAS 或固定包大小本身作为论文创新。 + +H1/H2/H3 分别检验传输选择、跨版本复用和多 workspace 共享。每项都要求正确性及成本分解；REAPI 批量 CAS、工作集聚合和版本化惰性文件系统是必须核对的先例。未运行实际实现时只做机制比较，不宣称胜过原系统。 + ### 5.2 A/B/C 架构 bakeoff 三个方案定义见 [版本 spec §9](monorepo-versioning.md#9-架构-adr先验证再选择共享挂载方案)。先在 C 上提供正确性基线；A/B 的首轮原型时间箱暂定各 3–5 人日（工程估算，不是排期承诺）。到期产出 ADR：可以进入生产、继续调查或淘汰；没有功能/隔离/恢复证据不扩展为生产重构。 @@ -132,6 +136,8 @@ workload 至少覆盖 Buck2、Bazel、Cargo、CMake/Ninja、百万文件 sparse 每个 claim 对应 experiment ID、原始文件、聚合脚本和图表；预注册 outlier/失败处理，报告绝对值、相对值、median/tail/CI 和失败率。agent 同时报告任务成功率。存储报告 resident/cached/upper/object logical/physical，root-trie 创建成本和服务器租约/GC 成本单列。 +传输实验额外控制 Mega 包缓存与客户端对象/kernel cache 的冷热；统计构包 CPU、后端 IO、首次对象完成和整个构建时间。纯按需、执行前已有提示、完整未来 trace 的 offline oracle 分开；按项目及时间切分调优/测试数据，不能用本次测试访问记录预先生成自己的包。既有 1,000 文件/8 包数字只适用于需求已知且可按 128 个聚合的示例，不是在线访问保证。 + artifact 包含一键 smoke、fake backend、固定 namespace fixtures、环境 manifest、故障矩阵、raw JSONL/CSV 和画图脚本。需要 FUSE/mount namespace 的测试在具备条件的 runner 中执行，普通 CI 运行无 FUSE 的协议/schema/调度/恢复模型测试。 ## 7. 第一轮可执行拆分与验证 diff --git a/docs/versioned-readonly-layer-design.md b/docs/versioned-readonly-layer-design.md index ba23dfa..95012ed 100644 --- a/docs/versioned-readonly-layer-design.md +++ b/docs/versioned-readonly-layer-design.md @@ -61,7 +61,7 @@ build-new → workspace B → V101 固定版本之后，ScorpioFS 先取得目录条目的准确大小和 blob ID，再检查本地内容缓存。并发读取缺少的小文件时，将它们组成一个请求，让 Mega 返回 tar + zstd 小包；热点目录也可选择服务端已经准备好的包。 -例如 1,000 个 8 KiB 源码文件，全部缺失时可以按 128 个文件一包请求 8 个包。V101 只改了其中 10 个时，就只请求这 10 个，继续使用已验证的 990 个缓存对象。目录分页另计，包数不是构建加速比。 +例如 1,000 个 8 KiB 源码文件，全部缺失且需求已知/足够并发时，可以按 128 个文件一包请求 8 个包。纯按需 FUSE 可能逐个暴露需求，不保证这个包数。V101 只改了其中 10 个时，就只请求这 10 个，继续使用已验证的 990 个缓存对象。目录分页另计，包数不是构建加速比。 包内逐个对象进行 Git 哈希验证并写入对象缓存，不直接解包到 workspace。缓存以文件为单位，包布局变化不会让已有内容失效。一次只打开一个文件时立即读取；预取和批量聚合均有时间、内存和带宽上限。 @@ -69,6 +69,8 @@ build-new → workspace B → V101 协议提案及两端职责见 Mega 的 [通俗设计](https://github.com/gitmono-dev/mega/blob/codex/namespace-snapshot-spec/docs/scorpiofs-transfer-design.md) 和 [传输 Spec](https://github.com/gitmono-dev/mega/blob/codex/namespace-snapshot-spec/docs/spec/scorpiofs-transfer-v1.md)。两仓实现使用同一契约和 fixtures，ScorpioFS 不维护第二套有分歧的服务端字段定义。本节是待实现设计，不表示当前 lower 已支持批量、分块或受控版本切换。 +论文需要把并发单文件、固定批量和按缓存/成本选择的策略公平对比。tar.zstd、CAS 或按需加载本身不作为新颖性结论；正确性、服务端构包成本、真实构建时间和失败区间共同验证。详见 Mega 的 [研究设计](https://github.com/gitmono-dev/mega/blob/codex/namespace-snapshot-spec/docs/spec/scorpiofs-transfer-research.md)。 + ## Dicfuse、Antares 与 Libra ~~~text