diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index ce867515d..a56f0d35d 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -113,7 +113,7 @@ compatibility guard is pinned by `compat_global_config_schema_future`. | sparse-view | intentionally-different | Read-only sparse VIEW filter (lore.md 2.2, Libra-only extension) — the NON-declined complement of git sparse-checkout (the materializing `sparse-checkout` command and `clone --sparse` stay declined, D10). `sparse-view set|add|list|enable|disable|clear|status` stores an allowlist of gitignore-syntax include patterns (`!pat` carves holes; allowlist last-match-wins, no ancestor-dominance) that scope what `ls-files` and the WORKING-TREE `diff` DISPLAY. It NEVER mutates the working tree, writes no skip-worktree bits, and NEVER filters the changes-to-be-committed set: `status` content is unfiltered (only a one-line advisory) and `diff --staged` / `diff A..B` are unfiltered, so nothing ever hides what a commit will record. Unmerged/conflicted entries are always shown. Patterns in the `sparse_view` SQLite table (owner `internal::sparse`), toggle in the per-worktree `sparse_view_meta` projection (the scope-less config_kv `sparse.enabled` key was retired by migration 2026072304); worktree-scoped since W1 (§C.4.1.1 — patterns and the toggle are per-worktree facts, legacy state adopts to main only absent linked worktrees, else the migration fails closed); disabled/empty = no-op (byte-identical output). `--json` on list/status. Deferred: cone mode, materialization | | alternates | intentionally-different | Object alternates (lore.md 2.3, Libra extension) — borrow objects from a shared/parent object store instead of copying. `alternates add|list|remove|prune` registers an object dir in the git-standard `objects/info/alternates` file (`prune` is the base-side retirement of borrower registrations whose repository is gone — never automatic, because an absent path is indistinguishable from an unmounted one); reads resolve through the chain (transitive, cycle-safe, depth-capped) on a LOCAL miss, FULL-BYTE OID-verified before a borrowed hit is returned (a tampered alternate can't poison a read). `exist` consults alternates so a borrowed-but-present object is not treated as missing. DELETION SAFETY (airtight): registering a base ALSO records this repo as a BORROWER (`objects/info/borrowers`); while any live borrower exists the base's `gc` and `cache evict` REFUSE to prune loose objects — a shared base can never delete a borrowed object. Guards: refuses a self-reference, a base with a different `core.objectformat`, and a TIERED (s3/r2) base (a local alternate can't reach its remote tier). `file obliterate` refuses a borrow-only object (never reaches into a parent's store); `fsck` reports a dangling alternate. Deferred: `clone --reference` copy-avoidance (needs fetch have-negotiation — the flags stay accepted no-ops), `--dissociate`, and 2.11 default shared-store | | deps | intentionally-different | File dependency graph (lore.md 3.1, Libra extension — Git has no file-dependency concept). `deps add|rm|list|why|tree` declares/queries typed, VERSIONED per-file edges `(from -> to, kind)`. Edges are stored per-commit under the reserved notes ref `refs/notes/deps` (owner `internal::deps::DependencyStore`, the sole reader/writer — mirrors the `refs/notes/metadata` pattern; no new SQLite table). Queries: direct deps / dependents (`--reverse`), transitive closure (`tree`, cycle-safe iterative BFS with a `--depth-limit`), and `why` (shortest dependency path). Paths are repo-relative, normalized (`./` stripped, `\`→`/`), and reject absolute / `..`-escape / empty. `--revision` targets a commit (default HEAD); `--json` on reads; add is idempotent; a fresh/empty graph reads empty (absence-tolerant). The `transitive_closure` API is the reusable seam for 3.2 (dependency-filtered clone/sync) and 3.3 (hydrating VFS). Cross-machine edge travel is delivered by 3.2: `fetch`/`pull --notes` imports `refs/notes/deps` from a LOCAL Libra source over a dedicated side-channel (a deps note is a loose blob + a SQLite `notes` row, not a commit-reachable object, so it does not ride the pack; import union-merges and re-validates every endpoint, and is per-note fault-tolerant), default OFF (Git parity). Deferred: network / foreign-Git / push-side notes travel (D17, see [_compatibility.md](docs/development/commands/_compatibility.md)), carry-forward onto new commits, rename-following, and automatic dependency inference (v1 edges are author-declared) | -| media | intentionally-different | FastCDC LFS media chunking CLIENT (lore.md §6, **feature-gated `fastcdc`, default OFF** — Libra media extension, Git has no equivalent; absent from the default binary). The honest v1 client substrate with NO live media server: `media chunk` content-defines chunks of a file (in-tree deterministic gear-hash FastCDC, frozen `fastcdc-v1` params) and builds a versioned manifest; `--store` persists chunks + manifest to a private `.libra/media/` store that is a physical SIBLING of `objects/` (a chunk is NEVER a Git object ID, §6.2). `media inspect` validates a manifest; `media verify` reassembles from the store and verifies the full `media_oid` before publishing (never a corrupt file). `media probe` capability-probes a remote (`libra/media/v1/capabilities`, §0.2 backoff) and reports the §6.4 transfer decision — chunked vs a SAFE standard-LFS fallback (no endpoint / disabled / incompatible / unknown-higher-version / 5xx-after-backoff → standard LFS; server-refuses-fallback + no-local-fallback → BLOCK, never a chunk-only half-write). `media_oid` is ALWAYS SHA-256 of the full content (independent of `core.objectformat`), byte-identical to a standard LFS pointer OID. `--json` on all. DEFERRED / FROZEN (the Libra-aware media SERVER, lore.md §6.5–6.8): real cross-machine chunked upload/download, capability + chunk + manifest-finalize endpoints, the Pending→Finalized lifecycle + GC/fsck/heal, every §6.7 anti-side-channel guarantee, and chunk-only repo policy — so against every reachable remote today the probe falls back to standard Git LFS | +| media | intentionally-different | Feature-gated FastCDC extension (`fastcdc`, default OFF): chunk, inspect, verify, probe; versioned manifests and raw SHA-256 chunks outside the Git object graph. Feature-enabled LFS transfers use Mega's authenticated user/repository-scoped media protocol for missing-chunk uploads, verified finalize with full standard-LFS fallback, and cached-chunk downloads with atomic destination replacement. Frozen in-tree `fastcdc-v1` parameters; `media_oid` is always full-content SHA-256. Other remotes use standard LFS; `lfs.fastcdc=false` disables the extension. Shared repository ACLs, server orphan GC/quota/fsck/heal/obliteration and range hydration remain deferred. See `docs/commands/media.md`. | | hydrate | intentionally-different | On-demand whole-object hydration (lore.md 3.3, Libra extension — the honest v1 of Lore's "hydrating VFS": an EXPLICIT `libra hydrate ...` command, NOT a transparent FUSE-on-access filesystem, which stays a `worktree-fuse` follow-up). Materializes a path's content (and, by default, its transitive forward deps via 3.1's closure — `--no-deps`/`--depth-limit` bound it) into the working tree, resolving each blob local → alternate (2.3) → remote durable tier; the read policy is honored (`--offline`/`--local` refuse a remote fetch). FAILURE-RECOVERY contract: each blob is OID-verified on a borrowed/remote hit (`--verify` also re-hashes the local path, healing on mismatch) and published via an atomic temp-write + rename, so a hydration that fails for ANY reason (missing object, unreachable remote, verify mismatch, interruption) leaves the pre-existing file UNTOUCHED — never a truncated/half-written file. An active sparse view (2.2) gates the FULL set (roots AND deps); out-of-view paths are refused unless `--ignore-sparse`. Already-present (byte-identical) content is a no-op skip; `--dry-run` writes nothing; `--fail-fast`/`--json`. Whole-object only (NO FastCDC byte-range). Cross-machine dep expansion now works once the graph is fetched via 3.2's `fetch`/`pull --notes` (local Libra source; network deferred, D17). Deferred: LFS-pointer blobs (their download is not yet atomic — skipped cleanly), symlink/gitlink entries, and transparent FUSE on-access hydration | | logfile | intentionally-different | Diagnostic for the tracing log-file sink (`logfile info`): reports the env-resolved path, rolling strategy (`LIBRA_LOG_ROTATION`), filter, and size; also enables `tracing-appender` time-rolled logs. Git has no equivalent (mirrors Lore's `logfile`); needs no repository. `--json`/`--machine` → `{ enabled, file, rotation, filter, size_bytes, file_count }`. Rotation splits logs by time (each file bounded); it does not prune old files, so total disk use needs external retention. Exit 0 | | completions | intentionally-different | Shell completion generator (`bash`/`zsh`/`fish`/`powershell`/`elvish`) built from the live clap command tree; Git ships completions via `contrib/completion` rather than a `git completions` subcommand. Prints the script to stdout (`--json`/`--machine` wraps it as `{ shell, script }`); needs no repository. Exit 0 / 129 (unknown or missing shell, Git-style usage error) | diff --git a/Cargo.toml b/Cargo.toml index 7c64bb01a..92232afb5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -20,7 +20,7 @@ test-live-cloud = [] # L3: tests hitting real D1/R2 endpoints test-provider = [] # deterministic hidden provider for local Code UI automation tests test-live-agent = [] # plan-20260713 live agent gate: real local claude/codex/opencode CLI data (dev machine acceptance; env LIBRA_RUN_LIVE_AGENT_GATE=1) test-upgrade = [] # plan-20260714 §A.11: auto-upgrade test hooks (trust-root/endpoint injection; requires LIBRA_TEST=1 at runtime; release workflow must NEVER enable this feature) -fastcdc = [] # lore.md §6: default-OFF FastCDC LFS media chunking client substrate (no new deps; reuses ring sha256; server protocol frozen) +fastcdc = [] # default-OFF FastCDC LFS transport (in-tree; reuses ring SHA-256) subagent-scaffold = [] # CEX-S2-10 schema-only scaffold (Step 2 sub-agent contracts; gated on CP-4 in production — see docs/development/tracing/agent.md "Step 2 audit closure") otlp = ["dep:opentelemetry", "dep:opentelemetry_sdk", "dep:opentelemetry-otlp", "dep:tracing-opentelemetry"] # lore.md 1.7: OTLP trace export (default binary unaffected) keyring = ["dep:keyring"] # lore.md 2.7: OS-keyring auth backend (release builds enable it; default dev builds unaffected) diff --git a/docs/commands/lfs.md b/docs/commands/lfs.md index 72f9368c9..1c6c73b51 100644 --- a/docs/commands/lfs.md +++ b/docs/commands/lfs.md @@ -28,6 +28,25 @@ lock API. Unlike Git, which requires a separate `git-lfs` extension installed as a smudge/clean filter, Libra integrates LFS natively. The LFS client, pointer file parsing, and attribute management are built into the `libra` binary. No additional installation or filter configuration is needed. +## Optional FastCDC transfers + +Build both Libra and Mega (`mono`) with `--features fastcdc` to enable the optional +media transport, and save a host-scoped Mono Bearer access token with +`libra auth login`. Both builds default to feature OFF. The LFS URL retains the +repository path (`.git/info/lfs`); capability discovery uses +`libra/media/v1/capabilities` below that URL. Set `libra config lfs.fastcdc false` +to disable the extension in a repository. + +Host-only HTTP remotes retain the legacy root LFS endpoints (for example, +`http://localhost:8000/locks`). Use a repository path for scoped FastCDC transfers. + +Normal LFS uploads send missing chunks and finalize a verified full object; +downloads reuse verified local chunks. Standard LFS pointers remain unchanged, +and unsupported remotes or missing manifests use full-object LFS. Mega isolates +the extension by authenticated user and repository path; this is not a complete +repository ACL implementation. See [`libra media`](media.md) for the transfer +flow and current retention, quota, and maintenance limitations. + ## Options `libra lfs` has no top-level options. All functionality is accessed through subcommands documented below. diff --git a/docs/commands/media.md b/docs/commands/media.md index b0d3c64c9..bbc1d468b 100644 --- a/docs/commands/media.md +++ b/docs/commands/media.md @@ -26,22 +26,80 @@ OID. ## Safe fallback -`media probe` reports one of: `chunked (fastcdc-v1)` (a fully compatible -Libra-aware media server), `standard-lfs (fallback)` with a reason (no capability -endpoint, disabled by server, incompatible algorithm, disabled by repo policy, -unknown higher version, or a server error after backoff), or `blocked` (the -server keeps no standard fallback object AND no local complete object exists — a -chunk-only upload is refused rather than silently produced). Against every -reachable remote today — none of which run the (frozen) Libra media server — the -decision is a standard Git LFS fallback. +`media probe` reports the remote's capabilities: `chunked (fastcdc-v1)` or +`standard-lfs (fallback)` with a reason such as no capability endpoint, disabled +server support, incompatible algorithm, insufficient required capabilities, +unknown protocol version, or a server error after backoff. It assumes the +repository permits chunking and a complete local fallback object is available; +it does **not** read `lfs.fastcdc` and does not report `blocked` under these +assumptions. A `chunked` probe result therefore does not prove that transfers are +enabled in this repository. + +Actual LFS transfers also apply `lfs.fastcdc` and require the server to retain a +complete standard-LFS fallback and accept manifests. Chunk-only advertisements +use basic LFS instead. Mega built with `--features fastcdc` implements the +authenticated extension; other remotes retain the standard Git LFS fallback. + +## Live LFS transfers with Mega + +Build Libra with `cargo build --features fastcdc` and, in the Mega repository, +build/start the HTTP server with +`cargo run -p mono --features fastcdc -- service http` using its normal server +configuration. Both builds default to feature OFF. The `libra` commands below +must use the feature-built binary (`target/debug/libra`, or `libra.exe` on +Windows); compiling does not replace a separately installed binary on PATH. + +Obtain a **Mono-issued access token** through Mega's existing authenticated +token-creation flow (`POST /api/v1/user/token/generate`). `libra auth login` +only stores that token locally; it does not issue a Mega token. A GitHub PAT or +browser session cookie is not a substitute for the Mono access token. + +For a local Mega HTTP server on port 8000, run in the Libra repository: + +```bash +libra config remote.origin.url http://localhost:8000/project/demo.git +libra auth login --host http://localhost:8000 +# Paste the Mono access token at the hidden prompt. +libra auth status --host http://localhost:8000 +libra config lfs.fastcdc true +libra media probe --remote origin +``` + +After compiling the feature, an unset `lfs.fastcdc` permits automatic negotiation; +`true` explicitly enables it and `false` disables it in that repository. The +stored token must match the remote's **host and port**. Use HTTPS for non-loopback +servers (for example `--host https://mega.example.com:8443`); HTTP token attachment +is allowed only for loopback. Pass only the origin to `--host`, without the +repository path, and do not put tokens in URLs. For scripts, feed the token on +stdin with `--with-token`; see [`libra auth`](auth.md). + +Keep the repository URL in `origin`. The LFS client preserves +`.git/info/lfs`; capability discovery appends `libra/media/v1/capabilities` +to that LFS URL. The Bearer header is attached automatically from the stored token. + +Normal LFS push/upload now prepares a versioned manifest, uploads only missing +chunks, and finalizes. Mega verifies chunk hashes, full SHA-256 and the frozen +FastCDC boundaries, writes the complete standard-LFS object, then publishes the +manifest. Repeating push resumes from missing chunks. Downloads use finalized +manifests, reuse verified local chunks, and atomically publish only verified full +content. Invalid manifests or corrupted remote chunks are errors and preserve the +existing destination. No manifest, unsupported capabilities or disabled feature +means standard full-object LFS. Objects exceeding the negotiated manifest/chunk +count limits also use basic upload before any manifest is sent. Chunk-only uploads +are not supported. Outside a Libra repository, the public LFS download client uses +basic LFS instead of creating a repository cache. + +Mega's initial extension isolates chunks by authenticated user and repository; +another user's data is fetched through the standard full-object fallback. It +requires Bearer access tokens and does not introduce a public chunk-hash API. +Manifests are limited to 10 MiB / 8192 chunks and chunks to 8 MiB. This is an +opt-in transport; deployments need explicit retention and quota planning. ## Deferred -The Libra-aware media **server** (real cross-machine chunked upload/download, -capability + chunk + manifest-finalize endpoints, the manifest lifecycle, -GC/fsck/heal, and every anti-side-channel guarantee) is frozen in lore.md §6.5–6.8 -and not part of this client v1. Chunk-only repo policy (dropping the standard LFS -fallback object) and range-based hydration are also deferred. +Shared repository ACLs, automatic orphan GC, quota accounting, server fsck/heal, +obliteration, chunk-only policy and byte-range hydration remain deferred. The +current implementation does not claim completion of all Lore §6.5–6.8 guarantees. ## Examples diff --git a/docs/commands/zh-CN/media.md b/docs/commands/zh-CN/media.md index 7d84c30c2..6015a2159 100644 --- a/docs/commands/zh-CN/media.md +++ b/docs/commands/zh-CN/media.md @@ -1,8 +1,8 @@ # `libra media` -FastCDC LFS media chunking client（lore.md §6）— 一个 **feature-gated** Libra 扩展（`fastcdc`，只编译进带 `--features fastcdc` 的构建；**默认二进制中不存在**）。它对媒体文件做内容定义分块，构建 versioned manifest，把 chunks 存入私有本地 store，重组并验证它们，并与远端协商 chunked-LFS 能力，在不支持时安全回退到标准 Git LFS。 +FastCDC LFS 媒体分块客户端（lore.md §6），是受 `fastcdc` 功能开关控制的 Libra 扩展，只有使用 `--features fastcdc` 构建才会编译，**默认二进制中不存在**。它按内容为媒体文件分块，构建带版本的 manifest，将块存入私有本地存储，重组并验证文件，并与远端协商分块 LFS 能力；远端不支持时回退到标准 Git LFS。 -`media` 是 Libra-only 扩展（`intentionally-different`）：Git 没有 media chunking 概念。Git 对象图永不被触碰 — chunk 永远不是 Git object ID，chunks/manifests 存放在私有 `.libra/media/` store 中，它是 `objects/` 的 sibling。`media_oid` 始终是完整文件的 SHA-256（独立于 `core.objectformat`），与标准 LFS pointer OID 字节一致。 +`media` 是 Libra 专有扩展（`intentionally-different`）：Git 没有媒体分块概念。它不修改 Git 对象图，chunk hash 不是 Git object ID；块和 manifest 存放在与 `objects/` 同级的私有 `.libra/media/` 中。`media_oid` 始终是完整文件的 SHA-256，独立于 `core.objectformat`，与标准 LFS pointer OID 一致。 ## 子命令 @@ -16,11 +16,40 @@ FastCDC LFS media chunking client（lore.md §6）— 一个 **feature-gated** L ## 安全回退 -`media probe` 报告以下之一：`chunked (fastcdc-v1)`（完全兼容的 Libra-aware media server）、`standard-lfs (fallback)` 并附带原因（无 capability endpoint、服务端禁用、算法不兼容、仓库策略禁用、未知更高版本，或 backoff 后的服务端错误），或 `blocked`（服务端没有保留标准 fallback 对象且本地也没有完整对象 — 拒绝生成 chunk-only upload，而不是静默生成）。对当前所有可达远端 — 都没有运行（冻结的）Libra media server — 决策都是标准 Git LFS fallback。 +`media probe` 只报告远端能力：`chunked (fastcdc-v1)`，或 `standard-lfs (fallback)` 并附带原因，例如没有能力端点、服务端禁用、算法不兼容、所需能力不足、协议版本不兼容或退避后的服务端错误。它假定仓库允许分块且本地存在完整 fallback，**不会读取 `lfs.fastcdc`**，在这些假定下也不会报告 `blocked`。因此，probe 输出 `chunked` 不等于当前仓库已经启用实际分块传输。 + +实际 LFS 传输还会检查 `lfs.fastcdc`，并要求服务端保留标准完整对象、允许 manifest。仅提供 chunk-only 的远端回退 basic LFS。以 `--features fastcdc` 构建的 Mega 实现了需要认证的扩展；其他远端继续使用标准 Git LFS。 + +## 与 Mega 联动传输 + +在 Libra 源码仓库执行 `cargo build --features fastcdc`；在 Mega 仓库按正常服务配置执行 `cargo run -p mono --features fastcdc -- service http` 构建并启动 HTTP 服务。两端默认构建均关闭该 feature。以下 `libra` 命令必须使用刚构建的二进制（`target/debug/libra`，Windows 为 `libra.exe`）；编译不会替换 PATH 中另行安装的版本。 + +先通过 Mega 现有的已登录用户令牌签发流程（`POST /api/v1/user/token/generate`）取得 **Mono 访问令牌**。`libra auth login` 只在本地保存已有令牌，不会替 Mega 签发令牌；GitHub PAT 或浏览器会话 cookie 不能代替 Mono access token。 + +以本机 8000 端口的 Mega HTTP 服务为例，在 Libra 仓库中执行： + +```bash +libra config remote.origin.url http://localhost:8000/project/demo.git +libra auth login --host http://localhost:8000 +# 在隐藏提示中粘贴 Mono 访问令牌。 +libra auth status --host http://localhost:8000 +libra config lfs.fastcdc true +libra media probe --remote origin +``` + +编入 feature 后，未设置 `lfs.fastcdc` 时默认允许自动协商；`true` 显式启用，`false` 在该仓库禁用传输扩展。令牌绑定的**主机和端口**必须与远端一致。非 loopback 服务必须使用 HTTPS，例如 `--host https://mega.example.com:8443`；HTTP 仅允许为 loopback 附加令牌。`--host` 只传 origin，不带仓库路径，不要将令牌放进 URL。脚本通过 `--with-token` 从 stdin 读取令牌，详见 [`libra auth`](../auth.md)。 + +`origin` 保留仓库 URL；LFS 客户端使用 `.git/info/lfs`，在该地址后追加 `libra/media/v1/capabilities` 探测能力，并自动将本地存储的令牌附加为 Bearer header。 + +正常 LFS push/upload 会准备 manifest、查询缺块、只上传缺失块，再请求 finalize。Mega 校验块的 SHA-256、完整文件 SHA-256 和冻结的 FastCDC 分块边界，保存标准 LFS 完整对象后才发布 manifest。重新 push 会再次查询缺块并续传。下载只使用已 finalize 的 manifest，复用校验通过的本地块，并在完整文件校验成功后原子替换目标。远端清单或块损坏会报错，保留已有目标文件；没有 manifest、能力不兼容或功能被禁用时使用标准完整对象 LFS。不支持仅保存块而丢弃完整对象的上传。 + +Mega 当前按「认证用户＋仓库路径」隔离块和 manifest，其他用户通过既有标准 LFS 完整对象路径下载。这些端点要求 Bearer 访问令牌，不提供公开的裸 chunk-hash 查询或下载；这并不等于实现了完整仓库 ACL。manifest 上限为 10 MiB / 8192 块，单块上限为 8 MiB。 + +Pending 描述符在 24 小时后过期，重新准备 manifest 可继续查询和上传缺块；过期数据不会自动回收。此扩展需要显式启用，部署前应规划保留策略与配额，不能对仍被已发布 manifest 引用的块直接设置生命周期删除。 ## 延后项 -Libra-aware media **server**（真实跨机器 chunked upload/download、capability + chunk + manifest-finalize endpoints、manifest lifecycle、GC/fsck/heal，以及所有 anti-side-channel 保证）已冻结在 lore.md §6.5–6.8，不属于此 client v1。Chunk-only repo policy（丢弃标准 LFS fallback 对象）和 range-based hydration 也已延后。 +共享仓库 ACL、自动孤儿块 GC、配额统计、服务端 fsck/heal、obliteration、仅存块策略和按字节范围水合尚未实现。当前传输扩展不代表已完成 Lore §6.5–6.8 的全部生产要求。 ## 示例 diff --git a/docs/development/commands/README.md b/docs/development/commands/README.md index 8fe300335..9e5c546d3 100644 --- a/docs/development/commands/README.md +++ b/docs/development/commands/README.md @@ -38,7 +38,7 @@ | [`branch`](branch.md) | `partial` | create/list/delete/rename/copy(`-c`/`-C`)/upstream set+unset/current/contains/points-at/merged/no-merged/sort(refname,version:refname,committerdate,creatordate,authordate,objectsize,objectname)/ignore-case/`--column`/`--no-column`(countermands `--column`, last wins)/`-v`(`--verbose`, `-vv` adds upstream tracking)/`--edit-description`(edit `branch..description` in an editor; empty unsets)/`--format`(for-each-ref atom rendering; replaces `* name`/`-v`/`--column`) supported; the remaining for-each-ref sort keys (the `*`-deref keys, tag-only) not exposed | | [`bundle`](bundle.md) | `partial` | Full v2 `create` from explicit revs or `--all/--branches/--tags`, preserving annotated tag objects; checksum-aware `verify`, `list-heads`, and hash-kind-correct bounded `unbundle` (objects only; no ref updates). System Git can clone the result. Prerequisite/thin/incremental create, clone-from-bundle, and exhaustive verify decode deferred | | [`cache`](cache.md) | `intentionally-different` | Inspect the tiered-storage / LRU cache configuration (`cache info`): resolved `LIBRA_STORAGE_TYPE`, whether tiered, and the `LIBRA_STORAGE_THRESHOLD` / `LIBRA_STORAGE_CACHE_SIZE` tunables; `--json`/`--machine` → `{ storage_type, tiered, threshold_bytes, cache_size_bytes }`. Git has no equivalent; needs no repo (lore.md §0.10) | -| [`media`](media.md) | `intentionally-different` | FastCDC LFS media chunking client (lore.md §6, **feature-gated `fastcdc`, default OFF**) — `chunk`/`inspect`/`verify`/`probe`. In-tree deterministic content-defined chunker + versioned manifest (`media_oid` always SHA-256), a private `.libra/media/` chunk store outside the Git object graph, and the §6.4 safe standard-LFS fallback negotiation. The Libra-aware media server (real chunked transfer, §6.5–6.8) is honestly frozen. Git has no equivalent | +| [`media`](media.md) | `intentionally-different` | FastCDC LFS extension (lore.md §6, **feature-gated `fastcdc`, default OFF**) — `chunk`/`inspect`/`verify`/`probe`, deterministic in-tree chunking, versioned manifests (`media_oid` always SHA-256), and a private `.libra/media/` cache outside the Git object graph. Matching feature-enabled Mega builds provide authenticated chunk upload, verified finalize with a complete standard-LFS fallback, and cached-chunk download. Shared repository ACLs, server orphan GC/quota/fsck/heal/obliteration and range hydration remain deferred. Git has no equivalent | | [`hydrate`](hydrate.md) | `intentionally-different` | On-demand whole-object hydration (lore.md 3.3) — materialize a path + its transitive deps into the working tree, resolved local→alternate→remote, atomic + OID-verified (never a corrupt file), sparse-gated. The honest v1 of Lore's hydrating VFS (an explicit command, not a FUSE VFS). Git has no equivalent | | [`deps`](deps.md) | `intentionally-different` | File dependency graph (lore.md 3.1) — declare/query typed, versioned per-file edges (`add`/`rm`/`list`/`why`/`tree`). Cycle-safe transitive closure over edges stored per-commit under `refs/notes/deps` (owner `internal::deps`). The reusable substrate for 3.2/3.3. Git has no equivalent | | [`alternates`](alternates.md) | `intentionally-different` | Object alternates (lore.md 2.3) — borrow objects from a shared store instead of copying. `alternates add|list|remove|prune` over the git-standard `objects/info/alternates` (`prune` retires borrower registrations whose repository is gone — explicit, because an absent path is indistinguishable from an unmounted one); reads resolve through the chain (verified). Airtight deletion safety: a shared base refuses to gc/evict borrowed objects (borrower registry). Git has no `alternates` command (git edits the file by hand) | diff --git a/docs/development/commands/media.md b/docs/development/commands/media.md index d9b2e99a4..dfcb9f7d7 100644 --- a/docs/development/commands/media.md +++ b/docs/development/commands/media.md @@ -1,74 +1,72 @@ -# `libra media` 开发设计（lore.md §6 FastCDC LFS media chunking） +# `libra media` 开发设计 ## 命令实现目标 -FastCDC content-defined chunking 的**诚实客户端 v1**（lore.md §6 的最后一项特性）。 -严格 feature-gated（`fastcdc`，**默认关闭**，`fastcdc = []` 纯 in-tree 无新依赖）， -对默认二进制/CI **零影响**。它交付客户端底座 + 能力协商/安全回退，**冻结** -Libra-aware media 服务端协议（§6.5–6.8）——因此对今天任何可达远端，能力探测都回退 -标准 Git LFS。 +保留默认关闭的 `fastcdc = []` 功能，复用既有确定性分块器、manifest 和本地 chunk store， +把 FastCDC 接到 Libra 的真实 LFS 上传/下载路径，并与 Mega 的可选服务端实现联动。 +不增加客户端依赖，不修改 Git 对象图或标准 LFS pointer 的 SHA-256 标识。 ## 对比 Git 与兼容性 -`intentionally-different`：Git 没有 media chunking 概念，也不理解 Libra chunk manifest。 -分类前提（§6.2）：Git object graph **不变**——FastCDC chunk **绝不**成为 Git object ID； -chunk/manifest 存于私有 `.libra/media/`（`objects/` 的物理兄弟目录，从不作为 loose object -遍历）。`media_oid` **恒为 SHA-256**（全文件哈希，独立于 `core.objectformat`），与标准 LFS -pointer 的 `oid sha256:…` 逐字节一致，保证 fallback 与端到端校验。 +`intentionally-different`：`media chunk/inspect/verify/probe` 是 Libra 扩展。 +默认构建仍使用标准 LFS；启用功能后，远端没有兼容能力或 manifest 时回退完整对象。 +`lfs.fastcdc=false` 可按仓库关闭传输扩展。所有远端现在都保留仓库路径， +例如 `/project/demo.git/info/lfs`，不再把 Mega 的地址截断到主机根路径。 ## 设计方案 -- **模块**：`src/utils/media/`（`chunker` / `manifest` / `chunk_store` / `capability` / - `negotiate`），全部 `#[cfg(feature = "fastcdc")]`；CLI 在 `src/command/media.rs`。 -- **chunker**：in-tree 确定性 gear-hash + normalized chunking，**冻结** v1 参数 - （MIN 512 KiB / AVG 2 MiB / MAX 8 MiB，固定 256 项 GEAR 表由 splitmix64 常量构建）。同字节 - → 逐字节相同的 chunk 边界。退化契约：空输入→0 chunk；小于 MIN→单个整块。 -- **manifest**：serde `MediaManifest`（version/algorithm/hash_algorithm/media_oid/media_size/ - chunks[]/created_by/fallback_oid），内容寻址存 `.libra/media/manifests/.json` - （`write_atomic`，**零迁移无 SQLite 表**）。frozen schema：`crc32c` 可选字段在 v1 **留空** - （`crc32fast` 是 IEEE CRC-32 非 Castagnoli，权威 per-chunk 完整性用 `chunk_hash` sha256）。 -- **chunk_store**：`MediaChunkStore` 存**原始字节**（无 Git ` \0`/zlib），读时按 - sha256 重校验；`reassemble` **先校验 media_oid 再 rename**（verify-then-publish，绝无坏文件）。 -- **capability/negotiate**：能力探测 `libra/media/v1/capabilities`（`BasicAuth::send` 挂令牌 - + `utils::backoff::retry_idempotent` §0.2 退避），纯 `negotiate()` 决策（§6.4 矩阵，全绿默认 - Chunked，任一疑点回退标准 LFS，服务端拒 fallback + 本地无 fallback → **Block** 绝不 chunk-only - 半写）。`ProbeOutcome` 区分 NoEndpoint / ServerErrorAfterBackoff / Ok。 - -## CLI 面 - -`libra media chunk [--store]` / `inspect ` / `verify |--media-oid ` / -`probe [--remote ]`，全部稳定 `--json`。不改任何既有命令默认语义。 - -## 诚实延后（服务端冻结，lore.md §6.5–6.8） - -全部 §6.5/6.6 服务端 endpoint、真实跨机 chunked 上传/下载、manifest Pending→Finalized→ -Obliterated 生命周期 + finalize CAS + 孤儿 chunk GC、**每一条 §6.7 反侧信道保证**（scope 绑定 -`chunks/exists`、跨仓存在性不可区分、未 finalize manifest 不可下载）、服务端 GC/fsck/heal chunk -维护（§6.8）、chunk-only 仓库策略（§6.9 默认保留标准完整 LFS fallback）、把探测接进 live LFS -`download_object`/`push_objects` 热路径、以及对纯 git 客户端的 `.gitattributes`/git-lfs filter -bridge。§6.10 前置 (1)–(5) 均已落地；(6) §6.7 拒绝矩阵集成测试随服务端一并落地。 +- 算法保持冻结：in-tree gear hash + normalized chunking，MIN 512 KiB / AVG 2 MiB / + MAX 8 MiB，固定 SplitMix64 GEAR 表。它不是第三方 crate 的 v2020 算法。 +- `MediaManifest` 字段保持 v1；新增拒绝零长度、超大块、溢出、错误 fallback_oid、 + 不支持的 checksum 和超过 8192 块的清单。JSON 上限为 10 MiB。 +- `capability` 在仓库 LFS URL 后追加 `libra/media/v1/capabilities`， + 使用 host-scoped Bearer token、请求超时、有界响应和既有退避。 +- `transfer::MediaClient` 上传先准备 manifest/查询缺块，只上传缺失内容，再 finalize； + 服务端逐块校验、完整 SHA-256 校验和 FastCDC 边界校验后保存完整 LFS fallback， + 最后发布可下载 manifest。失败重试重新查询缺块即可。 +- 下载只使用 Finalized manifest；按实际 offset/length 对应的内容块缓存恢复，不使用 + 旧等长分块的除法推算。缓存块读取时重算 SHA-256；远端坏块或坏清单拒绝发布。 +- `chunk_store::reassemble` 使用既有 `StreamingAtomicFile`，独占临时文件、 + 错误时自动清理、完整校验后原子覆盖目标。缓存仍在私有 `.libra/media` 中。 +- `LFSClient::upload_object/download_object` 的新调用严格在 feature gate 内。 + 标准 LFS batch 保持 basic，不向普通服务端发送扩展上传请求。 + +## Mega 协议与权限边界 + +端点位于 `.git/info/lfs/libra/media/v1`： + +| 方法 | 路径 | +|---|---| +| GET | /capabilities | +| POST | /manifests | +| PUT | /manifests/{id}/chunks/{hash} | +| POST | /manifests/{id}/finalize | +| GET | /manifests/by-media/{oid} | +| GET | /manifests/by-media/{oid}/chunks/{hash} | + +prepare 返回 manifest_id 和 missing_chunks。manifest_id 是紧凑 JSON 数组 +`[version,algorithm,hash_algorithm,media_oid,media_size,chunks]` 的 SHA-256， +不包含客户端 provenance。冻结边界保证同一内容的合法 manifest ID 一致。 + +Mega 新端点要求 Mono access token，并保留 URI 改写前的仓库路径。 +由于现有 Mega LFS 尚无完整仓库 ACL，本版按「认证用户＋仓库」隔离存储， +再由 manifest ID / media OID 限定对象范围。不同用户/仓库不能查询或读到彼此的块； +另一用户的下载回退既有完整 LFS 对象。没有公开的裸 chunk-hash GET。 +服务端必须以 `--features fastcdc` 显式构建，默认不暴露扩展端点。 ## 测试 -`--features fastcdc` 下：chunker 单测（确定性 + 退化契约）、manifest round-trip + 校验、 -negotiate() §6.4 全矩阵（含 all-green→Chunked 正例）、chunk-store 原始字节 + 重校验、 -capability 状态分类；集成测试 `tests/media_fastcdc_test.rs`（chunk--store/verify 往返、坏 chunk -干净失败、probe 不可达端点回退标准 LFS）。feature-gate 由 `compat_fastcdc_feature_gate_guard` -常驻钉住（默认不入 default、cfg 门控）。 - -## Examples - -```bash -libra media chunk big.psd # FastCDC-chunk a file; print the manifest summary -libra media chunk big.psd --store # also persist chunks + manifest to .libra/media -libra media inspect .libra/media/manifests/.json -libra media verify big.psd # reassemble from the store and verify the media_oid -libra media probe --remote origin # capability-probe; falls back to standard LFS -libra --json media chunk big.psd # structured JSON for agents -``` +既有分块/manifest/cache 单测；`media_fastcdc_test` 的 CLI 测试； +坏块不覆盖目标、普通服务端完整 LFS 回退测试； +手动 `mega_fastcdc_http_interop` 连接 Mega 的真实 HTTP 路由和令牌验证， +覆盖实际变长分块上传/下载、只补缺块、缓存恢复/修复、跨用户拒绝和空文件。 +两进程命令见 Mega 的 `docs/lfs-api.md`，共享 `MEGA_FASTCDC_READY_FILE`。 +测试结果以本次实际运行记录为准；不把编译失败或 skipped/ignored 计为通过。 -## 维护要求 +## 未完成项 -修改本命令前先读 [_general.md](_general.md) 与 [_compatibility.md](_compatibility.md)。改动需同步 -`COMPATIBILITY.md`、本文件、`docs/commands/media.md` 与测试。严格保持 feature-gate：任何新增 -依赖必须 `optional = true`，不得进入 default。 +本次交付传输链路，不宣称完成 Lore §6 的全部生产门禁。 +共享仓库 ACL、自动孤儿块 GC、quota、服务端 fsck/heal、obliteration、 +chunk-only 策略、字节范围水合、跨租户 dedup 均未开放。 +Pending 描述符 24 小时到期，过期数据不会自动回收；部署方需明确保留策略， +不得对仍被 Finalized manifest 共享的块设置无条件生命周期删除。 diff --git a/docs/development/gap/lore.md b/docs/development/gap/lore.md index 32a21c2c5..b959bf857 100644 --- a/docs/development/gap/lore.md +++ b/docs/development/gap/lore.md @@ -469,32 +469,33 @@ file metadata 与 MCP tree handle 可以在 LR-02 之后、LR-09 之前做：它 ## 6. 最后支持的特性：LFS FastCDC chunking -> **实施状态（客户端 v1 ✅ 已落地；服务端协议冻结；#461 未合入）**：诚实 v1 交付**严格 feature-gated -> （`fastcdc`，默认关闭，`fastcdc = []` 纯 in-tree 无新依赖，对默认二进制/CI 零影响）的 -> 客户端底座**——`libra media chunk/inspect/verify/probe`（`src/utils/media/` + -> `src/command/media.rs`）：in-tree 确定性 gear-hash FastCDC chunker（冻结 `fastcdc-v1` 参数 -> MIN 512K/AVG 2M/MAX 8M + splitmix64 GEAR 表）、版本化 manifest（`media_oid` 恒 SHA-256、 -> 独立于 `core.objectformat`、与标准 LFS pointer 逐字节一致；`crc32c` 字段 v1 留空避免把 -> IEEE crc32 烙进 frozen schema）、私有 `.libra/media/` 内容寻址 chunk store（`objects/` 兄弟、 -> chunk **绝不**成 Git object ID、读时 sha256 重校验）、verify-then-rename 重组（绝无坏文件）、 -> 以及 §6.4 **安全回退协商**（能力探测经 `retry_idempotent` §0.2 退避；`negotiate()` 纯函数： -> 全绿默认 Chunked、任一疑点回退标准 LFS、服务端拒 fallback + 本地无 fallback → **Block** 绝不 -> chunk-only 半写；`ProbeOutcome` 区分 NoEndpoint/ServerErrorAfterBackoff/Ok）。**冻结/延后**： -> §6.5–6.8 全部服务端协议（真实跨机 chunked 上传/下载、finalize 生命周期、GC/fsck/heal、 -> **每条 §6.7 反侧信道保证均为服务端义务**）+ chunk-only 策略 + 接进 live LFS 热路径 + 纯 git -> 客户端 bridge——故对今天任何可达远端，探测都回退标准 Git LFS。测试：media 单测（chunker -> 确定性+退化契约、manifest 往返+校验、negotiate 全矩阵含 all-green→Chunked 正例、chunk-store -> 原始字节+重校验、capability 分类）+ 集成 `tests/media_fastcdc_test.rs`（chunk--store/verify -> 往返、坏 chunk 干净失败、probe 不可达回退）+ `compat_fastcdc_feature_gate_guard` 常驻钉门控。 -> 规划经多智能体工作流（Understand→Design→三面对抗审阅：安全/门控 NEEDS_REVISION→已修 -> negotiate 优先级+正例、可行性 SOUND、诚实/侧信道 SOUND）+ Codex 两轮（backoff 非 §0.2 + -> banner guard cfg → 已修 → APPROVE）。以下 §6.1–6.10 为完整设计规范（客户端 + 冻结的服务端）。 +> **当前实施范围（2026-08-28）：客户端底座＋默认关闭的 Libra/Mega 传输扩展。** +> 已有 `libra media chunk/inspect/verify/probe`、冻结的 in-tree `fastcdc-v1` 分块器 +> （MIN 512 KiB / AVG 2 MiB / MAX 8 MiB、SplitMix64 GEAR 表）、版本化 manifest 和私有 +> `.libra/media/` 缓存；块及完整文件均使用 SHA-256，不修改 Git 对象图或 LFS pointer。 +> Libra 的 `src/utils/media/transfer.rs` 接入实际 LFS 上传/下载；Mega 提供仓库 LFS URL +> 下的 `libra/media/v1` 扩展端点。两端都须使用 `--features fastcdc` 构建，并配置绑定主机的 +> Mono Bearer 访问令牌。默认构建仍关闭，`lfs.fastcdc=false` 可在仓库中禁用传输。 > -> **2026-09-01**：[#461](https://github.com/libra-tools/libra/pull/461) 把既有 `fastcdc-v1` -> chunker/manifest/本地 chunk store 接到普通 LFS push/download（配套 Mega `#2178`），默认 -> 仍关闭、`lfs.fastcdc=false` 可关。**未合入 main**。0.9.0 要求合流进该 PR 或其后续卡的 -> 完成判据：0.11 query 匹配层级、「已有则不重传」、3.3 media/LFS 范围读；服务端 GC/fsck/heal/ -> obliterate 认识 chunk 仍等 #461 之后解冻 §6.5–6.8。禁止平行第二套 media 传输。 +> 当前上传把 prepare manifest 与缺块查询合并，只上传缺块，再 finalize。Mega 校验块、 +> 完整 SHA-256 和冻结边界，先保存标准 LFS 完整对象，再原子发布可下载 manifest。 +> 下载复用已校验的本地块，完整校验后才原子替换目标；无兼容能力或无 manifest 时回退 +> 标准完整对象。一旦选择 manifest，鉴权或完整性失败就报错，不以静默回退掩盖错误。 +> 不支持 chunk-only 或按字节范围水合。实际端点及流程见 +> [media 开发设计](../commands/media.md) 和[命令文档](../../commands/media.md)。 +> +> **尚未满足全部生产门禁。** Mega 现有 LFS 没有完整仓库 ACL，本扩展按「认证用户＋仓库路径」 +> 隔离，再由 manifest/media OID 限定访问；其他用户走既有完整对象路径。这不能替代共享仓库 +> ACL，也不宣称满足 §6.7 的全部授权和时序侧信道保证。manifest 限制为 10 MiB / 8192 块， +> 单块最大 8 MiB；Pending 描述符 24 小时到期，但不会自动回收存储，可重新 prepare 续传。 +> 自动孤儿 GC、配额统计、服务端 fsck/heal、obliteration、备份恢复联动和跨用户去重均待实现。 +> 部署方须制定保留策略，不得无条件删除仍被 Finalized manifest 共享的块。 +> +> 相关验证入口包括 media 单测、`tests/media_fastcdc_test.rs` 和 +> `compat_fastcdc_feature_gate_guard`；跨系统测试 `mega_fastcdc_http_interop` 需要显式启动 +> Mega 测试服务。测试是否通过以运行记录为准，不把 skipped/ignored 计为通过。 +> **以下 §6.1–6.10 仍是完整目标规范与后续生产验收要求，不是当前实现清单**；其中建议的独立 +> `chunks/exists`、预签名 URL、range、GC/ACL/quota/CAS 等接口或保证不能视为已实现。 ### 6.1 为什么必须最后做 diff --git a/src/command/media.rs b/src/command/media.rs index 0d8cda999..5596389ef 100644 --- a/src/command/media.rs +++ b/src/command/media.rs @@ -3,9 +3,8 @@ //! The honest, feature-gated (`fastcdc`) v1 CLIENT surface: chunk a media file, //! inspect/validate a manifest, reassemble+verify from the local chunk store, //! and probe a remote's chunked-LFS capability with the §6.4 safe-fallback -//! decision. It ships NO real cross-machine chunked transfer — the Libra-aware -//! media server (§6.5–6.8) is frozen; against every reachable remote the probe -//! resolves to standard Git LFS. This module is only the CLI surface; all logic +//! decision. Feature-enabled LFS transfers can use Mega's authenticated media +//! endpoints, with standard LFS fallback. This module is only the CLI surface; all logic //! lives in [`crate::utils::media`]. use clap::{Parser, Subcommand}; @@ -39,8 +38,8 @@ NOTES: FastCDC media chunking is a feature-gated Libra extension (lore.md §6). The media_oid is always SHA-256 of the full file (standard-LFS-compatible), and chunks live in a private .libra/media store outside the Git object graph. - Cross-machine chunked transfer requires a Libra-aware media server that is - not yet available; every real remote falls back to standard Git LFS."; + Cross-machine chunked transfer requires Mega built with --features fastcdc + and a stored access token. Other remotes fall back to standard Git LFS."; #[derive(Parser, Debug)] #[command(after_help = MEDIA_EXAMPLES)] diff --git a/src/command/rename_detect.rs b/src/command/rename_detect.rs index 144e3af94..616e0b190 100644 --- a/src/command/rename_detect.rs +++ b/src/command/rename_detect.rs @@ -2289,7 +2289,8 @@ mod tests { /// budget is independent" from "the object budget is broken in the same /// way". Here a real repository exists, the WORKTREE budget is spent to /// nothing, and an object read is then required to succeed on content it - /// actually returns. + /// actually returns. The worktree fixture uses a Unix symlink. + #[cfg(unix)] #[test] #[serial_test::serial] fn object_read_still_succeeds_after_the_worktree_budget_is_spent() { diff --git a/src/internal/ai/observed_agents/opencode_export.rs b/src/internal/ai/observed_agents/opencode_export.rs index eeeadad66..3c153ef06 100644 --- a/src/internal/ai/observed_agents/opencode_export.rs +++ b/src/internal/ai/observed_agents/opencode_export.rs @@ -1124,13 +1124,16 @@ fn pin_store_under(base: &std::path::Path) -> Result { #[cfg(test)] mod tests { + #[cfg(unix)] use std::os::unix::fs::PermissionsExt; use super::*; /// Write an executable fake exporter script (tests never touch a real /// `opencode`, GC-DR-07). The script body receives argv untouched, which - /// is exactly what the no-shell contract must preserve. + /// is exactly what the no-shell contract must preserve. This fixture + /// requires a POSIX shell and Unix executable permission bits. + #[cfg(unix)] fn fake_exporter(dir: &std::path::Path, body: &str) -> PathBuf { let path = dir.join("fake-opencode"); std::fs::write(&path, format!("#!/bin/sh\n{body}\n")).unwrap(); @@ -1140,6 +1143,7 @@ mod tests { /// Whether an executable named `name` is resolvable on `PATH` (used to skip /// tests that depend on an optional system tool such as `setsid`). + #[cfg(unix)] fn binary_on_path(name: &str) -> bool { std::env::var_os("PATH") .map(|path| { @@ -1157,13 +1161,15 @@ mod tests { #[tokio::test] async fn opencode_export_rejects_bad_session_id() { let dir = tempfile::tempdir().unwrap(); - let bin = fake_exporter(dir.path(), "echo '{}'"); + // Invalid IDs must be rejected without spawning any executable. + let bin = dir.path().join("unused-exporter"); for bad in ["", "../escape", "id with spaces", "a;b", "$(rm -rf /)"] { + let err = run_export_subprocess(&bin, bad, ExportLimits::default()) + .await + .expect_err("invalid session id must fail"); assert!( - run_export_subprocess(&bin, bad, ExportLimits::default()) - .await - .is_err(), - "session id {bad:?} must be rejected before spawn" + err.to_string().contains("invalid OpenCode session id"), + "session id {bad:?} must be rejected before spawn, got {err:#}" ); } } @@ -1171,6 +1177,7 @@ mod tests { /// opencode_export_argv_no_shell: metacharacters in a (valid-charset) /// session id reach the child as ONE argv element — no shell ever /// interprets them. The fake exporter prints its argv verbatim. + #[cfg(unix)] #[tokio::test] async fn opencode_export_argv_no_shell() { let dir = tempfile::tempdir().unwrap(); @@ -1183,6 +1190,7 @@ mod tests { /// opencode_export_bytes_path_byte_cap: over-cap output kills the run — /// error, never a silent truncation. + #[cfg(unix)] #[tokio::test] async fn opencode_export_byte_cap_fails_closed() { let dir = tempfile::tempdir().unwrap(); @@ -1199,6 +1207,7 @@ mod tests { /// A non-terminating writer is killed by the byte cap instead of being /// allowed to consume disk until the much later wall-clock deadline. + #[cfg(unix)] #[tokio::test] async fn opencode_export_byte_cap_kills_runaway_writer() { let dir = tempfile::tempdir().unwrap(); @@ -1221,6 +1230,7 @@ mod tests { /// A successful direct child cannot leave a background writer holding the /// inherited output descriptors after the result has been validated. + #[cfg(unix)] #[tokio::test] async fn opencode_export_rejects_surviving_descendant() { let dir = tempfile::tempdir().unwrap(); @@ -1252,6 +1262,7 @@ mod tests { /// over-cap bytes it writes to the inherited stdout are still refused — /// the byte cap is enforced on the bytes, not on group membership. Skips /// when `setsid` is unavailable. + #[cfg(unix)] #[tokio::test] async fn opencode_export_setsid_escapee_cannot_exceed_cap() { if !binary_on_path("setsid") { @@ -1418,6 +1429,7 @@ mod tests { } /// Deadline kills a hung exporter; the wait stays bounded. + #[cfg(unix)] #[tokio::test] async fn opencode_export_deadline_kills_hung_exporter() { let dir = tempfile::tempdir().unwrap(); @@ -1440,6 +1452,7 @@ mod tests { /// A failing exporter surfaces capped, redacted stderr — and secrets in /// stderr never appear raw in the error text. + #[cfg(unix)] #[tokio::test] async fn opencode_export_failure_redacts_stderr() { let dir = tempfile::tempdir().unwrap(); diff --git a/src/internal/operation_wrapper.rs b/src/internal/operation_wrapper.rs index 26253e26a..cf2a0e3e0 100644 --- a/src/internal/operation_wrapper.rs +++ b/src/internal/operation_wrapper.rs @@ -694,6 +694,7 @@ mod claim_owner_tests { ); } + #[cfg(unix)] #[test] fn a_pid_that_does_not_exist_is_gone() { let Some(machine) = claim_machine_identity() else { diff --git a/src/internal/protocol/lfs_client.rs b/src/internal/protocol/lfs_client.rs index f14ab6527..842aae1d0 100644 --- a/src/internal/protocol/lfs_client.rs +++ b/src/internal/protocol/lfs_client.rs @@ -75,6 +75,63 @@ pub enum LockListError { static LFS_CLIENT: OnceCell = OnceCell::const_new(); impl LFSClient { + #[cfg(feature = "fastcdc")] + async fn media_client( + &self, + local_fallback: bool, + ) -> anyhow::Result> { + if crate::utils::util::try_get_storage_path(None).is_ok() + && let Some(value) = ConfigKv::get_best_effort("lfs.fastcdc").await? + { + match value.value.to_ascii_lowercase().as_str() { + "false" | "no" | "off" | "0" => return Ok(None), + "true" | "yes" | "on" | "1" => (), + _ => return Err(anyhow!("lfs.fastcdc must be a boolean")), + } + } + crate::utils::media::transfer::MediaClient::discover( + self.client.clone(), + &self.lfs_url, + local_fallback, + ) + .await + } + + #[cfg(feature = "fastcdc")] + async fn try_media_upload( + &self, + oid: &str, + size: i64, + file: &Path, + remote_has_full_object: bool, + ) -> Result { + let result = async { + let Some(media) = self.media_client(true).await? else { + return Ok(false); + }; + let size = u64::try_from(size).context("negative LFS object size")?; + if remote_has_full_object { + // A prior finalize may have persisted the standard object but + // failed before publishing the manifest. Repair it only when + // a complete local source is available; ordinary basic LFS + // skips never require opening a local object. + let Ok(metadata) = tokio::fs::metadata(file).await else { + return Ok(false); + }; + if !metadata.is_file() || metadata.len() != size { + return Ok(false); + } + } + media.upload(oid, size, file).await + } + .await; + result.map_err(|error| LfsPushError { + path: Some(file.display().to_string()), + oid: Some(oid.to_owned()), + detail: format!("FastCDC upload failed: {error:#}"), + }) + } + /// Get LFSClient instance /// - DO NOT use `async_static!`: No IDE Code Completion & lagging pub async fn get() -> anyhow::Result<&'static LFSClient> { @@ -101,30 +158,11 @@ impl ProtocolClient for LFSClient { /// graceful path that also handles SCP-style SSH URLs and surfaces /// errors with context. fn from_url(repo_url: &Url) -> Self { - // The trailing slash is MUST, or `join()` method will replace the last segment. - // like: Url("/info/lfs").join("objects/batch") => "/info/objects/batch" - let lfs_server = lfs::generate_lfs_server_url(repo_url.to_string()) + "/"; // IMPORTANT - let lfs_server = Url::parse(&lfs_server).expect( - "LFSClient::from_url: derived LFS server URL did not parse (use LFSClient::new for SCP-style)", - ); - let client = Client::builder() - .redirect(super::https_client::no_downgrade_redirect_policy()) - .default_headers(lfs::LFS_HEADERS.clone()) // will be overwritten by `json()`, careful! - .build() - // INVARIANT (trait contract, see the impl doc above): the trait - // returns Self, so this must panic; LFSClient::new() is the - // fallible path. - .expect( - "LFSClient::from_url: reqwest client builder failed (likely missing TLS backend)", - ); - Self { - // Caution: DO NOT start with `/`, or path after domain will be replaced. - batch_url: lfs_server - .join("objects/batch") - .expect("'objects/batch' is a valid relative URL"), - lfs_url: lfs_server, - client, - } + // INVARIANT (trait contract): this constructor cannot return errors; + // callers needing recovery use `from_remote_url` or `new` instead. + Self::from_remote_url(repo_url.as_str()).expect( + "LFSClient::from_url: failed to construct LFS client (use from_remote_url for errors)", + ) } } @@ -141,23 +179,7 @@ impl LFSClient { `libra branch --set-upstream-to /`" ) })?; - // generate_lfs_server_url converts SCP-style SSH URLs (git@host:user/repo.git) - // to valid HTTPS URLs, so we pass the raw remote string directly instead of - // going through Url::parse which rejects SCP format with RelativeUrlWithoutBase. - let lfs_server = lfs::generate_lfs_server_url(url.clone()) + "/"; - let lfs_server = Url::parse(&lfs_server) - .with_context(|| format!("failed to derive LFS server URL from remote '{url}'"))?; - let client = Client::builder() - .redirect(super::https_client::no_downgrade_redirect_policy()) - .default_headers(lfs::LFS_HEADERS.clone()) - .build()?; - Ok(Self { - batch_url: lfs_server - .join("objects/batch") - .expect("'objects/batch' is a valid relative URL"), - lfs_url: lfs_server, - client, - }) + Self::from_remote_url(&url) } /// Build a client from an EXPLICIT remote URL (lore.md 2.8): the lock @@ -165,17 +187,24 @@ impl LFSClient { /// `remote.origin.url`), where [`Self::new`]'s current-branch resolution /// would refuse. pub fn from_remote_url(url: &str) -> anyhow::Result { - let lfs_server = lfs::generate_lfs_server_url(url.to_string()) + "/"; - let lfs_server = Url::parse(&lfs_server) - .with_context(|| format!("failed to derive LFS server URL from remote '{url}'"))?; + // Convert SCP-style SSH before parsing and append to the URL path, + // never after the query string. Keep the query on both endpoints. + let lfs_server = lfs::generate_lfs_server_url(url.to_string()); + let mut lfs_server = Url::parse(&lfs_server).with_context(|| { + format!( + "failed to derive LFS server URL from remote '{}'", + crate::utils::redact::redact_url_credentials(url) + ) + })?; + lfs_server.set_path(&format!("{}/", lfs_server.path().trim_end_matches('/'))); + let mut batch_url = lfs_server.clone(); + batch_url.set_path(&format!("{}objects/batch", lfs_server.path())); let client = Client::builder() .redirect(super::https_client::no_downgrade_redirect_policy()) .default_headers(lfs::LFS_HEADERS.clone()) .build()?; Ok(Self { - batch_url: lfs_server - .join("objects/batch") - .expect("'objects/batch' is a valid relative URL"), + batch_url, lfs_url: lfs_server, client, }) @@ -439,6 +468,13 @@ impl LFSClient { })?; println!("Uploading LFS file: {}", object.oid); + #[cfg(feature = "fastcdc")] + if self + .try_media_upload(&object.oid, object.size, file, false) + .await? + { + return Ok(true); + } let content_len = tokio::fs::metadata(file) .await .map_err(|e| LfsPushError { @@ -494,6 +530,13 @@ impl LFSClient { println!("Uploaded."); Ok(true) } else { + #[cfg(feature = "fastcdc")] + if self + .try_media_upload(&object.oid, object.size, file, true) + .await? + { + return Ok(true); + } tracing::debug!("LFS file {} already exists on remote server", object.oid); Ok(false) } @@ -602,6 +645,23 @@ impl LFSClient { anyhow!("LFS batch download response missing 'download' action for oid {oid}") })?; + #[cfg(feature = "fastcdc")] + if let Ok(storage) = util::try_get_storage_path(None) + && let Some(media) = self.media_client(false).await? + { + // The public LFSClient also supports callers outside a repository; + // those use basic LFS instead of a repository-only cache that panics. + let store = crate::utils::media::chunk_store::MediaChunkStore::at( + storage.join("media").join("chunks"), + ); + if media.download(oid, size, path.as_ref(), &store).await? { + if let Some((report, _)) = reporter.as_mut() { + report(100.0)?; + } + return Ok(()); + } + } + let mut is_chunked = false; // Chunk API — infer that all chunks share the same size, falling back to the // total object size when the server reports a single-chunk download. @@ -715,6 +775,11 @@ impl LFSClient { } pb.finish_and_clear(); } + // Tokio may still be completing the last write on its blocking pool. + // Do not report success until those bytes (and any write error) are visible. + file.flush() + .await + .context("failed to finish writing downloaded LFS object")?; let checksum = hex::encode(checksum.finish().as_ref()); if checksum == oid { println!("Downloaded."); @@ -727,6 +792,9 @@ impl LFSClient { file.set_len(0).await?; // clear file.seek(tokio::io::SeekFrom::Start(0)).await?; // ensure file.write_all(pointer.as_bytes()).await?; + file.flush() + .await + .context("failed to finish writing LFS fallback pointer")?; Err(anyhow!("Checksum mismatch, fallback to pointer file.")) } } @@ -934,6 +1002,24 @@ impl LFSClient { #[cfg(test)] mod tests { use super::*; + + #[test] + fn constructors_append_endpoints_before_the_query() { + let remote = "ssh://git@host.example:8443/project/demo.git/?tenant=one#ref"; + let client = LFSClient::from_remote_url(remote).unwrap(); + assert_eq!( + client.lfs_url.as_str(), + "https://host.example:8443/project/demo.git/info/lfs/?tenant=one" + ); + assert_eq!( + client.batch_url.as_str(), + "https://host.example:8443/project/demo.git/info/lfs/objects/batch?tenant=one" + ); + let trait_client = LFSClient::from_url(&Url::parse(remote).unwrap()); + assert_eq!(trait_client.lfs_url, client.lfs_url); + assert_eq!(trait_client.batch_url, client.batch_url); + } + #[test] fn test_request_vars() { let vars = RequestObject { @@ -1111,11 +1197,9 @@ mod tests { } fn test_lfs_client(base_url: &str) -> LFSClient { - LFSClient { - batch_url: Url::parse(&format!("{base_url}objects/batch")).unwrap(), - lfs_url: Url::parse(base_url).unwrap(), - client: Client::builder().no_proxy().build().unwrap(), - } + let mut client = LFSClient::from_remote_url(base_url).unwrap(); + client.client = Client::builder().no_proxy().build().unwrap(); + client } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] diff --git a/src/utils/lfs.rs b/src/utils/lfs.rs index 0e7e8c382..f9526dea2 100644 --- a/src/utils/lfs.rs +++ b/src/utils/lfs.rs @@ -122,63 +122,71 @@ pub fn format_pointer_string(oid: &str, size: u64) -> String { /// - like `https://git-server.com/foo/bar.git/info/lfs` /// - support ssh & https & git@ format fn generate_git_lfs_server_url(mut url: String) -> String { - if url.ends_with('/') { - url.pop(); - } - if !url.ends_with(".git") { - url.push_str(".git"); - } - url.push_str("/info/lfs"); - + let ssh_url = url.starts_with("ssh://"); if url.starts_with("git@") { // git@git-server.com:foo/bar.git - url = "https://".to_string() + &url[4..].replace(":", "/"); - } else if url.starts_with("ssh://") { + let remote = &url[4..]; + let separator = if remote.starts_with('[') { + remote.find("]:").map(|index| index + 1) + } else { + remote.find(':') + }; + if let Some(separator) = separator { + url = format!( + "https://{}/{}", + &remote[..separator], + &remote[separator + 1..] + ); + } + } else if ssh_url { // ssh://git-server.com/foo/bar.git url = "https://".to_string() + &url[6..]; } - url + let Ok(mut parsed) = Url::parse(&url) else { + return url; + }; + if ssh_url { + // SSH usernames/passwords are not HTTP credentials. Leaving `git@` + // here would also suppress the host-scoped HTTP token lookup. + // INVARIANT: a parsed HTTPS URL with a host supports userinfo setters. + let _ = parsed.set_username(""); + let _ = parsed.set_password(None); + } + let path = parsed.path().trim_end_matches('/'); + let path = if path.ends_with("/info/lfs") { + path.to_owned() + } else if path.is_empty() { + "/info/lfs".to_owned() + } else if path.ends_with(".git") { + format!("{path}/info/lfs") + } else { + format!("{path}.git/info/lfs") + }; + parsed.set_path(&path); + parsed.set_fragment(None); + parsed.to_string() } /// Generate Mono LFS Server Url from repo Url. -/// - Just get domain with port -/// ### Example -/// https://github.com/git-lfs/git-lfs/blob/main/docs/api/locking.md -> https://github.com -/// -/// http://localhost:8000/xxx/yyy -> http://localhost:8000 -/// -/// Falls back to the original URL string if parsing fails or the URL has no -/// host (e.g. `file:///path`). Callers will then either accept the literal -/// URL or surface a downstream error. +/// Preserve repository scope for Mega's standard `/info/lfs` router. +/// Example: `http://localhost:8000/project/demo` becomes +/// `http://localhost:8000/project/demo.git/info/lfs`. +/// A host-only HTTP remote keeps the legacy root LFS endpoints. fn generate_mono_lfs_server_url(url: String) -> String { - let parsed = match Url::parse(&url) { - Ok(parsed) => parsed, - Err(err) => { - tracing::warn!( - url = %url, - error = %err, - "failed to re-parse remote URL while deriving mono LFS URL; using as-is" - ); - return url; - } - }; - let Some(host) = parsed.host() else { - tracing::warn!( - url = %url, - "remote URL has no host; using as-is for mono LFS URL" - ); - return url; - }; - match parsed.port() { - None => format!("{}://{host}", parsed.scheme()), - Some(port) => format!("{}://{host}:{port}", parsed.scheme()), + if let Ok(mut parsed) = Url::parse(&url) + && matches!(parsed.scheme(), "http" | "https") + && parsed.path().trim_end_matches('/').is_empty() + { + parsed.set_fragment(None); + return parsed.to_string(); } + generate_git_lfs_server_url(url) } /// Generate LFS Server Url from repo Url. /// - Automatically detect git or mono repo by domain -/// - Caution: without trailing slash `/` +/// - Callers normalize the trailing slash before joining endpoint paths. pub fn generate_lfs_server_url(url_str: String) -> String { let url = match Url::parse(&url_str) { Ok(url) => url, @@ -502,6 +510,29 @@ mod tests { let url = "ssh://github.com/libra-tools/mega.git".to_owned(); assert_eq!(generate_lfs_server_url(url), LFS_SERVER_URL); + + let url = "ssh://git@github.com/libra-tools/mega.git".to_owned(); + assert_eq!(generate_lfs_server_url(url), LFS_SERVER_URL); + } + + #[test] + fn lfs_url_preserves_query_and_only_removes_ssh_credentials() { + assert_eq!( + generate_lfs_server_url( + "ssh://git:unused@host.example:8443/repo.git/?tenant=one#ref".to_owned() + ), + "https://host.example:8443/repo.git/info/lfs?tenant=one" + ); + assert_eq!( + generate_lfs_server_url( + "https://user:token@host.example/repo.git/info/lfs/?tenant=one".to_owned() + ), + "https://user:token@host.example/repo.git/info/lfs?tenant=one" + ); + assert_eq!( + generate_lfs_server_url("git@[::1]:project/demo.git".to_owned()), + "https://[::1]/project/demo.git/info/lfs" + ); } #[test] @@ -509,7 +540,7 @@ mod tests { const LFS_SERVER_URL: &str = "https://gitmono.com/libra-tools/mega.git/info/lfs"; assert_eq!( generate_lfs_server_url(LFS_SERVER_URL.to_owned()), - "https://gitmono.com" + LFS_SERVER_URL ); const LOCAL_LFS_SERVER_URL: &str = "http://localhost:8000/xxx/yyy"; assert_eq!( @@ -518,8 +549,19 @@ mod tests { ); assert_eq!( generate_lfs_server_url(LOCAL_LFS_SERVER_URL.to_owned()), - "http://localhost:8000" + "http://localhost:8000/xxx/yyy.git/info/lfs" ); + for (remote, expected) in [ + ("http://127.0.0.1:8000", "http://127.0.0.1:8000/"), + ("http://localhost:8000/", "http://localhost:8000/"), + ("https://gitmono.com", "https://gitmono.com/"), + ( + "http://[::1]:8000/?tenant=one#ref", + "http://[::1]:8000/?tenant=one", + ), + ] { + assert_eq!(generate_lfs_server_url(remote.to_owned()), expected); + } } #[test] diff --git a/src/utils/media/capability.rs b/src/utils/media/capability.rs index a09cebebc..0f90a3203 100644 --- a/src/utils/media/capability.rs +++ b/src/utils/media/capability.rs @@ -1,13 +1,12 @@ //! Remote media-capability probe (lore.md §6.4). //! //! GETs `/libra/media/v1/capabilities` with the host-scoped bearer token, -//! wrapped in the §0.2 bounded backoff for 429/5xx (`BasicAuth::send` attaches -//! auth but does NOT itself retry 5xx — Codex P1). Classifies the result into a +//! wrapped in the §0.2 bounded backoff for 429/5xx. Classifies the result into a //! [`ProbeOutcome`]; every ambiguity resolves to a SAFE outcome (a plain remote -//! reads as `NoEndpoint`, so negotiation falls back to standard LFS). Against -//! every reachable remote today — none of which expose the (frozen, unbuilt) -//! Libra media server — the probe returns `NoEndpoint`. +//! reads as `NoEndpoint`, so negotiation falls back to standard LFS). Discovery +//! preserves the repository's standard LFS URL and uses a host-scoped token. +use futures_util::StreamExt; use serde::Deserialize; use super::negotiate::ProbeOutcome; @@ -67,6 +66,18 @@ pub const CAPABILITIES_PATH: &str = "libra/media/v1/capabilities"; /// non-2xx-non-5xx status yields `NoEndpoint`; a 429/5xx that survives §0.2 /// retries yields `ServerErrorAfterBackoff`. pub async fn probe(base_url: &str) -> ProbeOutcome { + let Ok(client) = reqwest::Client::builder() + .redirect(crate::internal::protocol::https_client::no_downgrade_redirect_policy()) + .build() + else { + return ProbeOutcome::NoEndpoint; + }; + probe_with_client(base_url, client).await +} + +/// The supplied client must refuse HTTPS-to-HTTP redirects before forwarding +/// stored credentials. LFSClient and `probe` use the shared redirect policy. +pub async fn probe_with_client(base_url: &str, client: reqwest::Client) -> ProbeOutcome { let url = match join_capabilities_url(base_url) { Some(u) => u, None => return ProbeOutcome::NoEndpoint, @@ -79,24 +90,26 @@ pub async fn probe(base_url: &str) -> ProbeOutcome { }, None => None, }; - let client = reqwest::Client::new(); let result: Result = retry_idempotent(&RetryPolicy::default(), |_attempt| { let url = url.clone(); let token = token.clone(); let client = client.clone(); async move { - let mut req = client.get(url).header("Accept", "application/json"); + let mut req = client + .get(url) + .header("Accept", "application/json") + .timeout(std::time::Duration::from_secs(10)); if let Some(t) = &token { req = req.bearer_auth(t); } match req.send().await { Ok(resp) => match classify_status(resp.status().as_u16()) { - StatusClass::Success => match resp.json::().await { - Ok(caps) => RetryOutcome::Done(Ok(ProbeOutcome::Ok(caps))), + StatusClass::Success => match decode_capabilities(resp).await { + Some(caps) => RetryOutcome::Done(Ok(ProbeOutcome::Ok(caps))), // A 2xx with an undecodable body is not a usable // capability endpoint — fall back safely. - Err(_) => RetryOutcome::Done(Ok(ProbeOutcome::NoEndpoint)), + None => RetryOutcome::Done(Ok(ProbeOutcome::NoEndpoint)), }, StatusClass::NoEndpoint => RetryOutcome::Done(Ok(ProbeOutcome::NoEndpoint)), StatusClass::Retryable => RetryOutcome::Retry { @@ -123,11 +136,29 @@ pub async fn probe(base_url: &str) -> ProbeOutcome { result.unwrap_or(ProbeOutcome::ServerErrorAfterBackoff) } +async fn decode_capabilities(response: reqwest::Response) -> Option { + let mut data = Vec::new(); + let mut stream = response.bytes_stream(); + while let Some(bytes) = stream.next().await { + let bytes = bytes.ok()?; + if bytes.len() > (16 * 1024usize).saturating_sub(data.len()) { + return None; + } + data.extend_from_slice(&bytes); + } + serde_json::from_slice(&data).ok() +} + /// Join the media base URL with the capabilities path, tolerating a trailing /// slash. Returns `None` for an unparseable base. fn join_capabilities_url(base_url: &str) -> Option { - let trimmed = base_url.trim_end_matches('/'); - url::Url::parse(&format!("{trimmed}/{CAPABILITIES_PATH}")).ok() + let lfs_url = crate::utils::lfs::generate_lfs_server_url(base_url.to_owned()); + let mut url = url::Url::parse(&lfs_url).ok()?; + url.set_path(&format!( + "{}/{CAPABILITIES_PATH}", + url.path().trim_end_matches('/') + )); + Some(url) } #[cfg(test)] @@ -151,12 +182,19 @@ mod tests { let u = join_capabilities_url("https://host.example/repo").unwrap(); assert_eq!( u.as_str(), - "https://host.example/repo/libra/media/v1/capabilities" + "https://host.example/repo.git/info/lfs/libra/media/v1/capabilities" ); let u = join_capabilities_url("https://host.example/repo/").unwrap(); assert_eq!( u.as_str(), - "https://host.example/repo/libra/media/v1/capabilities" + "https://host.example/repo.git/info/lfs/libra/media/v1/capabilities" + ); + let u = + join_capabilities_url("ssh://git@host.example:8443/repo.git/info/lfs/?tenant=one#ref") + .unwrap(); + assert_eq!( + u.as_str(), + "https://host.example:8443/repo.git/info/lfs/libra/media/v1/capabilities?tenant=one" ); assert!(join_capabilities_url("not a url").is_none()); } diff --git a/src/utils/media/chunk_store.rs b/src/utils/media/chunk_store.rs index 5be0bc90f..846bcbcd1 100644 --- a/src/utils/media/chunk_store.rs +++ b/src/utils/media/chunk_store.rs @@ -45,6 +45,30 @@ pub struct MediaChunkStore { } impl MediaChunkStore { + pub fn put_manifest(&self, manifest: &MediaManifest) -> Result<(), MediaStoreError> { + let path = self + .root + .parent() + .unwrap_or(std::path::Path::new(".")) + .join("manifests") + .join(format!("{}.json", manifest.media_oid)); + let io_error = |source| MediaStoreError::Io { + path: path.display().to_string(), + source, + }; + manifest + .validate() + .map_err(|e| io_error(std::io::Error::other(e.to_string())))?; + let json = manifest + .to_json() + .map_err(|e| io_error(std::io::Error::other(e.to_string())))?; + atomic_write::write_atomic(&path, json.as_bytes(), atomic_write::sync_data_enabled()) + .map_err(io_error) + } + /// Use an explicit private cache root (also useful for independent clients). + pub fn at(root: PathBuf) -> Self { + Self { root } + } /// Open the store at the repo's media-chunks root (created lazily on write). pub fn open() -> Self { Self { @@ -63,14 +87,18 @@ impl MediaChunkStore { /// correct bytes rather than silently trusted. Writes RAW bytes (no Git /// header, no zlib) via the crash-safe temp+rename discipline. pub fn put_chunk(&self, bytes: &[u8]) -> Result { + if bytes.len() > super::chunker::MAX_SIZE { + return Err(MediaStoreError::Io { + path: self.root.display().to_string(), + source: std::io::Error::other("chunk exceeds FastCDC size limit"), + }); + } let chunk_hash = sha256_hex(bytes); let path = self.chunk_path(&chunk_hash); if path.exists() { // Content-addressed: if the stored bytes still hash to `chunk_hash` // it is already durably present; otherwise fall through to rewrite. - if let Ok(existing) = std::fs::read(&path) - && sha256_hex(&existing) == chunk_hash - { + if self.get_chunk(&chunk_hash).is_ok() { return Ok(chunk_hash); } } @@ -96,7 +124,26 @@ impl MediaChunkStore { return Err(MediaStoreError::InvalidHash(chunk_hash.to_string())); } let path = self.chunk_path(chunk_hash); - let bytes = match std::fs::read(&path) { + if path + .metadata() + .is_ok_and(|m| m.len() > super::chunker::MAX_SIZE as u64) + { + return Err(MediaStoreError::Io { + path: path.display().to_string(), + source: std::io::Error::other("cached chunk exceeds FastCDC size limit"), + }); + } + let bytes = match std::fs::File::open(&path).and_then(|file| { + let mut bytes = Vec::new(); + file.take(super::chunker::MAX_SIZE as u64 + 1) + .read_to_end(&mut bytes)?; + if bytes.len() > super::chunker::MAX_SIZE { + return Err(std::io::Error::other( + "cached chunk exceeds FastCDC size limit", + )); + } + Ok(bytes) + }) { Ok(b) => b, Err(e) if e.kind() == std::io::ErrorKind::NotFound => { return Err(MediaStoreError::Missing(chunk_hash.to_string())); @@ -140,59 +187,51 @@ pub fn reassemble( source: std::io::Error::other(e.to_string()), })?; - if let Some(parent) = dest.parent() { - std::fs::create_dir_all(parent).map_err(|source| MediaStoreError::Io { - path: parent.display().to_string(), - source, - })?; - } - // Write + verify into a sibling temp; only rename over `dest` after the - // media_oid verifies. EVERY error path (io / missing chunk / corrupt chunk / - // digest mismatch / rename failure) removes the temp so no partial bytes - // survive — the contract is "temp cleaned on any error". - let tmp = dest.with_extension("libra-media-tmp"); - let build = || -> Result<(), MediaStoreError> { - let mut digest = Context::new(&SHA256); - let file = std::fs::File::create(&tmp).map_err(|source| MediaStoreError::Io { - path: tmp.display().to_string(), - source, - })?; - let mut writer = std::io::BufWriter::new(file); - for entry in &manifest.chunks { - let bytes = store.get_chunk(&entry.chunk_hash)?; - digest.update(&bytes); - writer - .write_all(&bytes) - .map_err(|source| MediaStoreError::Io { - path: tmp.display().to_string(), - source, - })?; - } - writer.flush().map_err(|source| MediaStoreError::Io { - path: tmp.display().to_string(), - source, - })?; - drop(writer); - let actual = hex::encode(digest.finish().as_ref()); - if actual != manifest.media_oid { - return Err(MediaStoreError::MediaOidMismatch { - expected: manifest.media_oid.clone(), - actual, - }); - } - Ok(()) + let io_error = |source| MediaStoreError::Io { + path: dest.display().to_string(), + source, }; - if let Err(e) = build() { - let _ = std::fs::remove_file(&tmp); - return Err(e); + // A leaf such as `asset.bin` has an empty parent. Resolve it before passing + // it to the atomic writer, which requires a real staging/target directory. + let target = std::path::absolute(dest).map_err(io_error)?; + let parent = target + .parent() + .filter(|p| !p.as_os_str().is_empty()) + .unwrap_or(std::path::Path::new(".")); + let mut writer = match std::fs::metadata(&target) { + Ok(metadata) => crate::utils::atomic_stream::StreamingAtomicFile::new_in_with_permissions( + parent, + atomic_write::sync_data_enabled(), + metadata.permissions(), + ), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + crate::utils::atomic_stream::StreamingAtomicFile::new_in( + parent, + atomic_write::sync_data_enabled(), + ) + } + Err(error) => return Err(io_error(error)), } - std::fs::rename(&tmp, dest).map_err(|source| { - let _ = std::fs::remove_file(&tmp); - MediaStoreError::Io { - path: dest.display().to_string(), - source, + .map_err(io_error)?; + let mut digest = Context::new(&SHA256); + for entry in &manifest.chunks { + let bytes = store.get_chunk(&entry.chunk_hash)?; + if bytes.len() as u64 != entry.length { + return Err(io_error(std::io::Error::other( + "chunk length does not match manifest", + ))); } - })?; + digest.update(&bytes); + writer.write_all(&bytes).map_err(io_error)?; + } + let actual = hex::encode(digest.finish().as_ref()); + if actual != manifest.media_oid { + return Err(MediaStoreError::MediaOidMismatch { + expected: manifest.media_oid.clone(), + actual, + }); + } + writer.persist(&target).map_err(io_error)?; Ok(()) } @@ -205,6 +244,12 @@ pub fn read_span( length: u64, ) -> Result, MediaStoreError> { use std::io::{Seek, SeekFrom}; + if length > super::chunker::MAX_SIZE as u64 { + return Err(MediaStoreError::Io { + path: "".into(), + source: std::io::Error::other("chunk exceeds FastCDC size limit"), + }); + } file.seek(SeekFrom::Start(offset)) .map_err(|source| MediaStoreError::Io { path: "".to_string(), diff --git a/src/utils/media/manifest.rs b/src/utils/media/manifest.rs index 8cf448b94..6047713bd 100644 --- a/src/utils/media/manifest.rs +++ b/src/utils/media/manifest.rs @@ -21,6 +21,8 @@ use super::{ /// Manifest schema version (bumped on an incompatible on-disk change). pub const MANIFEST_VERSION: u32 = 1; +pub const MAX_MANIFEST_SIZE: usize = 10 * 1024 * 1024; +pub const MAX_CHUNKS: usize = 8192; /// One chunk entry in the manifest. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -138,6 +140,9 @@ impl MediaManifest { /// Parse + fully validate a manifest from JSON text. pub fn from_json(text: &str) -> Result { + if text.len() > MAX_MANIFEST_SIZE { + return Err(ManifestError::Invalid("manifest exceeds size limit".into())); + } let manifest: MediaManifest = serde_json::from_str(text).map_err(|e| ManifestError::Serde(e.to_string()))?; manifest.validate()?; @@ -149,10 +154,36 @@ impl MediaManifest { serde_json::to_string_pretty(self).map_err(|e| ManifestError::Serde(e.to_string())) } + /// Protocol identity shared with Mega. Provenance and the fallback pointer + /// do not change the identity of the frozen content-defined chunk layout. + pub fn id(&self) -> Result { + self.validate()?; + let bytes = serde_json::to_vec(&( + self.version, + &self.algorithm, + &self.hash_algorithm, + &self.media_oid, + self.media_size, + &self.chunks, + )) + .map_err(|error| ManifestError::Serde(error.to_string()))?; + Ok(super::sha256_hex(&bytes)) + } + /// Validate the frozen invariants: version/algorithm/hash, a 64-hex /// `media_oid`, first-chunk-offset-0, contiguity, and that the chunk lengths /// sum to `media_size`. Returns an actionable error on any violation. pub fn validate(&self) -> Result<(), ManifestError> { + if self.chunks.len() > MAX_CHUNKS + || self + .fallback_oid + .as_ref() + .is_some_and(|oid| oid != &self.media_oid) + { + return Err(ManifestError::Invalid( + "too many chunks or mismatched fallback_oid".into(), + )); + } if self.version != MANIFEST_VERSION { return Err(ManifestError::Invalid(format!( "unsupported manifest version {} (this binary supports {MANIFEST_VERSION})", @@ -179,6 +210,11 @@ impl MediaManifest { } let mut expected_offset = 0u64; for (i, c) in self.chunks.iter().enumerate() { + if c.length == 0 || c.length > chunker::MAX_SIZE as u64 || c.checksum.is_some() { + return Err(ManifestError::Invalid(format!( + "chunk {i} has invalid length or unsupported checksum" + ))); + } if c.offset != expected_offset { return Err(ManifestError::Invalid(format!( "chunk {i} offset {} breaks contiguity (expected {expected_offset})", @@ -204,7 +240,9 @@ impl MediaManifest { c.encoded_length, c.length ))); } - expected_offset += c.length; + expected_offset = expected_offset + .checked_add(c.length) + .ok_or_else(|| ManifestError::Invalid("chunk offset overflow".into()))?; } if expected_offset != self.media_size { return Err(ManifestError::Invalid(format!( @@ -265,6 +303,19 @@ mod tests { assert!(!json.contains("checksum")); } + #[test] + fn rejects_zero_oversize_chunks_and_mismatched_fallback() { + for length in [0, u64::MAX, chunker::MAX_SIZE as u64 + 1] { + let mut m = sample(); + m.chunks[0].length = length; + m.chunks[0].encoded_length = length; + assert!(m.validate().is_err()); + } + let mut m = sample(); + m.fallback_oid = Some("f".repeat(64)); + assert!(m.validate().is_err()); + } + #[test] fn rejects_bad_version_algo_oid_and_contiguity() { let mut m = sample(); diff --git a/src/utils/media/mod.rs b/src/utils/media/mod.rs index ca828da8d..0174ba500 100644 --- a/src/utils/media/mod.rs +++ b/src/utils/media/mod.rs @@ -1,16 +1,14 @@ -//! FastCDC LFS media chunking — CLIENT substrate (lore.md §6). +//! FastCDC LFS media chunking and authenticated Mega transport (lore.md §6). //! //! This module is the honest v1 of lore.md §6 "LFS FastCDC chunking": a //! strictly feature-gated (`fastcdc`, default OFF) **client** layer that //! content-defines chunks of a media object, builds a versioned manifest, //! stores chunks in a local content-addressed store, reassembles + verifies, //! and negotiates a remote's chunked-vs-standard-LFS capability with an airtight -//! safe fallback. It ships ZERO real cross-machine chunked transfer: the -//! Libra-aware media SERVER (§6.5–6.8 endpoints, chunk upload/download, -//! manifest finalize, GC/fsck/heal, and every §6.7 anti-side-channel guarantee) -//! is a separate deliverable that is honestly FROZEN in lore.md §6 — against -//! every reachable remote today the capability probe resolves to standard Git -//! LFS fallback. +//! safe fallback. [`transfer`] connects the feature-enabled LFS upload/download +//! paths to Mega's opt-in media server. Shared repository ACLs, server-side GC, +//! obliteration, and byte-range hydration remain separate work; see the media +//! command documentation for the current owner/repository isolation boundary. //! //! ## Invariants (load-bearing) //! @@ -33,6 +31,7 @@ pub mod chunk_store; pub mod chunker; pub mod manifest; pub mod negotiate; +pub mod transfer; use ring::digest::{Context, SHA256}; diff --git a/src/utils/media/transfer.rs b/src/utils/media/transfer.rs new file mode 100644 index 000000000..e70c8d527 --- /dev/null +++ b/src/utils/media/transfer.rs @@ -0,0 +1,261 @@ +//! Real FastCDC upload/download over the authenticated Mega media extension. +//! Standard LFS remains the fallback before a transfer starts. Once a manifest +//! is selected, integrity/authentication failures fail closed. +use std::{collections::HashSet, path::Path, time::Duration}; + +use anyhow::{Context, Result, bail}; +use futures_util::StreamExt; +use reqwest::{Client, Method, RequestBuilder, Response, StatusCode}; +use serde::{Deserialize, de::DeserializeOwned}; +use tokio::io::{AsyncReadExt, AsyncSeekExt}; +use url::Url; + +use super::{ + capability, + chunk_store::{self, MediaChunkStore}, + chunker, is_sha256_hex, + manifest::{MAX_MANIFEST_SIZE, MediaManifest}, + negotiate::{self, ProbeOutcome, TransferDecision}, + sha256_hex, +}; + +#[derive(Deserialize)] +struct PrepareResponse { + manifest_id: String, + missing_chunks: Vec, +} + +#[derive(Deserialize)] +struct ManifestResponse { + manifest_id: String, + manifest: MediaManifest, +} + +pub struct MediaClient { + client: Client, + base: Url, + token: Option, + max_manifest: usize, +} + +impl MediaClient { + pub async fn discover( + client: Client, + lfs_url: &Url, + local_fallback: bool, + ) -> Result> { + let outcome = capability::probe_with_client(lfs_url.as_str(), client.clone()).await; + // Live transfers require the server to retain a complete basic LFS + // object. A chunk-only advertisement cannot disable a valid basic + // download that the batch endpoint already offered. + if let ProbeOutcome::Ok(caps) = &outcome + && (!caps.supports_standard_lfs_fallback || caps.max_manifest_size == 0) + { + return Ok(None); + } + match negotiate::negotiate(&outcome, true, local_fallback) { + TransferDecision::StandardLfs { .. } => return Ok(None), + TransferDecision::Block { reason } => { + bail!("FastCDC transfer blocked: {}", reason.as_str()) + } + TransferDecision::Chunked { .. } => (), + } + let ProbeOutcome::Ok(caps) = outcome else { + return Ok(None); + }; + let mut base = lfs_url.clone(); + base.set_path(&format!( + "{}/libra/media/v1/", + lfs_url.path().trim_end_matches('/') + )); + base.set_fragment(None); + let token = match crate::internal::auth::HostScope::from_request_url(&base) { + Some(scope) => match crate::internal::auth::lookup(&scope).await { + crate::internal::auth::Lookup::Valid { token, .. } => Some(token), + _ => None, + }, + None => None, + }; + Ok(Some(Self { + client, + base, + token, + max_manifest: caps.max_manifest_size.min(MAX_MANIFEST_SIZE as u64) as usize, + })) + } + + fn request(&self, method: Method, path: &str) -> Result { + let mut url = self.base.clone(); + url.set_path(&format!("{}{path}", self.base.path())); + let mut request = self + .client + .request(method, url) + .header("Accept", "application/json") + .timeout(Duration::from_secs(120)); + if let Some(token) = &self.token { + request = request.bearer_auth(token); + } + Ok(request) + } + + async fn bytes(response: Response, limit: usize) -> Result> { + let response = response + .error_for_status() + .context("FastCDC request rejected by remote")?; + let mut bytes = Vec::new(); + let mut stream = response.bytes_stream(); + while let Some(part) = stream.next().await { + let part = part.context("failed to read FastCDC response")?; + if part.len() > limit.saturating_sub(bytes.len()) { + bail!("FastCDC response exceeds size limit"); + } + bytes.extend_from_slice(&part); + } + Ok(bytes) + } + + async fn json(&self, response: Response) -> Result { + serde_json::from_slice(&Self::bytes(response, self.max_manifest).await?) + .context("invalid FastCDC response JSON") + } + + async fn finalized_manifest(&self, oid: &str, size: u64) -> Result> { + if !is_sha256_hex(oid) { + bail!("invalid LFS object SHA-256"); + } + let response = self + .request(Method::GET, &format!("manifests/by-media/{oid}"))? + .send() + .await?; + if response.status() == StatusCode::NOT_FOUND { + return Ok(None); + } + let response: ManifestResponse = self.json(response).await?; + let manifest = response.manifest; + if response.manifest_id != manifest.id()? + || manifest.media_oid != oid + || manifest.media_size != size + { + bail!("FastCDC manifest does not match requested LFS object"); + } + Ok(Some(manifest)) + } + + /// Returns false before sending a manifest if this object's layout exceeds + /// the extension's limits; the caller may then use its basic upload action. + pub async fn upload(&self, oid: &str, size: u64, path: &Path) -> Result { + // Also repairs the crash window where a previous finalize wrote the + // complete LFS fallback but failed before publishing the manifest. + if self.finalized_manifest(oid, size).await?.is_some() { + return Ok(true); + } + if size > (super::manifest::MAX_CHUNKS * chunker::MAX_SIZE) as u64 { + return Ok(false); + } + let source = path.to_path_buf(); + let (mut manifest, _) = + tokio::task::spawn_blocking(move || MediaManifest::build_from_file(source)).await??; + if manifest.chunks.len() > super::manifest::MAX_CHUNKS { + return Ok(false); + } + manifest.validate()?; + if manifest.media_oid != oid || manifest.media_size != size { + bail!("local LFS object size or SHA-256 mismatch"); + } + manifest.fallback_oid = Some(oid.to_owned()); + let body = serde_json::to_vec(&manifest)?; + if body.len() > self.max_manifest { + return Ok(false); + } + let response = self + .request(Method::POST, "manifests")? + .header("Content-Type", "application/json") + .body(body) + .send() + .await?; + let prepared: PrepareResponse = self.json(response).await?; + if prepared.manifest_id != manifest.id()? { + bail!("remote returned a manifest ID that does not match the uploaded manifest"); + } + let mut seen = HashSet::new(); + let mut source = tokio::fs::File::open(path) + .await + .context("cannot reopen LFS source")?; + for hash in prepared.missing_chunks { + let chunk = manifest + .chunks + .iter() + .find(|chunk| chunk.chunk_hash == hash) + .ok_or_else(|| anyhow::anyhow!("remote requested a chunk outside the manifest"))?; + if !seen.insert(hash.clone()) { + continue; + } + source.seek(std::io::SeekFrom::Start(chunk.offset)).await?; + let mut bytes = vec![0; chunk.length as usize]; + source.read_exact(&mut bytes).await?; + if sha256_hex(&bytes) != hash { + bail!("LFS source changed while uploading"); + } + self.request( + Method::PUT, + &format!("manifests/{}/chunks/{hash}", prepared.manifest_id), + )? + .header("Content-Type", "application/octet-stream") + .body(bytes) + .send() + .await? + .error_for_status() + .context("FastCDC chunk upload rejected")?; + } + self.request( + Method::POST, + &format!("manifests/{}/finalize", prepared.manifest_id), + )? + // Finalize reconstructs and checks the entire file, unlike a bounded + // chunk request. Large LFS objects need a longer verification window. + .timeout(Duration::from_secs(60 * 60)) + .send() + .await? + .error_for_status() + .context("FastCDC finalize failed; rerun push to resume")?; + Ok(true) + } + + /// Returns false only when no finalized manifest exists. Valid local chunks + /// are reused, so a failed/interrupted download resumes at variable boundaries. + pub async fn download( + &self, + oid: &str, + size: u64, + path: &Path, + store: &MediaChunkStore, + ) -> Result { + let Some(manifest) = self.finalized_manifest(oid, size).await? else { + return Ok(false); + }; + for chunk in &manifest.chunks { + if let Ok(bytes) = store.get_chunk(&chunk.chunk_hash) + && bytes.len() as u64 == chunk.length + { + continue; + } + let bytes = Self::bytes( + self.request( + Method::GET, + &format!("manifests/by-media/{oid}/chunks/{}", chunk.chunk_hash), + )? + .send() + .await?, + chunker::MAX_SIZE, + ) + .await?; + if bytes.len() as u64 != chunk.length || sha256_hex(&bytes) != chunk.chunk_hash { + bail!("FastCDC chunk size or SHA-256 mismatch"); + } + store.put_chunk(&bytes)?; + } + chunk_store::reassemble(&manifest, store, path)?; + store.put_manifest(&manifest)?; + Ok(true) + } +} diff --git a/tests/INDEX.md b/tests/INDEX.md index 4c398c03f..60eadac51 100644 --- a/tests/INDEX.md +++ b/tests/INDEX.md @@ -28,6 +28,7 @@ | `compat_live_compat_workflow` | 1 | Guards optional live AI/cloud workflow remains manual/scheduled and secret-gated | `.github/workflows/live-compat.yml` | | `compat_branch_lossy_wrapper_guard` | 1 | Guards branch-name lossy conversion wrapper | `src/internal/branch.rs` | | `compat_lfs_client_production_unwrap_guard` | 1 | Bans `unwrap()/expect()` in `internal/protocol/lfs_client.rs` | `src/internal/protocol/lfs_client.rs` | +| `media_fastcdc_test` | 1 / manual interop | Feature-gated chunk/cache/verify/probe tests; ignored `mega_fastcdc_http_interop` exercises real Mega HTTP upload, dedup, resume and download (requires `MEGA_FASTCDC_READY_FILE`) | `src/utils/media/`, `src/internal/protocol/lfs_client.rs` | | `compat_config_production_unwrap_guard` | 1 | Bans `unwrap()/expect()` in `internal/config.rs` | `src/internal/config.rs` | | `compat_head_production_unwrap_guard` | 1 | Bans `unwrap()/expect()` in `internal/head.rs` | `src/internal/head.rs` | | `compat_util_production_unwrap_guard` | 1 | Bans `unwrap()/expect()` in `common_utils.rs` / `utils/` | `src/common_utils.rs`, `src/utils/` | diff --git a/tests/SERIAL_REGISTRY.tsv b/tests/SERIAL_REGISTRY.tsv index 8b9992440..9ae0c3e22 100644 --- a/tests/SERIAL_REGISTRY.tsv +++ b/tests/SERIAL_REGISTRY.tsv @@ -100,6 +100,7 @@ machine_pull_emits_single_json_line lane:env mutates or depends on the process e manifest_roles_oids_and_lengths_match_actual_blobs lane:cloud_live+cwd+env+hash_kind+workspace_failpoints fail-closed body (TA-01 global verdict) mechanically expanded by TA-03 to the FULL named key set — an unkeyed #[serial] locks only the empty-string key, so global exclusion strength requires every resource key: cloud_live+cwd+env+hash_kind+workspace_failpoints mcp_created_task_is_observable_through_web_sse lane:cloud_live+cwd+env+hash_kind+workspace_failpoints fail-closed body (TA-01 global verdict) mechanically expanded by TA-03 to the FULL named key set — an unkeyed #[serial] locks only the empty-string key, so global exclusion strength requires every resource key: cloud_live+cwd+env+hash_kind+workspace_failpoints mcp_http_and_stdio_expose_identical_tool_set lane:cloud_live+cwd+env+hash_kind+workspace_failpoints fail-closed body (TA-01 global verdict) mechanically expanded by TA-03 to the FULL named key set — an unkeyed #[serial] locks only the empty-string key, so global exclusion strength requires every resource key: cloud_live+cwd+env+hash_kind+workspace_failpoints +mega_fastcdc_http_interop lane:cwd changes the process working directory (ChangeDirGuard/set_current_dir) metadata_model_field_prefers_event_model lane:cloud_live+cwd+env+hash_kind+workspace_failpoints fail-closed body (TA-01 global verdict) mechanically expanded by TA-03 to the FULL named key set — an unkeyed #[serial] locks only the empty-string key, so global exclusion strength requires every resource key: cloud_live+cwd+env+hash_kind+workspace_failpoints oversized_message_is_rejected_before_reaching_runtime lane:cloud_live+cwd+env+hash_kind+workspace_failpoints fail-closed body (TA-01 global verdict) mechanically expanded by TA-03 to the FULL named key set — an unkeyed #[serial] locks only the empty-string key, so global exclusion strength requires every resource key: cloud_live+cwd+env+hash_kind+workspace_failpoints package_install_auto_enables_non_mutating_but_defers_mutating lane:cwd changes the process working directory (ChangeDirGuard/set_current_dir) diff --git a/tests/media_fastcdc_test.rs b/tests/media_fastcdc_test.rs index 9688647ee..7f4549ed4 100644 --- a/tests/media_fastcdc_test.rs +++ b/tests/media_fastcdc_test.rs @@ -1,13 +1,10 @@ //! Integration tests for `libra media` — the feature-gated FastCDC LFS media //! chunking client (lore.md §6). Compiled only under `--features fastcdc`. //! -//! Verifies the client substrate end-to-end without any server: chunk+store, -//! reassemble+verify (incl. a corrupt-chunk failure that writes no output), a -//! valid `--json` envelope, manifest inspect, and the §6.4 safe fallback of the -//! capability probe against an unreachable endpoint (→ standard LFS). -//! -//! Layer: L1 (tempdir + isolated HOME; the only "network" is a connection to a -//! refused loopback port, which resolves immediately to a no-endpoint fallback). +//! Covers local chunk/store/verify, bounded HTTP transfers against loopback +//! fixtures, integrity failures, and ordinary LFS fallback. An ignored test +//! connects the real Libra client to Mega's production media router. +//! Layer: L1 by default (temporary directories and local loopback only). #![cfg(feature = "fastcdc")] use std::{ @@ -20,6 +17,486 @@ fn media_bin() -> &'static str { env!("CARGO_BIN_EXE_libra") } +fn supported_capabilities(fallback: bool) -> serde_json::Value { + serde_json::json!({ + "version": "1", "chunked_lfs": true, + "chunk_algorithms": ["fastcdc-v1"], "hash_algorithms": ["sha256"], + "max_chunk_size": 8 * 1024 * 1024, "max_manifest_size": 10 * 1024 * 1024, + "supports_batch_exists": true, "supports_range_read": false, + "supports_standard_lfs_fallback": fallback + }) +} + +#[tokio::test] +async fn invalid_remote_manifest_or_chunk_preserves_existing_destination() { + use std::sync::{ + Arc, + atomic::{AtomicUsize, Ordering}, + }; + + use axum::{Json, Router, routing::get}; + use libra::utils::media::{ + chunk_store::MediaChunkStore, manifest::MediaManifest, transfer::MediaClient, + }; + + let dir = tempfile::tempdir().unwrap(); + let source = dir.path().join("source"); + fs::write(&source, b"correct media").unwrap(); + let (manifest, _) = MediaManifest::build_from_file(&source).unwrap(); + for wrong_identity in [false, true] { + let id = if wrong_identity { + "a".repeat(64) + } else { + manifest.id().unwrap() + }; + let returned = serde_json::json!({"manifest_id": id, "manifest": manifest}); + let chunk_requests = Arc::new(AtomicUsize::new(0)); + let counted = chunk_requests.clone(); + let app = Router::new() + .route( + "/repo.git/info/lfs/libra/media/v1/capabilities", + get(|| async { Json(supported_capabilities(true)) }), + ) + .route( + "/repo.git/info/lfs/libra/media/v1/manifests/by-media/{oid}", + get(move || { + let response = returned.clone(); + async move { Json(response) } + }), + ) + .route( + "/repo.git/info/lfs/libra/media/v1/manifests/by-media/{oid}/chunks/{hash}", + get(move || { + counted.fetch_add(1, Ordering::SeqCst); + async { "corrupt media" } + }), + ) + .layer(axum::middleware::from_fn( + |request: axum::extract::Request, next: axum::middleware::Next| async move { + assert_eq!(request.uri().query(), Some("tenant=test")); + next.run(request).await + }, + )); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base = url::Url::parse(&format!( + "http://{}/repo.git/info/lfs/?tenant=test", + listener.local_addr().unwrap() + )) + .unwrap(); + let task = tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + let client = MediaClient::discover( + reqwest::Client::builder().no_proxy().build().unwrap(), + &base, + false, + ) + .await + .unwrap() + .unwrap(); + let dest = dir.path().join("dest"); + fs::write(&dest, b"keep me").unwrap(); + let store = MediaChunkStore::at(dir.path().join("chunks")); + let error = client + .download(&manifest.media_oid, manifest.media_size, &dest, &store) + .await + .unwrap_err(); + assert!( + error.to_string().contains(if wrong_identity { + "manifest does not match" + } else { + "chunk size or SHA-256 mismatch" + }), + "{error:#}" + ); + assert_eq!( + chunk_requests.load(Ordering::SeqCst), + usize::from(!wrong_identity) + ); + assert_eq!(fs::read(dest).unwrap(), b"keep me"); + task.abort(); + } +} + +#[tokio::test] +async fn ordinary_lfs_server_falls_back_to_full_transfer() { + use std::sync::{ + Arc, + atomic::{AtomicBool, Ordering}, + }; + + use axum::{ + Json, Router, + body::Bytes, + routing::{get, post, put}, + }; + use libra::{internal::protocol::lfs_client::LFSClient, utils::media::transfer::MediaClient}; + + // No extension, chunk-only policy, and an extension whose manifest limit + // is too small must all retain the standard complete-object path. + for (advertise, fallback, small_manifest) in [ + (false, true, false), + (true, false, false), + (true, true, true), + ] { + let data = "plain LFS bytes"; + let oid = + hex::encode(ring::digest::digest(&ring::digest::SHA256, data.as_bytes()).as_ref()); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let remote = format!("http://{}/repo.git", listener.local_addr().unwrap()); + let base = format!("{remote}/info/lfs/"); + let object_base = format!("{base}objects/"); + let uploaded = Arc::new(AtomicBool::new(false)); + let saved = uploaded.clone(); + let mut app = Router::new() + .route("/repo.git/info/lfs/objects/batch", post(move |Json(request): Json| { + let object_base = object_base.clone(); + async move { + let action = request["operation"].as_str().unwrap(); + let requested_oid = request["objects"][0]["oid"].as_str().unwrap(); + let href = format!("{object_base}{requested_oid}"); + Json(serde_json::json!({"transfer":"basic","objects":[{ + "oid":requested_oid, "size":data.len(), "actions":{(action):{"href":href,"expires_at":""}} + }]})) + } + })) + .route("/repo.git/info/lfs/objects/{oid}", get(move || async move { data }).merge(put(move |body: Bytes| { + assert_eq!(body.as_ref(), data.as_bytes()); + saved.store(true, Ordering::SeqCst); + async { axum::http::StatusCode::OK } + }))); + if advertise { + app = app.route( + "/repo.git/info/lfs/libra/media/v1/capabilities", + get(move || async move { + let mut caps = supported_capabilities(fallback); + if small_manifest { + caps["max_manifest_size"] = serde_json::json!(32); + } + Json(caps) + }), + ); + } + let task = tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + let mut client = LFSClient::from_remote_url(&remote).unwrap(); + client.client = reqwest::Client::builder().no_proxy().build().unwrap(); + // A chunk-only advertisement must not block either basic operation. + assert_eq!( + MediaClient::discover(client.client.clone(), &client.lfs_url, false) + .await + .unwrap() + .is_some(), + small_manifest, + ); + let dir = tempfile::tempdir().unwrap(); + let source = dir.path().join("source"); + fs::write(&source, data).unwrap(); + assert!(client.push_object(&oid, &source).await.unwrap()); + assert!(uploaded.load(Ordering::SeqCst)); + let dest = dir.path().join("dest"); + client + .download_object(&oid, data.len() as u64, &dest, None) + .await + .unwrap(); + assert_eq!(fs::read(&dest).unwrap(), data.as_bytes()); + // A legal SHA-256 OID for different content reaches the checksum error + // path. Its replacement pointer must be visible as soon as we return. + let wrong_oid = hex::encode( + ring::digest::digest(&ring::digest::SHA256, b"a different LFS object").as_ref(), + ); + let error = client + .download_object(&wrong_oid, data.len() as u64, &dest, None) + .await + .expect_err("incorrect object bytes must fail checksum verification"); + assert!(error.to_string().contains("Checksum mismatch"), "{error:#}"); + assert_eq!( + fs::read_to_string(&dest).unwrap(), + libra::utils::lfs::format_pointer_string(&wrong_oid, data.len() as u64), + "the checksum error must flush the complete fallback pointer before returning" + ); + task.abort(); + } +} + +#[test] +fn relative_reassembly_target_is_replaced_only_after_verification() { + use libra::utils::media::{ + chunk_store::{self, MediaChunkStore}, + manifest::MediaManifest, + }; + let dir = tempfile::tempdir().unwrap(); + let source = dir.path().join("source"); + fs::write(&source, b"verified replacement").unwrap(); + let (mut manifest, _) = MediaManifest::build_from_file(&source).unwrap(); + let store = MediaChunkStore::at(dir.path().join("chunks")); + store.put_chunk(b"verified replacement").unwrap(); + // Use an actual leaf path without changing the process working directory. + let target = tempfile::NamedTempFile::new_in(".") + .unwrap() + .into_temp_path(); + let relative = Path::new(target.file_name().unwrap()); + fs::write(relative, b"old contents").unwrap(); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + fs::set_permissions(relative, fs::Permissions::from_mode(0o750)).unwrap(); + } + chunk_store::reassemble(&manifest, &store, relative).unwrap(); + assert_eq!(fs::read(relative).unwrap(), b"verified replacement"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!( + fs::metadata(relative).unwrap().permissions().mode() & 0o777, + 0o750 + ); + } + manifest.media_oid = "a".repeat(64); + assert!(chunk_store::reassemble(&manifest, &store, relative).is_err()); + assert_eq!(fs::read(relative).unwrap(), b"verified replacement"); +} + +/// Run against Mega's real production media router (isolated test database). +/// See Mega docs/lfs-api.md for the two-process invocation. +#[tokio::test] +#[ignore = "requires Mega serve_libra_interop and MEGA_FASTCDC_READY_FILE"] +#[serial_test::serial(cwd)] +async fn mega_fastcdc_http_interop() { + use libra::{ + internal::protocol::lfs_client::LFSClient, + utils::{ + media::{chunk_store::MediaChunkStore, manifest::MediaManifest, transfer::MediaClient}, + test::ChangeDirGuard, + }, + }; + use reqwest::header::{AUTHORIZATION, HeaderMap, HeaderValue}; + + let ready = std::env::var("MEGA_FASTCDC_READY_FILE").expect("MEGA_FASTCDC_READY_FILE required"); + let connection: serde_json::Value = serde_json::from_slice(&fs::read(ready).unwrap()).unwrap(); + let lfs_url = url::Url::parse(connection["lfs_url"].as_str().unwrap()).unwrap(); + let mut headers = HeaderMap::new(); + headers.insert( + AUTHORIZATION, + HeaderValue::from_str(&format!("Bearer {}", connection["token"].as_str().unwrap())) + .unwrap(), + ); + let http = reqwest::Client::builder() + .no_proxy() + .default_headers(headers) + .build() + .unwrap(); + let media = MediaClient::discover(http.clone(), &lfs_url, true) + .await + .unwrap() + .expect("Mega must negotiate FastCDC"); + let dir = tempfile::tempdir().unwrap(); + ok(&["init"], dir.path()); + let _cwd = ChangeDirGuard::new(dir.path()); + let remote = lfs_url.as_str().strip_suffix("/info/lfs/").unwrap(); + let mut lfs = LFSClient::from_remote_url(remote).unwrap(); + lfs.client = http.clone(); + let source = dir.path().join("source.bin"); + let mut seed = 0x1234_5678_9abc_def0u64; + let data: Vec = (0..12 * 1024 * 1024) + .map(|_| { + seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + (seed >> 33) as u8 + }) + .collect(); + fs::write(&source, &data).unwrap(); + let (manifest, _) = MediaManifest::build_from_file(&source).unwrap(); + assert!( + manifest + .chunks + .windows(2) + .any(|c| c[0].length != c[1].length) + ); + let base = lfs_url.join("libra/media/v1/").unwrap(); + let prepared: serde_json::Value = http + .post(base.join("manifests").unwrap()) + .json(&manifest) + .send() + .await + .unwrap() + .error_for_status() + .unwrap() + .json() + .await + .unwrap(); + let id = prepared["manifest_id"].as_str().unwrap(); + let chunk = &manifest.chunks[0]; + http.put( + base.join(&format!("manifests/{id}/chunks/{}", chunk.chunk_hash)) + .unwrap(), + ) + .body(data[..chunk.length as usize].to_vec()) + .send() + .await + .unwrap() + .error_for_status() + .unwrap(); + // A restart resumes from the server's persisted missing-chunk response. + assert!(lfs.push_object(&manifest.media_oid, &source).await.unwrap()); + let published: serde_json::Value = http + .get( + base.join(&format!("manifests/by-media/{}", manifest.media_oid)) + .unwrap(), + ) + .send() + .await + .unwrap() + .error_for_status() + .unwrap() + .json() + .await + .unwrap(); + assert_eq!(published["manifest_id"], manifest.id().unwrap()); + let dedup: serde_json::Value = http + .post(base.join("manifests").unwrap()) + .json(&manifest) + .send() + .await + .unwrap() + .error_for_status() + .unwrap() + .json() + .await + .unwrap(); + assert!(dedup["missing_chunks"].as_array().unwrap().is_empty()); + assert!(lfs.push_object(&manifest.media_oid, &source).await.unwrap()); + let cache_root = dir.path().join(".libra/media/chunks"); + let store = MediaChunkStore::at(cache_root.clone()); + // Simulate a previously downloaded first chunk, then reconstruct the rest. + store.put_chunk(&data[..chunk.length as usize]).unwrap(); + let output = dir.path().join("download.bin"); + fs::write(&output, b"previous contents").unwrap(); + lfs.download_object(&manifest.media_oid, manifest.media_size, &output, None) + .await + .unwrap(); + assert_eq!(fs::read(&output).unwrap(), data); + assert!( + dir.path() + .join(".libra/media/manifests") + .join(format!("{}.json", manifest.media_oid)) + .exists(), + "ordinary LFS download must select FastCDC and persist its manifest" + ); + let cached = cache_root + .join(&chunk.chunk_hash[..2]) + .join(&chunk.chunk_hash[2..]); + fs::write(cached, b"corrupt cache").unwrap(); + lfs.download_object(&manifest.media_oid, manifest.media_size, &output, None) + .await + .unwrap(); + assert_eq!(fs::read(&output).unwrap(), data); + let bob = reqwest::Client::builder() + .no_proxy() + .default_headers(HeaderMap::from_iter([( + AUTHORIZATION, + HeaderValue::from_static("Bearer test-bob"), + )])) + .build() + .unwrap(); + let mut bob_lfs = LFSClient::from_remote_url(remote).unwrap(); + bob_lfs.client = bob.clone(); + let bob = MediaClient::discover(bob, &lfs_url, false) + .await + .unwrap() + .unwrap(); + assert!( + !bob.download(&manifest.media_oid, manifest.media_size, &output, &store) + .await + .unwrap() + ); + bob_lfs + .download_object(&manifest.media_oid, manifest.media_size, &output, None) + .await + .unwrap(); + assert_eq!( + fs::read(&output).unwrap(), + data, + "other users retain complete standard LFS access" + ); + + // Reproduce the finalize crash window: a complete basic object is present, + // but this user's manifest has not been published. Batch now omits upload + // actions; the normal push path must still repair the missing manifest. + let recover = dir.path().join("recover.bin"); + fs::write(&recover, b"complete fallback without a manifest").unwrap(); + let (recover_manifest, _) = MediaManifest::build_from_file(&recover).unwrap(); + let batch: serde_json::Value = http + .post(lfs.batch_url.clone()) + .json(&serde_json::json!({ + "operation":"upload", "transfers":["basic"], "hash_algo":"sha256", + "objects":[{"oid":recover_manifest.media_oid,"size":recover_manifest.media_size}] + })) + .send() + .await + .unwrap() + .error_for_status() + .unwrap() + .json() + .await + .unwrap(); + http.put( + batch["objects"][0]["actions"]["upload"]["href"] + .as_str() + .unwrap(), + ) + .body(fs::read(&recover).unwrap()) + .send() + .await + .unwrap() + .error_for_status() + .unwrap(); + assert_eq!( + http.get( + base.join(&format!( + "manifests/by-media/{}", + recover_manifest.media_oid + )) + .unwrap() + ) + .send() + .await + .unwrap() + .status(), + reqwest::StatusCode::NOT_FOUND + ); + assert!( + lfs.push_object(&recover_manifest.media_oid, &recover) + .await + .unwrap() + ); + http.get( + base.join(&format!( + "manifests/by-media/{}", + recover_manifest.media_oid + )) + .unwrap(), + ) + .send() + .await + .unwrap() + .error_for_status() + .unwrap(); + let empty = dir.path().join("empty.bin"); + fs::write(&empty, []).unwrap(); + let (manifest, _) = MediaManifest::build_from_file(&empty).unwrap(); + assert!(lfs.push_object(&manifest.media_oid, &empty).await.unwrap()); + assert!( + media + .download(&manifest.media_oid, 0, &output, &store) + .await + .unwrap() + ); + assert!(fs::read(&output).unwrap().is_empty()); +} + fn run(args: &[&str], cwd: &Path) -> Output { let home = cwd.join(".libra-test-home"); fs::create_dir_all(home.join(".config")).unwrap();