From 7929f4f1aacf518a04fad24f129215b7468d2105 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 13:42:02 +0800 Subject: [PATCH 01/16] =?UTF-8?q?feat(build.mcpp):=20=E6=8C=87=E4=BB=A4?= =?UTF-8?q?=E5=AE=9A=E4=B9=89=E6=94=B6=E6=95=9B=E6=88=90=E4=B8=80=E5=BC=A0?= =?UTF-8?q?=E8=A1=A8=20+=20=E5=8D=8F=E8=AE=AE=E7=89=88=E6=9C=AC/=E7=BC=93?= =?UTF-8?q?=E5=AD=98=20epoch/=E8=BF=90=E8=A1=8C=E4=B8=8A=E9=99=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `build.mcpp` 机制的架构地基。设计见 `.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md`(本次实现步 0 + 步 1)。 ## 新增一条 directive 从改 9 处降到改 1 处 一条指令原本要在九个地方定义:`Directives` 字段、`parse_line` 分派、 `write_cache`、`read_cache`、`apply`、`cache_fresh` 的产物存在性校验、 `prepare.cppm` 的 `DirectiveMark`、`markDirectiveTail`、 `foldDirectiveTailIntoPrivateBuild`,再加内置 `mcpp` 模块的类型化包装。 `prepare.cppm:2611` 的注释还**自己承认**这个拆分仍不完整 (「Link/source/fingerprint residues stay at the call sites」)。 这是本仓库反复付过学费的「同一决策在 N 处推导」形态(#233/#240/#242/#344): 它不在你新增指令时失败,而是过一阵子在别的地方失败。 现在一条指令 = 新模块 `src/build/directives.cppm` 里 `kTable` 的**一行**, 解析 / 缓存读写 / 落盘应用 / 声明产物契约 / 私有作用域折叠全部由该行驱动。 **作用域是必填字段**:`include-dir` 之所以私有,是「构建期程序不得静默拓宽包的 公开接口」这条供应链规则,把它变成表里一列意味着下一条指令不回答这个问题就加不进来。 为什么是新模块而不是继续写 `build_program.cppm`:那个文件的匿名 namespace 在 clang 22 + C++20 modules + -O2 下会**误编译自己的邻居**(PR#332 证实一个从未被 调用的新函数就足以破坏 `contract_env`,PR#334 复现)。`mcpp.build.hostprogram` 当初就是为此拆出去的。 ## S1 线协议版本 用 `import mcpp;` 的程序在 `main` 之前自动声明 `mcpp:protocol=`(占位符替换 注入,不硬编码 ⇒ 声明值与引擎校验值不可能漂移)。于是: - 声明的协议**高于**本 mcpp 所理解 → 拒绝执行并给升级提示。原行为是警告后忽略, 那会让「构建成功了但那个 flag 根本没到」静默发生。 - 双方已被证明一致 ⇒ **未知指令即错误**(同版本内只可能是拼写错误)。 手写 printf 的程序什么都不声明,**保留**警告并忽略。这条不对称就是兼容性契约: 那一面冻结在现有 11 条指令上,新能力只在类型化 API 里落地。 ## S2 缓存语义 epoch 缓存里的指令是命中时**原样重放**的,所以引擎改变某条指令的**解释**时旧条目必须 失效。纪律照抄 `cache_key::kCacheEpoch`(只在旧条目真不可用时 bump,与 mcpp 版本 号解耦)。一条本 mcpp 不认识的 `d` 记录同样作废整条条目 —— 只重放认识的那部分, 等于应用了程序所要求的一个**真子集**。 ## S3 运行上限 默认 600s,超时杀掉并让构建失败,错误**点名是哪个包**并说明怎么改。原先没有任何 上限:死循环或卡在网络读上的构建程序会让整个构建挂死且毫无诊断。 编译刻意不设上限 —— 与 `mcpp test` 同一条不对称纪律(run 有限 / build 无限): 编译跑得久通常正当(首次构建 std 模块就是分钟级),杀掉只会产生莫名其妙的失败。 `capture_exec_deadline` 顺带补上 `cwd` 形参 —— 没有它,加超时会**静默改变**构建 程序相对写入的落点。 ## 其它 - 内带 xlings 升级到 2026.8.5.1(13 个 pin 由 check_version_pins.sh 机器校验) - 版本 2026.8.5.1 ## 验证 - 单测 56/56(新增 `BuildDirectives` 21 例:表完整性、解析、协议三分支、 缓存 round-trip、apply 路由、私有折叠) - e2e:12 个 build.mcpp / 生成源用例全绿,新增 186 号覆盖协议三分支 + 超时 + 缓存四条失效路径 - `03_multi_module` 在本机失败,但已发布的 2026.8.4.1 **同样失败** ⇒ 与本次无关 --- ...5-build-mcpp-extensibility-architecture.md | 401 +++++++++ ...5-issue355-dependency-host-tools-design.md | 810 ++++++++++++++++++ .github/actions/bootstrap-mcpp/action.yml | 2 +- .github/actions/setup-macos-llvm/action.yml | 2 +- .github/workflows/bootstrap-macos.yml | 2 +- .github/workflows/ci-fresh-install.yml | 6 +- .github/workflows/ci-linux-e2e.yml | 2 +- .github/workflows/cross-build-test.yml | 4 +- .github/workflows/release.yml | 14 +- CHANGELOG.md | 29 + docs/07-build-mcpp.md | 41 +- docs/zh/07-build-mcpp.md | 31 +- mcpp.toml | 2 +- src/build/build_program.cppm | 269 +++--- src/build/directives.cppm | 453 ++++++++++ src/build/hostprogram.cppm | 26 + src/build/prepare.cppm | 61 +- src/platform/process.cppm | 16 +- src/version.cppm | 2 +- src/xlings.cppm | 2 +- .../e2e/186_build_mcpp_protocol_and_bound.sh | 151 ++++ tests/unit/test_build_directives.cpp | 338 ++++++++ 22 files changed, 2432 insertions(+), 232 deletions(-) create mode 100644 .agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md create mode 100644 .agents/docs/2026-08-05-issue355-dependency-host-tools-design.md create mode 100644 src/build/directives.cppm create mode 100755 tests/e2e/186_build_mcpp_protocol_and_bound.sh create mode 100644 tests/unit/test_build_directives.cpp diff --git a/.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md b/.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md new file mode 100644 index 00000000..4be40dfe --- /dev/null +++ b/.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md @@ -0,0 +1,401 @@ +# build.mcpp 机制架构设计：一个 hook，多种节点 + +> 状态：**步 0 + 步 1 已实施（2026.8.5.1）**；步 2–6 待 review。实施记录见 §9。 +> 范围：`build.mcpp` 作为**扩展机制**的长期形态 —— 不是某个具体特性 +> 关联：#355（依赖产出的 host 工具，是本路线的第一块地基）、#241、L3 原始设计 +> （`.agents/docs/2026-06-30-l3-build-mcpp-implementation-design.md`） +> 涉及（预估）：`src/build/build_program.cppm`、`src/build/hostprogram.cppm`、 +> `src/build/prepare.cppm`、`src/build/plan.cppm`、`src/build/ninja_backend.cppm` + +--- + +## 0. 结论摘要 + +今天的 `build.mcpp` 把**两件本质不同的事**塞进了同一个点： + +- **配置**（configuration）——「这次构建长什么样」：探测宿主、选源码、定 flag。 +- **施工**（build work）——「把这批输入变成那批输出」：codegen、transpile、静态检查、后处理。 + +它天生适合前者，被迫承担后者。**所有已知痛点都是这个混淆的推论**——codegen 没有增量、 +不能并行、失败无法归因到具体输入、拿不到依赖产出的工具（#355）。用户提到的两个未来 +场景（编译前静态检查、把某语言编译成 C++）**都是施工**，直接塞进今天的机制会把这四个 +痛点原样继承一遍。 + +架构主张三条： + +| # | 主张 | 反面 | +|---|---|---| +| **A** | **一个 hook，多种节点** —— 不增加生命周期钩子的数量，增加钩子**输出**的表达力。后处理 = 「以最终产物为输入的节点」，不是新钩子 | 加 pre/post/pre-link/post-link：N 个钩子 = N 套生命周期 × N 份缓存语义 × N 种失败模式，且钩子之间的顺序还要再定义一次 | +| **B** | **声明期必须能算出「文件名集合」，不必算出「文件内容」** —— 这条 mcpp 特有的硬约束（来自 plan/fingerprint/模块扫描）把「什么能进图、什么必须 eager」划得一清二楚 | 假装所有 codegen 都能进图；或反过来假装都不能 | +| **C** | **扩展靠包，不靠 DSL** —— 可复用规则（protobuf codegen、某语言 transpiler、lint 预设）以**普通 mcpp 包**分发，`build.mcpp` `import` 它们的 host 模块 | 引入 Lua/Starlark/YAML 规则 DSL：第二门语言，直接违背 mcpp「用 C++ 写构建」的立身之本 | + +一个可测量的「不优雅」指标：**今天新增一条 directive 要改 9 处代码**（§1.3 清单）。 +架构目标之一是把它降到 **1 处（一张表）**。 + +结构性发现：**主张 C 要求「build.mcpp 能 import 依赖提供的 host 模块」，而 #355 要求 +「build.mcpp 能拿到依赖提供的 host 二进制」—— 这是同一套 host 子构建机制的两种产物 +（`lib` 与 `bin`）。#355 不是孤立特性，它是这条架构路线的第一块地基。** + +在扩表之前，有 **5 个现存的稳定性/兼容性缺口**必须先补（§4）。 + +--- + +## 1. 今天的机制：精确画像 + +### 1.1 一句话 + +一个 host 程序，在 prepare 期跑一次，通过**行式 stdout** 输出**对 `buildConfig` 的补丁**。 + +### 1.2 三张表 + +**生命周期位置**（`src/build/prepare.cppm`） + +| # | 阶段 | 位置 | +|---|---|---| +| 0 | manifest 解析 | — | +| 1 | `[generated_files]` 物化 | `:1664`（**依赖解析之前**，可产出 `build.mcpp` 自身） | +| 2 | 依赖解析 / feature 激活 | `:1725`–`:3830` | +| 3 | **依赖的 build.mcpp** | `:3841` | +| 4 | **root 的 build.mcpp** | `:3985` | +| 5 | modgraph 扫描 / fingerprint | `:4100`–`:4220` | +| 6 | BuildPlan | `:4239` | +| 7 | build.ninja → 编译链接 | `execute.cppm:292` | +| 8 | 分发（`mcpp pack`） | `src/pack/` | + +**用户逻辑今天只能在 3/4。5–8 完全进不去。** + +**输出通道与作用域**（`build_program.cppm`） + +| directive | 落点 | 作用域 | +|---|---|---| +| `cxxflag` / `cflag` / `cfg` | `buildConfig.{cxx,c}flags` → `privateBuild` | 本包 TU | +| `link-lib` / `link-search` | `buildConfig.ldflags` → 转发到 root | 最终链接 | +| `generated` / `source` | `buildConfig.sources` + `modules.sources` | 编译集 | +| `include-dir[-after]` | `privateBuild.includeDirs[After]` | **本包私有**（Cargo 纪律：构建程序不得拓宽包的公开接口） | +| `rerun-if-changed` / `-env-changed` | 缓存键 | — | + +**缓存键**（`build_program.cppm:486-496`） + +`hash(build.mcpp 源) × hash(编译器身份 + 链接策略) × hash(契约 env) × 声明的文件 × 声明的 env` +→ 命中则**重放缓存的 directive**，不再运行程序。 + +### 1.3 新增一条 directive 的成本：9 处 + +以 `include-dir` 为例： + +| # | 位置 | 内容 | +|---|---|---| +| 1 | `build_program.cppm:103` | `Directives` 结构体字段 | +| 2 | `build_program.cppm:157` | `parse_line` 分派 | +| 3 | `build_program.cppm:336` | `write_cache` 序列化 | +| 4 | `build_program.cppm:379` | `read_cache` 反序列化 | +| 5 | `build_program.cppm:432` | `apply` 落到 manifest | +| 6 | `build_program.cppm:401` | `cache_fresh` 的存在性校验（仅产物类） | +| 7 | `prepare.cppm:2613` | `DirectiveMark` 字段 | +| 8 | `prepare.cppm:2617` + `:2628` | `markDirectiveTail` / `foldDirectiveTailIntoPrivateBuild` | +| 9 | `hostprogram.cppm:kMcppModuleSource` | 类型化 API | + +外加文档 ×4（en/zh × 05/07）。而 `prepare.cppm:2611` 的注释**自己承认**这还不完整： + +``` +// Link/source/fingerprint residues stay at the call sites — they +// genuinely differ between root and dep (see each). +``` + +也就是说 link/source 类的新 directive **还要再改 2 个调用点**。这正是这个代码库反复吃亏 +的「同一决策多处推导」形态，只不过这次是**提前**看见它。 + +--- + +## 2. 症结：配置 vs 施工 + +| | 配置（configuration） | 施工（build work） | +|---|---|---| +| 回答什么 | 这次构建**长什么样** | 把**这些**输入变成**那些**输出 | +| 输出物 | 一份**描述**（对构建计划的补丁） | **文件**，或一个成败判定 | +| 运行频率 | 每次解析一次 | 每个输入一次 | +| 需要的性质 | 全局视野、可缓存重放 | 增量、并行、可归因、可缓存 | +| 归属层 | prepare | ninja | +| 今天在哪 | `build.mcpp` ✅ 天生合适 | `build.mcpp` ❌ 被迫 eager 全量 | + +四个已知痛点全是推论： + +1. **无增量**：改一个 `.proto` → 整个 build.mcpp 重跑 → 全量重生成 +2. **无并行**：一个进程串行做完所有生成 +3. **失败无归因**：程序退出非 0，用户看到的是「build.mcpp exited with 1」，不是「foo.proto 第 3 行」 +4. **拿不到依赖的工具**（#355）：施工需要工具，而工具由构建产出 —— 配置期还没有构建 + +用户提到的两个未来场景，按这张表分类： + +- **编译前静态代码检查** → 施工（输入=最终源码集，输出=成败）。今天连表达都表达不了： + build.mcpp 跑的时候源码集还没定，它自己就是源码集的贡献者之一。 +- **把某语言编译成 C++** → 施工（N 输入 → M 输出）。今天只能 eager 全量转译， + 改一个文件全量重来。 + +--- + +## 3. 架构主张 + +### 3.1 主张 A：一个 hook，多种节点 + +**反对增加钩子。** 每个生命周期钩子的真实成本是：一套运行时机 + 一份 env 契约 + +一份缓存/失效语义 + 一种失败模式 + 一节文档 + N 个 e2e。而且 N 个钩子之间的**相对顺序** +还要单独定义一次。这是乘法，不是加法。 + +**主张：保留唯一的配置钩子，让它声明节点。** + +后处理（strip/sign/打包）在这个模型里不是「post 钩子」，而是「一个以最终产物为输入的 +节点」——它因此**免费获得**增量（产物没变就不重跑）、并行、以及正确的失效语义。 +上一轮讨论里「post 钩子会二次 strip / 二次签名」的自我失效难题，在节点模型下根本不存在， +因为 ninja 的输出不是输入。 + +统一后的**唯一新原语**： + +``` +action { + id : string // 诊断与去重 + inputs : [path] // 必须存在，或由另一个 action 产出 + outputs : [path] // 声明期即确定（见主张 B） + command : [argv] // 工具路径由程序自己给（如 #355 的 dep_bin） + role : source | check | artifact + deps : [action-id] // 可选 +} +``` + +**三个 role 不是三种机制，是同一条边的三种接线方式**： + +| role | 输出接到哪 | 典型 | +|---|---|---| +| `source` | 进编译集（等价于今天的 `generated=`，但**延迟到 ninja 期执行**） | protoc、transpiler | +| `check` | 输出是一个 stamp；默认挂 `default` 与编译**并行**，`blocking=true` 时成为编译边的前置 | clang-tidy、格式检查、ABI 检查 | +| `artifact` | 输入是 link 产物，输出进 `dist/` | codesign、appimage、size budget | + +`role=check` 默认并行而非阻塞：静态检查串行化整个编译是不可接受的代价，而「构建最终失败」 +的效果一样。`blocking` 留给「不通过就不该浪费编译时间」的场景。 + +### 3.2 主张 B：声明期的能力边界 + +> **INV-D：配置期必须能算出「输出文件名的集合」，不必算出「文件内容」。** + +这条不是设计选择，是 mcpp 的**结构性约束**：BuildPlan、fingerprint、 +`compile_commands.json`、以及非 dyndep 模式下的模块 topo 序，都在 prepare 期定型， +而它们全都需要知道「有哪些源文件」。 + +推论表 —— 这张表就是「用 action 还是用 eager」的判据： + +| 场景 | 输出名可预测？ | 归属 | +|---|---|---| +| `.proto → .pb.cc/.pb.h` | ✅ 命名规则确定 | **action**（增量） | +| transpiler X→C++（1:1 或规则确定） | ✅ | **action** | +| lint / 静态检查（输出=stamp） | ✅ | **action** | +| strip / sign / 打包 | ✅ | **action** | +| 生成**模块接口** `.cppm` | 文件名 ✅，但模块名与 import 边要**内容** | **eager**（今天的路径）—— 但见下 | +| 输出数量/名字由工具运行时决定 | ❌ | **eager**；或加一个「聚合成单一已知产物」的步骤 | + +**这条判据最重要的副作用**：它让今天的 eager 路径（`generated=`/`source=`）从 +「历史包袱」变成**有明确适用面的一等路径**。两条路径都有存在理由，文档能一句话讲清 +用哪条 —— 而不是「新的更好，旧的别用」。 + +**`.cppm` 那一行值得单独跟进**（不在本设计定稿）：`ninja_backend.cppm:214-222` +显示 **dyndep 已是默认**（`MCPP_NINJA_DYNDEP=0` 才退回静态 deps），模块依赖边本来就在 +ninja 期用 `rule cxx_scan`(`:844`) + `rule cxx_dyndep`(`:486`) 解析。所以「生成的 `.cppm` +必须 eager」的真正阻塞点，是 plan 仍消费 prepare 期的 `topoOrder`（`prepare.cppm:4239`）。 +若能确认该 topoOrder 在 dyndep 模式下是冗余的，这条限制可以解除，transpiler 场景就能 +完全进图。**这是一个值得单独核实的问题，不要在本设计里假设答案。** + +### 3.3 主张 C：扩展靠包，不靠 DSL + +「把某语言编译成 C++」这类规则应该**写一次、到处用**，而不是每个包把同一段 build.mcpp +复制一遍。三条路： + +| 路 | 做法 | 判断 | +|---|---|---| +| (a) | 每个包自己写 build.mcpp | 不可复用；规则的 bug 要修 N 遍 | +| (b) | 引入规则 DSL（xmake 的 Lua rule、Bazel 的 Starlark） | **拒绝**：引入第二门语言，直接违背 mcpp「用 C++ 写构建、不引入第二语言」的立身之本 | +| (c) | **规则以普通 mcpp 包分发，build.mcpp `import` 它的 host 模块** | ✅ 推荐 | + +```cpp +// build.mcpp +import mcpp; +import mcpp.rules.protobuf; // 一个普通的 mcpp 包，为 HOST 编译 + +int main() { + mcpp::rules::protobuf::generate({ + .protos = "proto/**/*.proto", + .out_dir = "gen", + .grpc = true, + }); +} +``` + +规则包因此**有版本、能测试、能发布、用 C++ 写、零新语法**，走的是 mcpp 已有的全部 +包管理机制（index、语义化版本、feature、lockfile）。 + +**这条要求一个引擎能力**：build.mcpp 能 `import` 依赖提供的、为 host 编译的模块。 +今天 build.mcpp 只能 import 两个东西 —— 内置的 `mcpp` 模块和 `std` +（`build_program.cppm:541-543`）。 + +而这正是 **#355 的同一套机制**： + +| 需要什么 | 产物类型 | 机制 | +|---|---|---| +| #355：调用依赖产出的工具 | `bin` | host 子构建 → tool store → 路径 | +| 主张 C：import 依赖提供的规则 | `lib`（host BMI + 对象） | host 子构建 → tool store → BMI/对象路径 | + +**⇒ #355 不是一个孤立特性，它是这条架构路线的第一块地基。** +这是本轮分析最重要的结构性发现：它把 #355 从「gRPC 的权宜之计」提升为「扩展性架构的 +必经之路」，并且说明 #355 的 tool store 设计必须留出「条目里不止有 `bin/`」的余地。 + +--- + +## 4. 稳定性与兼容性：扩表之前先补 5 个缺口 + +这些不是未来问题，是**今天就存在**的。在 directive 表变长之前补，成本最低。 + +| # | 缺口 | 证据 | 后果 | 处置 | +|---|---|---|---|---| +| **S1** | **协议无版本** | `build_program.cppm:162`：未知 directive → `warning` + 忽略 | 为新版 mcpp 写的 build.mcpp 在旧 mcpp 上**静默丢语义**（构建成功，行为不对） | 程序声明 `mcpp:protocol=N`（由内置模块自动发）；引擎见到未知 directive 且程序声明的 N ≥ 自身支持 → **硬错误**并指明「升级 mcpp」 | +| **S2** | **build.mcpp cache 无语义 epoch** | `write_cache` 只写 `program`/`compiler`/`ctx` 三行 | 引擎改了某条 directive 的**解释**，旧缓存条目被按新语义重放 —— 静默错误 | 加 `epoch` 行，纪律照抄 `cache_key.cppm::kCacheEpoch`（「仅当旧条目不可用时才 bump，且与 mcpp 版本号解耦」） | +| **S3** | **build.mcpp 无超时** | 用的是 `capture_exec`；带超时的 `capture_exec_deadline` 就在隔壁（`process.cppm:105`） | 一个死循环 / 等网络的 build.mcpp 让构建**永久挂起**，且没有任何诊断 | 给默认超时（可 `--build-program-timeout` 覆盖），超时错误点名是哪个包的 build.mcpp。对照 `mcpp test` 已有的两档超时纪律 | +| **S4** | **裸 `printf` 是唯一真正的兼容风险** | 内置 `mcpp` 模块与引擎在**同一个二进制里**，永远同步；`#include ` + 手写 `mcpp:` 字符串不是 | 用户手写的字符串会随协议演进腐烂，且 S1 的守卫看不见它（它不会发 `protocol=`） | 文档把 `import mcpp;` 定为**唯一演进面**；裸 printf **冻结**在现有 11 条，不再新增。生态现状支持这一步（grpc-m / opencv-m 都已是 `import mcpp` 风格） | +| **S5** | **directive 的作用域在多处推导** | §1.3 的 9 处清单；`prepare.cppm:2611` 注释自认不完整 | 加新语义时**必然漏一处**，且失败在很远的地方 | 收敛成**一张 directive 定义表**：`{ 名字, 落哪个字段, 作用域(private/link/source), 是否进 cache, 是否进 fingerprint }`；parse / write_cache / read_cache / apply / fold 全部**由表驱动** | + +**S5 是「优雅简洁」这个诉求的量化目标：9 处 → 1 处。** 它也是引入 `action` 原语的 +**前置条件** —— 在一个要改 9 处的结构上加一个字段数是 6 的新 directive，是自找麻烦。 + +另外两条**已经做对、要保持**的纪律，写下来以免将来被推翻： + +- **契约 env 无条件进缓存键**（`build_program.cppm:305-312`）：target/profile/feature 变了 + 必须重跑，这条正确性**不能**依赖作者记得写 `rerun-if-env-changed`。 +- **include-dir 永远私有**（`build_program.cppm:99-102`）：构建期程序**不得**拓宽包的 + 公开接口。这是供应链面的判据，不是风格偏好。任何新 directive 都要先回答 + 「它会不会拓宽公开接口」。 + +--- + +## 5. 目标架构：分层视图 + +``` +L0 声明层（mcpp.toml） + 静态可解析：依赖图、feature、target 条件、targets + ↑ 程序不可改写 —— lockfile / LSP / 审计的前提 +L1 配置层（build.mcpp）—— 唯一的 hook + 跑一次；输出 = 「对 L0 的补丁」+「对 L2 的节点声明」 + 能力：探测宿主 / 选源码 / 定 flag / 声明 action / import 规则包 +L2 图层（ninja） + 增量、并行、可缓存、可归因 + 节点：compile / link / action(source | check | artifact) +L3 分发层（mcpp pack） + 与构建解耦 +``` + +一条方向性规则，是既有 INV-1 的一般化： + +> **L1 只能向下写（声明 L2 的节点），不能向上写（改 L0 的依赖图）。** + +它同时解释了为什么 #355 的工具请求必须写在 `mcpp.toml`（L0）而不是 build.mcpp（L1）： +「我需要某个包产出的某个工具」是对依赖图提出的需求。 + +--- + +## 6. 演进路径 + +每一步都是**加法**，没有任何一步需要改变现有 build.mcpp 的行为 —— +这是把兼容性当一等约束的直接结果。 + +| 步 | 内容 | 前置 | 破坏性 | 状态 | +|---|---|---|---|---| +| 0 | 补 S1–S4：协议版本 / cache epoch / 超时 / 冻结裸 printf | — | 无（S4 是文档 + 停止扩表） | **已实施 2026.8.5.1** | +| 1 | S5：directive 定义表收敛 | — | 无（纯内部重构，可用「产物逐字节相同」验证） | **已实施 2026.8.5.1** | +| 2 | #355：host 工具 + 工作目录外置 | 0, 1 | 无 | 待 review | +| 3 | `action` 原语，先只做 `role=source`（即 codegen 进图） | 2 | 无（新增） | 待 review（§8 未决） | +| 4 | `role=check`（静态分析）、`role=artifact`（后处理） | 3 | 无（新增） | 待 review | +| 5 | build.mcpp 可 `import` 依赖提供的 host 模块 → **规则包生态** | 2 | 无（新增） | 待 review | +| 6 | 核实并（若成立）解除「生成的 `.cppm` 必须 eager」限制 | 3 | 无 | 待核实 | + +步 0/1 值得优先，因为它们**成本最低而收益随时间递增**：directive 表越长，补的代价越大。 + +--- + +## 7. 明确不做 + +| 不做 | 理由 | +|---|---| +| 引入第二门语言的规则 DSL（Lua / Starlark / YAML） | 违背 mcpp「用 C++ 写构建」的立身之本；主张 C 用包机制拿到了同样的复用性 | +| 增加第二个生命周期 hook（pre / post / pre-link / post-link） | 主张 A：钩子数是乘法成本；节点模型覆盖同样的用例且自带增量 | +| 让 build.mcpp 改依赖图 | L1 不向上写；静态可解析性是 lockfile / LSP / 审计的前提 | +| build.mcpp 升级为返回 LazyPath 的完整构建图 DSL（Zig 形态） | `action` 原语已拿到主要收益；完整形态要把行式协议换成结构化图协议，与既有生态不兼容 | +| 包自定义 manifest 键 | 既有 Appendix A「closed syntax, open vocabulary」原则 | + +--- + +## 8. 需要 review 决定的开放问题 + +1. **`action.command` 的表达力边界**：只允许 argv + **封闭的引擎变量词表** + （`$in` / `$out` / `${mcpp.compile_db}` / `${mcpp.out_dir}` / `${mcpp.target_file:}`）， + 还是允许任意 shell？封闭词表更可移植（Windows 无 shell 假设）、更可缓存， + 但会有人抱怨不够用。**倾向封闭。** +2. **结构化 directive 的线格式**：`action` 有 6 个字段，行式 `key=value` 会很难看。 + 建议在既有平坦协议上**扩展**一条 `mcpp:action={json}`（引擎侧已有 `mcpp.libs.json`）， + 而内置模块负责编码 —— 这与 S4「`import mcpp;` 是唯一演进面」自洽。**是否接受 JSON 载荷？** +3. **`role=check` 的默认挂载**：默认并行（本文建议）还是默认阻塞？ + 是否需要一个 `mcpp check` 只跑 check 节点？ +4. **规则包的命名空间与稳定性承诺**：`mcpp.rules.*` 是保留前缀吗？谁来维护第一批 + （protobuf / clang-tidy）？ +5. **步 6 的前置核实**：dyndep 已是默认，那么 `prepare.cppm:4239` 传给 `make_plan` + 的 `topoOrder` 在 dyndep 模式下是否已经是冗余的？这决定 transpiler 场景能否完全进图。 + +--- + +## 9. 实施记录（步 0 + 步 1，2026.8.5.1） + +**范围决策**：本次只实施步 0 与步 1。步 3–5 会新增**公开协议面**（`mcpp:action=`）， +一经发布即成为兼容承诺，而 §8 的开放问题（命令表达力边界、结构化载荷格式、 +`role=check` 默认挂载）尚未 review —— 在未定案的情况下把它发进生态，正是本文 +§4 想要防的那类债。 + +### 9.1 落地清单 + +| 项 | 位置 | +|---|---| +| directive 定义表（S5） | **新增** `src/build/directives.cppm` —— `kTable` 一行即一条指令；解析 / 缓存读写 / apply / 声明产物契约 / 私有折叠全部表驱动 | +| 协议版本（S1） | `directives.cppm::kProtocolVersion`、`protocol_error()`；内置模块在 `hostprogram.cppm` 里以 `@PROTOCOL@` 占位符**替换**注入（不硬编码，杜绝漂移） | +| cache epoch（S2） | `directives.cppm::kCacheEpoch`；`build_program.cppm` 的 `write_cache`/`read_cache`/`cache_fresh` | +| 运行上限（S3） | `directives.cppm::run_timeout()`；`build_program.cppm` 改用 `capture_exec_deadline` | +| `capture_exec_deadline` 补 `cwd` | `src/platform/process.cppm` | +| mark/fold 迁出 | `prepare.cppm` 的 `DirectiveMark`/`markDirectiveTail`/`foldDirectiveTailIntoPrivateBuild` → `directives.cppm` 的 `Mark`/`mark()`/`fold_private_tail()` | +| 文档 | `docs/07-build-mcpp.md` + `docs/zh/07`：演进面对照表、协议、epoch、运行上限 | +| 测试 | `tests/unit/test_build_directives.cpp`（21 例）、`tests/e2e/186_build_mcpp_protocol_and_bound.sh` | + +### 9.2 与设计的三处偏差 + +1. **`Def` 多了 `missingPrefix`/`missingSuffix` 两列。** 初版把两条产物型指令的 + 缺失诊断收敛成一句通用文案，`143_build_mcpp_source_directive.sh` 立刻变红 —— + 它断言的是 `selected source`。这个断言是对的而不是过时的：`generated=` 说的是 + 「我**写了**这个文件」，`source=` 说的是「我**选中了**这个已存在的文件」， + 两者是不同的契约，用户需要被告知自己违反了哪一个。**表驱动不等于文案统一**， + 所以把差异也放进表里。 + +2. **`run_timeout()` 放在 `directives.cppm` 而不是 `build_program.cppm`。** + 后者的匿名 namespace 在 clang 22 下会误编译邻居（PR#332/#334）， + 「别再往那个匿名 ns 加代码」是硬约束。于是该模块的定位是「**build.mcpp 契约** + ——指令表 + 协议版本 + 缓存 epoch + 运行上限」，四者都是 mcpp 与构建程序之间的 + 约定，放在一起是自洽的。 + +3. **未知缓存记录也作废整条条目。** 设计只写了 epoch。实施时补上：一条本 mcpp + 不认识的 `d` 记录（更新的 mcpp 写的）若只跳过它，等于应用了程序所要求的一个 + **真子集** —— 与 S1 拒绝未知指令是同一条判据。 + +### 9.3 验证 + +- 单测 56/56 通过（新增 `BuildDirectives` 21 例） +- e2e：12 个 build.mcpp / 生成源相关用例全绿，新增 186 号通过 +- 三条协议分支实测：`import mcpp;` + 未知指令 → 硬错误；裸 printf + 未知指令 → + 警告并继续；`protocol=999` → 拒绝并提示升级 +- 缓存四条失效路径实测：干净命中 / 缺 epoch / epoch 不符 / 未知 `d` 标签 +- 超时实测：3 秒上限对死循环程序在 3.1s 内触发，错误点名包名与 env 覆盖方式 + +> **一个测试方法论坑**：`build.mcpp` 的缓存命中被工程级 fast path（`try_fast_build`） +> 挡在前面 —— 无变更的第二次构建走 fast path，**根本不会读 build.mcpp 缓存**。 +> 验证缓存行为必须先 `touch` 一个源文件把 fast path 打掉，否则会把「fast path 生效」 +> 误读成「缓存未命中」。我第一次就是这么误判的。 diff --git a/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md b/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md new file mode 100644 index 00000000..e1b0242e --- /dev/null +++ b/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md @@ -0,0 +1,810 @@ +# issue #355：依赖产出的 host 工具（codegen 工具链缺口） + +> 状态：**设计待 review，未实施**（2026-08-05 追加 §12 行业调研，并据此调整了两处，见 §12.4） +> 关联：#355（本条）、#241（`MCPP_DEP__DIR`）、#274（显式 ninja 目标）、#344（cache 地址） +> 涉及（预估）：`src/build/prepare.cppm`、`src/build/build_program.cppm`、 +> `src/build/hostprogram.cppm`、`src/pm/dep_spec.cppm`、`src/manifest/toml.cppm`、 +> `src/manifest/xpkg.cppm`、`src/build/compile_commands.cppm`、`src/home.cppm`、 +> 新增 `src/build/tool_store.cppm` +> 生态侧：`mcpp-index` 的 `compat.protobuf`、`mcpplibs/grpc-m` + +--- + +## 0. 结论摘要 + +issue 的判断是对的：**`build.mcpp` 唯一缺的就是「工具在哪」**。但代码核实后， +「把工具交给消费者」这件事在当前架构里不是加一个环境变量能完成的，它撞上了**四条 +彼此独立的结构性约束**，其中一条直接否掉了「把依赖的 bin target 塞进主 ninja 图」 +这一最直观的方案： + +| # | 约束 | 出处 | 后果 | +|---|---|---|---| +| C1 | **`build.mcpp` 在 prepare 期运行，那时 ninja 图还不存在** | `prepare.cppm:3841`（dep 循环）/ `:3985`（root）；plan 在 `:4239`、`build.ninja` 更在 `execute.cppm:292` 之后 | 任何「由主构建产出工具」的方案都**时序上不成立** | +| C2 | 主构建图**只为 target 编译**；工具必须能在 host 上跑 | `plan.cppm:497` `naming_for(tc)`；`prepare.cppm:1683` `host_tc_for_build_program` | 交叉编译下主图的产物**无法执行** | +| C3 | 依赖包的 `[targets]` **从不进入 link 计划**（唯一例外是 `kind="shared"`） | `plan.cppm:714` 只遍历 root 的 `manifest.targets`；`:922-937` `sharedDepTargets` | 依赖的 bin 目标今天**根本不被构建** | +| C4 | **注册表包根是共享的、可能只读，明文禁止写入** | `build_program.cppm:41-45`、`:233`；docs/07 §「Dependencies' build.mcpp」 | 「把依赖当 root 再构建一次」不能就地写 | + +由 C1 + C2 得到本方案的形状：**工具只能由一次「以该依赖包为 root、面向 host 的 +嵌套构建」在 prepare 期产出，产物落到全局 tool store，再以环境变量交给 +`build.mcpp`。** 这正是 Cargo 的 `[build-dependencies]`（host 图与 target 图分离） +与 Bazel 的 exec configuration 的形状，不是新发明。 + +由 C4 得到一条**必须先做的前置项**：mcpp 目前有 5 处写入「工程根」的地方 +（`target/`、`mcpp.lock`、`compile_commands.json`、`.mcpp/`、`target/.build-mcpp`）， +嵌套构建必须能把它们整体外置。这一项独立有价值（只读源码树、CI 缓存、发行版打包）。 + +三个决策点的回答（对应 issue「需要设计决策的点」）： + +1. **触发条件** → 消费方在 `[dependencies]` 里显式声明 `tools = ["protoc"]`。 + 默认不构建。成本门（libprotoc 的 157 TU）复用**已有的** + `[features].sources` + `[targets].required_features`，不需要新机制。 +2. **交叉下必须是 HOST 产物** → 是，且比 issue 说的更强：工具子构建**整条链** + （它自己的依赖、它的工具链）都按 host 解析，且**允许**与主构建用不同工具链—— + 因为**可执行文件与主构建之间没有任何 ABI 接触**。这一条是本方案便宜的根本原因。 +3. **与 `[xlings] deps` 的边界** → `[xlings] deps` = mcpp 包图**之外**的宿主工具 + （make/cmake/python）；本方案 = 包图**之内**、由包自己产出的工具。两者不重叠， + `xim:` 预编译工具包仍然是「源码不可构建」时的合法出路。 + +一句话总结方案：**tool store 是接口，「怎么把条目填满」是 provider 细节。** +本次只实现 `build-from-source` 这一个 provider；`prebuilt-asset`（描述符自带 +per-host 预编译资产，把 protoc 从「编 400 TU」降到「下 5MB」）是**设计好的扩展点， +本次不实现**。 + +--- + +## 1. 精确机制（代码级） + +### 1.1 依赖的 `[targets]` 从不进入 link 计划 + +`src/build/plan.cppm:713-714`： + +```cpp +// 4. Link units (one per [targets.X]) +for (auto& t : manifest.targets) { // ← manifest = ROOT 的 manifest +``` + +依赖包只贡献 **CompileUnit**（`cu.packageName` 标记归属），它们的 `.o` 被无差别地 +灌进 root 的 link unit。依赖自己的 `[targets]` 在整个 plan 里**只有一个例外**被读到—— +`kind="shared"`： + +`src/build/plan.cppm:922-937`： + +```cpp +for (std::size_t i = 1; i < packages.size(); ++i) { + for (auto const& t : p.manifest.targets) { + if (t.kind != mcpp::manifest::Target::SharedLibrary) continue; + sharedDepPackages.insert(qname); + sharedDepTargets.push_back(SharedDepTarget{...}); +``` + +这是「依赖的 target 被提升为 link unit」的**既有先例**，形状上完全可以照抄给 +`kind="bin"`。但它救不了 #355——见 1.2。 + +### 1.2 时序：`build.mcpp` 跑在 ninja 图存在之前（**这条决定方案形状**） + +`prepare_build` 的实际顺序： + +``` +依赖解析 / feature 激活 prepare.cppm:~3400-3830 + ↓ +G2: 依赖的 build.mcpp 循环 prepare.cppm:3841-3903 + ↓ +L3: root 的 build.mcpp prepare.cppm:3985-4040 + ↓ +modgraph 扫描 / fingerprint prepare.cppm:~4100-4220 + ↓ +BuildPlan 生成 prepare.cppm:4239 make_plan() + ↓ +run_build_plan → 写 build.ninja execute.cppm:292 ← ninja 图到这里才成形 +``` + +`build.mcpp` 要调用 protoc 生成 `*.pb.cc`，而这些生成源必须在 **modgraph 扫描之前** +落盘（docs/07 明文：「BEFORE the modgraph scan (so its generated=/source= sources +are picked up)」）。所以： + +> **凡是由主 ninja 图产出的东西，`build.mcpp` 都用不上。** + +把依赖的 `kind="bin"` 提升成主图 link unit（1.1 的先例）因此**不能解决 #355**。 +它是另一个独立特性（「消费者想把依赖的 bin 当交付物一起装出来」），有价值， +但不在本设计范围内。 + +### 1.3 host ≠ target：主图只为 target 编译 + +`plan.cppm:497` `const auto naming = naming_for(tc);` —— 整个 plan 的产物命名、 +flags、链接都取自解析出的 target 工具链。交叉构建下（`--target x86_64-windows-gnu`） +主图产出的是 PE，protoc 必须是当前机器能跑的 ELF。 + +mcpp 里**已经有**这条语义的正确实现，就是 `build.mcpp` 自己： + +`src/build/prepare.cppm:1683-1723` `host_tc_for_build_program()` —— 「把同一个 +toolchain spec **去掉 target 轴**再解析一次」，并且 `build_program.cppm:576-581` +把这条列为 load-bearing： + +``` +// build.mcpp is compiled AND run on the machine doing the build, so a std BMI +// built for the target would produce a helper that cannot execute — the same +// host≠target mistake the mingw-cross work had to fix in four separate places. +``` + +**工具子构建复用同一条规则即可**，不需要新的 host 解析逻辑。 + +### 1.4 依赖包没有 per-target 源码分区 + +`src/manifest/types.cppm:75-94`：`Target` 只有 `name/kind/main/soname/cflags/ +cxxflags/defines/requiredFeatures`——**没有 `sources`**。一个包的全部源码编成一个 +对象池，link unit 按 kind 取用。 + +所以「给 compat.protobuf 加一个 protoc bin target」会把 libprotoc 的 157 TU 灌进 +**所有**消费者的库构建里。这正是 issue 决策点 1 担心的成本。 + +但成本门**已经存在**，不需要新机制： + +- `[features.].sources` —— feature 门控源码集（`xpkg.cppm:1364` 起、 + `toml.cppm` 同形；e2e `106_feature_gated_sources_toml.sh` 覆盖） +- `[targets.].required_features` —— target 门（`types.cppm:93`， + gate 在 `prepare.cppm:4056-4062`） + +组合起来： + +```toml +[features.protoc] +sources = ["*/src/google/protobuf/compiler/**/*.cc"] # 157 TU，默认不编 + +[targets.protoc] +kind = "bin" +main = "src/google/protobuf/compiler/main.cc" +required_features = ["protoc"] +``` + +**engine 侧唯一的缺口**：`src/manifest/xpkg.cppm:1336-1350` 的 targets 解析器只认 +`kind` / `main` / `soname`，**不认 `required_features`**（`toml.cppm:473` 认）。 +Form B 描述符（`compat.protobuf` 正是 Form B）因此今天写不出这个门。这是必须补的 +一行级改动。 + +### 1.5 注册表包根共享 / 可能只读 + +`src/build/build_program.cppm:41-45`、`:233-237` 明文： + +``` +// Dependencies MUST point this into the CONSUMING project's tree — a registry +// package root is shared and may be read-only. +``` + +而一次完整的 `mcpp build` 会往工程根写 **5 处**： + +| 写入物 | 位置 | 出处 | +|---|---|---| +| `target///` | `root/target/...` | `prepare.cppm:199-210` `target_dir()` | +| `mcpp.lock` | `root/mcpp.lock` | `prepare.cppm:4690` | +| `compile_commands.json` | `plan.projectRoot/...` | `compile_commands.cppm:208` | +| `.mcpp/.xlings.json` | `root/.mcpp` | `prepare.cppm:1782-1790`（仅当有 `[indices]`/`[xlings]`） | +| `target/.build-mcpp/` | `root/target/.build-mcpp` | `build_program.cppm:234-237` | + +嵌套构建若就地跑，会**同时违反 1.5 的明文不变量**、污染 registry、并让两个并发 +构建在 `/target/` 上打架。所以「工作目录外置」是硬前置项，不是优化。 + +### 1.6 `prepare_build` 以 cwd 为根 + +`src/build/prepare.cppm:842`： + +```cpp +auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); +``` + +工作区 fan-out（`cmd_build.cppm:68-81`）也是靠 `package_filter` 在**同一个 cwd 根** +下选成员，不是靠换根。要以「registry 里的某个包」为根构建，必须给 `prepare_build` +一个显式的根参数。好消息：`prepare_build` 全函数**只有一处 `static`** +（`:4413` 一个 `static const std::string kEmpty`），没有跨调用状态，**递归重入是安全的**。 + +### 1.7 现状复核：gRPC 这条链到底缺什么 + +- `mcpplibs/grpc-m` 是 **Form A**（自带 `mcpp.toml`），`compat.protobuf` 是 **Form B**。 +- `compat.protobuf` 描述符自己写着（`pkgs/c/compat.protobuf.lua:20-23`）： + 「does NOT build `libprotoc` (a further 157 TUs) and ships no protoc binary」。 +- `grpc-m/README.md`「Code generation」明文：「mcpp has no mechanism for handing a + dependency's built binaries to a consumer. So generated stubs are checked in」， + 并要求用户自备 protoc **35.1** 与 gRPC **1.83.0** 的插件——**版本匹配责任 + 今天完全落在用户身上**，且错配是运行期错误。 + +issue 对现状的描述与代码一致，无需修正。 + +--- + +## 2. 设计判据（不变量） + +本方案接受下列判据，后文每个决策都能追溯到其中一条： + +- **INV-1 依赖图保持声明式。** 「我需要某个包产出的某个工具」是对依赖图提出的 + 需求，必须写在 `mcpp.toml`，不能由 `build.mcpp` 在运行期申请。 + （docs/07 已有明文：「It cannot add a registry dependency — keep your dependency + graph declarative in mcpp.toml … build.mcpp is for *leaf* decisions」。） +- **INV-2 工具永远是 host 产物。** 与 `--target` 无关，与主构建的 linkage/profile + 无关。判据不是「跟主构建一致」，而是「能在这台机器上执行」。 +- **INV-3 可执行文件与主构建零 ABI 接触。** 因此工具子构建**可以**用与主构建不同的 + 工具链、不同的 profile、不同的依赖版本解析结果——这不是妥协，是本方案便宜的原因。 + （对照：`kind="lib"` 依赖绝不允许这样。） +- **INV-4 不写 registry 包根。** 见 1.5 的既有明文。 +- **INV-5 单一版本轴。** 工具的版本 = 产出它的那个包的版本。不引入第二条版本轴 + （这正是 issue 反对 `xim:grpc-tools@X` 的理由，方案必须真正做到）。 +- **INV-6 默认零成本。** 没有消费者要工具时，行为与今天**逐字节相同**。 + +--- + +## 3. 方案总览 + +``` + ┌──────────────────────────────────────┐ + mcpp.toml │ tool store（全局，跨工程共享） │ + [dependencies] │ /tool//@/ │ + protobuf = { │ /bin/protoc │ + version="35.1", └──────────────────────────────────────┘ + tools=["protoc"] } ▲ │ + │ │ provider 填充 │ 命中即取 + │ INV-1 声明式 │ ▼ + ▼ ┌─────┴──────┐ MCPP_DEP_PROTOBUF_BIN_PROTOC + prepare：工具供给 pass │ ①build- │ │ + （在 build.mcpp 之前） │ from-src │ ▼ + │ ②prebuilt │ build.mcpp: + │ (未实现) │ mcpp::dep_bin("protobuf","protoc") + └────────────┘ → 生成 *.pb.cc + → mcpp::generated(...) +``` + +**tool store 是接口**：条目的形状（`/bin/` + `entry.json`）与 +「谁把它填满」解耦。本次只实现 provider ①。 + +--- + +## 4. 分层实施 + +### Phase 0（前置，必须）：工作目录外置 + +**目标**：把 1.5 表里的 5 个写入点收敛到**一个决策点**。 + +新增（建议放 `src/project.cppm` 或新 `src/build/workdirs.cppm`）： + +```cpp +struct WorkDirs { + std::filesystem::path source; // 包根：只读语义，只用来找源码/manifest + std::filesystem::path work; // 所有写入落在这里；默认 == source + std::filesystem::path targetRoot() const { return work / "target"; } + std::filesystem::path lockPath() const { return work / "mcpp.lock"; } + std::filesystem::path compileDb() const { return work / "compile_commands.json"; } + std::filesystem::path projectEnv() const { return work / ".mcpp"; } + std::filesystem::path buildMcppDir() const { return work / "target" / ".build-mcpp"; } +}; +``` + +改动： + +- `prepare.cppm:199` `target_dir(tc, fp, root)` → `target_dir(tc, fp, wd)` +- `prepare.cppm:4690` lock 写入 → `wd.lockPath()` +- `compile_commands.cppm:208` → 由 `BuildPlan` 携带（新增 `plan.compileDbPath`）， + 不再从 `plan.projectRoot` 推导 +- `prepare.cppm:1786` `ensure_project_index_dir(cfg, root, ...)` → 传 `wd.projectEnv()` +- `build_program.cppm:234-237` `build_dir()` 的默认分支 → `wd.buildMcppDir()` + +`BuildOverrides` 新增两个字段： + +```cpp +std::filesystem::path project_root; // 空 = find_manifest_root(cwd)（今天的行为） +std::filesystem::path work_dir; // 空 = project_root（今天的行为） +``` + +**默认值让所有既有路径逐字节不变**（INV-6）。同时顺手解决 `prepare.cppm:842` 的 +cwd 硬编码（1.6）。 + +> 副产品：`mcpp build --work-dir ` 对只读源码树、CI 缓存、发行版打包都直接 +> 有用。是否暴露成 CLI 由 review 决定；**引擎内部必须有**。 + +### Phase 1（本次核心）：`build-from-source` provider + +#### 1.1 消费方语法 + +```toml +[dependencies] +protobuf = { version = "35.1", tools = ["protoc"] } +grpc = { version = "1.83.0", tools = ["grpc_cpp_plugin"] } +``` + +- `DependencySpec` 新增 `std::vector tools;`（`src/pm/dep_spec.cppm`） +- `manifest/toml.cppm`：`:534` 的 key 白名单加 `"tools"`，`:564` 旁按 `features` + 的同形写法解析数组，`:632` 的错误文案同步 +- `manifest/xpkg.cppm:1488` 的 `deps` 解析器**目前只接受字符串值** + （`name = "version"`）。必须扩成「字符串 **或** 表」： + + ```lua + deps = { + ["compat.protobuf"] = { version = "35.1", tools = { "protoc" } }, + } + ``` + + 否则 Form B 描述符永远无法请求工具。（注释里说的「richer Lua parser」指的是 + **namespaced 子表**；这里只是值位置的表，`cur.skip_table()` 已有同形能力。） +- `manifest/xpkg.cppm:1336-1350` targets 解析器补 `required_features`（见 1.4） + +#### 1.2 工具请求的聚合 + +`DependencyEdge`（`prepare.cppm` 内部结构）新增： + +```cpp +std::vector requestedTools; // 与 requestedFeatures 完全同形 +``` + +聚合规则与 feature 一致：**同一个依赖包的所有入边取并集**。理由与 #242/#243 +同源——请求必须来自权威边图，而不是只扫 root 的直接依赖，否则「grpc 的 mcpp.toml +请求 protoc」这类**传递请求会被静默丢掉**。 + +#### 1.3 供给 pass 的插入点 + +放在 `prepare.cppm` 的 feature 激活之后（`:3830` `apply(packages[i], ...)` 结束）、 +**G2 依赖 build.mcpp 循环之前**（`:3832`）： + +``` +feature 激活 (:3812-3830) + ↓ +【新】工具供给 pass ← 本方案 + ↓ +G2: 依赖 build.mcpp (:3841) ← 拿得到 MCPP_DEP_*_BIN_* + ↓ +L3: root build.mcpp (:3985) ← 同上 +``` + +伪码： + +```cpp +// key: (依赖包 index) → 请求的工具名（去重、排序） +std::map> toolRequests; +for (auto const& edge : dependencyEdges) + for (auto const& t : edge.requestedTools) + toolRequests[edge.dependencyPackageIndex].insert(t); + +// consumerIndex → (envVarName → 绝对路径) +std::map>> toolEnv; + +for (auto const& [depIdx, tools] : toolRequests) { + for (auto const& tool : tools) { + auto p = mcpp::build::provision_tool(packages[depIdx], tool, hostAxes, cfg); + if (!p) return std::unexpected(...); // 硬失败，绝不静默降级 + // 回填给所有请求了它的消费者 + } +} +``` + +#### 1.4 一次工具子构建做什么 + +`provision_tool(pkg, toolName, ...)`： + +1. **定位 target**：在 `pkg.manifest.targets` 里找 `name == toolName && + kind == Binary`。找不到 → 报错并列出该包所有 `kind="bin"` target 名。 +2. **算 feature 集**：`pkg` 的 `[features].default` ∪ 该 target 的 + `required_features` 的闭包。**注意方向反转**，见 §5.2。 +3. **算 store key**（§5.5）。命中 → 直接返回路径，**零构建**。 +4. **未命中 → 嵌套构建**： + + ```cpp + BuildOverrides sub; + sub.project_root = pkg.root; // Phase 0 + sub.work_dir = storeDir / "build"; // Phase 0，INV-4 + sub.target_triple = ""; // ← host（INV-2） + sub.profile = "release"; // 固定，见 §5.4 + sub.features = join(featureSet); + auto ctx = prepare_build(false, false, {}, sub); + BuildOptions o; o.ninjaTargets = { linkUnitOutputOf(toolName) }; // #274 已有能力 + run_build_plan(*ctx, ...); + ``` + +5. **收口**：把产物 `mv` 进 `/bin/`，写 `entry.json` + （记录全部 key 输入，照 `bmi_cache.cppm:55-58` 的「hash 相等也要逐字段比」纪律）， + 然后**删除 `/build/`**（protoc 的几百个 `.o` 没有保留价值——key 覆盖了 + 全部输入，命中就不会重建）。 +6. 全流程走**临时目录 + rename**（照 `stage.cppm` 的纪律），并对 `` 加 + 文件锁，避免两个并发 mcpp 撞同一个 key。 + +#### 1.5 环境契约与 API + +新增契约变量： + +``` +MCPP_DEP__BIN_ = 工具可执行文件的绝对路径 +``` + +`src/build/hostprogram.cppm` 的 `kMcppModuleSource` 加： + +```cpp +// #355: 依赖产出的 host 工具的绝对路径，未提供则返回 ""。 +inline const char* dep_bin(const char* pkg, const char* tool) { ... } +``` + +`sanitize` 沿用 `has_feature` / `dep_dir` 的同一套（大写、非字母数字 → `_`）。 +碰撞处理照抄 `build_program.cppm:289-301`：**保留首个 + 警告**，绝不静默 last-wins。 + +这些变量与 `depDirs` 一样进入 `contract_env` → `contract_hash` +（`build_program.cppm:308`），所以**工具路径变化（= 工具被重建）自动触发 +`build.mcpp` 重跑**，作者无需写 `rerun-if-changed`。这一条是正确性，不是优化。 + +### Phase 2（**本次不实现**，仅锁定扩展点）：`prebuilt-asset` provider + +protoc 的现实是 upstream **为每个平台发布预编译二进制**。允许描述符声明 +per-host 预编译资产，就能把「编 400 TU」降成「下 5MB」，而**版本轴仍然是同一条** +（资产写在同一个包版本的描述符里），INV-5 不破。 + +形状（示意，不在本次定稿）： + +```lua +tool_assets = { + protoc = { + ["x86_64-linux-gnu"] = { url = "...protoc-35.1-linux-x86_64.zip", + sha256 = "...", path = "bin/protoc" }, + }, +} +``` + +消费方语法、store 布局、环境变量**完全不变**——这就是「store 是接口」的价值。 +本次只需保证 Phase 1 的实现不把 provider 语义焊死在 `provision_tool` 里面。 + +### Phase 3（可选）：`mcpp tool` CLI + +```bash +mcpp tool protobuf:protoc -- -I proto --cpp_out=gen proto/helloworld.proto +``` + +价值：(a) 让 grpc-m README 里那段「你得自己找一个 35.1 的 protoc」直接消失； +(b) 给 e2e 一个不经过 `build.mcpp` 的独立断言面。成本很低（供给 pass 已经写好）。 + +--- + +## 5. 关键语义细则 + +### 5.1 为什么触发条件放在消费方而不是生产方 + +生产方（包作者）能声明的只有「我有这个 bin target」——`[targets]` 已经是声明式的 +公开产物表，再加一个 `export = true` 是纯仪式。而**成本是消费方承担的** +（157 TU 的编译时间落在消费者机器上），所以决定权必须在消费方。 + +同时 INV-1 要求它写在 `mcpp.toml` 而不是 `build.mcpp` 里：请求一个工具 = 向依赖图 +索取一个额外产物，与「加一个依赖」同级。 + +> **拒绝的写法**：顶层 `[tools] protoc = "compat.protobuf:protoc"` 别名表。 +> 它引入第三套命名空间（本地别名 → 包:target），而 `tools = [...]` 挂在依赖上 +> 天然复用已有的包名解析（含 namespace-stripped 短名），且 `dep_bin("protobuf", +> "protoc")` 与 `dep_dir("protobuf")` 对称——正是 issue 要的形状。 + +### 5.2 `required_features` 在子构建里方向反转（**必须写进文档**） + +docs/05 今天写的是：「A gate only — it does not activate features」。 + +在工具子构建里，被请求的 target **就是**构建目标，所以它的 `required_features` +成为**输入**而非门。这不是「同一决策两处推导」——字段的含义只有一个 +（「这个 target 需要这些 feature」），只是两个调用方的**解析方向**相反： + +| 调用方 | 方向 | 行为 | +|---|---|---| +| 主构建 | feature 集已定 → 问哪些 target 可发射 | 门（`prepare.cppm:4056`） | +| 工具子构建 | target 已定 → 问需要哪些 feature | 激活 | + +docs/05 的那句话必须补一个从句，否则读者会认为工具子构建违反了文档。 + +### 5.3 为什么是「每工具一个路径变量」而不是 issue 提的 `_BIN_DIR` + +issue 建议 `MCPP_DEP__BIN_DIR`。实测下来 per-tool 的**路径**更好： + +- **消歧**：store 条目的 key 含 target 名（不同 target 的 `required_features` + 不同 → feature 集不同 → 必然是不同条目）。若坚持一个 `_BIN_DIR`，就得让 key + 改按「本次请求的工具集合」聚合，于是 `{protoc}` 与 `{protoc,x}` 变成两次 + 完整构建；或者再造一层聚合目录（软链/拷贝）。两者都是纯粹的额外机制。 +- **`.exe` 后缀**：`dep_bin` 在 host 上编译，可以自己补后缀；给 dir 则每个 + `build.mcpp` 都要自己拼平台后缀，必然有人写错。 +- **工具的相邻数据**（如 protoc 的 well-known `.proto`）本来就在**包源码树**里， + 用已有的 `dep_dir("protobuf")` 拿，不需要 bin 目录。 + +### 5.4 target / toolchain / profile / linkage 的取值 + +| 轴 | 取值 | 理由 | +|---|---|---| +| target triple | **host**（`sub.target_triple = ""`） | INV-2 | +| 工具链 | **该包自己的 `[toolchain]`**，缺省才回落到主构建 spec 去掉 target 轴（`host_tc_for_build_program` 的同一规则） | INV-3：可执行文件与主构建零 ABI 接触，作者自己 pin 的工具链才是他验证过的 | +| profile | **固定 `release`** | 不随主构建 dev/release 摆动，否则同一台机器上 store 至少翻倍；工具跑得快对大 `.proto` 集有实际收益 | +| linkage / `--static` | **不继承** | 主构建的 `--static` 是对**交付物**的要求，与工具无关；musl 主机上的 static 默认仍由该包自己的解析决定 | +| 依赖解析 | 子构建**独立解析**（它自己的 `[dependencies]`，按 host） | INV-3。交叉构建下这本来就是必须的 | + +> 这里有一个必须写进文档的推论：**同一个包可能在一次 `mcpp build` 里被构建两次** +> ——一次作为 target 库（主图），一次作为 host 工具（子构建）。它们是两个互不 +> 影响的产物，不共享对象。native 构建下这看起来像浪费，但**正确性上无法合并** +> （feature 集不同：主图没有 `protoc` feature），而且合并会让 native 与 cross +> 走两条不同的路径——这正是这个代码库反复吃亏的「同一决策两处推导」。 + +### 5.5 store 布局与 key + +布局（挂在既有 cache root 下，`mcpp cache gc/clean` 可顺带接管）： + +``` +/tool//@// + bin/ + entry.json +``` + +key 直接复用 `src/build/cache_key.cppm` 的轴，避免第二套 key 语义： + +| 轴 | 内容 | 复用 | +|---|---|---| +| A 工具链 | host 编译器 id/version/driver identity、host triple、stdlib | `build_axes()` | +| B 语言 | C++ 标准 + flag、dialect flags、C 标准 | `build_axes()` | +| C profile | 固定 release（仍入 key，便于将来放开） | `build_axes()` | +| D 身份 | index 名、包 FQN、版本、**target 名** | `PackageAxes` + 新增字段 | +| E 自身配置 | 解析后的 feature 集、`[build]` flags、generated_files… | `PackageAxes` | +| F 上游 | 子构建依赖闭包的 key（Merkle，递归） | `cache_key` 已有 | +| G epoch | 新增 `kToolStoreEpoch` | 新增 | + +**F 轴必须递归**（照 `cache_key.cppm` 抬头的论证）：protobuf 的某个依赖换版本 +而 protoc 不重建，就是一个**静默错产物**。 + +### 5.6 递归、环、深度 + +工具包自己的 `build.mcpp` 可以再请求工具（合法：grpc 的 build.mcpp 要 protoc）。 +用一个请求栈做环检测，并给一个深度上限（建议 4）+ 明确错误： + +``` +error: tool provisioning cycle: + root → mcpplibs.grpc:grpc_cpp_plugin → compat.protobuf:protoc → mcpplibs.grpc:... +``` + +### 5.7 失败与可观测性 + +- 目标不存在 / 不是 `kind="bin"` → 报错并列出该包的全部 bin target +- 该包在 host 平台上不受支持（`[package].platforms`）→ 报错点名 host triple 与 + 该包声明的 platforms，而不是让它在第 300 个 TU 上炸 +- 首次构建必须显式告知代价（否则用户以为 mcpp 卡死）： + + ``` + Building host tool 'protoc' from compat.protobuf@35.1 + (once per (package version × host toolchain); cached at ~/.mcpp/build-cache/v1/tool/…) + ``` + +- 子构建输出默认折叠（`ui::set_quiet`），**失败时全量回放**，并前缀标注是哪个 + 工具的子构建——否则用户会以为是自己的工程编译失败。 + +--- + +## 6. 与 `[xlings] deps` / `xim:` 的边界（issue 决策点 3） + +| 机制 | 面向 | 版本轴 | 何时用 | +|---|---|---|---| +| `[xlings] deps` | mcpp 包图**之外**的宿主工具：make、cmake、python、nasm | 独立（xim） | 工具不是任何 mcpp 包的产物 | +| **本方案 `tools = [...]`** | 包图**之内**、由该包源码产出 | **与包同轴**（INV-5） | 工具是某个 mcpp 包的 `kind="bin"` target | +| `xim:` 预编译工具包 + `[xlings] deps` | 源码在 mcpp 里构建不出来的工具 | 独立 | 兜底，仍然合法 | + +docs/05 §2.13 与 docs/07 都要补这张表——issue 说得对，「值得写清楚各自的适用面」。 + +--- + +## 7. 拒绝的替代方案 + +| 方案 | 为什么拒绝 | +|---|---| +| **把依赖的 `kind="bin"` 提升为主图 link unit**（照抄 `sharedDepTargets`） | 撞 C1（时序：`build.mcpp` 早于 ninja 图）+ C2（交叉下产物不可执行）。**它是另一个独立特性**，可以单独做，但解决不了 #355 | +| **主图内嵌 host 子图 + ninja 里做 codegen 边** | 生成源必须在 modgraph 扫描（prepare 期）之前存在；ninja 期生成要走 dyndep。工程量数量级更大，而 `build.mcpp` 的机制已经够用 | +| **签入生成产物**（grpc-m 现状） | 对模板/example 正确，对真实用户不成立（`.proto` 会改），且版本匹配责任推给用户 | +| **`xim:grpc-tools` 独立包** | 破 INV-5：第二条版本轴，protoc 与 protobuf 运行时错配是最难查的一类问题。仍作为「源码构建不出来」的兜底保留 | +| **顶层 `[tools]` 别名表** | 第三套命名空间；`tools = [...]` 挂依赖上更省 | +| **`build.mcpp` 运行期申请工具**（`mcpp:need-tool=`） | 破 INV-1，且形成「跑一遍→发现缺→装→再跑一遍」的重入循环 | +| **子构建用子进程 `mcpp build`** | 隔离性确实更好，但需要额外的 CLI 面（`--project-root`/`--work-dir` 必须公开）、重复的工具链探测与索引读取，且错误只能靠文本回传。`prepare_build` 无跨调用状态（1.6），进程内递归更省且错误结构化。**若 review 更看重隔离性，这是可切换的实现细节，接口不变** | + +--- + +## 8. 改动清单（估算） + +**引擎** + +| 文件 | 改动 | 规模 | +|---|---|---| +| `src/build/prepare.cppm` | `BuildOverrides` 两字段；`:842` 换根；`:199` `target_dir`；`:4690` lock；`:1786` projectEnv；`DependencyEdge.requestedTools`；供给 pass（`:3830` 后） | 中 | +| **新** `src/build/tool_store.cppm` | store 布局、key、entry.json、锁、`provision_tool` | 中 | +| `src/build/build_program.cppm` | `BuildProgramEnv.toolPaths`；`contract_env` 新变量 + 碰撞守卫；`build_dir()` 接 WorkDirs | 小 | +| `src/build/hostprogram.cppm` | `mcpp::dep_bin` 加进 `kMcppModuleSource` | 小 | +| `src/build/cache_key.cppm` | `PackageAxes` 加 target 名；`kToolStoreEpoch` | 小 | +| `src/build/compile_commands.cppm` | 输出路径由 plan 携带 | 小 | +| `src/pm/dep_spec.cppm` | `DependencySpec.tools` | 小 | +| `src/manifest/toml.cppm` | `tools` 解析 + 白名单 + 错误文案 | 小 | +| `src/manifest/xpkg.cppm` | `deps` 值支持表形；targets 补 `required_features` | 小-中 | +| `src/cli/cmd_build.cppm`（可选） | `--work-dir`；Phase 3 的 `mcpp tool` | 小 | + +**文档**：`docs/05-mcpp-toml.md`（`tools`、`required_features` 的方向反转、 +§2.13 边界表）、`docs/07-build-mcpp.md`（`dep_bin`、环境契约表、host 语义、 +「同一个包可能被构建两次」）、两份 `docs/zh/` 同步。 + +**生态**：`compat.protobuf` 加 `[features.protoc]` + `[targets.protoc]`； +`grpc-m` 加 `[features.codegen]`（`src/compiler/**`）+ `[targets.grpc_cpp_plugin]`， +并把 README 的「Code generation」从「手工签入」改成 `tools = [...]` + `build.mcpp`。 + +--- + +## 9. 测试计划 + +**单测**：store key 的稳定性与敏感性（改 feature / 改上游 key / 改 host 工具链 +必须换 key；改主构建 `--target` / profile **不得**换 key）；env 变量 sanitize +与碰撞守卫。 + +**e2e**（新增，遵守 `# requires:` 必须在第 2 行的既有规则）： + +1. **最小闭环**：fixture 包 A 有 `kind="bin"` 的 `codegen`（一个把 `.txt` 变成 + `.cpp` 的十行程序）；工程 B 声明 `a = { path = "...", tools = ["codegen"] }`， + `build.mcpp` 调它生成源并 `mcpp::generated()`。断言构建通过且产物行为正确。 +2. **默认零成本**：同一个 fixture，不写 `tools` → 断言 `codegen` **没有**被构建 + （store 目录为空），且构建输出与基线一致（INV-6）。 +3. **成本门**：`required_features` 门控的 target，不激活时子构建报错并列出可用 + target；激活时通过。 +4. **store 命中**：连跑两次，第二次断言无编译、输出出现 cached 语义。 +5. **交叉语义（本方案最重要的一条断言）**：复用 `102_mingw_cross_wine.sh` 的环境， + `mcpp build --target x86_64-windows-gnu`，断言 + **产物是 PE 而工具是 host ELF**（`file` / 魔数）。这条不过 = INV-2 没落地。 +6. **环检测**：两个 fixture 互相请求对方的工具 → 断言报出环而不是挂死。 + +> 教训回填（`link-argv-max-arg-strlen` / `explicit-ninja-goals-two-regressions`）： +> **CI 全绿不等于生态可用**。合入前必须在本机跑一次真实的 `compat.protobuf` +> protoc 子构建（400+ TU 的真实规模），验证时间、磁盘、并发锁三项。 + +--- + +## 10. 风险与未决 + +| 风险 | 说明 | 处置 | +|---|---|---| +| **首次构建时间** | protoc ≈ libprotobuf + libprotoc；grpc_cpp_plugin 更大。分钟级 | 全局 store 摊薄；显式进度文案；Phase 2 的 prebuilt provider 是真正的解法 | +| **磁盘** | 每个 (包版本 × host 工具链) 一份 | 构建成功即删 `build/`；接入 `mcpp cache gc` | +| **子构建的 `[xlings]`** | 若工具包声明了 `[xlings] deps`，会写 `.mcpp/` —— Phase 0 已把它导向 work dir，但 xlings 侧的沙箱语义需实测 | 实施前先验（记忆里 `HOME=` 不是 xlings 沙箱，必须用 `XLINGS_HOME` 且**先验证再用**） | +| **并发** | 两个工程同时首次要 protoc | store 目录文件锁 + 临时目录 rename | +| **`prepare_build` 重入** | 函数 4784 行；虽无 static，但 `ui` 全局 quiet 是进程级 | 用 RAII 保存/恢复 quiet；若 review 认为风险仍高，切子进程实现（§7 末行） | +| **lockfile 可复现性** | 消费者的 `mcpp.lock` 目前不记录工具 store key | 本次不做，列为后续（工具是可执行文件，不进链接，复现性影响低于库依赖） | +| **Form B 描述符表达力** | `deps` 值改表形是解析器扩展，可能牵出别的 Form B 用法 | 只在值位置支持表，不动 namespaced 子表（保持原注释的边界） | + +--- + +## 11. 落地顺序建议 + +1. **Phase 0** 单独一个 PR（默认值不变 → diff 可用「构建产物逐字节相同」验证） +2. **Phase 1 引擎** 一个 PR（含最小 fixture e2e 1/2/4/6） +3. **交叉 e2e（第 5 条）** 与 Phase 1 同 PR —— 它是 INV-2 的唯一证明，不能后补 +4. **索引侧**：`compat.protobuf` 的 `protoc` target（先验证 protoc 真能被 mcpp + 从源码构建出来——这一步有独立风险，见下） +5. **grpc-m**：`grpc_cpp_plugin` + README 改写 + example 去掉签入产物 +6. **Phase 3 `mcpp tool`**（可选） + +> **第 4 步的独立风险**：`compat.protobuf` 今天明文「does NOT build libprotoc」。 +> libprotoc 能否被 mcpp 无 CMake 构建出来（是否有 `.h.in` / 生成步骤 / 额外依赖） +> **尚未核实**。这一步应先做一次纯手工验证再开 PR；若失败，Phase 2 的 +> prebuilt provider 会从「优化」升级为「protoc 唯一可行路径」，需要提前重排。 +> —— §12.4 的「逃生舱」把这条风险从**阻塞项降级为优化项**。 + +--- + +## 12. 行业调研（2026-08-05） + +### 12.1 两条正交的轴 + +| 轴 | 问题 | 行业状态 | +|---|---|---| +| **A：host/target 分离** | 工具该为哪台机器构建、怎么声明、怎么交付 | **已收敛**，八个系统答案一致 | +| **B：codegen 放在哪一层** | pre-pass（构建前跑一次的程序） vs 图节点（build graph 里的一条边） | **未收敛**；只有 Cargo 站 pre-pass，其余全站图节点 | + +本设计站在轴 A 的行业共识上。mcpp 在轴 B 上目前是 Cargo 派（`build.mcpp` = `build.rs`）， +这是一个独立的、更长期的方向问题，见 §12.5。 + +### 12.2 轴 A：各家机制对照 + +| 系统 | 机制 | 声明位置 | 版本轴 | 交付方式 | +|---|---|---|---|---| +| **Nix** | `depsBuild{Build,Host,Target}` / `depsHost{Host,Target}` / `depsTargetTarget` 六格 + splicing；`nativeBuildInputs` = `depsBuildHost` | 消费方依赖列表（**按类型分格**） | 同一 nixpkgs 求值 | `pkgsBuildHost.*` 进 PATH | +| **vcpkg** | 依赖上 `"host": true`；`VCPKG_CROSSCOMPILING`；host-only port 用 `"native"` supports 表达式 | 消费方 `vcpkg.json` 的依赖项 | 同一 registry baseline | `CURRENT_HOST_INSTALLED_DIR`；`VCPKG_USE_HOST_TOOLS` 加进 `CMAKE_PROGRAM_PATH` | +| **Conan 2** | `requires` (host context) + `tool_requires` (build context)，双 profile `-pr:b`/`-pr:h` | 消费方 recipe | **`protobuf/` 占位符锁同版本** | 环境 / `VirtualBuildEnv` | +| **Cargo** | 稳定：`[build-dependencies]` 恒为 host；不稳定：artifact deps（RFC 3028，**仍 nightly `-Z bindeps`**） | 消费方 `Cargo.toml` 的依赖项 | 同一 lock | `CARGO_BIN_FILE__` / `CARGO_BIN_DIR_` | +| **xmake** | `add_deps("pkgconf", {host = true})`；**`if package:is_binary() then requireinfo.host = true`**（二进制包自动是 host 包） | 依赖边 | 同一 repo | `package:addenv("PATH","bin")` | +| **Bazel** | attribute 上 `cfg = "exec"`（原 `cfg="host"`，迁移中）；genrule 的 `tools` 属性 | **依赖边（attribute）级** | 同一 WORKSPACE | 直接作为 action 的 executable | +| **Meson** | 全 API 一个 `native: true`（`executable` / `find_program` / `dependency` / compiler）；cross file 分 build/host machine | 调用点 | — | `custom_target` 直接引用 | +| **CMake** | **无一等机制**：`add_executable(t IMPORTED)` + `if(NOT CMAKE_CROSSCOMPILING)`；各项目自造（Qt `QT_HOST_PATH`、LLVM `LLVM_NATIVE_TOOL_DIR`、protobuf `Protobuf_PROTOC_EXECUTABLE`） | 每项目一套变量 | 用户自己保证 | 变量指路径 | + +### 12.3 提炼出的最佳实践，及本设计的符合度 + +| # | 实践 | 本设计 | +|---|---|---| +| 1 | host 工具是**依赖边的属性**，声明在消费方 | ✅ `tools = [...]` | +| 2 | **单一版本轴**（Conan `` 是这条的成名解法） | ✅ INV-5，且天然成立（tools 挂在同一条依赖上） | +| 3 | **binary 包/target 默认即 host**（xmake） | ⚠️ 见 §12.4 采纳 | +| 4 | **native 不特殊化**（同一条代码路径，不因 host==target 分叉） | ✅ INV-2 | +| 5 | 工具通过**路径/env** 交付，不通过链接 | ✅ `MCPP_DEP__BIN_` | +| 6 | **给一个逃生舱**：允许指定现成的 host 工具，跳过构建 | ❌ 见 §12.4 采纳 | +| 7 | codegen 应是**图节点**而非 pre-pass | ❌ mcpp 是 Cargo 派，见 §12.5 | + +**最有信息量的两个反面数据点：** + +- **Cargo 的 artifact dependencies（RFC 3028）自 2021 年 accept 至今仍未稳定**，只能 + `-Z bindeps`。说明「让依赖交出二进制」的语义细节极多（多 artifact 类型、profile + 继承、feature 统一、跨 target 的重复构建）。mcpp 应当**抄它的结论，不抄它的规模**： + 只做 `bin`，不做 `cdylib`/`staticlib`；不做多 target 请求。 +- **xmake 有 host 包机制，但 protoc 在交叉下依然是断的**： + `xmake-repo/packages/p/protobuf-cpp/xmake.lua:99` 只在 `not package:is_cross()` + 时把 `bin` 加进 PATH，`:218` 交叉时直接 `os.tryrm(installdir("bin/*.exe"))`；而 + `xmake/rules/protobuf/proto.lua:36` 用 `find_tool("protoc", {envs})` 从 **PATH 查找**， + 不是图依赖。说明难的不是机制本身，而是**把机制真的接到 codegen 规则上**—— + mcpp 若按本设计落地，这一点上会领先 xmake。 + +### 12.4 据此对本设计做的两处调整 + +**调整 A：`kind="bin"` 的 target 在工具语境下默认按 host 解析（xmake 实践 3）。** +本设计原本就把「工具永远是 host 产物」写进 INV-2，这里只是把 xmake 的措辞采纳为 +文档表述：`tools = [...]` 请求的必然是 host 产物，**不提供 target 侧的变体** +（对照 Cargo 的 `target = "target"` 逃生舱——那是 Cargo 因为 artifact deps 还要服务 +「把二进制打进产物」这个用例才需要的；mcpp Phase 1 不做那个用例，就不需要这个轴）。 + +**调整 B（新增，实践 6）：逃生舱 —— 允许指定现成的 host 工具。** + +```toml +[tools.overrides] # 或等价的 CLI / 环境形式 +"compat.protobuf:protoc" = "/usr/bin/protoc" +``` + +以及等价的环境变量 `MCPP_TOOL__=`（CI / 发行版打包友好）。 +命中覆盖时 **完全跳过子构建**，直接把该路径填进 `MCPP_DEP_*_BIN_*`。 + +为什么这一条必须进 Phase 1 而不是以后再说： + +- 它是**全行业统一的逃生舱**（vcpkg `VCPKG_HOST_TRIPLET`、CMake + `LLVM_NATIVE_TOOL_DIR`、Qt `QT_HOST_PATH`、Cargo `target = "target"`）， + 没有它，用户在「工具编不出来 / 编太慢 / 我已经有一个」时无路可走； +- 它把 §11 第 4 步那条**未核实风险**（libprotoc 能否被 mcpp 无 CMake 构建） + 从**阻塞项降级为优化项**：即使 libprotoc 一时构建不出来， + `[tools.overrides]` + upstream 官方 protoc 也能让整条链先跑通； +- 它同时是 Phase 2（prebuilt-asset provider）的**手动版本**——两者共用同一个 + store 出口，验证了「store 是接口」这个分层是真的。 + +代价：override 的路径不进 store key，因此**不参与可复现性**。必须在文档里点名 +这是逃生舱而不是常规路径，且 `mcpp doctor` 应报告当前生效的所有 override。 + +### 12.5 轴 B：mcpp 的长期方向（**不在本设计范围**，仅记录评估） + +Zig 的形态是这条轴的标杆，而 `build.mcpp` 的祖宗正是 `build.zig`： + +```zig +const tool = b.addExecutable(.{ .root_module = b.createModule(.{ .target = b.graph.host }) }); +const run = b.addRunArtifact(tool); +run.addFileArg(b.path("input.json")); +const out = run.addOutputFileArg("generated.zig"); // → LazyPath +exe.root_module.addAnonymousImport("generated", .{ .root_source_file = out }); +``` + +关键差异：`LazyPath` 让产物路径**在执行时才解析**，于是 codegen 是**图里的一条边**， +自动获得增量、并行、缓存。mcpp 抄了 build.zig 的「用同语言写构建程序」， +**没抄「构建程序产出的是图节点，而不是副作用」**。 + +对 mcpp 而言，中间态是可行的，且**可行性已在代码里核实**： + +- mcpp 已有 **ninja 期** 的 P1689 扫描（`ninja_backend.cppm:844` `rule cxx_scan`、 + `:486` `rule cxx_dyndep`），模块依赖边本来就在 ninja 期解析； +- `.pb.cc` / `.pb.h` **不是模块**，它们只需要「文件名集合在 prepare 期可预测」， + 而这正是一条声明式 codegen 规则能给的（输入 glob + 命名规则 → 输出名集合）； +- 只有生成的 `.cppm`（模块接口）需要 prepare 期就有内容 —— 这类保持走 + `build.mcpp` / `generated_files` 现有路径。 + +因此中期方向是 **`[codegen]` 声明式规则**：prepare 期只算文件名集合（不执行）， +ninja 期执行、增量、并行；工具从本设计的 tool store 取。收益是 pre-pass 永远给不了的 +（改一个 `.proto` 只重生成一个）。 + +再往前一步（`build.mcpp` 升级为构建图 DSL，返回 LazyPath）需要把行式 stdout 协议 +换成结构化图协议，与现有 directive 协议不兼容。**评估结论：不做**—— +声明式 `[codegen]` 已经拿到其中 ~80% 的收益。 + +### 12.6 参考 + +- Nix：（`depsBuild*` 六格与 splicing） +- vcpkg： +- Conan 2： +- Cargo RFC 3028：； + 跟踪 issue +- Cargo build scripts： +- xmake：`xmake/modules/private/action/require/impl/package.lua:656-660`、 + `xmake/rules/protobuf/proto.lua:36`、`xmake-repo/packages/p/protobuf-cpp/xmake.lua:99,218` +- Bazel exec transition： +- Meson cross： +- CMake cross：； + LLVM `LLVM_NATIVE_TOOL_DIR` +- Zig build system： diff --git a/.github/actions/bootstrap-mcpp/action.yml b/.github/actions/bootstrap-mcpp/action.yml index 8886c232..0c1f2613 100644 --- a/.github/actions/bootstrap-mcpp/action.yml +++ b/.github/actions/bootstrap-mcpp/action.yml @@ -25,7 +25,7 @@ inputs: # `package.name`, so one of the two was simply unreachable — and which one # depended on the machine, which is why CI failed on `compat:lua` on # Windows and `mcpplibs.capi:lua` on Linux. Never pin below that. - default: '2026.8.4.1' + default: '2026.8.5.1' cache-target: description: also restore/save target/ (build artifacts + BMIs) required: false diff --git a/.github/actions/setup-macos-llvm/action.yml b/.github/actions/setup-macos-llvm/action.yml index 80a5d12b..8564a7c7 100644 --- a/.github/actions/setup-macos-llvm/action.yml +++ b/.github/actions/setup-macos-llvm/action.yml @@ -15,7 +15,7 @@ inputs: # Floor imposed by the index, not a routine bump — see # .github/actions/bootstrap-mcpp/action.yml for why 0.4.69 is required # (two packages named `lua` in one repo need openxlings/xlings#381). - default: '2026.8.4.1' + default: '2026.8.5.1' runs: using: composite diff --git a/.github/workflows/bootstrap-macos.yml b/.github/workflows/bootstrap-macos.yml index b6efca8f..77bd107c 100644 --- a/.github/workflows/bootstrap-macos.yml +++ b/.github/workflows/bootstrap-macos.yml @@ -17,7 +17,7 @@ jobs: # Dormant (workflow_dispatch only), but kept in step with the rest — # check_version_pins.sh holds it there. Floor: 0.4.69, below which the # index cannot resolve two packages that share a short name. - XLINGS_VERSION: '2026.8.4.1' + XLINGS_VERSION: '2026.8.5.1' steps: - uses: actions/checkout@v4 diff --git a/.github/workflows/ci-fresh-install.yml b/.github/workflows/ci-fresh-install.yml index 860a2928..bc7a83dd 100644 --- a/.github/workflows/ci-fresh-install.yml +++ b/.github/workflows/ci-fresh-install.yml @@ -152,7 +152,7 @@ jobs: env: XLINGS_NON_INTERACTIVE: '1' run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.4.1 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.5.1 echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - name: Install mcpp and config mirror @@ -292,7 +292,7 @@ jobs: - name: Install xlings + mcpp run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.4.1 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.5.1 # Deliberately NOT writing to $GITHUB_PATH here. On container # images that declare no PATH in their config (opensuse/ # tumbleweed), appending a single dir to GITHUB_PATH makes the @@ -363,7 +363,7 @@ jobs: # (older ones carry minos=15 and refuse to start). # v0.4.51+: in-process sha256 — this image has no sha256sum # binary, so pinned fetches failed before it. - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.4.1 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.5.1 echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - name: Install mcpp and config mirror diff --git a/.github/workflows/ci-linux-e2e.yml b/.github/workflows/ci-linux-e2e.yml index 6ebeea9a..0d29cbc3 100644 --- a/.github/workflows/ci-linux-e2e.yml +++ b/.github/workflows/ci-linux-e2e.yml @@ -123,7 +123,7 @@ jobs: - name: Bootstrap xlings + released mcpp run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.4.1 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.5.1 export PATH="$HOME/.xlings/subos/current/bin:$PATH" xlings update xlings install mcpp -y -g diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 17021772..1a5cce45 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -118,7 +118,7 @@ jobs: # release assets were uploaded in a broken state (records present, # blobs missing → 404 on GET); re-uploaded clean. The stale-INDEX # half is handled by the marker-clear below. - XLINGS_VERSION: '2026.8.4.1' + XLINGS_VERSION: '2026.8.5.1' run: | tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" curl -fsSL -o "/tmp/${tarball}" \ @@ -255,7 +255,7 @@ jobs: - name: Bootstrap mcpp via xlings env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.8.4.1' + XLINGS_VERSION: '2026.8.5.1' run: | tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" curl -fsSL -o "/tmp/${tarball}" \ diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b59763b5..ccdd8e40 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -96,7 +96,7 @@ jobs: # Pin xlings to a known-good version. The upstream install # script always grabs `latest` (no version override), so we # download + self-install manually to avoid broken releases. - XLINGS_VERSION: '2026.8.4.1' + XLINGS_VERSION: '2026.8.5.1' run: | if [ ! -x "$HOME/.xlings/subos/default/bin/xlings" ]; then tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" @@ -288,7 +288,7 @@ jobs: - name: Bootstrap mcpp via xlings env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.8.4.1' + XLINGS_VERSION: '2026.8.5.1' run: | tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" curl -fsSL -o "/tmp/${tarball}" \ @@ -358,11 +358,11 @@ jobs: # below are pinned to the same version as XLINGS_VERSION; they are # NOT interpolated from it, so check_version_pins.sh scans for them # explicitly (they were absent from the old lock-step comment). - XLA="xlings-2026.8.4.1-linux-aarch64.tar.gz" + XLA="xlings-2026.8.5.1-linux-aarch64.tar.gz" if curl -fsSL -o "/tmp/$XLA" \ - "https://github.com/openxlings/xlings/releases/download/v2026.8.4.1/$XLA"; then + "https://github.com/openxlings/xlings/releases/download/v2026.8.5.1/$XLA"; then tar -xzf "/tmp/$XLA" -C /tmp - XLBIN=$(find /tmp/xlings-2026.8.4.1-linux-aarch64 -path '*/bin/xlings' -type f | head -1) + XLBIN=$(find /tmp/xlings-2026.8.5.1-linux-aarch64 -path '*/bin/xlings' -type f | head -1) if [ -n "$XLBIN" ]; then mkdir -p "$STAGING/$WRAPPER/registry/bin" cp "$XLBIN" "$STAGING/$WRAPPER/registry/bin/xlings" @@ -440,7 +440,7 @@ jobs: - name: Bootstrap mcpp via xlings env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.8.4.1' + XLINGS_VERSION: '2026.8.5.1' run: | if [ ! -x "$HOME/.xlings/subos/default/bin/xlings" ]; then WORK=$(mktemp -d) @@ -622,7 +622,7 @@ jobs: shell: bash env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.8.4.1' + XLINGS_VERSION: '2026.8.5.1' run: | # Captured before the `cd` below, in POSIX form: this step never # returns to the workspace, and GITHUB_WORKSPACE is a backslash diff --git a/CHANGELOG.md b/CHANGELOG.md index 06689781..99151ab5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,35 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [2026.8.5.1] — 2026-08-05 + +`build.mcpp` 机制的**架构地基**:把「一条指令是什么」收敛成一张表,并补上三个今天就存在的稳定性缺口。架构分析见 `.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md`(本次实现其中的步 0 与步 1)。 + +### 改进 + +- **新增一条 directive 从改 9 处降到改 1 处。** 一条指令原本要在九个地方定义:`Directives` 结构体字段、`parse_line` 分派、`write_cache` 序列化、`read_cache` 反序列化、`apply` 落到 manifest、`cache_fresh` 的产物存在性校验、`prepare.cppm` 的 `DirectiveMark` 字段、`markDirectiveTail`、`foldDirectiveTailIntoPrivateBuild`,再加内置 `mcpp` 模块的类型化包装。`prepare.cppm` 自己的注释还承认这个拆分**仍不完整**(「link/source/fingerprint residues stay at the call sites」)。 + + 这是本仓库反复付过学费的「同一决策在 N 处推导」形态(#233/#240/#242/#344):它**不在你新增指令时失败**,而是过一阵子在别的地方失败。现在一条指令就是新模块 `src/build/directives.cppm` 里 `kTable` 的**一行**,解析、缓存读写、落盘应用、声明产物契约、私有作用域折叠全部由该行驱动。 + + **作用域是必填字段**,不是可选注释:`include-dir` 之所以是 `PackagePrivate`,是「构建期程序不得静默拓宽包的公开接口」这条供应链规则(Cargo 纪律),把它变成表里一列意味着**下一条指令不回答这个问题就加不进来**。 + + > 为什么是新模块而不是继续写 `build_program.cppm`:那个文件的匿名 namespace 在 clang 22 + C++20 modules + `-O2` 下会**误编译自己的邻居**(PR#332 证实一个**从未被调用**的新函数就足以破坏 `contract_env`,PR#334 复现)。`mcpp.build.hostprogram` 当初就是为此拆出去的。 + +- **`build.mcpp` 有了线协议版本(S1)。** 用 `import mcpp;` 的程序在 `main` 之前自动声明 `mcpp:protocol=`(你不用自己写)。于是: + + - 程序声明的协议**高于**本 mcpp 所理解的 → **拒绝执行**并给出升级提示。原先的行为是警告后忽略,那会让「构建成功了但那个 flag 根本没到」——最难查的一类构建 bug——静默发生。 + - 既然双方已被证明一致,**未知指令即错误**:同一协议版本内它只可能是拼写错误。 + + 手写 `printf("mcpp:…")` 的程序什么都不声明,**保留**历史上的「警告并忽略」。这条不对称就是兼容性契约本身:那一面**冻结在现有 11 条指令**上,新能力只在类型化 API 里落地。 + +- **`build.mcpp` 缓存带上了语义 epoch(S2)。** 缓存条目里的指令是命中时**原样重放**的,所以当引擎对某条指令的**解释**改变时,旧条目必须失效而不是被按新含义重放。纪律照抄 `cache_key::kCacheEpoch`:只在旧条目真的不可用时 bump,且**与 mcpp 版本号解耦**(否则每次发布都会让所有构建程序白重跑一遍)。一条本 mcpp 不认识的 `d` 记录(更新的 mcpp 写的)同样让整条条目作废——只重放认识的那部分,等于应用了程序所要求的一个**真子集**。 + +- **`build.mcpp` 的运行有了时间上限(S3)。** 默认 600 秒,超时杀掉并让构建失败,错误**点名是哪个包**、并说明怎么改。原先没有任何上限:一个死循环或卡在网络读上的构建程序会让整个构建**挂死且毫无诊断**。 + + **编译**这一步刻意不设上限——与 `mcpp test` 同一条不对称纪律(run 有限 / build 无限):编译跑得久通常是正当的(首次构建 `std` 模块就是分钟级),杀掉它只会产生莫名其妙的失败;构建**程序**跑得久通常是卡住了。`capture_exec_deadline` 顺带补上了 `cwd` 形参——没有它,加超时会**静默改变**构建程序相对写入的落点。 + +- **内带 xlings 升级到 `2026.8.5.1`**(自 `2026.8.4.1`)。13 个 pin 点由 `check_version_pins.sh` 机器校验并全部更新。 + ## [2026.8.4.1] — 2026-08-04 ### 修复 diff --git a/docs/07-build-mcpp.md b/docs/07-build-mcpp.md index 78ae65c3..57369b1c 100644 --- a/docs/07-build-mcpp.md +++ b/docs/07-build-mcpp.md @@ -101,6 +101,34 @@ int main() { The raw stdout protocol above remains the low-level substrate; `import mcpp;` is the typed layer over it. +### `import mcpp;` is the surface that evolves (mcpp 2026.8.5.1+) + +Two ways to talk to mcpp, and they carry **different compatibility promises**: + +| | `import mcpp;` | hand-written `printf("mcpp:…")` | +|---|---|---| +| Compatibility | The module is **bundled in the mcpp binary** and recompiled by the mcpp that runs it, so program and engine can never disagree | Your string is frozen text; nothing checks it against the engine | +| New directives | Arrive as new functions | **Will not be added** | +| Unknown directive | **Hard error** | Warning, then ignored | + +Programs using `import mcpp;` automatically announce the protocol version they +were built against (`mcpp:protocol=`, emitted before `main` runs — you never +write it yourself). mcpp uses that two ways: + +- A program announcing a **newer** protocol than mcpp understands is **refused**, + with an upgrade hint. Continuing would silently drop directives the build + depends on — and "the build succeeded but the flag never arrived" is the + worst class of build bug. +- Because the two sides then provably agree, an **unrecognized directive is an + error** rather than a warning: within one protocol version it can only be a + typo. + +A `printf`-style program announces nothing, so it keeps the historical +warn-and-ignore behaviour. That surface is **frozen at the eleven directives in +the table above** — it still works and will keep working, but new capabilities +land only in the typed API. Prefer `import mcpp;` for anything you intend to +maintain. + ### `import std;` (mcpp 2026.8.2.1+) A `build.mcpp` may `import std;` (and `import std.compat;`), alone or together @@ -204,7 +232,10 @@ directives and re-runs only when something it depends on changed: - the toolchain, - any file you declared with `rerun-if-changed`, - any env var you declared with `rerun-if-env-changed`, -- (or a `generated` output / `source=` selection went missing). +- (or a `generated` output / `source=` selection went missing), +- (or the cache was written by an mcpp that interpreted a directive differently + — the entry carries a format **epoch**, and a foreign one re-runs the program + once instead of replaying values under the wrong meaning). So **declare your inputs**: if your program reads `config.h` or the `USE_FAST` variable, emit `mcpp:rerun-if-changed=config.h` / `mcpp:rerun-if-env-changed=USE_FAST`. @@ -224,3 +255,11 @@ When nothing changed you'll see `build.mcpp up to date (cached)`; otherwise - **CWD is the project root**, so relative paths (`src/generated.cpp`) land where you expect. - A non-zero exit from `build.mcpp` aborts the build and prints its output. +- **The run is bounded** (mcpp 2026.8.5.1+): a build program gets **600 s** by + default, after which mcpp kills it and fails the build naming the package. + Override with `MCPP_BUILD_PROGRAM_TIMEOUT=` (`0` = no limit). The + **compile** is deliberately *not* bounded — the same asymmetry `mcpp test` + uses: a long compile is usually legitimate (a first-run `std` module build is + minutes) and killing it produces a baffling failure, while a long-running + build *program* is usually stuck, and an unbounded one hangs the whole build + with no diagnostic at all. diff --git a/docs/zh/07-build-mcpp.md b/docs/zh/07-build-mcpp.md index f62e53fb..05899a34 100644 --- a/docs/zh/07-build-mcpp.md +++ b/docs/zh/07-build-mcpp.md @@ -93,6 +93,28 @@ int main() { 上面的裸 stdout 协议仍是底层基底;`import mcpp;` 是其上的类型化层。 +### `import mcpp;` 才是会演进的那一面(mcpp 2026.8.5.1+) + +和 mcpp 对话有两条路,它们的**兼容性承诺不同**: + +| | `import mcpp;` | 手写 `printf("mcpp:…")` | +|---|---|---| +| 兼容性 | 该模块**内置在 mcpp 二进制里**,由运行它的那个 mcpp 现场编译,程序与引擎不可能不一致 | 你的字符串是冻结的文本,没有任何东西替你校验 | +| 新指令 | 以新函数的形式到来 | **不会再新增** | +| 未知指令 | **硬错误** | 警告后忽略 | + +用 `import mcpp;` 的程序会自动声明它编译时对应的协议版本(`mcpp:protocol=`, +在 `main` 之前发出——你不需要自己写)。mcpp 用它做两件事: + +- 程序声明的协议**高于** mcpp 所理解的 → **拒绝执行**,并给出升级提示。继续跑会 + 静默丢掉构建依赖的指令,而「构建成功了但那个 flag 根本没到」是最难查的一类问题。 +- 既然双方已被证明一致,**未知指令就是错误**而不是警告:在同一个协议版本内, + 它只可能是拼写错误。 + +`printf` 风格的程序什么都不声明,因此保留历史上的「警告并忽略」行为。这一面 +**冻结在上表的 11 条指令**上——它仍然能用、也会继续能用,但新能力只在类型化 API 里 +落地。**要长期维护的程序请用 `import mcpp;`。** + ### `import std;`(mcpp 2026.8.2.1+) `build.mcpp` 可以 `import std;`(以及 `import std.compat;`),单用或与 @@ -186,7 +208,9 @@ mcpp **不会**每次构建都重跑 `build.mcpp`。它会缓存程序产出的 - 工具链, - 任何用 `rerun-if-changed` 声明的文件, - 任何用 `rerun-if-env-changed` 声明的环境变量, -- (或某个 `generated` 产物 / `source=` 选中的文件丢失了)。 +- (或某个 `generated` 产物 / `source=` 选中的文件丢失了), +- (或该缓存是由一个对某条指令解释不同的 mcpp 写下的——条目带一个格式 **epoch**, + 遇到不认识的 epoch 就重跑一次,而不是把值按错误的含义重放)。 所以请**声明你的输入**:如果程序读了 `config.h` 或 `USE_FAST` 变量,就分别 emit `mcpp:rerun-if-changed=config.h` / `mcpp:rerun-if-env-changed=USE_FAST`。这用一份明确的 @@ -203,3 +227,8 @@ mcpp **不会**每次构建都重跑 `build.mcpp`。它会缓存程序产出的 [05 - mcpp.toml 工程文件指南](05-mcpp-toml.md)。 - **当前工作目录是工程根目录**,因此相对路径(`src/generated.cpp`)会落在你预期的位置。 - `build.mcpp` 非零退出会中止构建并打印其输出。 +- **运行有时间上限**(mcpp 2026.8.5.1+):构建程序默认有 **600 秒**,超时后 mcpp 杀掉它 + 并让构建失败,错误里会点名是哪个包。用 `MCPP_BUILD_PROGRAM_TIMEOUT=<秒>` 覆盖 + (`0` = 不限)。**编译**这一步刻意**不设**上限——与 `mcpp test` 同一条不对称纪律: + 编译跑得久通常是正当的(首次构建 `std` 模块就是分钟级),杀掉它只会产生莫名其妙的 + 失败;而构建**程序**跑得久通常是卡住了,不设上限就会让整个构建挂死且毫无诊断。 diff --git a/mcpp.toml b/mcpp.toml index be8f572d..561204f3 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.8.4.1" +version = "2026.8.5.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index fc714c73..e8406b8f 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -19,6 +19,7 @@ import mcpp.platform.process; import mcpp.toolchain.cppfly; // std_flag (dialect- and c++fly-aware -std= spelling) import mcpp.toolchain.dialect; // CommandDialect — gnu vs cl.exe spellings import mcpp.toolchain.fingerprint; // hash_file / hash_string (FNV-1a, 16 hex) +import mcpp.build.directives; // the directive definition table (own module: see its header) import mcpp.build.hostprogram; // bundled `mcpp` module compile (own module: see its header) import mcpp.toolchain.hostflags; // the shared host-compile flag producer import mcpp.toolchain.linkmodel; // shared C-library / clang-cfg-bypass model @@ -76,103 +77,21 @@ namespace { namespace fs = std::filesystem; -// Parsed directives in apply order. Stored verbatim in the cache so a cache hit -// reapplies the exact same edits without re-running the program. -struct Directives { - std::vector cxxflags; // -> buildConfig.cxxflags - std::vector cflags; // -> buildConfig.cflags - // -> buildConfig.ldflags, already spelled for the host dialect - // (-l/-L for GNU, name.lib//LIBPATH: for cl.exe) — see parse_line. - std::vector ldflags; - std::vector defines; // cfg= -> define prefix, into BOTH c/cxx flags - std::vector generated; // relative source paths - // source= — select a PRE-EXISTING file (tarball payload / vendored tree) - // into the compile set. Downstream identical to generated=; the semantic - // difference is intent only: the program did not write this file, it chose - // it. Relative paths resolve against the PACKAGE ROOT in both root and - // dependency mode (a payload file lives in the package tree, never in - // OUT_DIR) — unlike generated=, whose dep-mode relatives resolve to genBase. - std::vector sources; - // include-dir[/-after]= — private include directories for THIS package's - // own TUs (-I / -idirafter). Normalized to absolute at parse time (abs - // taken as-is, relative against the package root, same as link-search). - // Cargo discipline: NEVER propagated as a usage requirement — an include - // dir that consumers must see belongs in the declarative descriptor, not - // in a build-time program (supply-chain surface: a build program must not - // silently widen a package's public interface). - std::vector includeDirs; - std::vector includeDirsAfter; - std::vector rerunFiles; // declared file inputs - std::vector rerunEnv; // declared env-var inputs -}; +namespace dirs = mcpp::build::directives; -std::string trim(std::string_view s) { - std::size_t b = 0, e = s.size(); - while (b < e && (s[b] == ' ' || s[b] == '\t' || s[b] == '\r')) ++b; - while (e > b && (s[e - 1] == ' ' || s[e - 1] == '\t' || s[e - 1] == '\r')) --e; - return std::string(s.substr(b, e - b)); -} +// The directive model — what a directive IS, how it parses, how it is cached +// and applied — lives in mcpp.build.directives as a single table. This file +// only orchestrates: compile, run, cache, validate. See that module's header +// for why it is separate (both the nine-sites problem and the clang 22 +// anonymous-namespace miscompile that forbids growing this one). +using Directives = dirs::Directives; +using dirs::Slot; // Resolve a possibly-relative path against the project root, returning an // absolute lexically-normal path (no filesystem touch, so it works for dirs that // the program is about to create as well as existing ones). std::string abs_against_root(const fs::path& root, std::string_view p) { - fs::path pp(p); - if (pp.is_relative()) pp = root / pp; - return pp.lexically_normal().string(); -} - -// Parse one stdout line. Returns true if it was a recognized (or unknown-but- -// `mcpp:`) directive; false for ordinary program chatter. -// `dial` decides how `link-lib` / `link-search` are spelled. The `mcpp:` -// protocol itself is declarative — a build program says WHICH library it -// needs, never how the local compiler driver names one — so the translation -// belongs here at the boundary, not in the program. -// -// Storing the translated form in Directives (and therefore in the build.mcpp -// cache) is safe because the cache key already hashes the compiler: switching -// toolchains invalidates the entry before any spelling from the old dialect -// could be replayed under the new one. -bool parse_line(const fs::path& root, const mcpp::toolchain::CommandDialect& dial, - std::string_view raw, Directives& d) { - std::string line = trim(raw); - constexpr std::string_view kPfx = "mcpp:"; - if (!line.starts_with(kPfx)) return false; - std::string_view body = std::string_view(line).substr(kPfx.size()); - auto eq = body.find('='); - std::string key = std::string(body.substr(0, eq)); - std::string val = eq == std::string_view::npos ? std::string() : std::string(body.substr(eq + 1)); - - if (key == "cxxflag") d.cxxflags.push_back(val); - else if (key == "cflag") d.cflags.push_back(val); - else if (key == "link-lib") d.ldflags.push_back( - mcpp::toolchain::lib_flag_for(dial, val)); - else if (key == "link-search") d.ldflags.push_back( - std::string(dial.libSearchPrefix) - + abs_against_root(root, val)); - else if (key == "cfg") d.defines.push_back( - std::string(dial.definePrefix) + val); - else if (key == "generated") d.generated.push_back(val); - else if (key == "source") d.sources.push_back(val); - else if (key == "include-dir") d.includeDirs.push_back(abs_against_root(root, val)); - else if (key == "include-dir-after") - d.includeDirsAfter.push_back(abs_against_root(root, val)); - else if (key == "rerun-if-changed") d.rerunFiles.push_back(val); - else if (key == "rerun-if-env-changed") d.rerunEnv.push_back(val); - else mcpp::ui::warning(std::format("build.mcpp: ignoring unknown directive 'mcpp:{}'", key)); - return true; -} - -void parse_output(const fs::path& root, const mcpp::toolchain::CommandDialect& dial, - std::string_view out, Directives& d) { - std::size_t pos = 0; - while (pos <= out.size()) { - std::size_t nl = out.find('\n', pos); - std::string_view ln = out.substr(pos, nl == std::string_view::npos ? std::string_view::npos : nl - pos); - parse_line(root, dial, ln, d); - if (nl == std::string_view::npos) break; - pos = nl + 1; - } + return dirs::abs_against(root, p); } std::string env_value(const std::string& name) { @@ -217,13 +136,17 @@ std::vector host_base_flags(const mcpp::toolchain::Toolchain& tc, } // ── Cache (line-based; one record per line, internal format) ─────────────── +// epoch // program // compiler +// ctx // in // env -// d cxxflag|cflag|ldflag|define|generated|source|include-dir|include-dir-after -// The leading program/compiler/in/env lines are the re-run key; the `d` lines -// are the directives to reapply on a hit. +// d +// The leading epoch/program/compiler/ctx/in/env lines are the re-run key; the +// `d` lines are the directives to reapply on a hit. The `d` tag vocabulary is +// owned by mcpp.build.directives::kTable and is NOT spelled here — that list +// used to be duplicated in four places and drifted. // build.mcpp artifacts live under target/ (the build output tree), not in the // project: target/.build-mcpp/{build.mcpp.bin, build.mcpp.cache}. A stable subdir @@ -317,33 +240,32 @@ void write_cache(const fs::path& bdir, const fs::path& root, const Directives& d) { std::ofstream os(cache_path(bdir), std::ios::trunc); if (!os) return; // best-effort: a failed cache write only loses the optimization + // The epoch guards against a semantics change: the `d` lines below are + // replayed verbatim on a hit, so if this mcpp interprets a directive + // differently than the one that wrote them, the entry must not be reused. + os << "epoch " << dirs::kCacheEpoch << '\n'; os << "program " << programHash << '\n'; os << "compiler " << compilerHash << '\n'; os << "ctx " << ctxHash << '\n'; - for (auto const& f : d.rerunFiles) + for (auto const& f : d.at(Slot::RerunFiles)) os << "in " << mcpp::toolchain::hash_file(abs_against_root(root, f)) << ' ' << f << '\n'; - for (auto const& e : d.rerunEnv) + for (auto const& e : d.at(Slot::RerunEnv)) os << "env " << mcpp::toolchain::hash_string(env_value(e)) << ' ' << e << '\n'; - auto emit = [&](std::string_view kind, const std::vector& v) { - for (auto const& x : v) os << "d " << kind << ' ' << x << '\n'; - }; - emit("cxxflag", d.cxxflags); - emit("cflag", d.cflags); - emit("ldflag", d.ldflags); - emit("define", d.defines); - emit("generated", d.generated); - emit("source", d.sources); - emit("include-dir", d.includeDirs); - emit("include-dir-after", d.includeDirsAfter); + dirs::serialize(os, d); } struct CacheRecord { + int epoch = 0; // 0 = pre-epoch entry (written before this guard existed) std::string programHash; std::string compilerHash; std::string ctxHash; // contract env (target/profile/features/out-dir) std::vector> inputs; // (hash, path) std::vector> envs; // (hash, name) Directives directives; + // A `d` record whose tag this mcpp does not know — the entry was written + // by a newer mcpp. Replaying the rest would apply a strict subset of what + // the program asked for, so the whole entry is discarded instead. + bool unknownRecord = false; bool loaded = false; }; @@ -358,7 +280,12 @@ CacheRecord read_cache(const fs::path& bdir) { if (sp == std::string::npos) continue; std::string tag = line.substr(0, sp); std::string rest = line.substr(sp + 1); - if (tag == "program") r.programHash = rest; + if (tag == "epoch") { + int n = 0; + if (std::from_chars(rest.data(), rest.data() + rest.size(), n).ec == std::errc{}) + r.epoch = n; + } + else if (tag == "program") r.programHash = rest; else if (tag == "compiler") r.compilerHash = rest; else if (tag == "ctx") r.ctxHash = rest; else if (tag == "in" || tag == "env") { @@ -370,14 +297,8 @@ CacheRecord read_cache(const fs::path& bdir) { auto sp2 = rest.find(' '); if (sp2 == std::string::npos) continue; std::string kind = rest.substr(0, sp2), val = rest.substr(sp2 + 1); - if (kind == "cxxflag") r.directives.cxxflags.push_back(val); - else if (kind == "cflag") r.directives.cflags.push_back(val); - else if (kind == "ldflag") r.directives.ldflags.push_back(val); - else if (kind == "define") r.directives.defines.push_back(val); - else if (kind == "generated") r.directives.generated.push_back(val); - else if (kind == "source") r.directives.sources.push_back(val); - else if (kind == "include-dir") r.directives.includeDirs.push_back(val); - else if (kind == "include-dir-after") r.directives.includeDirsAfter.push_back(val); + if (!dirs::accept_cache_record(r.directives, kind, val)) + r.unknownRecord = true; } } r.loaded = true; @@ -389,6 +310,8 @@ bool cache_fresh(const fs::path& root, const CacheRecord& c, const std::string& programHash, const std::string& compilerHash, const std::string& ctxHash) { if (!c.loaded) return false; + if (c.epoch != dirs::kCacheEpoch) return false; // pre-epoch entries rerun once + if (c.unknownRecord) return false; if (c.programHash != programHash) return false; if (c.compilerHash != compilerHash) return false; if (c.ctxHash != ctxHash) return false; // pre-G3 caches (no ctx line) rerun once @@ -396,43 +319,15 @@ bool cache_fresh(const fs::path& root, const CacheRecord& c, if (mcpp::toolchain::hash_file(abs_against_root(root, path)) != h) return false; for (auto const& [h, name] : c.envs) if (mcpp::toolchain::hash_string(env_value(name)) != h) return false; - // A declared generated output / selected source that vanished invalidates - // the cache. - for (auto const& g : c.directives.generated) - if (!fs::exists(abs_against_root(root, g))) return false; - for (auto const& s : c.directives.sources) - if (!fs::exists(abs_against_root(root, s))) return false; - return true; -} - -void apply(mcpp::manifest::Manifest& m, const Directives& d) { - auto& bc = m.buildConfig; - bc.cxxflags.insert(bc.cxxflags.end(), d.cxxflags.begin(), d.cxxflags.end()); - bc.cflags.insert(bc.cflags.end(), d.cflags.begin(), d.cflags.end()); - bc.ldflags.insert(bc.ldflags.end(), d.ldflags.begin(), d.ldflags.end()); - // cfg defines apply to both C and C++ translation units. - bc.cflags.insert(bc.cflags.end(), d.defines.begin(), d.defines.end()); - bc.cxxflags.insert(bc.cxxflags.end(), d.defines.begin(), d.defines.end()); - // Generated + selected (source=) sources join the source set. BOTH lists: - // the scanner walks the legacy modules.sources mirror — pushing only - // bc.sources left a generated file outside the base globs invisible to the - // scan (latent since L3). - for (auto const& g : d.generated) { - bc.sources.push_back(g); - m.modules.sources.push_back(g); + // A declared output that vanished invalidates the cache. Driven off the + // table's mustExistAfterRun so a future output-shaped directive is covered + // without editing this function. + for (auto const& def : dirs::kTable) { + if (!def.mustExistAfterRun) continue; + for (auto const& p : c.directives.at(def.slot)) + if (!fs::exists(abs_against_root(root, p))) return false; } - for (auto const& s : d.sources) { - bc.sources.push_back(s); - m.modules.sources.push_back(s); - } - // Include dirs (already absolute from parse_line). PRIVATE by design: for - // the root they join buildConfig before the package snapshot; for a - // dependency the caller (prepare.cppm dep loop) mirrors them into the - // dep's privateBuild only — never into publicUsage (see Directives note). - for (auto const& p : d.includeDirs) - bc.includeDirs.emplace_back(p); - for (auto const& p : d.includeDirsAfter) - bc.includeDirsAfter.push_back(p); + return true; } } // namespace @@ -490,7 +385,7 @@ std::expected run_build_program( // directives, no run. CacheRecord cache = read_cache(bdir); if (cache_fresh(root, cache, programHash, compilerHash, ctxHash)) { - apply(m, cache.directives); + dirs::apply(m, cache.directives); mcpp::ui::info("build.mcpp", "up to date (cached)"); return {}; } @@ -717,15 +612,49 @@ std::expected run_build_program( // Run with cwd = package root so the program's relative file writes (e.g. // mcpp:generated sources) land in the project, not in mcpp's invocation // dir. The MCPP_* contract env is injected into the CHILD only. + // + // Bounded, unlike the compile above. The asymmetry is deliberate and the + // same one `mcpp test` settled on: a COMPILE that runs long is usually + // legitimate (a first-run std module build is minutes), and killing it + // produces a baffling failure; a build PROGRAM that runs long is usually + // stuck — waiting on a network read or spinning — and without a bound the + // whole build hangs with no diagnostic at all. mcpp::ui::info("build.mcpp", "running"); - auto rres = mcpp::platform::process::capture_exec({bin.string()}, childEnv, root.string()); + bool timedOut = false; + auto rres = mcpp::platform::process::capture_exec_deadline( + {bin.string()}, childEnv, dirs::run_timeout(), &timedOut, root.string()); + if (timedOut) { + return std::unexpected(std::format( + "build.mcpp for '{}' exceeded its {}s time limit and was killed.\n" + " Raise or disable it with MCPP_BUILD_PROGRAM_TIMEOUT= " + "(0 = no limit).\n" + " Output so far:\n{}", + m.package.name.empty() ? std::string("") + : m.package.name, + dirs::run_timeout().count() / 1000, rres.output)); + } if (rres.exit_code != 0) { return std::unexpected(std::format( "build.mcpp exited with {} (build aborted):\n{}", rres.exit_code, rres.output)); } Directives d; - parse_output(root, dial, rres.output, d); + dirs::accept_output(d, dial, root, rres.output); + + // Protocol gate (S1). A program that announced a version this mcpp cannot + // speak, or that emitted an unknown directive INSIDE a version both sides + // speak, is a hard error — "warn and ignore" would turn a missing + // directive into a silently different build. A program that announced + // nothing is a hand-written printf program (the frozen surface) and keeps + // the historical warn-and-ignore behaviour. + if (auto perr = dirs::protocol_error(d)) { + return std::unexpected(*perr); + } + if (d.protocol == 0) { + for (auto const& k : d.unknownKeys) + mcpp::ui::warning(std::format( + "build.mcpp: ignoring unknown directive 'mcpp:{}'", k)); + } // Dependency mode (genBase set): relative `generated=` paths resolve // against OUT_DIR-style genBase, not the (possibly read-only, shared) @@ -733,30 +662,28 @@ std::expected run_build_program( // `source=` paths are NOT rewritten: they name pre-existing files in the // package tree (MCPP_MANIFEST_DIR-relative), never OUT_DIR products. if (!env.genBase.empty()) { - for (auto& g : d.generated) { + for (auto& g : d.at(Slot::Generated)) { fs::path gp(g); if (gp.is_relative()) g = (env.genBase / gp).lexically_normal().string(); } } - // Missing declared generated outputs are a hard error (declared-output contract). - for (auto const& g : d.generated) { - if (!fs::exists(abs_against_root(root, g))) { - return std::unexpected(std::format( - "build.mcpp declared generated source '{}' but it does not exist after the run", g)); - } - } - // A `source=` selection must already exist — the program selects a file it - // did NOT write (payload / vendored tree); a missing one is a typo or a - // broken payload, surfaced now instead of as a later glob no-match. - for (auto const& s : d.sources) { - if (!fs::exists(abs_against_root(root, s))) { - return std::unexpected(std::format( - "build.mcpp selected source '{}' (mcpp:source=) but no such file exists", s)); + // Declared-output contract, driven off the table's mustExistAfterRun: + // generated= — the program said it wrote this file; if it did not, the + // build would fail far away as a glob no-match. + // source= — the program SELECTED a pre-existing file (payload / + // vendored tree); a missing one is a typo or a broken + // payload, surfaced now. + for (auto const& def : dirs::kTable) { + if (!def.mustExistAfterRun) continue; + for (auto const& p : d.at(def.slot)) { + if (fs::exists(abs_against_root(root, p))) continue; + return std::unexpected(std::format("build.mcpp {} '{}' {}", + def.missingPrefix, p, def.missingSuffix)); } } - apply(m, d); + dirs::apply(m, d); write_cache(bdir, root, programHash, compilerHash, ctxHash, d); return {}; } diff --git a/src/build/directives.cppm b/src/build/directives.cppm new file mode 100644 index 00000000..ac602d9c --- /dev/null +++ b/src/build/directives.cppm @@ -0,0 +1,453 @@ +// mcpp.build.directives — the ONE definition of what a `build.mcpp` directive is. +// +// WHY THIS MODULE EXISTS +// +// A directive used to be defined in nine places: the Directives struct field, +// parse_line's dispatch, write_cache's emit, read_cache's parse, apply's fold +// into the manifest, cache_fresh's declared-output check, prepare.cppm's +// DirectiveMark field, markDirectiveTail, and foldDirectiveTailIntoPrivateBuild +// — plus the bundled `mcpp` module's typed wrapper. prepare.cppm's own comment +// admitted the split was still incomplete ("Link/source/fingerprint residues +// stay at the call sites"). That is the "same decision derived in N places" +// shape this codebase has paid for repeatedly (#233/#240/#242/#344): it does +// not fail when you add the directive, it fails later, somewhere else. +// +// Here a directive is ONE row in kTable. Parsing, cache serialization, cache +// deserialization, application to the manifest, the declared-output contract, +// and the private-scope fold are all driven off that row. +// +// WHY IT IS A SEPARATE MODULE RATHER THAN MORE OF build_program.cppm +// +// Not taste — a miscompile. build_program.cppm's anonymous namespace corrupts +// its own neighbours under clang 22 + C++20 modules + -O2: PR#332 established +// that an UNUSED helper added there was enough to break `contract_env`, and +// PR#334 reproduced it. mcpp.build.hostprogram was split out for exactly this +// reason and says so in its header. The rule is "stop growing that namespace", +// so the table lives here. +// +// SCOPE IS A REQUIRED FIELD, ON PURPOSE +// +// Every row must state its Scope. `include-dir` being PackagePrivate is not a +// style choice — it is the supply-chain rule that a build-time program must +// not silently widen a package's public interface (Cargo discipline). Making +// Scope a field means the next directive cannot be added without someone +// answering that question. +// +// See .agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md §4 (S5). + +export module mcpp.build.directives; + +import std; +import mcpp.manifest; +import mcpp.toolchain.dialect; + +export namespace mcpp::build::directives { + +// ── Protocol version ─────────────────────────────────────────────────────── +// +// The wire version this engine speaks. The bundled `mcpp` module announces the +// version it was built against (`mcpp:protocol=`) before main runs, so a +// program and the engine that compiled it always agree — the announcement only +// ever disagrees when a *cached* helper binary outlives an engine change, which +// is precisely the case worth catching. +// +// Bump when the meaning of an existing directive changes, or when a new +// directive is added that a program may rely on. An engine seeing a HIGHER +// number than this must refuse: it cannot know what it is being asked to do, +// and "warn and ignore" would turn that into a silently different build. +inline constexpr int kProtocolVersion = 1; + +// ── Cache-format epoch ───────────────────────────────────────────────────── +// +// Bump ONLY when previously written build.mcpp.cache entries become unusable — +// the record shape changed, or a directive's *interpretation* changed so that +// replaying a cached value would no longer mean what it meant when written. +// Deliberately NOT the mcpp release number: folding the whole version in would +// re-run every build program on every release for nothing. Same discipline as +// mcpp.build.cache_key::kCacheEpoch. +inline constexpr int kCacheEpoch = 1; + +// ── Run bound ────────────────────────────────────────────────────────────── +// +// How long a build program may RUN before mcpp kills it. The compile is +// deliberately left unbounded — the same asymmetry `mcpp test` settled on +// (run 300s / build 0): a long compile is usually legitimate (a first-run std +// module build is minutes) and killing it produces a baffling failure, while a +// long-running build PROGRAM is usually stuck, and without a bound the whole +// build hangs with no diagnostic at all. +// +// MCPP_BUILD_PROGRAM_TIMEOUT overrides, in seconds; 0 disables the bound. +inline constexpr int kDefaultRunTimeoutSecs = 600; + +std::chrono::milliseconds run_timeout(); + +// ── The table ────────────────────────────────────────────────────────────── + +// Where a directive's value accumulates. One slot may be fed by several wire +// names (link-lib and link-search both produce link flags). +enum class Slot : std::size_t { + CxxFlags = 0, + CFlags, + LdFlags, + Defines, + Generated, + Sources, + IncludeDirs, + IncludeDirsAfter, + RerunFiles, + RerunEnv, + Count +}; +inline constexpr std::size_t kSlotCount = static_cast(Slot::Count); + +// Who sees the value. The field that must be answered for every new directive. +enum class Scope { + PackagePrivate, // only this package's own TUs — never propagated to consumers + LinkGlobal, // reaches the final link of whatever consumes this package + SourceSet, // joins the compile set + RerunKey, // not a build input at all; only feeds the re-run key +}; + +// How the raw wire value is normalized before it is stored. Applied ONCE, at +// parse time, so the cache holds the already-spelled form (safe: the cache key +// hashes the compiler identity, so a dialect switch invalidates the entry +// before any old spelling could be replayed under a new dialect). +enum class Transform { + Verbatim, + LibFlag, // dialect lib_flag_for (-lfoo | foo.lib) + LibSearchPath, // dialect libSearchPrefix + absolute path + DefinePrefix, // dialect definePrefix + value + AbsPath, // absolute, lexically normal +}; + +struct Def { + std::string_view wire; // the `mcpp:=` name + std::string_view tag; // cache-record tag; empty = not persisted as a directive + Slot slot; + Scope scope; + Transform transform; + // Declared-output contract: the value names a file that MUST exist after + // the program ran, and whose disappearance invalidates the cache. + bool mustExistAfterRun; + // The diagnostic when it does not, as "build.mcpp '' + // ". Two fields rather than one generic sentence because the two + // output-shaped directives mean genuinely different things — `generated=` + // says "I WROTE this", `source=` says "I SELECTED this pre-existing file" + // — and a user debugging one needs to be told which contract they broke. + // Required (non-empty) whenever mustExistAfterRun is set. + std::string_view missingPrefix; + std::string_view missingSuffix; + int sinceProtocol; +}; + +inline constexpr std::array kTable{{ + // wire tag slot scope transform must missingPrefix missingSuffix since + {"cxxflag", "cxxflag", Slot::CxxFlags, Scope::PackagePrivate, Transform::Verbatim, false, "", "", 1}, + {"cflag", "cflag", Slot::CFlags, Scope::PackagePrivate, Transform::Verbatim, false, "", "", 1}, + {"link-lib", "ldflag", Slot::LdFlags, Scope::LinkGlobal, Transform::LibFlag, false, "", "", 1}, + {"link-search", "ldflag", Slot::LdFlags, Scope::LinkGlobal, Transform::LibSearchPath, false, "", "", 1}, + {"cfg", "define", Slot::Defines, Scope::PackagePrivate, Transform::DefinePrefix, false, "", "", 1}, + {"generated", "generated", Slot::Generated, Scope::SourceSet, Transform::Verbatim, true, "declared generated source", "but it does not exist after the run", 1}, + {"source", "source", Slot::Sources, Scope::SourceSet, Transform::Verbatim, true, "selected source", "(mcpp:source=) but no such file exists", 1}, + {"include-dir", "include-dir", Slot::IncludeDirs, Scope::PackagePrivate, Transform::AbsPath, false, "", "", 1}, + {"include-dir-after", "include-dir-after", Slot::IncludeDirsAfter, Scope::PackagePrivate, Transform::AbsPath, false, "", "", 1}, + {"rerun-if-changed", "", Slot::RerunFiles, Scope::RerunKey, Transform::Verbatim, false, "", "", 1}, + {"rerun-if-env-changed","", Slot::RerunEnv, Scope::RerunKey, Transform::Verbatim, false, "", "", 1}, +}}; + +// ── Collected output of one run ──────────────────────────────────────────── + +struct Directives { + std::array, kSlotCount> slots{}; + // The protocol the program announced. 0 = it never announced one, which + // means a hand-written `printf("mcpp:...")` program (the frozen surface). + int protocol = 0; + // `mcpp:` keys this engine does not know. Whether that is fatal depends on + // `protocol` — see unknown_directive_error(). + std::vector unknownKeys; + + std::vector& at(Slot s) { return slots[static_cast(s)]; } + const std::vector& at(Slot s) const { return slots[static_cast(s)]; } +}; + +// ── Lookups ──────────────────────────────────────────────────────────────── + +const Def* find_by_wire(std::string_view wire); +const Def* find_by_tag(std::string_view tag); + +// ── Path helper (shared with the caller's existence checks) ──────────────── + +std::string abs_against(const std::filesystem::path& base, std::string_view p); + +// ── Parse ────────────────────────────────────────────────────────────────── + +enum class LineResult { + NotADirective, // ordinary program chatter + Accepted, + Protocol, // `mcpp:protocol=` + Unknown, // a `mcpp:` key this engine does not know +}; + +// Parse ONE stdout line into `d`. `root` resolves relative paths for the +// path-shaped transforms; `dial` spells the link/define flags. +LineResult accept_line(Directives& d, const mcpp::toolchain::CommandDialect& dial, + const std::filesystem::path& root, std::string_view raw); + +void accept_output(Directives& d, const mcpp::toolchain::CommandDialect& dial, + const std::filesystem::path& root, std::string_view out); + +// Non-empty when the run must be rejected: either the program speaks a newer +// protocol than this engine, or it declared a protocol and still emitted a +// directive this engine does not know (inside a version both sides agree on, +// an unknown key is a bug, not a forward-compat situation). +// +// A program that never announced a protocol keeps the historical +// warn-and-ignore behaviour: it is a hand-written printf program, frozen at +// protocol 1, and its unknown keys are typos rather than future syntax. +std::optional protocol_error(const Directives& d); + +// ── Cache serialization ──────────────────────────────────────────────────── + +// `d ` lines, in table order. +void serialize(std::ostream& os, const Directives& d); + +// One `d ` record. Returns false for an unknown tag (a cache +// written by a newer mcpp) — the caller treats that as a stale entry. +bool accept_cache_record(Directives& d, std::string_view tag, std::string_view value); + +// ── Apply ────────────────────────────────────────────────────────────────── + +// Fold the collected directives into the manifest's buildConfig. The single +// place that knows which manifest channel each slot feeds. +void apply(mcpp::manifest::Manifest& m, const Directives& d); + +// ── Private-scope fold (was prepare.cppm's DirectiveMark / fold pair) ────── +// +// Lives here because "which compile-visible channels a PackagePrivate +// directive lands in" is a property of the table, not of the call site. The +// caller records a Mark before running the program and folds the tail after. + +struct Mark { + std::size_t cflags = 0, cxxflags = 0, includeDirs = 0, includeDirsAfter = 0; +}; + +Mark mark(const mcpp::manifest::Manifest& m); + +// Fold the PackagePrivate tail into a UsageRequirements-shaped destination. +// Templated so this module does not have to import the scanner (which would +// close a module cycle) — the destination only needs the four vectors. +template +void fold_private_tail(Usage& dst, const mcpp::manifest::Manifest& ran, const Mark& t) { + auto const& bc = ran.buildConfig; + dst.cflags.insert(dst.cflags.end(), + bc.cflags.begin() + static_cast(t.cflags), + bc.cflags.end()); + dst.cxxflags.insert(dst.cxxflags.end(), + bc.cxxflags.begin() + static_cast(t.cxxflags), + bc.cxxflags.end()); + auto append_unique = [](auto& v, const std::filesystem::path& p) { + if (std::find(v.begin(), v.end(), p) == v.end()) v.push_back(p); + }; + for (auto it = bc.includeDirs.begin() + static_cast(t.includeDirs); + it != bc.includeDirs.end(); ++it) + append_unique(dst.includeDirs, *it); + for (auto it = bc.includeDirsAfter.begin() + static_cast(t.includeDirsAfter); + it != bc.includeDirsAfter.end(); ++it) + append_unique(dst.includeDirsAfter, *it); +} + +} // namespace mcpp::build::directives + +namespace mcpp::build::directives { + +namespace fs = std::filesystem; + +const Def* find_by_wire(std::string_view wire) { + for (auto const& d : kTable) + if (d.wire == wire) return &d; + return nullptr; +} + +const Def* find_by_tag(std::string_view tag) { + if (tag.empty()) return nullptr; + for (auto const& d : kTable) + if (d.tag == tag) return &d; // first row wins; rows sharing a tag share a slot + return nullptr; +} + +std::string abs_against(const fs::path& base, std::string_view p) { + fs::path pp(p); + if (pp.is_relative()) pp = base / pp; + return pp.lexically_normal().string(); +} + +std::chrono::milliseconds run_timeout() { + int secs = kDefaultRunTimeoutSecs; + if (const char* v = std::getenv("MCPP_BUILD_PROGRAM_TIMEOUT")) { + std::string_view sv(v); + int parsed = 0; + if (std::from_chars(sv.data(), sv.data() + sv.size(), parsed).ec == std::errc{} + && parsed >= 0) + secs = parsed; + } + return std::chrono::milliseconds(static_cast(secs) * 1000); +} + +namespace { + +std::string trim(std::string_view s) { + std::size_t b = 0, e = s.size(); + while (b < e && (s[b] == ' ' || s[b] == '\t' || s[b] == '\r')) ++b; + while (e > b && (s[e - 1] == ' ' || s[e - 1] == '\t' || s[e - 1] == '\r')) --e; + return std::string(s.substr(b, e - b)); +} + +std::string transformed(const Def& def, std::string_view raw, + const mcpp::toolchain::CommandDialect& dial, + const fs::path& root) { + switch (def.transform) { + case Transform::Verbatim: return std::string(raw); + case Transform::LibFlag: return mcpp::toolchain::lib_flag_for(dial, raw); + case Transform::LibSearchPath: return std::string(dial.libSearchPrefix) + + abs_against(root, raw); + case Transform::DefinePrefix: return std::string(dial.definePrefix) + std::string(raw); + case Transform::AbsPath: return abs_against(root, raw); + } + return std::string(raw); +} + +} // namespace + +LineResult accept_line(Directives& d, const mcpp::toolchain::CommandDialect& dial, + const fs::path& root, std::string_view raw) { + std::string line = trim(raw); + constexpr std::string_view kPfx = "mcpp:"; + if (!line.starts_with(kPfx)) return LineResult::NotADirective; + std::string_view body = std::string_view(line).substr(kPfx.size()); + auto eq = body.find('='); + std::string key = std::string(body.substr(0, eq)); + std::string val = eq == std::string_view::npos ? std::string() + : std::string(body.substr(eq + 1)); + + if (key == "protocol") { + int n = 0; + auto* first = val.data(); + auto* last = val.data() + val.size(); + if (std::from_chars(first, last, n).ec == std::errc{}) d.protocol = n; + return LineResult::Protocol; + } + + const Def* def = find_by_wire(key); + if (!def) { + if (std::find(d.unknownKeys.begin(), d.unknownKeys.end(), key) + == d.unknownKeys.end()) + d.unknownKeys.push_back(key); + return LineResult::Unknown; + } + d.at(def->slot).push_back(transformed(*def, val, dial, root)); + return LineResult::Accepted; +} + +void accept_output(Directives& d, const mcpp::toolchain::CommandDialect& dial, + const fs::path& root, std::string_view out) { + std::size_t pos = 0; + while (pos <= out.size()) { + std::size_t nl = out.find('\n', pos); + std::string_view ln = out.substr( + pos, nl == std::string_view::npos ? std::string_view::npos : nl - pos); + accept_line(d, dial, root, ln); + if (nl == std::string_view::npos) break; + pos = nl + 1; + } +} + +std::optional protocol_error(const Directives& d) { + if (d.protocol > kProtocolVersion) { + return std::format( + "build.mcpp speaks directive protocol {}, but this mcpp understands " + "at most {}.\n" + " The package was written for a newer mcpp — upgrade with " + "`mcpp self update`.\n" + " (Continuing would silently drop directives this build " + "depends on.)", + d.protocol, kProtocolVersion); + } + if (d.protocol > 0 && !d.unknownKeys.empty()) { + std::string list; + for (auto const& k : d.unknownKeys) + list += (list.empty() ? "" : ", ") + ("mcpp:" + k); + return std::format( + "build.mcpp emitted directive(s) this mcpp does not know: {}.\n" + " The program announced protocol {}, which this mcpp also " + "speaks, so an unrecognized directive is a typo rather than newer " + "syntax.", + list, d.protocol); + } + return std::nullopt; +} + +void serialize(std::ostream& os, const Directives& d) { + // Table order, and one pass per row rather than per slot: rows sharing a + // slot (link-lib / link-search) share a tag, so emitting per row would + // duplicate them. + std::array done{}; + for (auto const& def : kTable) { + if (def.tag.empty()) continue; + auto idx = static_cast(def.slot); + if (done[idx]) continue; + done[idx] = true; + for (auto const& v : d.at(def.slot)) + os << "d " << def.tag << ' ' << v << '\n'; + } +} + +bool accept_cache_record(Directives& d, std::string_view tag, std::string_view value) { + const Def* def = find_by_tag(tag); + if (!def) return false; + d.at(def->slot).emplace_back(value); + return true; +} + +void apply(mcpp::manifest::Manifest& m, const Directives& d) { + auto& bc = m.buildConfig; + auto const& cxx = d.at(Slot::CxxFlags); + auto const& c = d.at(Slot::CFlags); + auto const& ld = d.at(Slot::LdFlags); + auto const& defines = d.at(Slot::Defines); + + bc.cxxflags.insert(bc.cxxflags.end(), cxx.begin(), cxx.end()); + bc.cflags.insert(bc.cflags.end(), c.begin(), c.end()); + bc.ldflags.insert(bc.ldflags.end(), ld.begin(), ld.end()); + // cfg defines colour BOTH language channels — the one slot that fans out. + bc.cflags.insert(bc.cflags.end(), defines.begin(), defines.end()); + bc.cxxflags.insert(bc.cxxflags.end(), defines.begin(), defines.end()); + + // Generated + selected sources join the source set. BOTH lists: the + // scanner walks the legacy modules.sources mirror, so pushing only + // bc.sources leaves a generated file outside the base globs invisible to + // the scan (latent since L3). + for (auto slot : {Slot::Generated, Slot::Sources}) { + for (auto const& s : d.at(slot)) { + bc.sources.push_back(s); + m.modules.sources.push_back(s); + } + } + + // Already absolute from the AbsPath transform. PRIVATE by design: for the + // root these join buildConfig before the package snapshot; for a + // dependency the caller mirrors them into privateBuild only, never into + // publicUsage. + for (auto const& p : d.at(Slot::IncludeDirs)) + bc.includeDirs.emplace_back(p); + for (auto const& p : d.at(Slot::IncludeDirsAfter)) + bc.includeDirsAfter.emplace_back(p); +} + +Mark mark(const mcpp::manifest::Manifest& m) { + return Mark{ m.buildConfig.cflags.size(), + m.buildConfig.cxxflags.size(), + m.buildConfig.includeDirs.size(), + m.buildConfig.includeDirsAfter.size() }; +} + +} // namespace mcpp::build::directives diff --git a/src/build/hostprogram.cppm b/src/build/hostprogram.cppm index fa53306d..b0c9a941 100644 --- a/src/build/hostprogram.cppm +++ b/src/build/hostprogram.cppm @@ -13,6 +13,7 @@ export module mcpp.build.hostprogram; import std; +import mcpp.build.directives; // kProtocolVersion — the announced value has ONE source import mcpp.platform; import mcpp.platform.process; import mcpp.toolchain.dialect; @@ -87,6 +88,26 @@ inline const char* dep_dir(const char* name) { return env_or(buf); } } +// ── Protocol announcement ─────────────────────────────────────────────── +// Emitted before main() runs, so a program that uses `import mcpp;` never has +// to remember to declare anything. The engine uses it two ways: it refuses a +// program that speaks a NEWER protocol than it understands, and — because the +// two sides then provably agree — it treats an unrecognized directive as an +// error rather than warning and silently dropping it. +// +// A hand-written `printf("mcpp:...")` program emits no announcement, which is +// exactly right: that surface is frozen at protocol 1 and keeps the historical +// warn-and-ignore behaviour. +// +// Namespace-scope `static` in the module purview: internal linkage, one object +// in mcpp.o, whose dynamic initializer runs from .init_array. mcpp.o is always +// on the link line, so it always fires. +namespace mcpp_detail { +struct ProtocolAnnouncer { + ProtocolAnnouncer() { std::printf("mcpp:protocol=%d\n", @PROTOCOL@); } +}; +static ProtocolAnnouncer mcpp_protocol_announcer; +} )CPP"; // Compile the bundled `mcpp` module into `bdir` and return the extra flags the @@ -139,6 +160,11 @@ build_mcpp_module(const fs::path& bdir, const fs::path& compiler, std::string moduleSrc(kMcppModuleSource); if (auto p = moduleSrc.find("@MODULE@"); p != std::string::npos) moduleSrc.replace(p, std::string_view("@MODULE@").size(), "export module"); + // Substituted rather than hardcoded so the announced version can never + // drift from the one the engine checks against. + if (auto p = moduleSrc.find("@PROTOCOL@"); p != std::string::npos) + moduleSrc.replace(p, std::string_view("@PROTOCOL@").size(), + std::to_string(mcpp::build::directives::kProtocolVersion)); { std::ofstream os(cppm, std::ios::trunc); os << moduleSrc; if (!os) return std::unexpected(std::string("could not write mcpp module source")); } diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index c6f416d6..874d6599 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -33,6 +33,7 @@ import mcpp.toolchain.triple; import mcpp.build.plan; import mcpp.build.cache_key; import mcpp.build.build_program; +import mcpp.build.directives; // directive table: mark / fold_private_tail import mcpp.lockfile; import mcpp.config; import mcpp.xlings; @@ -2601,42 +2602,26 @@ prepare_build(bool print_fingerprint, return changed; }; - // ONE owner of "which compile-visible channels a build.mcpp directive - // lands in" — shared by the dep loop and the root call site so the - // next directive kind cannot be threaded through one and silently - // missed in the other (the #242 two-derivations failure shape). - // apply() mutates the manifest the program ran against; this folds - // the NEW tail (recorded sizes → end) into the package's - // usage-resolved privateBuild, which is what its TUs actually read. - // Link/source/fingerprint residues stay at the call sites — they - // genuinely differ between root and dep (see each). - struct DirectiveMark { std::size_t c, cx, inc, incAfter; }; + // "Which compile-visible channels a build.mcpp directive lands in" is a + // property of the DIRECTIVE TABLE, not of this call site, so both the mark + // and the fold now live with the table in mcpp.build.directives. This pair + // used to be defined here and was already incomplete — the comment it + // replaced admitted that link/source residues stayed at the call sites, + // which is the #242 two-derivations shape. + // + // The fold is PRIVATE by design (Cargo discipline — a build-time program + // must not widen the package's public interface): privateBuild only, never + // publicUsage. The after-dirs ride the typed #249 channel, which owns the + // per-dialect degradations (cl.exe /I, NASM -I). + using DirectiveMark = mcpp::build::directives::Mark; auto markDirectiveTail = [](const mcpp::manifest::Manifest& mm) { - return DirectiveMark{ mm.buildConfig.cflags.size(), - mm.buildConfig.cxxflags.size(), - mm.buildConfig.includeDirs.size(), - mm.buildConfig.includeDirsAfter.size() }; + return mcpp::build::directives::mark(mm); }; auto foldDirectiveTailIntoPrivateBuild = - [&](auto& pkg, const mcpp::manifest::Manifest& ran, - const DirectiveMark& t) + [](auto& pkg, const mcpp::manifest::Manifest& ran, + const DirectiveMark& t) { - auto const& bc = ran.buildConfig; - pkg.privateBuild.cflags.insert(pkg.privateBuild.cflags.end(), - bc.cflags.begin() + t.c, bc.cflags.end()); - pkg.privateBuild.cxxflags.insert(pkg.privateBuild.cxxflags.end(), - bc.cxxflags.begin() + t.cx, bc.cxxflags.end()); - // include-dir[/-after] directives are PRIVATE (design §3.1: Cargo - // discipline — a build-time program must not widen the package's - // public interface): privateBuild only, never publicUsage. The - // after-dirs ride the typed #249 channel, which owns the - // per-dialect degradations (cl.exe /I, NASM -I). - for (auto it = bc.includeDirs.begin() + t.inc; - it != bc.includeDirs.end(); ++it) - appendUniquePath(pkg.privateBuild.includeDirs, *it); - for (auto it = bc.includeDirsAfter.begin() + t.incAfter; - it != bc.includeDirsAfter.end(); ++it) - appendUniquePath(pkg.privateBuild.includeDirsAfter, *it); + mcpp::build::directives::fold_private_tail(pkg.privateBuild, ran, t); }; @@ -4033,16 +4018,20 @@ prepare_build(bool print_fingerprint, // as the old pre-snapshot ordering implicitly did. pkg0.manifest.buildConfig.cflags.insert( pkg0.manifest.buildConfig.cflags.end(), - bcRoot.cflags.begin() + mark.c, bcRoot.cflags.end()); + bcRoot.cflags.begin() + static_cast(mark.cflags), + bcRoot.cflags.end()); pkg0.manifest.buildConfig.cxxflags.insert( pkg0.manifest.buildConfig.cxxflags.end(), - bcRoot.cxxflags.begin() + mark.cx, bcRoot.cxxflags.end()); + bcRoot.cxxflags.begin() + static_cast(mark.cxxflags), + bcRoot.cxxflags.end()); pkg0.manifest.buildConfig.includeDirs.insert( pkg0.manifest.buildConfig.includeDirs.end(), - bcRoot.includeDirs.begin() + mark.inc, bcRoot.includeDirs.end()); + bcRoot.includeDirs.begin() + static_cast(mark.includeDirs), + bcRoot.includeDirs.end()); pkg0.manifest.buildConfig.includeDirsAfter.insert( pkg0.manifest.buildConfig.includeDirsAfter.end(), - bcRoot.includeDirsAfter.begin() + mark.incAfter, + bcRoot.includeDirsAfter.begin() + + static_cast(mark.includeDirsAfter), bcRoot.includeDirsAfter.end()); // Link flags → the final link reads *m (already applied); keep the // linkUsage snapshot and fingerprint metadata equivalent too. diff --git a/src/platform/process.cppm b/src/platform/process.cppm index 01446e8e..7e220772 100644 --- a/src/platform/process.cppm +++ b/src/platform/process.cppm @@ -106,7 +106,8 @@ RunResult capture_exec_deadline( const std::vector& argv, const std::vector>& extraEnv, std::chrono::milliseconds deadline, - bool* timed_out); + bool* timed_out, + std::string_view cwd = {}); // Run `command` silently (discard stdout/stderr). // On POSIX, stdin is automatically redirected from /dev/null. @@ -598,10 +599,11 @@ RunResult capture_exec_deadline( const std::vector& argv, const std::vector>& extraEnv, std::chrono::milliseconds deadline, - bool* timed_out) + bool* timed_out, + std::string_view cwd) { if (timed_out) *timed_out = false; - if (deadline.count() <= 0) return capture_exec(argv, extraEnv); + if (deadline.count() <= 0) return capture_exec(argv, extraEnv, cwd); RunResult result; if (argv.empty()) { result.exit_code = 127; return result; } #if defined(__linux__) || defined(__APPLE__) @@ -618,6 +620,12 @@ RunResult capture_exec_deadline( posix_spawn_file_actions_t fa; ::posix_spawn_file_actions_init(&fa); + // Same cwd contract as capture_exec: a timed child must land in the same + // directory an untimed one would, or adding a timeout would silently + // change where a build program's relative writes go. + std::string cwdStore(cwd); + if (!cwdStore.empty()) + ::posix_spawn_file_actions_addchdir_np(&fa, cwdStore.c_str()); ::posix_spawn_file_actions_adddup2(&fa, fds[1], 1); ::posix_spawn_file_actions_adddup2(&fa, fds[1], 2); ::posix_spawn_file_actions_addclose(&fa, fds[0]); @@ -665,7 +673,7 @@ RunResult capture_exec_deadline( } } #else - return capture_exec(argv, extraEnv); + return capture_exec(argv, extraEnv, cwd); #endif } diff --git a/src/version.cppm b/src/version.cppm index 3e2f4738..51d39889 100644 --- a/src/version.cppm +++ b/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.8.4.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.8.5.1"; } // namespace mcpp diff --git a/src/xlings.cppm b/src/xlings.cppm index 15cbdfb2..94e9db79 100644 --- a/src/xlings.cppm +++ b/src/xlings.cppm @@ -44,7 +44,7 @@ namespace pinned { // in lock-step by hand; that list was already missing both composite // actions, which is how CI's sandbox sat on 0.4.30 unnoticed while // everything else had moved on. Don't reintroduce a hand-maintained list. - inline constexpr std::string_view kXlingsVersion = "2026.8.4.1"; + inline constexpr std::string_view kXlingsVersion = "2026.8.5.1"; inline constexpr std::string_view kNasmVersion = "3.02"; } diff --git a/tests/e2e/186_build_mcpp_protocol_and_bound.sh b/tests/e2e/186_build_mcpp_protocol_and_bound.sh new file mode 100755 index 00000000..c792afcf --- /dev/null +++ b/tests/e2e/186_build_mcpp_protocol_and_bound.sh @@ -0,0 +1,151 @@ +#!/usr/bin/env bash +# requires: gcc +# 186_build_mcpp_protocol_and_bound.sh — the build.mcpp CONTRACT hardening: +# wire-protocol version, the cache's semantic epoch, and the run bound. +# +# What each part protects against: +# * protocol — a build.mcpp written for a newer mcpp used to have its +# unknown directives WARNED about and dropped, producing a +# silently different build. It must now refuse. +# * legacy — a hand-written printf program announces nothing; its surface +# is frozen, so its unknown keys stay a warning (a typo), not +# an error. This asymmetry is the compatibility contract and +# has to be pinned by a test or it will be "simplified" away. +# * epoch — a cache entry written under a different directive +# interpretation must not be replayed under this one. +# * run bound — a build program that hangs used to hang the whole build with +# no diagnostic at all. +# +# See .agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md §4. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p app/src +cd app + +cat > mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" +EOF + +cat > src/main.cpp <<'EOF' +#ifndef FROM_BUILD_MCPP +#error "define missing" +#endif +int main() { return 0; } +EOF + +fresh() { rm -rf target; } + +# ── 1. import mcpp; announces a protocol, so an unknown directive is fatal ── +cat > build.mcpp <<'EOF' +#include +import mcpp; +int main() { + mcpp::cxxflag("-DFROM_BUILD_MCPP=1"); + std::printf("mcpp:no-such-directive=1\n"); +} +EOF +fresh +if "$MCPP" build > b1.log 2>&1; then + cat b1.log; echo "FAIL: unknown directive from an announcing program was accepted"; exit 1 +fi +grep -q "no-such-directive" b1.log || { + cat b1.log; echo "FAIL: error does not name the offending directive"; exit 1; } +# The message must explain WHY it is fatal here but not for a printf program, +# otherwise the asymmetry reads as a bug. +grep -qi "protocol" b1.log || { + cat b1.log; echo "FAIL: error does not mention the protocol"; exit 1; } + +# ── 2. A hand-written printf program keeps warn-and-ignore ────────────────── +cat > build.mcpp <<'EOF' +#include +int main() { + std::printf("mcpp:cxxflag=-DFROM_BUILD_MCPP=1\n"); + std::printf("mcpp:no-such-directive=1\n"); +} +EOF +fresh +"$MCPP" build > b2.log 2>&1 || { cat b2.log; echo "FAIL: legacy printf program was rejected"; exit 1; } +grep -q "ignoring unknown directive" b2.log || { + cat b2.log; echo "FAIL: legacy program lost its warn-and-ignore behaviour"; exit 1; } + +# ── 3. A program claiming a newer protocol is refused with an upgrade hint ── +cat > build.mcpp <<'EOF' +#include +int main() { + std::printf("mcpp:protocol=999\n"); + std::printf("mcpp:cxxflag=-DFROM_BUILD_MCPP=1\n"); +} +EOF +fresh +if "$MCPP" build > b3.log 2>&1; then + cat b3.log; echo "FAIL: a newer-protocol program was accepted"; exit 1 +fi +grep -q "999" b3.log || { cat b3.log; echo "FAIL: error does not name the claimed protocol"; exit 1; } +grep -qi "upgrade" b3.log || { cat b3.log; echo "FAIL: error gives no actionable next step"; exit 1; } + +# ── 4. Run bound: a hanging program is killed and the error says so ───────── +cat > build.mcpp <<'EOF' +#include +#include +int main() { + std::printf("mcpp:cxxflag=-DFROM_BUILD_MCPP=1\n"); + std::fflush(stdout); + for (;;) ::sleep(60); +} +EOF +fresh +start=$(date +%s) +if MCPP_BUILD_PROGRAM_TIMEOUT=3 "$MCPP" build > b4.log 2>&1; then + cat b4.log; echo "FAIL: a hanging build.mcpp did not fail the build"; exit 1 +fi +elapsed=$(( $(date +%s) - start )) +[ "$elapsed" -lt 60 ] || { echo "FAIL: the bound did not fire (took ${elapsed}s)"; exit 1; } +grep -q "time limit" b4.log || { cat b4.log; echo "FAIL: no timeout diagnostic"; exit 1; } +# It must name the package: in a workspace or a dependency graph, "build.mcpp +# hung" is useless without knowing whose. +grep -q "'app'" b4.log || { cat b4.log; echo "FAIL: timeout error does not name the package"; exit 1; } +grep -q "MCPP_BUILD_PROGRAM_TIMEOUT" b4.log || { + cat b4.log; echo "FAIL: timeout error does not say how to change the bound"; exit 1; } + +# ── 5. Cache: the entry carries an epoch, and a foreign one invalidates it ── +cat > build.mcpp <<'EOF' +#include +int main() { std::printf("mcpp:cxxflag=-DFROM_BUILD_MCPP=1\n"); } +EOF +fresh +"$MCPP" build > b5.log 2>&1 || { cat b5.log; echo "FAIL: build failed"; exit 1; } +CACHE=target/.build-mcpp/build.mcpp.cache +[ -f "$CACHE" ] || { echo "FAIL: no build.mcpp cache written"; exit 1; } +grep -q '^epoch ' "$CACHE" || { cat "$CACHE"; echo "FAIL: cache carries no epoch"; exit 1; } +cp "$CACHE" "$TMP/good.cache" + +# Touching a source defeats the whole-project fast path so prepare (and with it +# the build.mcpp cache) actually runs. +touch src/main.cpp +"$MCPP" build > b6.log 2>&1 || { cat b6.log; echo "FAIL: build failed"; exit 1; } +grep -q "up to date (cached)" b6.log || { + cat b6.log; echo "FAIL: an unchanged build.mcpp was re-run"; exit 1; } + +sed 's/^epoch .*/epoch 987654/' "$TMP/good.cache" > "$CACHE" +touch src/main.cpp +"$MCPP" build > b7.log 2>&1 || { cat b7.log; echo "FAIL: build failed"; exit 1; } +grep -q "build.mcpp running" b7.log || { + cat b7.log; echo "FAIL: a foreign cache epoch did not force a re-run"; exit 1; } + +# A `d` record this mcpp cannot interpret (a cache written by a NEWER mcpp) +# must invalidate the entry too — replaying the rest would apply a strict +# subset of what the program asked for. +cp "$TMP/good.cache" "$CACHE" +echo "d some-future-tag /x" >> "$CACHE" +touch src/main.cpp +"$MCPP" build > b8.log 2>&1 || { cat b8.log; echo "FAIL: build failed"; exit 1; } +grep -q "build.mcpp running" b8.log || { + cat b8.log; echo "FAIL: an unknown cache record did not force a re-run"; exit 1; } + +echo "OK" diff --git a/tests/unit/test_build_directives.cpp b/tests/unit/test_build_directives.cpp new file mode 100644 index 00000000..6bd26451 --- /dev/null +++ b/tests/unit/test_build_directives.cpp @@ -0,0 +1,338 @@ +#include + +import std; +import mcpp.build.directives; +import mcpp.manifest; +import mcpp.toolchain.dialect; + +// The directive table is the single definition of what a `build.mcpp` +// directive IS. Before it existed, a directive was defined in nine places and +// adding one meant editing all of them — the failure mode being that you edit +// eight, and the ninth surfaces much later as a flag that silently never +// reached the compiler. These tests hold the table's invariants so that a new +// row cannot be half-added. + +namespace dirs = mcpp::build::directives; + +namespace { + +const mcpp::toolchain::CommandDialect& gnu() { + return mcpp::toolchain::gnu_dialect(); +} + +dirs::Directives parse(std::string_view text, + const std::filesystem::path& root = "/pkg") { + dirs::Directives d; + dirs::accept_output(d, gnu(), root, text); + return d; +} + +} // namespace + +// ── Table integrity ──────────────────────────────────────────────────────── + +TEST(BuildDirectives, EveryRowIsInternallyConsistent) { + for (auto const& def : dirs::kTable) { + EXPECT_FALSE(def.wire.empty()); + EXPECT_GT(def.sinceProtocol, 0) << def.wire; + EXPECT_LE(def.sinceProtocol, dirs::kProtocolVersion) << def.wire; + // A RerunKey directive feeds only the re-run key, so it must NOT be + // persisted as a `d` record; everything else must be, or a cache hit + // would silently apply less than the program asked for. + if (def.scope == dirs::Scope::RerunKey) + EXPECT_TRUE(def.tag.empty()) << def.wire; + else + EXPECT_FALSE(def.tag.empty()) << def.wire; + // The declared-output contract needs a diagnostic that says WHICH + // contract was broken. + if (def.mustExistAfterRun) { + EXPECT_FALSE(def.missingPrefix.empty()) << def.wire; + EXPECT_FALSE(def.missingSuffix.empty()) << def.wire; + } + } +} + +TEST(BuildDirectives, WireNamesAreUnique) { + std::set seen; + for (auto const& def : dirs::kTable) + EXPECT_TRUE(seen.insert(def.wire).second) << "duplicate wire " << def.wire; +} + +TEST(BuildDirectives, RowsSharingATagShareASlot) { + // Cache deserialization resolves a tag to exactly one slot, so two rows + // with the same tag but different slots would round-trip to the wrong + // channel. link-lib and link-search legitimately share `ldflag`. + std::map slotOfTag; + for (auto const& def : dirs::kTable) { + if (def.tag.empty()) continue; + auto [it, fresh] = slotOfTag.try_emplace(def.tag, def.slot); + if (!fresh) EXPECT_EQ(it->second, def.slot) << def.tag; + } +} + +TEST(BuildDirectives, LookupsAgree) { + for (auto const& def : dirs::kTable) { + ASSERT_NE(dirs::find_by_wire(def.wire), nullptr) << def.wire; + EXPECT_EQ(dirs::find_by_wire(def.wire)->slot, def.slot) << def.wire; + if (!def.tag.empty()) { + ASSERT_NE(dirs::find_by_tag(def.tag), nullptr) << def.tag; + EXPECT_EQ(dirs::find_by_tag(def.tag)->slot, def.slot) << def.tag; + } + } + EXPECT_EQ(dirs::find_by_wire("no-such-directive"), nullptr); + // The empty tag belongs to the rerun rows and must never resolve, or a + // malformed cache line would be silently accepted into a slot. + EXPECT_EQ(dirs::find_by_tag(""), nullptr); +} + +// ── Parsing ──────────────────────────────────────────────────────────────── + +TEST(BuildDirectives, NonDirectiveLinesAreIgnored) { + auto d = parse("hello\nmcpp is not a directive\n \n"); + for (std::size_t i = 0; i < dirs::kSlotCount; ++i) + EXPECT_TRUE(d.slots[i].empty()); + EXPECT_TRUE(d.unknownKeys.empty()); + EXPECT_EQ(d.protocol, 0); +} + +TEST(BuildDirectives, TransformsAreAppliedOnceAtParseTime) { + auto d = parse("mcpp:cxxflag=-Wall\n" + "mcpp:link-lib=z\n" + "mcpp:link-search=vendor/lib\n" + "mcpp:cfg=HAVE_X\n" + "mcpp:include-dir=inc\n" + "mcpp:include-dir-after=/abs/inc\n"); + EXPECT_EQ(d.at(dirs::Slot::CxxFlags), (std::vector{"-Wall"})); + // link-lib and link-search share the ldflags slot, in emission order. + EXPECT_EQ(d.at(dirs::Slot::LdFlags), + (std::vector{"-lz", "-L/pkg/vendor/lib"})); + EXPECT_EQ(d.at(dirs::Slot::Defines), (std::vector{"-DHAVE_X"})); + // Relative resolves against the package root; absolute is taken as-is. + EXPECT_EQ(d.at(dirs::Slot::IncludeDirs), (std::vector{"/pkg/inc"})); + EXPECT_EQ(d.at(dirs::Slot::IncludeDirsAfter), + (std::vector{"/abs/inc"})); +} + +TEST(BuildDirectives, DialectDecidesTheSpelling) { + dirs::Directives d; + dirs::accept_output(d, mcpp::toolchain::msvc_dialect(), "/pkg", + "mcpp:link-lib=z\nmcpp:cfg=HAVE_X\n"); + EXPECT_EQ(d.at(dirs::Slot::LdFlags), (std::vector{"z.lib"})); + EXPECT_EQ(d.at(dirs::Slot::Defines), (std::vector{"/DHAVE_X"})); +} + +TEST(BuildDirectives, ValuesMayContainSpacesAndEqualsSigns) { + // Everything after the FIRST '=' is the value — a path with a space or a + // define with an '=' must survive intact. + auto d = parse("mcpp:cxxflag=-DMSG=\"a b\"\n"); + EXPECT_EQ(d.at(dirs::Slot::CxxFlags), + (std::vector{"-DMSG=\"a b\""})); +} + +// ── Protocol ─────────────────────────────────────────────────────────────── + +TEST(BuildDirectives, UnknownKeyWithoutAnAnnouncementIsTolerated) { + // A hand-written printf program announces nothing; its surface is frozen, + // so an unknown key is a typo the engine warns about rather than a + // forward-compat situation it must refuse. + auto d = parse("mcpp:no-such-thing=1\n"); + EXPECT_EQ(d.protocol, 0); + EXPECT_EQ(d.unknownKeys, (std::vector{"no-such-thing"})); + EXPECT_FALSE(dirs::protocol_error(d).has_value()); +} + +TEST(BuildDirectives, UnknownKeyWithAnAnnouncementIsFatal) { + auto d = parse("mcpp:protocol=1\nmcpp:no-such-thing=1\n"); + EXPECT_EQ(d.protocol, 1); + auto err = dirs::protocol_error(d); + ASSERT_TRUE(err.has_value()); + EXPECT_NE(err->find("no-such-thing"), std::string::npos); +} + +TEST(BuildDirectives, NewerProtocolIsFatalAndSaysWhatToDo) { + auto d = parse("mcpp:protocol=999\n"); + auto err = dirs::protocol_error(d); + ASSERT_TRUE(err.has_value()); + EXPECT_NE(err->find("999"), std::string::npos); + // The message has to tell the user the actionable thing, not just that + // something is wrong. + EXPECT_NE(err->find("upgrade"), std::string::npos); +} + +TEST(BuildDirectives, CurrentProtocolIsAccepted) { + auto d = parse(std::format("mcpp:protocol={}\nmcpp:cxxflag=-Wall\n", + dirs::kProtocolVersion)); + EXPECT_FALSE(dirs::protocol_error(d).has_value()); + EXPECT_EQ(d.at(dirs::Slot::CxxFlags), (std::vector{"-Wall"})); +} + +TEST(BuildDirectives, UnknownKeysAreDeduplicated) { + auto d = parse("mcpp:zzz=1\nmcpp:zzz=2\nmcpp:yyy=3\n"); + EXPECT_EQ(d.unknownKeys, (std::vector{"zzz", "yyy"})); +} + +// ── Cache round-trip ─────────────────────────────────────────────────────── + +TEST(BuildDirectives, SerializeDeserializeRoundTrip) { + auto d = parse("mcpp:cxxflag=-Wall\n" + "mcpp:cflag=-std=c11\n" + "mcpp:link-lib=z\n" + "mcpp:link-search=vendor/lib\n" + "mcpp:cfg=HAVE_X\n" + "mcpp:generated=src/gen.cpp\n" + "mcpp:source=vendor/pick.cpp\n" + "mcpp:include-dir=inc\n" + "mcpp:include-dir-after=after\n"); + + std::ostringstream os; + dirs::serialize(os, d); + + dirs::Directives back; + std::istringstream is(os.str()); + std::string line; + while (std::getline(is, line)) { + ASSERT_TRUE(line.starts_with("d ")); + auto rest = line.substr(2); + auto sp = rest.find(' '); + ASSERT_NE(sp, std::string::npos); + EXPECT_TRUE(dirs::accept_cache_record(back, rest.substr(0, sp), + rest.substr(sp + 1))); + } + + // Every persisted slot survives verbatim; the rerun slots deliberately do + // not (they are re-derived from their own cache records). + for (auto const& def : dirs::kTable) { + if (def.tag.empty()) continue; + EXPECT_EQ(back.at(def.slot), d.at(def.slot)) << def.wire; + } +} + +TEST(BuildDirectives, RerunSlotsAreNotPersistedAsDirectives) { + auto d = parse("mcpp:rerun-if-changed=config.h\n" + "mcpp:rerun-if-env-changed=USE_FAST\n"); + EXPECT_EQ(d.at(dirs::Slot::RerunFiles), (std::vector{"config.h"})); + EXPECT_EQ(d.at(dirs::Slot::RerunEnv), (std::vector{"USE_FAST"})); + std::ostringstream os; + dirs::serialize(os, d); + EXPECT_TRUE(os.str().empty()); +} + +TEST(BuildDirectives, UnknownCacheTagIsRejectedRatherThanDropped) { + // A cache written by a NEWER mcpp carries tags this one does not know. + // Silently skipping them would apply a strict subset of what the program + // asked for; the caller turns `false` into "entry is stale". + dirs::Directives d; + EXPECT_FALSE(dirs::accept_cache_record(d, "some-future-tag", "value")); + EXPECT_TRUE(dirs::accept_cache_record(d, "cxxflag", "-Wall")); +} + +// ── Apply ────────────────────────────────────────────────────────────────── + +TEST(BuildDirectives, ApplyRoutesEachSlotToItsManifestChannel) { + auto d = parse("mcpp:cxxflag=-Wall\n" + "mcpp:cflag=-std=c11\n" + "mcpp:link-lib=z\n" + "mcpp:cfg=HAVE_X\n" + "mcpp:generated=src/gen.cpp\n" + "mcpp:source=vendor/pick.cpp\n" + "mcpp:include-dir=inc\n" + "mcpp:include-dir-after=after\n"); + + mcpp::manifest::Manifest m; + dirs::apply(m, d); + auto const& bc = m.buildConfig; + + EXPECT_NE(std::find(bc.cxxflags.begin(), bc.cxxflags.end(), "-Wall"), + bc.cxxflags.end()); + EXPECT_NE(std::find(bc.cflags.begin(), bc.cflags.end(), "-std=c11"), + bc.cflags.end()); + EXPECT_NE(std::find(bc.ldflags.begin(), bc.ldflags.end(), "-lz"), + bc.ldflags.end()); + // A cfg define colours BOTH language channels — the one slot that fans out. + EXPECT_NE(std::find(bc.cflags.begin(), bc.cflags.end(), "-DHAVE_X"), + bc.cflags.end()); + EXPECT_NE(std::find(bc.cxxflags.begin(), bc.cxxflags.end(), "-DHAVE_X"), + bc.cxxflags.end()); + // generated= and source= must reach BOTH source lists: the scanner walks + // the legacy modules.sources mirror, and a file missing from it is + // invisible to the module scan. + for (auto const& s : {"src/gen.cpp", "vendor/pick.cpp"}) { + EXPECT_NE(std::find(bc.sources.begin(), bc.sources.end(), s), + bc.sources.end()) << s; + EXPECT_NE(std::find(m.modules.sources.begin(), m.modules.sources.end(), s), + m.modules.sources.end()) << s; + } + EXPECT_NE(std::find(bc.includeDirs.begin(), bc.includeDirs.end(), + std::filesystem::path("/pkg/inc")), + bc.includeDirs.end()); + EXPECT_NE(std::find(bc.includeDirsAfter.begin(), bc.includeDirsAfter.end(), + std::filesystem::path("/pkg/after")), + bc.includeDirsAfter.end()); +} + +TEST(BuildDirectives, ApplyAppendsRatherThanReplaces) { + mcpp::manifest::Manifest m; + m.buildConfig.cxxflags.push_back("-O2"); + dirs::apply(m, parse("mcpp:cxxflag=-Wall\n")); + EXPECT_EQ(m.buildConfig.cxxflags, + (std::vector{"-O2", "-Wall"})); +} + +// ── Private-scope fold ───────────────────────────────────────────────────── + +namespace { +// UsageRequirements-shaped, so the fold can be exercised without importing +// the scanner into a unit test. +struct FakeUsage { + std::vector includeDirs; + std::vector includeDirsAfter; + std::vector cflags; + std::vector cxxflags; +}; +} // namespace + +TEST(BuildDirectives, FoldMovesOnlyTheTailAndOnlyPrivateChannels) { + mcpp::manifest::Manifest m; + m.buildConfig.cxxflags.push_back("-O2"); // pre-existing, not a directive + m.buildConfig.includeDirs.emplace_back("/pre"); + + auto before = dirs::mark(m); + dirs::apply(m, parse("mcpp:cxxflag=-Wall\n" + "mcpp:cfg=HAVE_X\n" + "mcpp:link-lib=z\n" + "mcpp:include-dir=inc\n")); + + FakeUsage priv; + dirs::fold_private_tail(priv, m, before); + + // Only what the program added, and only the private channels. + EXPECT_EQ(priv.cxxflags, (std::vector{"-Wall", "-DHAVE_X"})); + EXPECT_EQ(priv.cflags, (std::vector{"-DHAVE_X"})); + EXPECT_EQ(priv.includeDirs, + (std::vector{"/pkg/inc"})); + // Link flags are NOT private — they reach the final link through their own + // path, and folding them here would double-apply them. + EXPECT_TRUE(priv.includeDirsAfter.empty()); +} + +TEST(BuildDirectives, FoldIsIdempotentOnIncludeDirs) { + // Include dirs are unique-appended: the same dir emitted twice, or a fold + // replayed, must not grow the list. + mcpp::manifest::Manifest m; + auto before = dirs::mark(m); + dirs::apply(m, parse("mcpp:include-dir=inc\nmcpp:include-dir=inc\n")); + FakeUsage priv; + dirs::fold_private_tail(priv, m, before); + dirs::fold_private_tail(priv, m, before); + EXPECT_EQ(priv.includeDirs, + (std::vector{"/pkg/inc"})); +} + +// ── Run bound ────────────────────────────────────────────────────────────── + +TEST(BuildDirectives, RunTimeoutDefaultsToABoundAndIsOverridable) { + // Default: bounded. An unbounded build program is how a build hangs with + // no diagnostic at all. + EXPECT_GT(dirs::run_timeout().count(), 0); + EXPECT_EQ(dirs::run_timeout().count(), dirs::kDefaultRunTimeoutSecs * 1000); +} From 3cafdb5b5c37fa2d2bec8ba28ef066c5664ba4f8 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 13:46:46 +0800 Subject: [PATCH 02/16] fix(ci): bootstrap pin named a version the index no longer serves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ci-aarch64-fresh-install` has been red on main since 2026-08-03 with [error] xlings: version '2026.8.3.2' not found for 'mcpp' [error] available: 2026.8.4.1 `.xlings.json` 的自举 pin 是**自举起点**,故意滞后、不随发布走(docs/09 §4), 所以平时不该动它 —— 但它的一条硬约束是**必须命名一个可安装的版本**。索引已不再 提供 2026.8.3.2,于是那条约束被打破了,这正是必须 bump 的场合(而不是「发版顺手 bump」那种误用)。 改到 2026.8.4.1:错误信息本身证明它可安装,且 ≤ 正在构建的 2026.8.5.1, check_version_pins.sh 的方向约束满足。 与本 PR 的其余改动无关 —— 是先前就红的,顺手修掉。 --- .xlings.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.xlings.json b/.xlings.json index 5adb177c..4cd7c236 100644 --- a/.xlings.json +++ b/.xlings.json @@ -1,5 +1,5 @@ { "workspace": { - "mcpp": "2026.8.3.2" + "mcpp": "2026.8.4.1" } } From ca1d40eaad3fa0fe2800bed00f07ada877a77120 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:16:28 +0800 Subject: [PATCH 03/16] =?UTF-8?q?feat:=20=E4=BE=9D=E8=B5=96=E4=BA=A7?= =?UTF-8?q?=E5=87=BA=E7=9A=84=20host=20=E5=B7=A5=E5=85=B7=E3=80=81?= =?UTF-8?q?=E6=9E=84=E5=BB=BA=E5=9B=BE=E8=8A=82=E7=82=B9=E3=80=81=E8=A7=84?= =?UTF-8?q?=E5=88=99=E5=8C=85(=E6=9E=B6=E6=9E=84=E6=96=87=E6=A1=A3?= =?UTF-8?q?=E6=AD=A5=202=E2=80=936)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 实现 `.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md` 的步 2–6, 接在同一文档步 0+1 之后。核心是把 build.mcpp 里「**配置**」与「**施工**」这两件被 混在一起的事分开:程序继续回答「这次构建长什么样」,而「把这批输入变成那批输出」 交给构建图。 ## 步 2 — 依赖产出的 host 工具(#355) 一个包能构建出消费者构建期需要的二进制(protoc / grpc_cpp_plugin / flatc / 转译器), 但消费者拿不到:`dep_dir()` 给的是源码树,依赖的 `kind="bin"` target 从不被构建。 **为什么不能是主图里的节点**:时序。build.mcpp 跑在 prepare 内,BuildPlan 还不存在、 build.ninja 更在其后 —— 主图产出的东西对需要它的程序永远来得太晚;叠上交叉编译连 架构都不对。所以走**嵌套的、面向 host 的子构建** + 全局 store(Cargo build-dependencies / Bazel exec configuration / vcpkg host:true / Conan tool_requires 的形状)。 **为什么便宜**:工具是可执行文件,与主构建零 ABI 接触 ⇒ 子构建可用工具包自己的 toolchain / profile / 依赖解析,不必与消费者一致。 单一版本轴(工具版本=依赖版本,protoc 与运行时错配结构上不可表达)、默认关闭、 成本门复用已有 features+required_features、全局 store 按 版本×host 工具链×feature× 依赖闭包 缓存。逃生舱 `[tools.overrides]` / `MCPP_TOOL_*` 直接指现成二进制并**跳过 构建** —— 每个同类系统都有这一条,且**刻意不进 cache key**。 前提是**工作目录外置**:一次 build 往工程根写 5 处,而「注册表包根共享/可能只读/ 绝不写入」是 build_program.cppm 自 G2 起的明文不变量。五处一起搬 —— 只搬一部分比 一处不搬更糟。 ## 步 3+4 — `mcpp:action=` 构建图节点 一个原语三种接线(role 只决定输出接到哪):source 进编译集、check 产 stamp 且**默认 与编译并行**、artifact 的输入是链接产物。顺序全由 ninja 的文件依赖决定,不需要 phase 机制 —— 这也是 artifact 不会像朴素 post 钩子那样把自己重复施加一遍的原因。 **必须写出输出文件名**(INV-D):prepare 期就定死源码集/fingerprint/模块图。畸形 action 是硬错误。命令是 argv 而非 shell 字符串,插值只有封闭的四个。 ## 步 5 — `host-module = true` 规则包 规则以普通 mcpp 库包分发,消费者 `import mcpp.rules.x;`。有版本、能测试、能发布, 用 **C++** 写 —— 不引入第二门语言(xmake Lua rule / Bazel Starlark)。 **关键**:规则模块与 build.mcpp **同一条命令、同一套 flag** 编译。不是优化 —— BMI 只 对与它在 standard/dialect/编译器身份上一致的编译可用,两次独立解析的构建没有理由 一致,而不一致表现为 `module X CRC mismatch` 而非清楚的错误。 ## 步 6 — 生成的 .cppm 核实结论:真正的阻塞点不是 topoOrder 的**顺序**(那只是名字普查 + 发射次序),而是 未扫描的文件**根本没有 graph.units 条目**。解法用代码库已有的答案 ——「声明而非发现」: action 的 `.provides()/.imports()` 让 mcpp 播下带该声明的占位文件,prepare 期扫描与 生成器将产出的内容一致,build 期由编译器自己的 P1689 复核。 ## 顺带修 `.xlings.json` 自举 pin 指向索引已不再提供的 2026.8.3.2 —— `ci-aarch64-fresh-install` 自 2026-08-03 在 main 上就是红的。自举 pin 平时不该动,但它有一条硬约束是必须命名 可安装的版本,这条被打破时正是该 bump 的场合。 ## 验证 - 单测 56/56 - e2e 19/21 通过;失败的 `07_static_library`(本机 binutils payload 的 ar 跑不起来)与 `09_path_dependency`(ninja missing dep BMI)在**已发布的 2026.8.4.1 上同样失败** ⇒ 环境性,非回归 - 新增 3 个 e2e:187(host 工具:端到端 / 成本门 / 默认关闭 / 错名报可用列表 / override)、 188(三种 role + 增量 + 失败的 check 让构建失败 + 畸形 action 被拒)、 189(规则包:导入生效 / 编辑规则触发重跑 / 缺 lib root 的诊断) --- CHANGELOG.md | 32 +++ docs/05-mcpp-toml.md | 86 +++++- docs/07-build-mcpp.md | 89 ++++++ src/build/build_program.cppm | 71 ++++- src/build/compile_commands.cppm | 4 +- src/build/directives.cppm | 125 +++++++- src/build/hostprogram.cppm | 185 ++++++++++++ src/build/ninja_backend.cppm | 43 ++- src/build/plan.cppm | 9 + src/build/prepare.cppm | 446 ++++++++++++++++++++++++++++- src/build/tool_store.cppm | 236 +++++++++++++++ src/manifest/toml.cppm | 27 +- src/manifest/types.cppm | 61 ++++ src/manifest/xpkg.cppm | 67 ++++- src/pm/dep_spec.cppm | 28 ++ tests/e2e/187_dep_host_tool.sh | 161 +++++++++++ tests/e2e/188_build_actions.sh | 151 ++++++++++ tests/e2e/189_host_module_rules.sh | 108 +++++++ 18 files changed, 1913 insertions(+), 16 deletions(-) create mode 100644 src/build/tool_store.cppm create mode 100755 tests/e2e/187_dep_host_tool.sh create mode 100755 tests/e2e/188_build_actions.sh create mode 100755 tests/e2e/189_host_module_rules.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 99151ab5..20e4cd72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,8 +30,40 @@ **编译**这一步刻意不设上限——与 `mcpp test` 同一条不对称纪律(run 有限 / build 无限):编译跑得久通常是正当的(首次构建 `std` 模块就是分钟级),杀掉它只会产生莫名其妙的失败;构建**程序**跑得久通常是卡住了。`capture_exec_deadline` 顺带补上了 `cwd` 形参——没有它,加超时会**静默改变**构建程序相对写入的落点。 +### 新增 + +- **依赖产出的 host 工具:`tools = ["protoc"]`(#355)。** 一个包能构建出消费者在**构建期**需要的二进制(protoc、grpc_cpp_plugin、flatc、moc、转译器),但消费者此前完全拿不到它 —— `mcpp::dep_dir()` 给的是**源码树**,而依赖的 `kind = "bin"` target **从不被构建**(`plan.cppm` 只遍历 root 的 targets;唯一的例外 `kind = "shared"` 是按 `--target` 构建的,当 host 工具用不了)。 + + **为什么它不能是主图里的一个节点**:时序。`build.mcpp` 跑在 prepare 内,那时 BuildPlan 还不存在、build.ninja 更在其后 —— 主图产出的东西对需要它的程序来说**永远来得太晚**。再叠上交叉编译,它连架构都不对。所以工具由**嵌套的、面向 host 的子构建**产出,落进全局 store。这正是 Cargo `[build-dependencies]` / Bazel exec configuration / vcpkg `"host": true` / Conan `tool_requires` 的形状。 + + **为什么它便宜**:工具是**可执行文件**,与主构建零 ABI 接触。子构建因此可以用工具包自己的 `[toolchain]`、自己的 profile、自己的依赖解析,不必与消费者一致 —— 对照 `kind = "lib"` 依赖,这几条**必须**一致。 + + **单一版本轴**:工具的版本就是依赖的版本,所以「protoc 与 protobuf 运行时错配」这类**运行期**才炸的问题结构上不可表达。默认关闭(成本由消费者付),成本门复用已有的 `[features]` + `required_features`。全局 store 按 包版本 × host 工具链 × feature × 自身依赖闭包 缓存。 + + **逃生舱** `[tools.overrides]` / `MCPP_TOOL__`:直接指一个现成二进制,**完全跳过构建**。每个同类系统都提供这一条(vcpkg `VCPKG_HOST_TRIPLET`、CMake `LLVM_NATIVE_TOOL_DIR`、Qt `QT_HOST_PATH`),理由一样 —— 源码在本机构建不出来的工具不能是死路。它**刻意不进 cache key**:逃生舱不是可复现输入。 + +- **`mcpp:action=`:声明构建图节点,而不是在 build.mcpp 里干活。** 在程序里直接写生成逻辑,是每次 prepare 跑一遍、全量、串行,失败报「build.mcpp exited 1」。**声明**成节点后它是图里的一条边 —— 增量、并行、失败可归因到具体那条边。 + + **一个原语,三种接线**(`role` 只决定输出接到哪,不是三套机制):`source` 进编译集(protoc、转译器)、`check` 产出 stamp 且**默认与编译并行**(clang-tidy、格式/ABI 检查;`blocking` 可改成前置)、`artifact` 的**输入**是链接产物(签名、打包、size budget)。顺序完全由 ninja 的文件依赖决定,不需要任何 phase 机制 —— 这也是为什么 `artifact` 不会像朴素的「post 钩子」那样把自己重复施加一遍。 + + **必须写出输出文件名**:mcpp 在 prepare 期就定死源码集、fingerprint 与模块图,名字未知的产物无法进图。内容可以晚到,名字不行。畸形 action 是**硬错误**而非静默跳过。生成**模块接口**时用 `.provides()/.imports()` 声明,mcpp 会按该声明播下占位文件让 prepare 期的扫描与生成器将要产出的内容一致 —— 与 `[modules].scan_overrides` 同一条「声明+验证」的取舍,build 期由编译器自己的 P1689 复核。 + + 命令是 **argv 而非 shell 字符串**(不假设存在 shell),插值只有封闭的四个:`${mcpp.out_dir}` / `${mcpp.bin_dir}` / `${mcpp.compile_db}` / `${mcpp.target_file:}`。 + +- **`host-module = true`:可复用的构建规则以普通包分发。** 「跑 protoc」这类规则应该写一次,而不是每个消费者的 build.mcpp 复制一遍。把它做成普通 mcpp 库包,消费者 `import mcpp.rules.protobuf;` 即可。规则因此**有版本、能测试、能发布**,走的是已有的包管理机制,而且是用 **C++** 写的 —— 不引入第二门语言(xmake 用 Lua rule、Bazel 用 Starlark),这正是 build.mcpp 存在的理由。 + + 实现上的关键:规则模块与 build.mcpp **在同一条命令里、用同一套 flag** 编译。这不是优化 —— BMI 只对「在 standard / dialect / 编译器身份上与它一致」的编译可用,分成两次独立解析的构建则毫无理由一致,而不一致的表现是 `module X CRC mismatch` 而不是一条清楚的错误。 + +- **工作目录可外置(`BuildOverrides::work_dir`)。** 一次 `mcpp build` 会往工程根写 5 处(`target/`、`mcpp.lock`、`compile_commands.json`、`.mcpp/`、`target/.build-mcpp`),而「注册表包根是共享的、可能只读、绝不写入」是 `build_program.cppm` 自 G2 起的明文不变量 —— 在此之前没有任何东西能对 build.mcpp 的临时目录之外兑现它。把「源码在哪」与「往哪写」拆开,是 host 工具子构建**能够存在**的前提。五处一起搬:只搬一部分比一处都不搬更糟,那等于照样写进共享目录、只是更不显眼。 + +### 改进 + - **内带 xlings 升级到 `2026.8.5.1`**(自 `2026.8.4.1`)。13 个 pin 点由 `check_version_pins.sh` 机器校验并全部更新。 +### 修复 + +- **自举 pin 指向了索引已不再提供的版本。** `ci-aarch64-fresh-install` 自 2026-08-03 起在 main 上就是红的:`version '2026.8.3.2' not found for 'mcpp'`。`.xlings.json` 的自举 pin 是**自举起点**、故意滞后、不随发布走(docs/09 §4),平时不该动 —— 但它有一条硬约束是**必须命名一个可安装的版本**,这条被打破时正是该 bump 的场合(而不是「发版顺手 bump」那种误用)。改到 `2026.8.4.1`。 + ## [2026.8.4.1] — 2026-08-04 ### 修复 diff --git a/docs/05-mcpp-toml.md b/docs/05-mcpp-toml.md index 542f630b..69db02fa 100644 --- a/docs/05-mcpp-toml.md +++ b/docs/05-mcpp-toml.md @@ -123,7 +123,7 @@ required_features = ["gui"] # only built when feature `gui` is |---|---| | `defines` | Preprocessor macros (`name` or `name=value`); desugar to `-D` on both the C and C++ entry compile. | | `cxxflags` / `cflags` | Extra compile flags for this target. Do **not** put `-std=...` here — use `[package].standard`. | -| `required_features` | The target is emitted only when **every** listed feature is active in the build; otherwise it is silently skipped. A gate only — it does not activate features (use `--features` / `[features].default`). | +| `required_features` | The target is emitted only when **every** listed feature is active in the build; otherwise it is silently skipped. A gate only — it does not activate features (use `--features` / `[features].default`). **One exception, and it is not a second rule:** when this target is requested as a host tool (`tools = [...]`, §2.14), the target is what was *asked for*, so its `required_features` become the sub-build's *inputs*. Same field, one meaning — the resolution just runs in the opposite direction. | > **Scope (important):** `defines` / `cxxflags` / `cflags` on a target apply **only to that > target's exclusive entry source** (its `main`) — never to shared module/impl objects, which @@ -964,6 +964,90 @@ build needs (`make`/`cmake`/`protoc`/…), pin tool versions per project, or set build-time env vars — without hand-editing `.xlings.json`. `[toolchain]` (§2.7) remains the ergonomic shorthand for the compiler; `[xlings.workspace]` is the general form. +### 2.14 Host tools from a dependency (mcpp 2026.8.5.1+) + +A package can build a binary its consumers need *at build time* — `protoc`, a +`grpc_cpp_plugin`, `flatc`, `moc`, a transpiler. Ask for it on the dependency: + +```toml +[dependencies] +protobuf = { version = "35.1", tools = ["protoc"] } +grpc = { version = "1.83.0", tools = ["grpc_cpp_plugin"] } +``` + +Each name must be a `kind = "bin"` target of that package. mcpp builds it **for +the build machine** and hands `build.mcpp` its absolute path as +`MCPP_DEP__BIN_` — read it with `mcpp::dep_bin("protobuf", "protoc")` +(see [07 — build.mcpp](07-build-mcpp.md)). + +Four properties worth knowing: + +- **Always a host binary.** Under `mcpp build --target ` the tool is + still built for *this* machine, because a code generator has to run here. It + is a separate, host-targeted sub-build — the tool package's own `[toolchain]` + and its own dependency resolution apply, and none of it has to agree with + your build. That is safe precisely because an executable has no ABI contact + with your code. +- **One version axis.** The tool's version *is* the dependency's version, so + a `protoc` that does not match its runtime is not expressible. (This is the + problem with packaging the tool separately, and it is the failure mode that + bites at run time rather than compile time.) +- **Default off.** Nothing is built unless someone asks; the cost is the + consumer's to pay. A package gates the expensive part with + `[features]` + `required_features` (protobuf's `protoc` needs libprotoc's + ~157 extra TUs, which the runtime's users must not compile). +- **Cached globally**, keyed on package version × host toolchain × features × + its own dependency closure — built once per machine, not once per project. + +#### `[tools.overrides]` — use a binary you already have + +```toml +[tools.overrides] +"compat.protobuf:protoc" = "/usr/bin/protoc" +``` + +or, without editing the manifest (CI, distro packaging): + +```bash +MCPP_TOOL_PROTOBUF_PROTOC=/usr/bin/protoc mcpp build +``` + +An override **skips the build entirely**. Every comparable system provides this +escape hatch (vcpkg's `VCPKG_HOST_TRIPLET`, CMake's `LLVM_NATIVE_TOOL_DIR`, +Qt's `QT_HOST_PATH`), and for the same reason: a tool that cannot be built from +source on this machine must not be a dead end. It is deliberately **not** part +of the cache key — an override is an escape hatch, not a reproducible input. + +#### `host-module = true` — reusable build rules as packages + +A rule (say "run protoc over these `.proto` files") should be written once, not +copy-pasted into every consumer's `build.mcpp`. Ship it as an ordinary mcpp +library package and import it: + +```toml +[dependencies] +"mcpp.rules.protobuf" = { version = "0.1.0", host-module = true } +``` + +```cpp +// build.mcpp +import mcpp; +import mcpp.rules.protobuf; +int main() { mcpp::rules::protobuf::generate(/* … */); } +``` + +mcpp compiles that package's lib-root module **for the host, in the same +command as `build.mcpp`** — which is what makes the BMI usable at all, since a +module interface is only importable by a compile that agrees with it on +standard, dialect and compiler identity. + +Rules are therefore versioned, testable and distributable through the package +manager you already have, written in **C++** — no second language, which is the +whole point of `build.mcpp` existing. + +*Limit:* the rule interface is compiled alone, so it may import `std` and the +bundled `mcpp` module, but not a third package. A rule package is a leaf. + ## Appendix A. Schema Ownership Principle (admission criteria for new fields) > **Closed syntax, open vocabulary**: whoever owns the parsing semantics defines the keys; whoever owns the domain knowledge defines the values. diff --git a/docs/07-build-mcpp.md b/docs/07-build-mcpp.md index 57369b1c..48dd8186 100644 --- a/docs/07-build-mcpp.md +++ b/docs/07-build-mcpp.md @@ -97,6 +97,95 @@ int main() { | `mcpp::source(p)` | `mcpp:source=` | | `mcpp::include_dir(d)` / `mcpp::include_dir_after(d)` | `mcpp:include-dir=` / `mcpp:include-dir-after=` | | `mcpp::rerun_if_changed(p)` / `mcpp::rerun_if_env_changed(v)` | the matching `rerun-*` directives | +| `mcpp::dep_bin(pkg, tool)` *(2026.8.5.1+)* | reads `MCPP_DEP__BIN_` — the absolute path of a **host tool** built by a dependency (see below) | +| `mcpp::action{…}.submit()` *(2026.8.5.1+)* | `mcpp:action=` — declares a **build-graph node** instead of doing the work here (see below) | + +### Host tools from a dependency (2026.8.5.1+) + +Declare the need in `mcpp.toml`, then call it: + +```toml +[dependencies] +protobuf = { version = "35.1", tools = ["protoc"] } +``` + +```cpp +// build.mcpp +import mcpp; +int main() { + const char* protoc = mcpp::dep_bin("protobuf", "protoc"); + // … invoke it, then declare what it produced … +} +``` + +mcpp builds that `kind = "bin"` target **for the build machine** (even under +`--target`), caches it globally, and hands you the path. The request lives in +`mcpp.toml` rather than here for the same reason a dependency does: asking the +graph for an extra artifact is a graph-level request, and the graph stays +statically analysable. See [05 §2.14](05-mcpp-toml.md) for the full contract, +including `[tools.overrides]`. + +### Declaring work instead of doing it: `mcpp::action` (2026.8.5.1+) + +Generating a source by writing it *here* is the easy path and the wrong one +past a certain size: it happens once per prepare, for the whole set, serially, +and a failure is reported as "build.mcpp exited 1". **Declare** the work and it +becomes an edge in the build graph — incremental, parallel, and attributable to +the edge that failed. + +```cpp +import mcpp; +int main() { + const std::string out = std::string(mcpp::out_dir()) + "/foo.pb.cc"; + mcpp::action a; + a.id = "protoc:foo"; + a.role = "source"; // "source" | "check" | "artifact" + a.arg(mcpp::dep_bin("protobuf", "protoc")) + .arg("--cpp_out=...").arg("proto/foo.proto") + .input("proto/foo.proto") + .output(out.c_str()) + .submit(); +} +``` + +Three roles, one primitive — `role` only decides where the edge's outputs +attach: + +| `role` | Outputs | Ordering | Typical | +|---|---|---|---| +| `source` | join the compile set | the compile edge consumes them | protoc, a transpiler | +| `check` | a stamp file | runs **alongside** compilation (set `blocking = true` to gate it) | clang-tidy, a format or ABI check | +| `artifact` | a new file | its *inputs* are link outputs, so it runs after the link | codesign, packaging, size budgets | + +No phase machinery is involved: ninja's own file dependencies do the +sequencing, which is also why an `artifact` action cannot double-apply itself +the way a naive "post-build hook" would. + +**You must name the output files.** mcpp fixes the source set, the fingerprint +and the module graph during prepare, so an output whose *name* is unknown +cannot be built. Content may arrive later; names may not. A malformed action is +a hard error, never a silent skip. + +For a generated **module interface**, declare its interface too: + +```cpp +a.output(gen.c_str()).provides("my.generated").imports("std").submit(); +``` + +mcpp seeds a placeholder carrying exactly that declaration so the prepare-time +scan agrees with what your generator will emit — the same assertion-plus- +verification trade `[modules].scan_overrides` makes, and the compiler's own +P1689 output checks it at build time. + +Commands are an **argv, not a shell string** (no shell is assumed — Windows has +none to rely on), and the only interpolations are a closed set: + +| Variable | Value | +|---|---| +| `${mcpp.out_dir}` | the build output directory | +| `${mcpp.bin_dir}` | where produced binaries land | +| `${mcpp.compile_db}` | path to `compile_commands.json` (what clang-tidy's `-p` wants) | +| `${mcpp.target_file:}` | the built file of target `` | The raw stdout protocol above remains the low-level substrate; `import mcpp;` is the typed layer over it. diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index e8406b8f..4944c0dc 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -54,6 +54,22 @@ struct BuildProgramEnv { // MCPP_DEP__DIR (same sanitizer as MCPP_FEATURE_) instead of // reverse-engineering the store layout. std::vector> depDirs; + // #355: HOST tools this package asked its dependencies for, as + // (env var name → absolute path to the executable) pairs. The caller has + // already resolved them (built, taken from the store, or an override), so + // this is purely the delivery channel. Rides the same contract env, hence + // the same re-run key: a rebuilt tool re-runs the program that uses it, + // with no `rerun-if-changed` needed from the author. + std::vector> toolPaths; + // #355 step 5: dependency-provided modules to compile FOR THE HOST and make + // importable from this build.mcpp — reusable build rules distributed as + // ordinary mcpp packages (`import mcpp.rules.protobuf;`) instead of a + // second, non-C++ rule DSL. + // + // (logical module name, absolute path to its interface unit). Compiled with + // the SAME flags as build.mcpp itself, in the same directory, which is what + // makes the BMI usable at all — see DependencySpec::hostModule. + std::vector> hostModules; }; // Compile + run `/build.mcpp` (if present) with `hostCompiler` (the resolved @@ -222,6 +238,22 @@ contract_env(const fs::path& root, const fs::path& outDir, const BuildProgramEnv it->second, dir.string())); } } + // #355: MCPP_DEP__BIN_ — absolute path to a host tool the + // consumer declared via `tools = [...]`. A PATH rather than a directory: + // the store keys an entry per (package, target), the typed reader can + // append the platform's exe suffix itself, and a tool's adjacent DATA + // (protoc's well-known .proto files, say) lives in the package tree, which + // dep_dir() already exposes. + for (auto const& [var, path] : env.toolPaths) { + auto [it, inserted] = depVarValue.try_emplace(var, path); + if (inserted) { + e.emplace_back(var, path); + } else if (it->second != path) { + mcpp::ui::warning(std::format( + "build.mcpp: tool name collides on {} (kept '{}', ignored '{}')", + var, it->second, path)); + } + } return e; } @@ -374,6 +406,15 @@ std::expected run_build_program( // Fold the policy into the compiler identity: a helper produced under an // older link policy must be rebuilt, not reused from the cache. std::string compilerIdentity = hostCompiler.string(); + // Host modules change what the helper links, so they belong in the identity + // the cache keys on — otherwise adding or removing a rule package would + // replay a cached run compiled without it. + for (auto const& [logical, ifacePath] : env.hostModules) { + compilerIdentity += "\nhost-module="; + compilerIdentity += logical; + compilerIdentity += "@"; + compilerIdentity += mcpp::toolchain::hash_file(ifacePath); + } compilerIdentity += "\nbuild-program-link="; compilerIdentity += muslStaticHelper ? "musl-static-v1" : mingwStaticHelper ? "mingw-static-v1" @@ -458,6 +499,27 @@ std::expected run_build_program( mcppModuleObject = std::move(mf->object); } + // #355 step 5: dependency-provided host modules (reusable build rules + // shipped as ordinary packages). Compiled HERE, with `base` and `std_flag` + // — the same flags the build.mcpp compile below gets — because a BMI is + // only usable by a compile that agrees with it. Doing this in a separate + // sub-build would leave that agreement to chance, and disagreement shows + // up as `module X CRC mismatch`, not as a clear error. + std::vector hostModuleObjects; + for (auto const& [logical, ifacePath] : env.hostModules) { + auto hm = build_host_module(bdir, hostCompiler, base, std_flag, tc, + compileEnv, logical, ifacePath, moduleFlags); + if (!hm) return std::unexpected(hm.error()); + for (auto& f : hm->useFlags) { + // GCC's marker is just `-fmodules`, already present when the + // bundled module was built; repeating it is harmless but noisy. + if (std::find(moduleFlags.begin(), moduleFlags.end(), f) + == moduleFlags.end()) + moduleFlags.push_back(f); + } + hostModuleObjects.push_back(std::move(hm->object)); + } + // ── `import std;` in build.mcpp ───────────────────────────────────────── // // mcpp asks projects to `import std;` everywhere and then made their build @@ -571,7 +633,7 @@ std::expected run_build_program( for (auto f : dial.forceCxxLangArgv) compileArgv.emplace_back(f); compileArgv.push_back(src.string()); } - if (usesModule || !stdObjects.empty()) { + if (usesModule || !stdObjects.empty() || !hostModuleObjects.empty()) { // Link the module objects. GNU drivers need the input language reset // first, or the .o that follows `-x c++` is handed to the frontend as // C++ source; cl.exe has no `-x` at all and infers from the extension. @@ -580,6 +642,7 @@ std::expected run_build_program( // answered with `D9002: ignoring unknown option '-x'`. if (!msvcHost) { compileArgv.push_back("-x"); compileArgv.push_back("none"); } if (usesModule) compileArgv.push_back(mcppModuleObject.string()); + for (auto& hmo : hostModuleObjects) compileArgv.push_back(hmo.string()); for (auto& so : stdObjects) compileArgv.push_back(so); } // Self-contained helper link — see the staticHostHelper doctrine above. @@ -650,6 +713,12 @@ std::expected run_build_program( if (auto perr = dirs::protocol_error(d)) { return std::unexpected(*perr); } + // Refuse a malformed action BEFORE applying anything: a half-applied + // action set is worse than none, and an action that silently does not + // exist surfaces as a missing generated source three edges away. + if (auto aerr = dirs::action_error(d); !aerr.empty()) { + return std::unexpected(aerr); + } if (d.protocol == 0) { for (auto const& k : d.unknownKeys) mcpp::ui::warning(std::format( diff --git a/src/build/compile_commands.cppm b/src/build/compile_commands.cppm index eeef67f8..f2438d3a 100644 --- a/src/build/compile_commands.cppm +++ b/src/build/compile_commands.cppm @@ -205,7 +205,9 @@ std::string merge_compile_commands( void write_compile_commands(const BuildPlan& plan, const CompileFlags& flags) { auto content = emit_compile_commands(plan, flags); - auto path = plan.projectRoot / "compile_commands.json"; + auto path = plan.compileDbPath.empty() + ? plan.projectRoot / "compile_commands.json" + : plan.compileDbPath; if (std::filesystem::exists(path)) { std::ifstream is(path); diff --git a/src/build/directives.cppm b/src/build/directives.cppm index ac602d9c..4798f048 100644 --- a/src/build/directives.cppm +++ b/src/build/directives.cppm @@ -38,6 +38,7 @@ export module mcpp.build.directives; import std; +import mcpp.libs.json; import mcpp.manifest; import mcpp.toolchain.dialect; @@ -96,6 +97,11 @@ enum class Slot : std::size_t { IncludeDirsAfter, RerunFiles, RerunEnv, + // Build-graph nodes (`mcpp:action=`). The value is a JSON payload rather + // than a scalar: an action has six fields, and a flat `key=value` line + // cannot carry them. The bundled `mcpp` module owns the encoding, which + // is exactly why the typed API is the only surface that grows (S4). + Actions, Count }; inline constexpr std::size_t kSlotCount = static_cast(Slot::Count); @@ -106,6 +112,7 @@ enum class Scope { LinkGlobal, // reaches the final link of whatever consumes this package SourceSet, // joins the compile set RerunKey, // not a build input at all; only feeds the re-run key + GraphNode, // declares an edge in the build graph; see manifest::BuildAction }; // How the raw wire value is normalized before it is stored. Applied ONCE, at @@ -140,7 +147,7 @@ struct Def { int sinceProtocol; }; -inline constexpr std::array kTable{{ +inline constexpr std::array kTable{{ // wire tag slot scope transform must missingPrefix missingSuffix since {"cxxflag", "cxxflag", Slot::CxxFlags, Scope::PackagePrivate, Transform::Verbatim, false, "", "", 1}, {"cflag", "cflag", Slot::CFlags, Scope::PackagePrivate, Transform::Verbatim, false, "", "", 1}, @@ -153,6 +160,7 @@ inline constexpr std::array kTable{{ {"include-dir-after", "include-dir-after", Slot::IncludeDirsAfter, Scope::PackagePrivate, Transform::AbsPath, false, "", "", 1}, {"rerun-if-changed", "", Slot::RerunFiles, Scope::RerunKey, Transform::Verbatim, false, "", "", 1}, {"rerun-if-env-changed","", Slot::RerunEnv, Scope::RerunKey, Transform::Verbatim, false, "", "", 1}, + {"action", "action", Slot::Actions, Scope::GraphNode, Transform::Verbatim, false, "", "", 1}, }}; // ── Collected output of one run ──────────────────────────────────────────── @@ -221,6 +229,35 @@ bool accept_cache_record(Directives& d, std::string_view tag, std::string_view v // place that knows which manifest channel each slot feeds. void apply(mcpp::manifest::Manifest& m, const Directives& d); +// Decode one `mcpp:action=` JSON payload. nullopt = malformed. +std::optional decode_action(std::string_view payload); + +// Non-empty when any declared action is malformed. A separate pass so the +// caller can refuse BEFORE applying anything — a half-applied action set is +// worse than none. +std::string action_error(const Directives& d); + +// Resolve an action's paths against `pkgRoot` and make its Source outputs +// exist, so the ordinary source scan can see them. +// +// A placeholder rather than a synthesised CompileUnit, because that reuses +// every existing mechanism: the glob finds it, the scanner reads it, the plan +// gives it an object path, and ninja overwrites it with the real content +// before the compile edge runs (the compile depends on the action's output). +// +// For a module interface the placeholder carries the DECLARED interface — +// `export module X;` plus its imports — so the prepare-time scan agrees with +// what the generator will emit. That is the same assertion-plus-verification +// trade `[modules].scan_overrides` makes: the declaration is checked against +// the compiler's own P1689 output at build time, so a wrong one is caught +// rather than silently believed. +// +// Never truncates an existing file: after the first build the real content is +// there, and rewriting it would make ninja think the input changed on every +// prepare. +void prepare_actions(std::vector& actions, + const std::filesystem::path& pkgRoot); + // ── Private-scope fold (was prepare.cppm's DirectiveMark / fold pair) ────── // // Lives here because "which compile-visible channels a PackagePrivate @@ -441,6 +478,92 @@ void apply(mcpp::manifest::Manifest& m, const Directives& d) { bc.includeDirs.emplace_back(p); for (auto const& p : d.at(Slot::IncludeDirsAfter)) bc.includeDirsAfter.emplace_back(p); + + // Build-graph nodes. Decoded here rather than at parse time so the cache + // stores the payload verbatim and a replay is byte-identical to a run. + for (auto const& payload : d.at(Slot::Actions)) { + if (auto a = decode_action(payload)) bc.actions.push_back(std::move(*a)); + } +} + +std::optional decode_action(std::string_view payload) { + try { + auto j = nlohmann::json::parse(payload); + mcpp::manifest::BuildAction a; + a.id = j.value("id", std::string{}); + auto role = j.value("role", std::string{"source"}); + a.role = role == "check" ? mcpp::manifest::BuildAction::Role::Check + : role == "artifact" ? mcpp::manifest::BuildAction::Role::Artifact + : mcpp::manifest::BuildAction::Role::Source; + auto arr = [&](const char* k, std::vector& dst) { + if (auto it = j.find(k); it != j.end() && it->is_array()) + for (auto const& v : *it) + if (v.is_string()) dst.push_back(v.get()); + }; + arr("inputs", a.inputs); + arr("outputs", a.outputs); + arr("command", a.command); + arr("provides", a.provides); + arr("imports", a.imports); + a.blocking = j.value("blocking", false); + a.description = j.value("description", std::string{}); + if (a.command.empty() || a.outputs.empty()) return std::nullopt; + if (a.id.empty()) a.id = a.outputs.front(); + return a; + } catch (...) { + return std::nullopt; + } +} + +std::string action_error(const Directives& d) { + for (auto const& payload : d.at(Slot::Actions)) { + if (decode_action(payload)) continue; + // A malformed action is a hard error, never a skip: an action that + // silently does not exist produces a build missing generated sources, + // and the user is left staring at a "no such file" three edges away. + return std::format( + "build.mcpp declared a malformed action.\n" + " Every action needs a non-empty `command` and at least one\n" + " declared `output` — mcpp fixes the source set during prepare,\n" + " so an output whose NAME is unknown cannot be built.\n" + " payload: {}", payload); + } + return {}; +} + +void prepare_actions(std::vector& actions, + const fs::path& pkgRoot) { + for (auto& a : actions) { + auto absolutize = [&](std::vector& v) { + for (auto& p : v) { + // An engine variable is resolved later, once the plan exists + // (outputDir depends on the fingerprint). Leave it alone. + if (p.find("${mcpp.") != std::string::npos) continue; + p = abs_against(pkgRoot, p); + } + }; + absolutize(a.inputs); + absolutize(a.outputs); + if (a.role != mcpp::manifest::BuildAction::Role::Source) continue; + for (auto const& o : a.outputs) { + if (o.find("${mcpp.") != std::string::npos) continue; + std::error_code ec; + fs::path p(o); + if (fs::exists(p, ec)) continue; // real content already there + fs::create_directories(p.parent_path(), ec); + std::ofstream os(p, std::ios::trunc); + if (!os) continue; + if (!a.provides.empty()) { + os << "// placeholder — replaced by action '" << a.id + << "' during the build\n"; + for (auto const& imp : a.imports) os << "import " << imp << ";\n"; + os << "export module " << a.provides.front() << ";\n"; + } + // A non-module output needs nothing: an empty TU scans as + // "provides nothing, imports nothing", which is what a plain + // generated .cpp/.cc is. + } + } } Mark mark(const mcpp::manifest::Manifest& m) { diff --git a/src/build/hostprogram.cppm b/src/build/hostprogram.cppm index b0c9a941..aeec6e00 100644 --- a/src/build/hostprogram.cppm +++ b/src/build/hostprogram.cppm @@ -50,6 +50,63 @@ inline void generated(const char* path) { std::printf("mcpp:generated= inline void source(const char* path) { std::printf("mcpp:source=%s\n", path); } inline void include_dir(const char* dir) { std::printf("mcpp:include-dir=%s\n", dir); } inline void include_dir_after(const char* dir) { std::printf("mcpp:include-dir-after=%s\n", dir); } +// ── Build-graph nodes (mcpp 2026.8.5.1+) ──────────────────────────────── +// Declare WORK instead of doing it. A build program is a good place to decide +// what the build looks like and a bad place to perform it: work done here is +// serial, whole-set, and reported as "build.mcpp exited 1". Declared as a node +// it becomes an edge in the build graph — incremental, parallel, attributable. +// +// You must name the OUTPUT FILES. mcpp fixes the source set, the fingerprint +// and the module graph during prepare, so an output whose name is unknown +// cannot be built. Content may arrive later; names may not. +struct action { + const char* id = ""; + const char* role = "source"; // "source" | "check" | "artifact" + const char* description = ""; + bool blocking = false; // check only: gate compilation on it + action& input(const char* p) { add(inputs_, p); return *this; } + action& output(const char* p) { add(outputs_, p); return *this; } + action& arg(const char* a) { add(command_, a); return *this; } + // Declare what a generated MODULE INTERFACE provides/imports. Same + // "declare instead of discover" trade [modules].scan_overrides makes, and + // what lets a generated .cppm exist as a graph node at all. + action& provides(const char* n) { add(provides_, n); return *this; } + action& imports(const char* n) { add(imports_, n); return *this; } + void submit() const { + std::printf("mcpp:action={\"id\":"); esc(id); + std::printf(",\"role\":"); esc(role); + std::printf(",\"description\":"); esc(description); + std::printf(",\"blocking\":%s", blocking ? "true" : "false"); + std::printf(",\"inputs\":[%s]", inputs_); + std::printf(",\"outputs\":[%s]", outputs_); + std::printf(",\"command\":[%s]", command_); + std::printf(",\"provides\":[%s]", provides_); + std::printf(",\"imports\":[%s]", imports_); + std::printf("}\n"); + } +private: + char inputs_[4096]{}, outputs_[4096]{}, command_[8192]{}, provides_[1024]{}, imports_[1024]{}; + static void esc(const char* s) { + std::putchar('"'); + for (const char* p = s; *p; ++p) { + if (*p == '"' || *p == '\\') std::putchar('\\'); + if (*p == '\n') { std::printf("\\n"); continue; } + std::putchar(*p); + } + std::putchar('"'); + } + static void add(char* buf, const char* s) { + unsigned long o = 0; while (buf[o]) ++o; + if (o) buf[o++] = ','; + buf[o++] = '"'; + for (const char* p = s; *p && o + 3 < 4096; ++p) { + if (*p == '"' || *p == '\\') buf[o++] = '\\'; + buf[o++] = *p; + } + buf[o++] = '"'; + buf[o] = 0; + } +}; inline void rerun_if_changed(const char* path) { std::printf("mcpp:rerun-if-changed=%s\n", path); } inline void rerun_if_env_changed(const char* var) { std::printf("mcpp:rerun-if-env-changed=%s\n", var); } // ── environment contract (read side; values injected by the engine) ───── @@ -87,6 +144,26 @@ inline const char* dep_dir(const char* name) { buf[o++] = '_'; buf[o++] = 'D'; buf[o++] = 'I'; buf[o++] = 'R'; buf[o] = 0; return env_or(buf); } +// mcpp#355: absolute path to a HOST tool built by a dependency — the binary +// behind one of its `kind = "bin"` targets. Returns "" unless the consumer +// declared it: = { version = "…", tools = ["protoc"] } +// The path already carries the platform's executable suffix. +inline const char* dep_bin(const char* pkg, const char* tool) { + char buf[256] = "MCPP_DEP_"; + unsigned long o = 9; + auto put = [&](const char* s) { + for (const char* p = s; *p && o + 8 < sizeof buf; ++p, ++o) { + char c = *p; + buf[o] = (c >= 'a' && c <= 'z') ? char(c - 'a' + 'A') + : ((c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')) ? c : '_'; + } + }; + put(pkg); + buf[o++] = '_'; buf[o++] = 'B'; buf[o++] = 'I'; buf[o++] = 'N'; buf[o++] = '_'; + put(tool); + buf[o] = 0; + return env_or(buf); +} } // ── Protocol announcement ─────────────────────────────────────────────── // Emitted before main() runs, so a program that uses `import mcpp;` never has @@ -150,6 +227,27 @@ struct McppModule { fs::path object; // linked alongside build.mcpp }; +// Compile ONE dependency-provided module interface for the host, into `bdir`, +// with the SAME flags build.mcpp itself gets. Returns how to name its BMI plus +// the object to link. +// +// Shares build_mcpp_module's per-family dispatch deliberately: a BMI is only +// usable by a compile that agrees with it on standard, dialect and compiler +// identity, and the cheapest way to guarantee that is to produce both from one +// set of flags rather than to check afterwards. +// +// Limitation, stated rather than hidden: the interface is compiled ALONE, so it +// may import `std` and the bundled `mcpp` module but not a third package. A +// rule package is a leaf by construction; a transitive host module graph would +// need the sub-build machinery and its own BMI-agreement story. +std::expected +build_host_module(const fs::path& bdir, const fs::path& compiler, + const std::vector& base, const std::string& stdFlag, + const mcpp::toolchain::Toolchain& tc, + const std::vector>& env, + std::string_view logicalName, const fs::path& interfacePath, + const std::vector& extraUseFlags); + std::expected build_mcpp_module(const fs::path& bdir, const fs::path& compiler, const std::vector& base, const std::string& stdFlag, @@ -233,4 +331,91 @@ build_mcpp_module(const fs::path& bdir, const fs::path& compiler, } +} // namespace mcpp::build + +namespace mcpp::build { + +std::expected +build_host_module(const fs::path& bdir, const fs::path& compiler, + const std::vector& base, const std::string& stdFlag, + const mcpp::toolchain::Toolchain& tc, + const std::vector>& env, + std::string_view logicalName, const fs::path& interfacePath, + const std::vector& extraUseFlags) { + std::error_code ec; + if (!fs::exists(interfacePath, ec)) { + return std::unexpected(std::format( + "host module '{}': no interface unit at {}\n" + " A package offering build rules must have a lib root " + "(src/.cppm or [lib] path).", + logicalName, interfacePath.string())); + } + // A filesystem-safe stem: a module name contains dots, which are fine in a + // path but make `foo.rules.o` read as an extension chain. + std::string stem(logicalName); + for (auto& c : stem) if (c == ':' || c == '/' || c == '\\') c = '-'; + + auto run = [&](std::vector argv, const char* what) + -> std::expected { + auto r = mcpp::platform::process::capture_exec(argv, env, bdir.string()); + if (r.exit_code != 0) + return std::unexpected(std::format( + "host module '{}' {} failed (exit {}):\n{}", + logicalName, what, r.exit_code, r.output)); + return {}; + }; + auto with_base = [&](std::vector head) { + for (auto& b : base) head.push_back(b); + for (auto& f : extraUseFlags) head.push_back(f); + return head; + }; + + const auto traits = mcpp::toolchain::bmi_traits(tc); + const auto& dial = mcpp::toolchain::dialect_for(tc); + McppModule out; + out.object = bdir / (stem + std::string(dial.objExt)); + + if (tc.compiler == mcpp::toolchain::CompilerId::MSVC) { + fs::path ifc = bdir / (stem + std::string(traits.bmiExt)); + std::vector argv{compiler.string()}; + for (auto f : dial.alwaysFlagsArgv) argv.emplace_back(f); + argv.push_back(stdFlag); + argv.push_back("/interface"); + for (auto f : dial.forceCxxLangArgv) argv.emplace_back(f); + argv.push_back("/c"); + argv.push_back(interfacePath.string()); + argv.push_back("/ifcOutput"); argv.push_back(ifc.string()); + argv.push_back(std::string(dial.outputObjPrefix) + out.object.string()); + if (auto r = run(with_base(std::move(argv)), "compile"); !r) + return std::unexpected(r.error()); + out.useFlags = mcpp::toolchain::bmi_reference_tokens( + std::format(" /reference {}=", logicalName), ifc); + return out; + } + + if (mcpp::toolchain::is_clang(tc)) { + fs::path pcm = bdir / (stem + std::string(traits.bmiExt)); + if (auto r = run(with_base({compiler.string(), stdFlag, "--precompile", + interfacePath.string(), "-o", pcm.string()}), + "precompile"); !r) + return std::unexpected(r.error()); + if (auto r = run(with_base({compiler.string(), stdFlag, "-c", + pcm.string(), "-o", out.object.string()}), + "object"); !r) + return std::unexpected(r.error()); + out.useFlags = mcpp::toolchain::bmi_reference_tokens( + std::format("-fmodule-file={}=", logicalName), pcm); + return out; + } + + // GCC: BMIs are implicit under /gcm.cache, so nothing to name — which + // is also why the compile has to happen in bdir (it already does). + if (auto r = run(with_base({compiler.string(), stdFlag, "-fmodules", "-c", + interfacePath.string(), "-o", out.object.string()}), + "compile"); !r) + return std::unexpected(r.error()); + out.useFlags = {"-fmodules"}; + return out; +} + } // namespace mcpp::build diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index e8de94af..a45ed3af 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -19,6 +19,7 @@ export module mcpp.build.ninja; import std; import mcpp.build.backend; +import mcpp.manifest; import mcpp.build.distribution; import mcpp.build.plan; import mcpp.build.flags; @@ -1299,7 +1300,46 @@ std::string emit_ninja_string(const BuildPlan& plan) { if (!plan.runtimeDeployFiles.empty()) append("\n"); - if (!plan.linkUnits.empty()) { + // ── Declared build-graph nodes (`mcpp:action=`) ───────────────────────── + // + // One rule + one edge per action. Ordering needs no special handling: a + // Source action's outputs ARE the compile edge's inputs, and an Artifact + // action's inputs ARE the link edge's outputs, so ninja's own file + // dependencies sequence everything. That is the entire reason this is a + // graph node rather than a phase — the alternative (a post hook) would + // have to re-derive the ordering by hand and would still lose incrementality. + std::string actionDefaults; + for (std::size_t i = 0; i < plan.actions.size(); ++i) { + auto const& a = plan.actions[i]; + std::string cmd; + for (auto const& tok : a.command) { + if (!cmd.empty()) cmd += ' '; + cmd += shell_quote_arg(tok); + } + append(std::format("rule mcpp_action_{}\n", i)); + append(std::format(" command = {}\n", cmd)); + append(std::format(" description = {} {}\n", + a.role == mcpp::manifest::BuildAction::Role::Check ? "CHECK" + : a.role == mcpp::manifest::BuildAction::Role::Artifact ? "ARTIFACT" + : "GENERATE", + a.description.empty() ? a.id : a.description)); + append("\n"); + std::string outs, ins; + for (auto const& o : a.outputs) outs += " " + escape_ninja_path(o); + for (auto const& in : a.inputs) ins += " " + escape_ninja_path(in); + append(std::format("build{} : mcpp_action_{}{}\n", outs, i, ins)); + append("\n"); + // A Source action's outputs are already reachable through the compile + // edges that consume them. Check and Artifact outputs are terminal, so + // without this nothing would ever ask for them — and under explicit + // ninja goals (#274) an edge reachable only via `default` is skipped, + // which is exactly how the soname aliases went missing in 0.0.104. + if (a.role != mcpp::manifest::BuildAction::Role::Source) + for (auto const& o : a.outputs) + actionDefaults += " " + escape_ninja_path(o); + } + + if (!plan.linkUnits.empty() || !actionDefaults.empty()) { std::string defaults; for (auto& lu : plan.linkUnits) { defaults += " " + escape_ninja_path(lu.output); @@ -1310,6 +1350,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { for (auto const& d : plan.runtimeDeployFiles) { defaults += " " + escape_ninja_path(d.dest); } + defaults += actionDefaults; append("default" + defaults + "\n"); } diff --git a/src/build/plan.cppm b/src/build/plan.cppm index 374ec108..1349e6b9 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -82,6 +82,11 @@ struct BuildPlan { std::filesystem::path projectRoot; // where mcpp.toml lives std::filesystem::path outputDir; // target/// + // Where compile_commands.json goes. Carried rather than derived from + // projectRoot: under BuildOverrides::work_dir the package root is a shared + // (possibly read-only) registry directory, and deriving the path would put + // an IDE database there. Empty → projectRoot, the historical default. + std::filesystem::path compileDbPath; std::filesystem::path stdBmiPath; // absolute path to prebuilt std.gcm std::filesystem::path stdObjectPath; // absolute path to prebuilt std.o std::filesystem::path stdCompatBmiPath; // absolute path to prebuilt std.compat.pcm @@ -95,6 +100,10 @@ struct BuildPlan { std::vector compileUnits; // topologically sorted std::vector linkUnits; + // Build-graph nodes declared by build programs (`mcpp:action=`). Paths are + // absolute and engine variables already substituted by the time they get + // here, so the backend only has to spell edges. + std::vector actions; std::vector runtimeLibraryDirs; // ONLY the dependency packages' [runtime] library_dirs (not toolchain/ // payload dirs). These are the dirs that must be baked into the produced diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 874d6599..a3dd1f81 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -34,6 +34,9 @@ import mcpp.build.plan; import mcpp.build.cache_key; import mcpp.build.build_program; import mcpp.build.directives; // directive table: mark / fold_private_tail +import mcpp.build.tool_store; // #355 host tools: store layout + key + overrides +import mcpp.build.backend; // BuildOptions for the tool sub-build +import mcpp.build.ninja; // make_ninja_backend — driving that sub-build import mcpp.lockfile; import mcpp.config; import mcpp.xlings; @@ -761,6 +764,32 @@ export std::string resolve_profile_name(const mcpp::manifest::Manifest& m, // Command-level overrides (--target / --static). // Empty defaults preserve pre-existing behaviour exactly. export struct BuildOverrides { + // Where the package being built LIVES (its mcpp.toml). Empty = walk up from + // the process cwd, which is what every user-facing invocation does. Set by + // the tool-provisioning pass, which builds a package that lives in the + // registry rather than under the cwd. + std::filesystem::path project_root; + // Where mcpp WRITES. Empty = the project root, which is the historical + // (and for a normal build, correct) behaviour. + // + // The two are separate because a registry package root is shared across + // projects and may be read-only — build_program.cppm has said so in a + // comment since G2, and until now nothing could honour it for anything + // bigger than build.mcpp's own scratch dir. Splitting "source" from "work" + // is what lets mcpp build such a package at all. + // + // EVERYTHING derived from it moves together: target/, mcpp.lock, + // compile_commands.json, .mcpp/, and build.mcpp's artifact dir. Moving + // only some would be worse than moving none — a half-redirected build + // writes into the shared root anyway, just less visibly. + std::filesystem::path work_dir; + // #355 tool provisioning re-enters prepare_build for the tool package. A + // tool package's own build.mcpp may legitimately want another tool (gRPC's + // wants protoc), so the depth cannot be 1 — but an unbounded chain is a + // bug, and hanging is a worse diagnostic than a named cycle. + int tool_depth = 0; + // The request chain, for that diagnostic. "root → grpc:grpc_cpp_plugin → …" + std::string tool_chain; std::string target_triple; // empty = host triple, fall through to [toolchain] bool force_static = false; // --static (or implied by musl target) std::string package_filter; // -p : only build this workspace member @@ -840,10 +869,21 @@ prepare_build(bool print_fingerprint, bool includeDevDeps = false, std::vector extraTargets = {}, BuildOverrides overrides = {}) { - auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); + auto root = overrides.project_root.empty() + ? mcpp::project::find_manifest_root(std::filesystem::current_path()) + : std::optional(overrides.project_root); if (!root) { return std::unexpected("no mcpp.toml found in current directory or any parent"); } + // Where mcpp writes. Defaults to the project root, so every existing + // invocation is byte-for-byte unchanged; the tool-provisioning pass points + // it at the tool store instead (BuildOverrides::work_dir). + const std::filesystem::path workRoot = + overrides.work_dir.empty() ? *root : overrides.work_dir; + { + std::error_code wdEc; + std::filesystem::create_directories(workRoot, wdEc); + } auto m = mcpp::manifest::load(*root / "mcpp.toml"); if (!m) return std::unexpected(m.error().format()); @@ -964,7 +1004,7 @@ prepare_build(bool print_fingerprint, // this function emits, both taken from the root manifest's [dependencies]. std::map gitLockAnchors; { - auto lockPath = *root / "mcpp.lock"; + auto lockPath = workRoot / "mcpp.lock"; if (std::filesystem::exists(lockPath)) { if (auto lock = mcpp::pm::load(lockPath); lock) { for (auto const& p : lock->packages) @@ -1788,7 +1828,7 @@ prepare_build(bool print_fingerprint, penv.subos = m->xlings.subos; for (auto const& [k, v] : m->xlings.workspace) penv.workspace.emplace_back(k, v); for (auto const& [k, v] : m->xlings.envs) penv.envs.emplace_back(k, v); - mcpp::config::ensure_project_index_dir(**cfg2, *root, m->indices, penv); + mcpp::config::ensure_project_index_dir(**cfg2, workRoot, m->indices, penv); // On first build, the project index data root may be empty because // ensure_project_index_dir only writes .xlings.json but does not @@ -2423,7 +2463,7 @@ prepare_build(bool print_fingerprint, // a hint about the known ≥2-repo xlings resolution gap. The // real fix lives in openxlings/xlings; this only surfaces WHY. auto xlingsJson = (useProjectEnv - ? (*root / ".mcpp") + ? (workRoot / ".mcpp") : (*cfg)->xlingsHome()) / ".xlings.json"; auto indexRepos = mcpp::pm::read_seeded_index_repos(xlingsJson); @@ -2566,8 +2606,25 @@ prepare_build(bool print_fingerprint, // silently dropped (resolution honors them per-edge; activation did not). std::vector requestedFeatures; bool defaultFeatures = true; + // #355: HOST tools this consumer asked the dependency for. Aggregated + // off the edge graph exactly like requestedFeatures — a transitive + // consumer's request must not be silently dropped, which is the + // #242/#243 failure shape. + std::vector requestedTools; }; std::vector dependencyEdges; + // #355: consumer package index → (env var, absolute path) for each host + // tool that consumer requested. Filled by the provisioning pass below; + // read by BOTH build.mcpp call sites (the dependency loop and the root), + // which is why it lives out here rather than inside the resolution block. + std::map>> + toolEnvByConsumer; + // #355 step 5: consumer package index → (logical module name, interface + // path) for each dependency that offers HOST build rules. Same fan-out + // shape as toolEnvByConsumer, and read by the same two call sites. + std::map>> + hostModulesByConsumer; auto parseVisibility = [](std::string_view visibility) { if (visibility == "private") @@ -2624,6 +2681,34 @@ prepare_build(bool print_fingerprint, mcpp::build::directives::fold_private_tail(pkg.privateBuild, ran, t); }; + // A declared build-graph node's Source outputs must be visible to the + // scan, so they are materialized as placeholders and joined to the source + // set here — the same two lists `generated=` feeds, for the same reason + // (the scanner walks the legacy modules.sources mirror). ninja overwrites + // the placeholder before the compile edge runs, because that compile + // depends on the action's output. + auto adoptActionOutputs = [](mcpp::manifest::Manifest& mm, + const std::filesystem::path& pkgRoot, + std::size_t firstNewAction) { + if (firstNewAction >= mm.buildConfig.actions.size()) return; + std::vector fresh( + mm.buildConfig.actions.begin() + + static_cast(firstNewAction), + mm.buildConfig.actions.end()); + mcpp::build::directives::prepare_actions(fresh, pkgRoot); + std::copy(fresh.begin(), fresh.end(), + mm.buildConfig.actions.begin() + + static_cast(firstNewAction)); + for (auto const& a : fresh) { + if (a.role != mcpp::manifest::BuildAction::Role::Source) continue; + for (auto const& o : a.outputs) { + if (o.find("${mcpp.") != std::string::npos) continue; + mm.buildConfig.sources.push_back(o); + mm.modules.sources.push_back(o); + } + } + }; + auto appendUniqueFlags = [](std::vector& flags, @@ -2723,6 +2808,7 @@ prepare_build(bool print_fingerprint, .visibility = visibility, .requestedFeatures = spec.features, .defaultFeatures = spec.defaultFeatures, + .requestedTools = spec.tools, }); }; @@ -3814,6 +3900,268 @@ prepare_build(bool print_fingerprint, apply(packages[i], req, depDefaultFeatures); } + // ── #355: HOST tool provisioning ──────────────────────────────────── + // + // Runs AFTER feature activation (a tool target's gate is a feature) and + // BEFORE any build.mcpp (which is what consumes the tools). That + // ordering is the whole point: build.mcpp runs inside prepare, so a + // tool produced by the main ninja graph would arrive far too late — + // and under --target it would be the wrong architecture besides. + // + // Each tool is built by re-entering prepare_build with the DEPENDENCY + // as the root and no --target, i.e. for the build machine. That is + // Cargo's [build-dependencies] / Bazel's exec configuration shape. + // It is affordable because an executable has zero ABI contact with the + // main build: the sub-build may use the tool package's own toolchain, + // its own profile, and its own resolution — none of it has to agree + // with the consumer. + { + // Aggregate off the authoritative edge graph, exactly like feature + // activation — a transitive consumer's request must not be + // silently dropped (#242/#243). + std::map> toolRequests; + for (auto const& edge : dependencyEdges) + for (auto const& t : edge.requestedTools) + toolRequests[edge.dependencyPackageIndex].insert(t); + + // #355 step 5: dependencies offering HOST build rules. Nothing is + // compiled here — the interface is handed to build_program.cppm, + // which compiles it in the SAME command as build.mcpp so the BMI + // and its consumer agree on standard, dialect and compiler by + // construction rather than by luck. + for (auto const& [depName, spec] : m->dependencies) { + if (!spec.hostModule) continue; + for (auto const& edge : dependencyEdges) { + if (edge.consumerPackageIndex != 0) continue; + auto const& depPkg = packages[edge.dependencyPackageIndex]; + auto const& canon = depPkg.manifest.package.name; + if (canon != depName && !depName.ends_with(canon) + && !canon.ends_with(depName)) continue; + auto rel = mcpp::manifest::resolve_lib_root_path(depPkg.manifest); + hostModulesByConsumer[0].emplace_back(canon, depPkg.root / rel); + break; + } + } + + if (overrides.tool_depth >= mcpp::build::tool_store::kMaxDepth + && !toolRequests.empty()) { + return std::unexpected(std::format( + "tool provisioning nested more than {} levels deep — this is " + "almost certainly a cycle.\n chain: {}", + mcpp::build::tool_store::kMaxDepth, overrides.tool_chain)); + } + + for (auto const& [depIdx, wanted] : toolRequests) { + auto& depPkg = packages[depIdx]; + const auto& depName = depPkg.manifest.package.name; + std::string depShort = depName; + if (auto dot = depName.rfind('.'); + dot != std::string::npos && dot + 1 < depName.size()) + depShort = depName.substr(dot + 1); + + for (auto const& toolName : wanted) { + // The target must exist and be a binary. Naming the + // alternatives matters: the consumer wrote a string, and a + // typo is the likeliest cause. + const mcpp::manifest::Target* tgt = nullptr; + std::string binList; + for (auto const& t : depPkg.manifest.targets) { + if (t.kind != mcpp::manifest::Target::Binary) continue; + if (!binList.empty()) binList += ", "; + binList += t.name; + if (t.name == toolName) tgt = &t; + } + if (!tgt) { + return std::unexpected(std::format( + "dependency '{}' has no `kind = \"bin\"` target named " + "'{}' (requested via tools = [...]).\n" + " available bin targets: [{}]", + depName, toolName, + binList.empty() ? std::string("none") : binList)); + } + + auto var = mcpp::build::tool_store::env_var_name(depName, toolName); + auto varShort = + mcpp::build::tool_store::env_var_name(depShort, toolName); + + auto record = [&](const std::filesystem::path& p) { + for (auto const& edge : dependencyEdges) { + if (edge.dependencyPackageIndex != depIdx) continue; + if (std::find(edge.requestedTools.begin(), + edge.requestedTools.end(), toolName) + == edge.requestedTools.end()) continue; + auto& v = toolEnvByConsumer[edge.consumerPackageIndex]; + v.emplace_back(var, p.string()); + if (varShort != var) v.emplace_back(varShort, p.string()); + } + }; + + // Escape hatch first: it is the cheapest resolution and the + // one a user reaches for precisely when building is not an + // option. Deliberately not part of the store key — see + // tool_store.cppm. + if (auto ovr = mcpp::build::tool_store::find_override( + *m, depName, depShort, toolName)) { + if (!std::filesystem::exists(*ovr)) { + return std::unexpected(std::format( + "tool override for '{}:{}' points at '{}', which " + "does not exist", depName, toolName, ovr->string())); + } + mcpp::ui::info("Tool", std::format( + "{}:{} → {} (override)", depName, toolName, ovr->string())); + record(*ovr); + continue; + } + + // Build it. The feature set is the tool package's own + // defaults PLUS the target's required_features — in a tool + // sub-build the target is what was ASKED FOR, so its + // requirements are inputs rather than a gate. (Same field, + // opposite resolution direction; docs/05 says so.) + std::vector feats = tgt->requiredFeatures; + auto closure = feature_closure(depPkg.manifest, feats, true); + + auto hostTc = host_tc_for_build_program(); + if (!hostTc) return std::unexpected(hostTc.error()); + + mcpp::build::tool_store::Key key; + key.indexName = depIdx >= 1 && depIdx - 1 < dep_cache_identities.size() + ? dep_cache_identities[depIdx - 1].indexName + : std::string(mcpp::pm::kDefaultNamespace); + key.packageName = depName; + key.version = depPkg.manifest.package.version; + key.targetName = toolName; + key.hostTriple = mcpp::toolchain::triple::host_triple().str(); + key.compilerIdentity = std::format("{}|{}|{}", + hostTc->second.label(), hostTc->second.version, + hostTc->first.string()); + key.profile = "release"; + key.features = closure; + std::ranges::sort(key.features); + for (auto const& edge : dependencyEdges) { + if (edge.consumerPackageIndex != depIdx) continue; + auto const& up = packages[edge.dependencyPackageIndex]; + key.upstreamKeys.push_back(std::format("{}@{}", + up.manifest.package.name, up.manifest.package.version)); + } + std::ranges::sort(key.upstreamKeys); + + const auto cacheRoot = mcpp::home::cache_root(); + const auto entry = mcpp::build::tool_store::entry_dir(cacheRoot, key); + const auto exeSuffix = std::string(mcpp::platform::exe_suffix); + const auto binOut = mcpp::build::tool_store::bin_path( + entry, toolName, exeSuffix); + + if (mcpp::build::tool_store::entry_valid(entry, key, toolName, + exeSuffix)) { + record(binOut); + continue; + } + + mcpp::ui::status("Building", std::format( + "host tool {}:{} from {} v{} (once per package version × " + "host toolchain)", depName, toolName, depName, + depPkg.manifest.package.version)); + + BuildOverrides sub; + sub.project_root = depPkg.root; + // Never the package root: it is shared across projects and + // may be read-only. This is the reason work_dir exists. + sub.work_dir = entry / "build"; + sub.target_triple = ""; // HOST — the whole point + sub.profile = "release"; + sub.cache_mode = overrides.cache_mode; + sub.tool_depth = overrides.tool_depth + 1; + sub.tool_chain = overrides.tool_chain.empty() + ? std::format("root → {}:{}", depName, toolName) + : std::format("{} → {}:{}", overrides.tool_chain, depName, + toolName); + for (auto const& f : closure) { + if (!sub.features.empty()) sub.features += ","; + sub.features += f; + } + + auto subCtx = prepare_build(/*print_fingerprint=*/false, + /*includeDevDeps=*/false, + /*extraTargets=*/{}, sub); + if (!subCtx) { + return std::unexpected(std::format( + "building host tool '{}:{}' failed: {}", + depName, toolName, subCtx.error())); + } + + // Build ONLY the requested target (#274 gave the backend + // explicit goals) — a tool request must not drag the whole + // package's other artifacts along. + std::filesystem::path goal; + for (auto const& lu : subCtx->plan.linkUnits) { + if (lu.targetName == toolName) { goal = lu.output; break; } + } + if (goal.empty()) { + return std::unexpected(std::format( + "host tool '{}:{}' produced no link unit — its " + "required_features may not be satisfiable on this " + "platform", depName, toolName)); + } + + auto be = mcpp::build::make_ninja_backend(); + mcpp::build::BuildOptions bopt; + bopt.ninjaTargets = { goal.generic_string() }; + auto br = be->build(subCtx->plan, bopt); + if (!br) { + return std::unexpected(std::format( + "building host tool '{}:{}' failed: {}\n{}", + depName, toolName, br.error().message, + br.error().diagnosticOutput)); + } + if (br->exitCode != 0) { + return std::unexpected(std::format( + "building host tool '{}:{}' failed (exit {})", + depName, toolName, br->exitCode)); + } + + // Publish into the store: build out of place, then move — + // the same discipline mcpp.build.stage follows, so a + // concurrent consumer never observes a half-written entry. + std::error_code cpEc; + auto produced = subCtx->plan.outputDir / goal; + if (!std::filesystem::exists(produced, cpEc)) { + return std::unexpected(std::format( + "host tool '{}:{}' built but '{}' is missing", + depName, toolName, produced.string())); + } + std::filesystem::create_directories(binOut.parent_path(), cpEc); + auto tmp = binOut; + tmp += ".tmp"; + std::filesystem::remove(tmp, cpEc); + std::filesystem::copy_file(produced, tmp, + std::filesystem::copy_options::overwrite_existing, cpEc); + if (cpEc) { + return std::unexpected(std::format( + "staging host tool '{}:{}' failed: {}", + depName, toolName, cpEc.message())); + } + std::filesystem::permissions(tmp, + std::filesystem::perms::owner_exec + | std::filesystem::perms::group_exec + | std::filesystem::perms::others_exec, + std::filesystem::perm_options::add, cpEc); + std::filesystem::rename(tmp, binOut, cpEc); + if (cpEc) { + return std::unexpected(std::format( + "publishing host tool '{}:{}' failed: {}", + depName, toolName, cpEc.message())); + } + mcpp::build::tool_store::write_entry(entry, key); + // The sub-build tree is large (protoc is several hundred + // objects) and the key covers every input, so a hit never + // needs it again. + std::filesystem::remove_all(entry / "build", cpEc); + record(binOut); + } + } + } + // ── G2: dependency build.mcpp (Cargo build.rs model) ──────────────── // Runs AFTER feature activation (the env contract exposes the dep's // active features) and BEFORE the modgraph scan (generated sources @@ -3841,7 +4189,7 @@ prepare_build(bool print_fingerprint, bpEnv.targetTriple = resolvedTargetCanonical; bpEnv.profile = effectiveProfile; bpEnv.features = feature_closure(pkg.manifest, req, depDefaultFeatures); - bpEnv.artifactsDir = *root / "target" / ".build-mcpp" / "deps" + bpEnv.artifactsDir = workRoot / "target" / ".build-mcpp" / "deps" / (dirSafe(pkg.manifest.package.name) + "@" + pkg.manifest.package.version); bpEnv.genBase = bpEnv.artifactsDir / "out"; // mcpp#241: expose this package's resolved dependencies (verdir / @@ -3863,9 +4211,16 @@ prepare_build(bool print_fingerprint, && dot + 1 < canon.size()) bpEnv.depDirs.emplace_back(canon.substr(dot + 1), depPkg.root); } + // #355: the host tools THIS package requested (resolved above). + if (auto tit = toolEnvByConsumer.find(i); tit != toolEnvByConsumer.end()) + bpEnv.toolPaths = tit->second; + bpEnv.hostModules = hostModulesByConsumer.count(i) + ? hostModulesByConsumer.at(i) + : std::vector>{}; auto& bcDep = pkg.manifest.buildConfig; const auto mark = markDirectiveTail(pkg.manifest); const auto ldN = bcDep.ldflags.size(); + const auto actN = bcDep.actions.size(); if (auto r = mcpp::build::run_build_program( pkg.manifest, pkg.root, host->first, host->second, pkg.manifest.cppStandard, bpEnv); @@ -3883,6 +4238,7 @@ prepare_build(bool print_fingerprint, // BFS walk, which ran before this pass — forward the new tail // (link-search paths are already absolute from parse_line). foldDirectiveTailIntoPrivateBuild(pkg, pkg.manifest, mark); + adoptActionOutputs(pkg.manifest, pkg.root, actN); m->buildConfig.ldflags.insert(m->buildConfig.ldflags.end(), bcDep.ldflags.begin() + ldN, bcDep.ldflags.end()); } @@ -3973,6 +4329,15 @@ prepare_build(bool print_fingerprint, mcpp::build::BuildProgramEnv bpEnv; bpEnv.targetTriple = resolvedTargetCanonical; bpEnv.profile = effectiveProfile; + // Set explicitly rather than relying on build_dir()'s root-relative + // default: under BuildOverrides::work_dir the package root is shared + // and may be read-only, and the default would write the compiled + // helper straight into it. Same value as the default when work_dir is + // unset, so an ordinary build is unchanged. + bpEnv.artifactsDir = workRoot / "target" / ".build-mcpp"; + // Root mode keeps genBase empty: a relative `generated=` from the ROOT + // package resolves against the package root (the documented contract), + // not against OUT_DIR. // Same expression as the pre-move call site (and same order), so the // contract hash — and therefore the build.mcpp cache — is unchanged // across the move for feature-identical builds. @@ -3989,10 +4354,17 @@ prepare_build(bool print_fingerprint, && dot + 1 < canon.size()) bpEnv.depDirs.emplace_back(canon.substr(dot + 1), depPkg.root); } + // #355: the host tools the ROOT package requested (consumer index 0). + if (auto tit = toolEnvByConsumer.find(0u); tit != toolEnvByConsumer.end()) + bpEnv.toolPaths = tit->second; + bpEnv.hostModules = hostModulesByConsumer.count(0u) + ? hostModulesByConsumer.at(0u) + : std::vector>{}; auto& bcRoot = m->buildConfig; const auto mark = markDirectiveTail(*m); const auto rldN = bcRoot.ldflags.size(), rsrcN = bcRoot.sources.size(), rmodN = m->modules.sources.size(); + const auto ractN = bcRoot.actions.size(); if (auto bp = mcpp::build::run_build_program( *m, *root, host->first, host->second, m->cppStandard, bpEnv); @@ -4003,6 +4375,10 @@ prepare_build(bool print_fingerprint, // Compile-visible tail → privateBuild: the shared fold (same owner // as the dep loop; the root's TUs read privateBuild). foldDirectiveTailIntoPrivateBuild(pkg0, *m, mark); + // Before the source residues are mirrored below: adopting an action's + // outputs APPENDS to bcRoot.sources, and those appends must be inside + // the tail that gets copied into the packages[0] snapshot the scan reads. + adoptActionOutputs(*m, *root, ractN); // Root residues — apply() mutated *m, but packages[0].manifest is a // value-copy snapshot taken at makePackageRoot, so everything the // scan/fingerprint read from the snapshot needs the tail mirrored: @@ -4206,7 +4582,7 @@ prepare_build(bool print_fingerprint, ctx.profile = effectiveProfile; ctx.cacheMode = cacheMode; ctx.projectRoot= *root; - ctx.outputDir = target_dir(*tc, fp, *root); + ctx.outputDir = target_dir(*tc, fp, workRoot); ctx.stdBmi = stdBmiPath; ctx.stdObject = stdObjectPath; // Every directory a package payload may legitimately have been INSTALLED @@ -4221,7 +4597,7 @@ prepare_build(bool print_fingerprint, const auto storeRoots = [&]() -> std::vector { std::vector roots; if (auto c = get_cfg()) roots.push_back((*c)->xlingsHome() / "data" / "xpkgs"); - for (auto& d : mcpp::config::project_xlings_data_roots(*root)) + for (auto& d : mcpp::config::project_xlings_data_roots(workRoot)) roots.push_back(d / "xpkgs"); return roots; }(); @@ -4230,6 +4606,60 @@ prepare_build(bool print_fingerprint, stdBmiPath, stdObjectPath, storeRoots); if (!planResult) return std::unexpected(planResult.error()); ctx.plan = std::move(*planResult); + ctx.plan.compileDbPath = workRoot / "compile_commands.json"; + + // ── Declared build-graph nodes → the plan ─────────────────────────────── + // + // Collected here rather than inside make_plan because the engine-variable + // vocabulary an action may reference includes values that only exist once + // the plan does (outputDir is fingerprint-derived; a target's file name is + // a link unit's output). + // + // The vocabulary is CLOSED on purpose. An action's command is an argv, not + // a shell string, and the only interpolations are these four — which is + // what makes an action portable (Windows has no shell to assume) and + // cacheable (nothing can smuggle in ambient state). + { + auto substitute = [&](std::string s) { + auto rep = [&](std::string_view what, const std::string& with) { + for (std::size_t p; (p = s.find(what)) != std::string::npos; ) + s.replace(p, what.size(), with); + }; + rep("${mcpp.out_dir}", ctx.plan.outputDir.string()); + rep("${mcpp.bin_dir}", (ctx.plan.outputDir / "bin").string()); + rep("${mcpp.compile_db}", ctx.plan.compileDbPath.string()); + constexpr std::string_view kTf = "${mcpp.target_file:"; + for (std::size_t p; (p = s.find(kTf)) != std::string::npos; ) { + auto close = s.find('}', p); + if (close == std::string::npos) break; + auto name = s.substr(p + kTf.size(), close - p - kTf.size()); + // The link unit's BUILD-DIR-RELATIVE output, not an absolute + // path. ninja identifies a file by the string an edge declares, + // and the link edge declares `bin/app`; an absolute reference + // to the same bytes is a DIFFERENT node, which ninja reports as + // "missing and no known rule to make it". Commands run with + // cwd = the build dir, so the relative form is also what the + // tool being invoked should receive. + std::string resolved; + for (auto const& lu : ctx.plan.linkUnits) + if (lu.targetName == name) + resolved = lu.output.generic_string(); + s.replace(p, close - p + 1, resolved); + } + return s; + }; + auto collect = [&](const mcpp::manifest::Manifest& mm) { + for (auto a : mm.buildConfig.actions) { + for (auto& x : a.inputs) x = substitute(x); + for (auto& x : a.outputs) x = substitute(x); + for (auto& x : a.command) x = substitute(x); + ctx.plan.actions.push_back(std::move(a)); + } + }; + collect(*m); + for (std::size_t i = 1; i < packages.size(); ++i) + collect(packages[i].manifest); + } ctx.plan.stdCompatBmiPath = stdCompatBmiPath; ctx.plan.stdCompatObjectPath = stdCompatObjectPath; @@ -4676,7 +5106,7 @@ prepare_build(bool print_fingerprint, lock.packages.push_back(std::move(lp)); } if (!lock.packages.empty() || !lock.indices.empty()) { - auto lockPath = *root / "mcpp.lock"; + auto lockPath = workRoot / "mcpp.lock"; (void)mcpp::lockfile::write(lock, lockPath); } } diff --git a/src/build/tool_store.cppm b/src/build/tool_store.cppm new file mode 100644 index 00000000..cbc1da3b --- /dev/null +++ b/src/build/tool_store.cppm @@ -0,0 +1,236 @@ +// mcpp.build.tool_store — where a dependency's HOST tools live, and how an +// entry gets filled. +// +// #355: a package can build a binary its consumers need at build time (protoc, +// grpc_cpp_plugin, flatc, moc, a transpiler). Until now a consumer had no way +// to reach it: `mcpp::dep_dir()` gives the dependency's SOURCE tree, and a +// dependency's `kind = "bin"` targets are never built at all (plan.cppm only +// walks the ROOT manifest's targets; the one exception, `kind = "shared"`, is +// built for the --target triple and so is useless as a host tool anyway). +// +// WHY IT CANNOT BE A NODE IN THE MAIN GRAPH +// +// Ordering. `build.mcpp` runs inside prepare (prepare.cppm's dep loop and root +// call site); the BuildPlan does not exist yet and build.ninja is written later +// still, in execute.cppm. Anything the main graph produces is therefore +// unavailable to the program that needs it. Add cross-compilation and it is not +// even the right binary: the main graph builds for --target, and a code +// generator has to run HERE. +// +// So the tool is produced by a NESTED build — the dependency package built as +// its own root, for the host, into a global store. That is Cargo's +// [build-dependencies], Bazel's exec configuration, vcpkg's `"host": true` and +// Conan's `tool_requires`, which is the whole industry's answer to this. +// +// WHAT MAKES IT CHEAP +// +// A tool is an EXECUTABLE, so it has zero ABI contact with the main build. +// The sub-build may therefore use the tool package's own [toolchain], its own +// profile, and its own resolution of its own dependencies — none of it has to +// agree with the consumer. (Contrast a `kind = "lib"` dependency, where every +// one of those must match.) That is what keeps this from needing a second +// coherent resolution universe. +// +// THE STORE IS THE INTERFACE +// +// An entry is `/bin/` plus an entry.json. How it gets filled is +// a provider detail: today `build-from-source` (a nested build) and `override` +// (the user pointed at an existing binary). A future `prebuilt-asset` provider +// — the descriptor ships a per-host binary, which is what protobuf upstream +// actually publishes — needs no change on the consumer side. +// +// See .agents/docs/2026-08-05-issue355-dependency-host-tools-design.md. + +export module mcpp.build.tool_store; + +import std; +import mcpp.libs.json; +import mcpp.manifest; +import mcpp.toolchain.fingerprint; + +export namespace mcpp::build::tool_store { + +// Bump ONLY when previously written entries become unusable (the layout or the +// key inputs changed shape). Deliberately not the mcpp release number: a tool +// binary's validity has nothing to do with mcpp's version. +inline constexpr int kEpoch = 1; + +// How deep a tool request may nest before mcpp calls it a cycle. A tool +// package's own build.mcpp may legitimately need another tool (gRPC's needs +// protoc), so the limit cannot be 1 — but an unbounded chain is a bug, and +// hanging is a worse diagnostic than a named cycle. +inline constexpr int kMaxDepth = 4; + +// One resolved tool: the package it came from, the target name, and the +// absolute path to the executable. +struct Tool { + std::string packageName; // canonical FQN, e.g. "compat.protobuf" + std::string targetName; // the [targets.X] name, e.g. "protoc" + std::filesystem::path path; // absolute path to the executable + bool fromOverride = false; +}; + +// Everything that decides whether two requests are the same tool. Kept as a +// struct so the key inputs are recorded verbatim in entry.json and compared +// field by field on a hit — hash equality alone is never trusted (the +// discipline mcpp.bmi_cache established). +struct Key { + int epoch = kEpoch; + std::string indexName; // "compat" / "mcpplibs" / ... + std::string packageName; // canonical FQN + std::string version; + std::string targetName; + std::string hostTriple; + std::string compilerIdentity; // the HOST toolchain that will build it + std::string profile; + std::vector features; // resolved closure, sorted + // The cache key of each direct dependency of the TOOL package, recursively + // (Merkle). Without it, bumping one of protobuf's own dependencies would + // leave a stale protoc in the store — a silently wrong artifact, which is + // the failure mode this project has paid for more than once. + std::vector upstreamKeys; +}; + +std::string key_hex(const Key& k); +nlohmann::json to_json(const Key& k); + +// /tool//@// +std::filesystem::path entry_dir(const std::filesystem::path& cacheRoot, const Key& k); +std::filesystem::path bin_path(const std::filesystem::path& entryDir, + std::string_view toolName, + std::string_view exeSuffix); + +// A complete, verified entry? (bin present AND entry.json records the same key +// inputs — never just the hash.) +bool entry_valid(const std::filesystem::path& entryDir, const Key& k, + std::string_view toolName, std::string_view exeSuffix); + +void write_entry(const std::filesystem::path& entryDir, const Key& k); + +// ── Overrides (the escape hatch every comparable system provides) ────────── +// +// vcpkg has VCPKG_HOST_TRIPLET, CMake projects have LLVM_NATIVE_TOOL_DIR and +// QT_HOST_PATH, Cargo has `target = "target"`. Without one, a user whose tool +// cannot be built from source — or who simply already has the right binary — +// has no way forward at all. +// +// Resolution order (first hit wins): +// 1. MCPP_TOOL__ (env; CI / distro packaging) +// 2. [tools.overrides] ":" = "" (manifest) +// +// An override deliberately does NOT enter the store key: it is an escape +// hatch, not a reproducible input, and pretending otherwise would let a local +// path silently decide a cached artifact's identity. +std::optional +find_override(const mcpp::manifest::Manifest& rootManifest, + std::string_view packageName, std::string_view shortName, + std::string_view toolName); + +// MCPP_FEATURE_-style sanitizer, shared with the env contract so a name is +// spelled the same on both sides. +std::string sanitize_env(std::string s); + +// The env var name a build.mcpp reads for this tool. +std::string env_var_name(std::string_view packageName, std::string_view toolName); + +} // namespace mcpp::build::tool_store + +namespace mcpp::build::tool_store { + +namespace fs = std::filesystem; + +std::string sanitize_env(std::string s) { + for (auto& c : s) + c = std::isalnum(static_cast(c)) + ? static_cast(std::toupper(static_cast(c))) : '_'; + return s; +} + +std::string env_var_name(std::string_view packageName, std::string_view toolName) { + return "MCPP_DEP_" + sanitize_env(std::string(packageName)) + + "_BIN_" + sanitize_env(std::string(toolName)); +} + +nlohmann::json to_json(const Key& k) { + nlohmann::json j; + j["epoch"] = k.epoch; + j["index"] = k.indexName; + j["package"] = k.packageName; + j["version"] = k.version; + j["target"] = k.targetName; + j["host_triple"] = k.hostTriple; + j["compiler_identity"] = k.compilerIdentity; + j["profile"] = k.profile; + j["features"] = k.features; + j["upstream_keys"] = k.upstreamKeys; + return j; +} + +std::string key_hex(const Key& k) { + return mcpp::toolchain::hash_string(to_json(k).dump()); +} + +fs::path entry_dir(const fs::path& cacheRoot, const Key& k) { + return cacheRoot / "tool" + / (k.indexName.empty() ? std::string("_") : k.indexName) + / std::format("{}@{}", k.packageName, k.version) + / key_hex(k); +} + +fs::path bin_path(const fs::path& entryDir, std::string_view toolName, + std::string_view exeSuffix) { + return entryDir / "bin" / (std::string(toolName) + std::string(exeSuffix)); +} + +bool entry_valid(const fs::path& entryDir, const Key& k, + std::string_view toolName, std::string_view exeSuffix) { + std::error_code ec; + if (!fs::exists(bin_path(entryDir, toolName, exeSuffix), ec)) return false; + std::ifstream is(entryDir / "entry.json"); + if (!is) return false; + try { + nlohmann::json recorded; + is >> recorded; + // Field-by-field, not hash-vs-hash: a directory named by a hash proves + // only that someone once computed that hash. This is the same rule + // bmi_cache follows and for the same reason. + return recorded == to_json(k); + } catch (...) { + return false; + } +} + +void write_entry(const fs::path& entryDir, const Key& k) { + std::error_code ec; + fs::create_directories(entryDir, ec); + std::ofstream os(entryDir / "entry.json", std::ios::trunc); + if (os) os << to_json(k).dump(2) << '\n'; +} + +std::optional +find_override(const mcpp::manifest::Manifest& rootManifest, + std::string_view packageName, std::string_view shortName, + std::string_view toolName) { + // 1. Environment — the CI / distro-packaging channel, and the one that + // works without editing a manifest you may not own. + for (auto const& name : { std::string(packageName), std::string(shortName) }) { + if (name.empty()) continue; + auto var = "MCPP_TOOL_" + sanitize_env(name) + "_" + + sanitize_env(std::string(toolName)); + if (const char* v = std::getenv(var.c_str()); v && *v) + return fs::path(v); + } + // 2. [tools.overrides] in the ROOT manifest. Accepts both the canonical + // and the namespace-stripped spelling, matching how `tools = [...]` + // itself may be written. + for (auto const& name : { std::string(packageName), std::string(shortName) }) { + if (name.empty()) continue; + auto it = rootManifest.toolOverrides.find( + std::format("{}:{}", name, toolName)); + if (it != rootManifest.toolOverrides.end() && !it->second.empty()) + return fs::path(it->second); + } + return std::nullopt; +} + +} // namespace mcpp::build::tool_store diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index 502c34c2..d5b8ed03 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -353,6 +353,15 @@ std::expected parse_string(std::string_view content, if (cval.is_string()) m.capabilityPins[cap] = cval.as_string(); } + // [tools.overrides] ":" = "" — #355 escape hatch. Use an + // existing host binary instead of building the dependency's tool target. + // Root-only, like [capabilities]: it is the consumer's environment being + // described, and a dependency has no business overriding it. + if (auto* tovr = doc->get_table("tools.overrides"); tovr && !tovr->empty()) { + for (auto& [k, v] : *tovr) + if (v.is_string()) m.toolOverrides[k] = v.as_string(); + } + // [generated_files] — "relative/path" = "file contents" (multiline // strings supported). Same mechanism as the index descriptor's // generated_files key: materialized into the package root before glob @@ -533,7 +542,8 @@ std::expected parse_string(std::string_view content, || k == "rev" || k == "tag" || k == "branch" || k == "features" || k == "default-features" || k == "workspace" || k == "visibility" - || k == "backend"; + || k == "backend" || k == "tools" + || k == "host-module"; }; auto looks_like_inline_dep_spec = [&](const t::Table& sub) { if (sub.empty()) return false; @@ -572,6 +582,19 @@ std::expected parse_string(std::string_view content, if (auto it = sub.find("default-features"); it != sub.end() && it->second.is_bool()) { spec.defaultFeatures = it->second.as_bool(); } + // #355: `tools = ["protoc"]` — HOST binaries this consumer wants from + // the dependency. Same shape as `features`, and deliberately on the + // dependency edge: requesting an extra artifact from the graph is a + // graph-level request, so it stays declarative in mcpp.toml. + if (auto it = sub.find("tools"); it != sub.end() && it->second.is_array()) { + for (auto& tv : it->second.as_array()) + if (tv.is_string()) spec.tools.push_back(tv.as_string()); + } + // #355 step 5: `host-module = true` — make this dependency's lib-root + // module importable from build.mcpp (reusable rules as packages). + if (auto it = sub.find("host-module"); it != sub.end() && it->second.is_bool()) { + spec.hostModule = it->second.as_bool(); + } // `backend = ""` — sugar for requesting the dependency's // `backend-` feature (library-level backend selection knob). if (auto it = sub.find("backend"); it != sub.end() && it->second.is_string()) { @@ -629,7 +652,7 @@ std::expected parse_string(std::string_view content, if (!looks_like_inline_dep_spec(sub)) { return std::unexpected(error(origin, std::format( "[{}.{}] must be a version string or table of " - "(path/version/git/rev/tag/branch/features/default-features/visibility)", + "(path/version/git/rev/tag/branch/features/default-features/visibility/tools)", section, key))); } if (auto r = fill_inline_spec(spec, section, key, sub); !r) return r; diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index d92c43ab..d5ddb588 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -206,6 +206,50 @@ inline void append(BuildInputs& dst, const BuildInputs& src) { src.includeDirsAfter.end()); } +// A build-graph node declared by a build program (`mcpp:action=`). +// +// The architectural point (see +// .agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md §3.1): +// build.mcpp answers "what does this build look like" — CONFIGURATION — and is +// a bad place to do WORK. Generating sources, linting, signing and packaging +// are work: they want to be incremental, parallel and attributable, which a +// once-per-prepare program can never be. So instead of DOING the work, the +// program DECLARES it, and it becomes an edge in the build graph. +// +// One primitive, three wirings. `role` is not three mechanisms — it is where +// the same edge's outputs attach: +// +// Source — outputs join the compile set (protoc, a transpiler) +// Check — outputs are a stamp; nothing consumes them (clang-tidy, a +// format or ABI check). Runs alongside compilation by default, +// because serialising every compile behind a linter is a cost +// nobody accepts and "the build still fails" is just as true. +// Artifact — inputs are link outputs (codesign, packaging, size budgets) +// +// INV-D, the constraint that makes this expressible at all: the declaration +// must name its OUTPUT FILES, not merely promise some. mcpp fixes the source +// set, the fingerprint, compile_commands.json and the module topo order during +// prepare, and all of them need to know which files exist. Content may arrive +// later; names may not. +struct BuildAction { + enum class Role { Source, Check, Artifact }; + + std::string id; // diagnostics + edge naming + Role role = Role::Source; + std::vector inputs; // absolute or package-relative + std::vector outputs; // ditto; declared, see INV-D + std::vector command; // argv; NOT a shell string + // Serialised module facts for a generated OUTPUT, when it is a module + // interface. Same "declare instead of discover" trade `[modules].scan_overrides` + // already makes — and the reason a generated `.cppm` does not need its + // content to exist during prepare. + std::vector provides; + std::vector imports; + // Check only: make compilation wait for this to pass. Off by default. + bool blocking = false; + std::string description; +}; + // `[build]` section — tunables for the build backend. // // M5.0: now also carries `sources` (moved from [modules]) and `include_dirs` @@ -240,6 +284,10 @@ struct BuildConfig : BuildInputs { // featureDefines above, which are interface switches). std::map> featureFlags; std::map generatedFiles; // Form B package-owned support files + // Build-graph nodes declared by this package's build program + // (`mcpp:action=`). Empty for every package that does not use one, so an + // ordinary build is untouched. + std::vector actions; bool staticStdlib = true; // #336 — the C++ runtime DISTRIBUTION contract: what the artifact promises // about the machine that runs it ("self-contained" | "toolchain-coupled" | @@ -521,6 +569,19 @@ struct Manifest { featureForwards; // Root-only: [capabilities] cap = "provider" pins (also fed by --cap). std::map capabilityPins; + // #355 `[tools.overrides]` — ":" → absolute path to an + // existing host binary, which mcpp uses INSTEAD of building the tool. + // + // The escape hatch every comparable system provides (vcpkg's + // VCPKG_HOST_TRIPLET, CMake's LLVM_NATIVE_TOOL_DIR, Qt's QT_HOST_PATH, + // Cargo's `target = "target"`). Without one, a user whose tool cannot be + // built from source — or who already has the right binary — has no way + // forward at all. + // + // Deliberately NOT part of the tool store key: an override is an escape + // hatch, not a reproducible input. `mcpp doctor` reports the ones in + // effect so a build that silently used one is still explainable. + std::map toolOverrides; // [target.] tables — empty if user didn't declare any. std::map targetOverrides; diff --git a/src/manifest/xpkg.cppm b/src/manifest/xpkg.cppm index b5645ae6..5291687d 100644 --- a/src/manifest/xpkg.cppm +++ b/src/manifest/xpkg.cppm @@ -1344,6 +1344,27 @@ synthesize_from_xpkg_lua(std::string_view luaContent, t.main = cur.read_string(); } else if (sub == "soname") { t.soname = cur.read_string(); + } else if (sub == "required_features") { + // #355: without this, a Form B descriptor could not + // express the cost gate that makes an optional host + // tool affordable — e.g. compat.protobuf's `protoc` + // needs libprotoc's ~157 extra TUs, which must NOT be + // compiled for the consumers that only want the + // runtime. mcpp.toml has read this since the target + // gate landed; the descriptor parser silently dropped + // it, so the target was either always or never built. + if (cur.peek() == '{') { + cur.consume('{'); + cur.skip_ws_and_comments(); + while (!cur.eof() && cur.peek() != '}') { + auto s = cur.read_string(); + if (!s.empty()) t.requiredFeatures.push_back(std::move(s)); + cur.skip_ws_and_comments(); + } + cur.consume('}'); + } else { + (void)cur.read_bareword(); + } } else { // unknown subfield — skip its value cur.skip_ws_and_comments(); @@ -1503,10 +1524,54 @@ synthesize_from_xpkg_lua(std::string_view luaContent, cur.skip_ws_and_comments(); if (!cur.consume('=')) break; cur.skip_ws_and_comments(); - auto dver = cur.read_string(); + // The value is either a bare version string (the long-standing + // form) or a table. #355 needs the table form so a Form B + // descriptor can request a HOST tool from a dependency: + // ["compat.protobuf"] = { version = "35.1", tools = {"protoc"} } + // Only the VALUE position becomes a table — namespaced + // subtables still are not supported, which is the limit the + // original comment was describing. + std::string dver; + std::vector dtools; + if (cur.peek() == '{') { + cur.consume('{'); + cur.skip_ws_and_comments(); + while (!cur.eof() && cur.peek() != '}') { + auto dk = cur.read_key(); + if (dk.empty()) break; + cur.skip_ws_and_comments(); + if (!cur.consume('=')) break; + cur.skip_ws_and_comments(); + if (dk == "version") { + dver = cur.read_string(); + } else if (dk == "tools" && cur.peek() == '{') { + cur.consume('{'); + cur.skip_ws_and_comments(); + while (!cur.eof() && cur.peek() != '}') { + auto s = cur.read_string(); + if (!s.empty()) dtools.push_back(std::move(s)); + cur.skip_ws_and_comments(); + } + cur.consume('}'); + } else { + // Record rather than swallow — a descriptor author + // writing an unsupported dep key deserves to be + // told, not to get a half-configured dependency. + m.xpkgUnknownKeys.push_back( + std::format("deps.{}.{}", dname, dk)); + if (cur.peek() == '{') cur.skip_table(); + else (void)cur.read_bareword(); + } + cur.skip_ws_and_comments(); + } + cur.consume('}'); + } else { + dver = cur.read_string(); + } if (!dname.empty()) { DependencySpec spec; spec.version = dver; + spec.tools = std::move(dtools); auto selector = mcpp::pm::resolve_dependency_selector( dname, mcpp::pm::DependencySelectorMode::OmittedMcpplibsPriority); diff --git a/src/pm/dep_spec.cppm b/src/pm/dep_spec.cppm index b12c0498..9c190ca9 100644 --- a/src/pm/dep_spec.cppm +++ b/src/pm/dep_spec.cppm @@ -40,6 +40,34 @@ struct DependencySpec { std::string gitRefKind; // "rev" / "tag" / "branch" (for clarity) std::string visibility = "public"; // public / private / interface std::vector features; // requested feature set (long-form dep spec) + // #355: HOST tools this consumer wants from the dependency — the names of + // its `kind = "bin"` targets. Requesting one makes mcpp build that target + // for the BUILD MACHINE (never the --target triple) and hand its path to + // build.mcpp as MCPP_DEP__BIN_. + // + // Declared here, on the dependency edge, rather than in build.mcpp: asking + // for an extra artifact from the graph is a graph-level request, and the + // graph stays statically analysable (lockfile / LSP / audit). It is also + // where the industry converged — vcpkg's `"host": true`, Conan's + // `tool_requires`, xmake's `add_deps(..., {host = true})`, Cargo's + // `[build-dependencies]` — all put it on the consumer's edge. + // + // Empty by default: the cost (e.g. protobuf's libprotoc is ~157 extra TUs) + // is paid by the consumer, so nothing is built unless someone asks. + std::vector tools; + // #355 step 5: compile this dependency's lib-root module interface FOR THE + // HOST and make it importable from the consumer's build.mcpp — the + // mechanism behind reusable build rules distributed as ordinary packages + // (`import mcpp.rules.protobuf;`), instead of a second, non-C++ rule DSL. + // + // Compiled ALONGSIDE build.mcpp with the SAME flags, not by a separate + // sub-build. That is not an optimisation: a BMI is only usable by a + // compile that agrees with it on standard, dialect flags and compiler + // identity, and two independently-resolved builds have no reason to. The + // shared-compile construction makes that agreement structural rather than + // something to verify — the same class of failure as `module X CRC + // mismatch`, which this project has paid for before. + bool hostModule = false; bool defaultFeatures = true; // consumer opt-out: `default-features = false` // suppresses the dep's own [features].default seed // (Cargo parity). Explicit `features = [...]` still apply. diff --git a/tests/e2e/187_dep_host_tool.sh b/tests/e2e/187_dep_host_tool.sh new file mode 100755 index 00000000..4436e5ab --- /dev/null +++ b/tests/e2e/187_dep_host_tool.sh @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# requires: gcc +# 187_dep_host_tool.sh — #355: a dependency's `kind = "bin"` target, built for +# the HOST and handed to the consumer's build.mcpp. +# +# Why this cannot be done any other way: build.mcpp runs inside prepare, before +# the BuildPlan exists and long before build.ninja is written, so a tool +# produced by the main graph arrives too late to be called. The tool is +# therefore built by a nested, host-targeted sub-build into a global store. +# +# Covered here: +# 1. end to end — the tool is built, `mcpp::dep_bin()` finds it, the source it +# generates is compiled and linked +# 2. the cost gate — a tool target behind `required_features` is built because +# the sub-build ACTIVATES those features (in a tool sub-build the target is +# what was asked for, so its requirements are inputs, not a gate) +# 3. default-off — a consumer that does not ask gets nothing built +# 4. a bad tool name fails with the available targets listed +# 5. the override escape hatch skips the build entirely +# +# See .agents/docs/2026-08-05-issue355-dependency-host-tools-design.md. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +# Isolate the tool store so a stale entry from a previous run cannot make a +# broken build look green. +export MCPP_HOME="$TMP/mcpphome" +mkdir -p "$MCPP_HOME" +# Keep the real registry (toolchains are expensive) but not its build cache. +if [ -d "$HOME/.mcpp/registry" ]; then + ln -s "$HOME/.mcpp/registry" "$MCPP_HOME/registry" +fi + +# ── the tool package ──────────────────────────────────────────────────────── +mkdir -p toolpkg/src +cat > toolpkg/mcpp.toml <<'EOF' +[package] +name = "toolpkg" +version = "0.1.0" + +[build] +sources = ["src/lib.cpp"] + +[features.codegen] +sources = ["src/codegen.cpp"] + +[targets.codegen] +kind = "bin" +main = "src/codegen.cpp" +required_features = ["codegen"] +EOF +cat > toolpkg/src/lib.cpp <<'EOF' +int toolpkg_lib() { return 1; } +EOF +# A minimal "code generator": writes a C++ source whose function returns 42. +cat > toolpkg/src/codegen.cpp <<'EOF' +#include +#include +int main(int argc, char** argv) { + if (argc < 2) return 2; + FILE* f = std::fopen(argv[1], "w"); + if (!f) return 3; + std::fprintf(f, "int generated_answer() { return 42; }\n"); + std::fclose(f); + return 0; +} +EOF + +# ── the consumer ──────────────────────────────────────────────────────────── +mkdir -p app/src +cat > app/mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" + +[dependencies] +toolpkg = { path = "../toolpkg", tools = ["codegen"] } +EOF +cat > app/src/main.cpp <<'EOF' +#include +int generated_answer(); +int main() { std::printf("ANSWER=%d\n", generated_answer()); } +EOF +cat > app/build.mcpp <<'EOF' +#include +#include +#include +import mcpp; +int main() { + const char* tool = mcpp::dep_bin("toolpkg", "codegen"); + if (!tool || !*tool) { std::fprintf(stderr, "no tool path\n"); return 1; } + std::string out = std::string(mcpp::out_dir()) + "/gen.cpp"; + std::string cmd = std::string("\"") + tool + "\" \"" + out + "\""; + if (std::system(cmd.c_str()) != 0) { std::fprintf(stderr, "tool failed\n"); return 1; } + mcpp::generated(out.c_str()); +} +EOF + +cd app +"$MCPP" build > b1.log 2>&1 || { cat b1.log; echo "FAIL: build with a dep host tool failed"; exit 1; } +grep -q "host tool" b1.log || { cat b1.log; echo "FAIL: no tool-build announcement"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" +[[ "$out" == "ANSWER=42" ]] || { echo "FAIL: generated source not linked: $out"; exit 1; } + +# The store keeps the tool, so a second build must not rebuild it. +rm -rf target +"$MCPP" build > b2.log 2>&1 || { cat b2.log; echo "FAIL: second build failed"; exit 1; } +grep -q "Building.*host tool" b2.log && { + cat b2.log; echo "FAIL: the tool was rebuilt despite a valid store entry"; exit 1; } + +# ── default-off: no `tools = [...]`, nothing gets built ───────────────────── +cd "$TMP" +mkdir -p plain/src +cat > plain/mcpp.toml <<'EOF' +[package] +name = "plain" +version = "0.1.0" + +[dependencies] +toolpkg = { path = "../toolpkg" } +EOF +cat > plain/src/main.cpp <<'EOF' +int main() {} +EOF +cd plain +"$MCPP" build > b3.log 2>&1 || { cat b3.log; echo "FAIL: plain consumer build failed"; exit 1; } +grep -q "host tool" b3.log && { + cat b3.log; echo "FAIL: a tool was provisioned for a consumer that never asked"; exit 1; } + +# ── a bad tool name names the alternatives ────────────────────────────────── +cd "$TMP" +sed 's/tools = \["codegen"\]/tools = ["nosuchtool"]/' app/mcpp.toml > app/mcpp.toml.new +mv app/mcpp.toml.new app/mcpp.toml +cd app && rm -rf target +if "$MCPP" build > b4.log 2>&1; then + cat b4.log; echo "FAIL: an unknown tool name was accepted"; exit 1 +fi +grep -q "nosuchtool" b4.log || { cat b4.log; echo "FAIL: error does not name the request"; exit 1; } +grep -q "codegen" b4.log || { + cat b4.log; echo "FAIL: error does not list the available bin targets"; exit 1; } + +# ── the override escape hatch ─────────────────────────────────────────────── +cd "$TMP" +sed 's/tools = \["nosuchtool"\]/tools = ["codegen"]/' app/mcpp.toml > app/mcpp.toml.new +mv app/mcpp.toml.new app/mcpp.toml +cat > fake_codegen.sh <<'EOF' +#!/usr/bin/env bash +printf 'int generated_answer() { return 7; }\n' > "$1" +EOF +chmod +x fake_codegen.sh +cd app && rm -rf target +MCPP_TOOL_TOOLPKG_CODEGEN="$TMP/fake_codegen.sh" "$MCPP" build > b5.log 2>&1 \ + || { cat b5.log; echo "FAIL: build with a tool override failed"; exit 1; } +grep -q "override" b5.log || { cat b5.log; echo "FAIL: the override was not reported"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" +[[ "$out" == "ANSWER=7" ]] || { echo "FAIL: the override was not actually used: $out"; exit 1; } + +echo "OK" diff --git a/tests/e2e/188_build_actions.sh b/tests/e2e/188_build_actions.sh new file mode 100755 index 00000000..6538d17e --- /dev/null +++ b/tests/e2e/188_build_actions.sh @@ -0,0 +1,151 @@ +#!/usr/bin/env bash +# requires: gcc +# 188_build_actions.sh — `mcpp:action=`: a build program DECLARES work instead +# of doing it, and the work becomes an edge in the build graph. +# +# The distinction this tests is the architectural one: a build program is a +# good place to decide what a build looks like (CONFIGURATION) and a bad place +# to perform it (WORK). Work done inside the program is serial, whole-set and +# reported as "build.mcpp exited 1". Declared as a node it is incremental, +# parallel and attributable to the edge that failed. +# +# All three wirings of the one primitive: +# source — outputs join the compile set, and are REGENERATED when an input +# changes (the property the eager path can never have) +# check — outputs are a stamp; a failing check fails the build +# artifact — inputs are link outputs, so ninja orders it after the link with +# no phase machinery at all +# +# Also: a malformed action is refused rather than silently skipped. +# +# See .agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md §3.1. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p app/src app/data +cd app + +cat > mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" +EOF + +cat > src/main.cpp <<'EOF' +#include +int generated_value(); +int main() { std::printf("VALUE=%d\n", generated_value()); } +EOF + +echo "11" > data/value.txt + +# The "generator": reads data/value.txt, writes a .cpp returning that number. +cat > gen.sh <<'EOF' +#!/usr/bin/env bash +printf 'int generated_value() { return %s; }\n' "$(cat "$1")" > "$2" +EOF +chmod +x gen.sh + +# A check that passes or fails depending on a marker file. +cat > check.sh <<'EOF' +#!/usr/bin/env bash +# The marker path is passed in: ninja runs commands with cwd = the BUILD dir, +# not the project root, so a bare relative name would never be found. +[ -f "$2" ] && { echo "check: refusing"; exit 1; } +: > "$1" +EOF +chmod +x check.sh + +cat > pack.sh <<'EOF' +#!/usr/bin/env bash +# $1 = the linked binary, $2 = the artifact to produce +cp "$1" "$2" +EOF +chmod +x pack.sh + +cat > build.mcpp <<'EOF' +#include +#include +import mcpp; +int main() { + const std::string root = mcpp::manifest_dir(); + const std::string out = std::string(mcpp::out_dir()) + "/gen.cpp"; + + mcpp::action a; + a.id = "generate"; + a.role = "source"; + a.arg((root + "/gen.sh").c_str()).arg((root + "/data/value.txt").c_str()).arg(out.c_str()) + .input((root + "/data/value.txt").c_str()) + .output(out.c_str()) + .submit(); + + mcpp::action c; + c.id = "lint"; + c.role = "check"; + c.arg((root + "/check.sh").c_str()).arg("${mcpp.out_dir}/lint.stamp") + .arg((root + "/FAIL_THE_CHECK").c_str()) + .output("${mcpp.out_dir}/lint.stamp") + .submit(); + + mcpp::action p; + p.id = "package"; + p.role = "artifact"; + p.arg((root + "/pack.sh").c_str()).arg("${mcpp.target_file:app}").arg("${mcpp.out_dir}/app.pack") + .input("${mcpp.target_file:app}") + .output("${mcpp.out_dir}/app.pack") + .submit(); +} +EOF + +# ── 1. all three roles in one build ───────────────────────────────────────── +"$MCPP" build > b1.log 2>&1 || { cat b1.log; echo "FAIL: build with actions failed"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^VALUE=' | tail -1)" +[[ "$out" == "VALUE=11" ]] || { echo "FAIL: generated source not linked: $out"; exit 1; } + +OUTDIR=$(find target -name 'app.pack' -printf '%h\n' 2>/dev/null | head -1) +[ -n "$OUTDIR" ] || { cat b1.log; echo "FAIL: artifact action produced nothing"; exit 1; } +[ -f "$OUTDIR/lint.stamp" ] || { echo "FAIL: check action produced no stamp"; exit 1; } + +# ── 2. incrementality — the property a pre-pass cannot have ──────────────── +# Changing the generator's INPUT must regenerate, without build.mcpp re-running. +echo "23" > data/value.txt +"$MCPP" build > b2.log 2>&1 || { cat b2.log; echo "FAIL: rebuild failed"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^VALUE=' | tail -1)" +[[ "$out" == "VALUE=23" ]] || { + cat b2.log; echo "FAIL: action did not re-run when its input changed: $out"; exit 1; } + +# ...and an unrelated rebuild must NOT re-run it (that is the whole point). +touch src/main.cpp +"$MCPP" build > b3.log 2>&1 || { cat b3.log; echo "FAIL: rebuild failed"; exit 1; } +grep -q "GENERATE" b3.log && { + cat b3.log; echo "FAIL: the action re-ran although its inputs were unchanged"; exit 1; } + +# ── 3. a failing check fails the build ───────────────────────────────────── +touch FAIL_THE_CHECK +rm -f "$OUTDIR/lint.stamp" +if "$MCPP" build > b4.log 2>&1; then + cat b4.log; echo "FAIL: a failing check did not fail the build"; exit 1 +fi +rm -f FAIL_THE_CHECK + +# ── 4. a malformed action is refused, not skipped ────────────────────────── +cat > build.mcpp <<'EOF' +#include +int main() { + std::printf("mcpp:protocol=1\n"); + // No `command`, no `outputs` — mcpp fixes the source set during prepare, + // so an output whose NAME is unknown cannot be built. + std::printf("mcpp:action={\"id\":\"broken\",\"role\":\"source\"}\n"); +} +EOF +rm -rf target +if "$MCPP" build > b5.log 2>&1; then + cat b5.log; echo "FAIL: a malformed action was accepted"; exit 1 +fi +grep -q "malformed action" b5.log || { + cat b5.log; echo "FAIL: no diagnostic for the malformed action"; exit 1; } + +echo "OK" diff --git a/tests/e2e/189_host_module_rules.sh b/tests/e2e/189_host_module_rules.sh new file mode 100755 index 00000000..1f27ea9b --- /dev/null +++ b/tests/e2e/189_host_module_rules.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# requires: gcc +# 189_host_module_rules.sh — reusable build rules distributed as ORDINARY mcpp +# packages: `host-module = true` makes a dependency's module importable from the +# consumer's build.mcpp. +# +# This is the answer to "a rule should be written once, not copy-pasted into +# every package's build.mcpp" — without introducing a second language. xmake +# reaches for Lua rules and Bazel for Starlark; mcpp's whole premise is that +# the build is written in C++, so a rule is a C++ module in a versioned package +# and rides the package manager that already exists. +# +# The load-bearing implementation detail this test pins: the rule module is +# compiled in the SAME command as build.mcpp, with the same flags. A BMI is +# only usable by a compile that agrees with it on standard, dialect and +# compiler identity — building it separately would leave that to chance, and +# disagreement surfaces as `module X CRC mismatch` rather than a clear error. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +# ── the rule package ──────────────────────────────────────────────────────── +# An ordinary mcpp library package whose lib root is the rule module. +mkdir -p rules/src +cat > rules/mcpp.toml <<'EOF' +[package] +name = "rules" +version = "0.1.0" + +[targets.rules] +kind = "lib" +EOF +cat > rules/src/rules.cppm <<'EOF' +module; +#include +export module rules; +export namespace rules { +// A rule: emits the directives its users would otherwise hand-write. Written +// ONCE here instead of copy-pasted into every consumer's build.mcpp. +inline void banner(const char* macro) { + std::printf("mcpp:cxxflag=-D%s=1\n", macro); +} +inline void define_answer(int n) { + std::printf("mcpp:cxxflag=-DRULE_ANSWER=%d\n", n); +} +} +EOF + +# ── the consumer ──────────────────────────────────────────────────────────── +mkdir -p app/src +cat > app/mcpp.toml <<'EOF' +[package] +name = "app" +version = "0.1.0" + +[dependencies] +rules = { path = "../rules", host-module = true } +EOF +cat > app/src/main.cpp <<'EOF' +#include +#ifndef RULE_ANSWER +#error "the rule package's directive never reached the compile" +#endif +#ifndef RULE_BANNER_OK +#error "the second rule directive never reached the compile" +#endif +int main() { std::printf("ANSWER=%d\n", RULE_ANSWER); } +EOF +cat > app/build.mcpp <<'EOF' +import mcpp; +import rules; // a DEPENDENCY's module, compiled for the host +int main() { + rules::banner("RULE_BANNER_OK"); + rules::define_answer(42); +} +EOF + +cd app +"$MCPP" build > b1.log 2>&1 || { cat b1.log; echo "FAIL: build with a host rule module failed"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" +[[ "$out" == "ANSWER=42" ]] || { + echo "FAIL: the rule package's directives did not reach the build: $out"; exit 1; } + +# Editing the RULE must re-run build.mcpp: the rule's content is part of what +# the helper was compiled from, so a cached run would silently keep the old +# behaviour — the exact failure the declared-input cache exists to prevent. +sed -i 's/define_answer(int n)/define_answer(int n_)/; s/RULE_ANSWER=%d\\n", n)/RULE_ANSWER=%d\\n", n_ + 1)/' ../rules/src/rules.cppm +touch src/main.cpp +"$MCPP" build > b2.log 2>&1 || { cat b2.log; echo "FAIL: rebuild after editing the rule failed"; exit 1; } +grep -q "build.mcpp running" b2.log || { + cat b2.log; echo "FAIL: editing the rule module did not re-run build.mcpp"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" +[[ "$out" == "ANSWER=43" ]] || { + echo "FAIL: the edited rule did not take effect: $out"; exit 1; } + +# A missing lib root must say so, not fail three edges later. +cd "$TMP" +mv rules/src/rules.cppm rules/src/elsewhere.cppm +cd app && rm -rf target +if "$MCPP" build > b3.log 2>&1; then + cat b3.log; echo "FAIL: a host module with no interface unit was accepted"; exit 1 +fi +grep -q "no interface unit" b3.log || { + cat b3.log; echo "FAIL: unhelpful diagnostic for a missing rule interface"; exit 1; } + +echo "OK" From f3c7214f69f76017b8837c855e9e85cb2c3b814b Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:19:32 +0800 Subject: [PATCH 04/16] =?UTF-8?q?docs:=20=E8=AE=B0=E5=BD=95=E6=AD=A5=202?= =?UTF-8?q?=E2=80=936=20=E7=9A=84=E5=AE=9E=E6=96=BD=E3=80=81=E4=BA=94?= =?UTF-8?q?=E4=B8=AA=E5=BC=80=E6=94=BE=E9=97=AE=E9=A2=98=E7=9A=84=E5=AE=9A?= =?UTF-8?q?=E6=A1=88=E4=B8=8E=E4=B8=A4=E4=B8=AA=E7=A1=AC=E5=9D=91?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 架构文档补 §10: - §8 五个开放问题全部定案(封闭命令词表 / JSON 载荷扩展 / check 默认并行 / mcpp.rules.* 只是约定 / 第 5 问问错了) - 两个实现坑写进文档而不是留在 commit 里: · `${mcpp.target_file:}` 必须解析成 build-dir 相对路径 —— ninja 按「边声明的 那个字符串」标识文件,指向同一份字节的绝对路径是**另一个节点** · 规则包的 BMI 不能走子构建 —— 两次独立解析的构建没有理由在 standard/dialect/ 编译器身份上一致,而不一致的表现是 CRC mismatch 不是清楚的错误 - 步 6 的核实结论:问的问题本身错了。阻塞点不是 topoOrder 的顺序,而是未扫描的 文件根本没有 graph.units 条目;解法用代码库已有的「声明而非发现」 --- ...5-build-mcpp-extensibility-architecture.md | 88 +++++++++++++++++-- 1 file changed, 82 insertions(+), 6 deletions(-) diff --git a/.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md b/.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md index 4be40dfe..9da2b72f 100644 --- a/.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md +++ b/.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md @@ -1,6 +1,6 @@ # build.mcpp 机制架构设计：一个 hook，多种节点 -> 状态：**步 0 + 步 1 已实施（2026.8.5.1）**；步 2–6 待 review。实施记录见 §9。 +> 状态：**步 0–6 全部已实施（2026.8.5.1）**。实施记录见 §9（步 0+1）与 §10（步 2–6）。 > 范围：`build.mcpp` 作为**扩展机制**的长期形态 —— 不是某个具体特性 > 关联：#355（依赖产出的 host 工具，是本路线的第一块地基）、#241、L3 原始设计 > （`.agents/docs/2026-06-30-l3-build-mcpp-implementation-design.md`） @@ -307,11 +307,11 @@ L3 分发层（mcpp pack） |---|---|---|---|---| | 0 | 补 S1–S4：协议版本 / cache epoch / 超时 / 冻结裸 printf | — | 无（S4 是文档 + 停止扩表） | **已实施 2026.8.5.1** | | 1 | S5：directive 定义表收敛 | — | 无（纯内部重构，可用「产物逐字节相同」验证） | **已实施 2026.8.5.1** | -| 2 | #355：host 工具 + 工作目录外置 | 0, 1 | 无 | 待 review | -| 3 | `action` 原语，先只做 `role=source`（即 codegen 进图） | 2 | 无（新增） | 待 review（§8 未决） | -| 4 | `role=check`（静态分析）、`role=artifact`（后处理） | 3 | 无（新增） | 待 review | -| 5 | build.mcpp 可 `import` 依赖提供的 host 模块 → **规则包生态** | 2 | 无（新增） | 待 review | -| 6 | 核实并（若成立）解除「生成的 `.cppm` 必须 eager」限制 | 3 | 无 | 待核实 | +| 2 | #355：host 工具 + 工作目录外置 | 0, 1 | 无 | **已实施 2026.8.5.1** | +| 3 | `action` 原语，先只做 `role=source`（即 codegen 进图） | 2 | 无（新增） | **已实施 2026.8.5.1** | +| 4 | `role=check`（静态分析）、`role=artifact`（后处理） | 3 | 无（新增） | **已实施 2026.8.5.1** | +| 5 | build.mcpp 可 `import` 依赖提供的 host 模块 → **规则包生态** | 2 | 无（新增） | **已实施 2026.8.5.1** | +| 6 | 核实并（若成立）解除「生成的 `.cppm` 必须 eager」限制 | 3 | 无 | **已核实并解决（§10.4）** | 步 0/1 值得优先，因为它们**成本最低而收益随时间递增**：directive 表越长，补的代价越大。 @@ -399,3 +399,79 @@ L3 分发层（mcpp pack） > 挡在前面 —— 无变更的第二次构建走 fast path，**根本不会读 build.mcpp 缓存**。 > 验证缓存行为必须先 `touch` 一个源文件把 fast path 打掉，否则会把「fast path 生效」 > 误读成「缓存未命中」。我第一次就是这么误判的。 + +--- + +## 10. 实施记录（步 2–6，2026.8.5.1） + +### 10.1 §8 五个开放问题的定案 + +| # | 问题 | 定案 | 理由 | +|---|---|---|---| +| 1 | `action.command` 的表达力 | **封闭词表**：argv + `${mcpp.out_dir}` / `${mcpp.bin_dir}` / `${mcpp.compile_db}` / `${mcpp.target_file:}` | 不假设存在 shell（Windows 没有可依赖的那个），且没有东西能夹带环境状态进来 —— 可移植性与可缓存性同一个理由 | +| 2 | 结构化载荷格式 | **在既有平坦协议上扩展 `mcpp:action={json}`** | action 有六个字段，平坦 `key=value` 表达不了；内置模块负责编码，与 S4「`import mcpp;` 是唯一演进面」自洽 | +| 3 | `role=check` 默认挂载 | **默认并行**，`blocking = true` 才前置 | 把整条编译串行化在 linter 后面是没人接受的代价，而「构建最终失败」的效果一样 | +| 4 | 规则包命名空间 | `mcpp.rules.*` 作为约定前缀，机制上不特殊 | 规则包就是普通包，特殊化它只会多一套规则 | +| 5 | dyndep 下 topoOrder 是否冗余 | **问题问错了** —— 见 §10.4 | | + +### 10.2 host 工具（步 2）：实现要点与两个坑 + +- **工作目录外置是硬前置**，且必须**五处一起搬**（`target/`、`mcpp.lock`、 + `compile_commands.json`、`.mcpp/`、`target/.build-mcpp`）。只搬一部分比一处都 + 不搬更糟：那等于照样写进共享的注册表包根，只是更不显眼。 +- **`${mcpp.target_file:NAME}` 必须解析成 build-dir 相对路径，不是绝对路径。** + ninja 用「边声明的那个字符串」标识文件，link 边声明的是 `bin/app`；指向同一份 + 字节的绝对路径是**另一个节点**，ninja 报 + `missing and no known rule to make it`。第一次实现取了绝对路径，e2e 立刻炸。 + 顺带确认了 action 命令的 cwd 是 build dir。 +- 子构建通过 `mcpp.build.ninja` 的 backend 驱动，**不能**走 `execute.cppm` + （它 import 了 prepare，反过来会成环）。`prepare_build` 全函数只有一处 `static`， + 递归重入是安全的。 + +### 10.3 `action`（步 3+4）：为什么占位文件是对的 + +role=source 的产物在 prepare 期不存在，而 modgraph 扫描要 glob 磁盘。选择是 +**播下占位文件**而不是凭空合成 CompileUnit：这样 glob 找得到它、scanner 读得到它、 +plan 给得出对象路径、ninja 在编译边之前用真实内容覆盖它（因为那条编译**依赖** +action 的输出）。整条链路复用现有机制，没有一处特判。 + +占位文件**绝不截断已存在的文件** —— 第一次构建之后那里是真实内容，重写它会让 ninja +以为输入每次 prepare 都变了。 + +### 10.4 步 6 的核实结论：问题问错了 + +设计里问的是「dyndep 模式下 `prepare.cppm:4239` 传给 `make_plan` 的 `topoOrder` +是否冗余」。核实后：`topoOrder` 在 `plan.cppm` 有两处用途 —— `:708` 的名字消歧 +普查（**与顺序无关**，只是遍历全部已扫描单元）和 `:829` 的 CompileUnit 发射次序。 + +**但真正的阻塞点根本不是顺序**，而是：一个没有被扫描过的文件**根本没有 +`graph.units` 条目**，于是没有 CompileUnit，于是不会被编译。顺序是不是冗余，与 +它无关。 + +解法用代码库**已有的**答案 ——「声明而非发现」，即 `[modules].scan_overrides` 早就 +做过的那个取舍：action 用 `.provides()/.imports()` 声明生成模块的接口，mcpp 播下 +带该声明的占位文件，prepare 期的扫描因此与生成器将要产出的内容一致，而 build 期 +由编译器自己的 P1689 复核这条声明。**限制解除，且没有动 topoOrder 一行。** + +### 10.5 规则包（步 5）：为什么不走 tool store + +最初的直觉是「像 host 工具一样，用子构建产出 BMI + 对象，放进 store」。那是**错的**： +BMI 只对「在 standard / dialect / 编译器身份上与它一致」的编译可用，而两次独立解析 +的构建**没有理由**一致 —— 消费者的 `standard` 与规则包自己的 `standard` 就可以不同。 +不一致的表现是 `module X CRC mismatch`，不是一条清楚的错误，而这个代码库为这一族 +问题付过多次学费。 + +改成**与 build.mcpp 同一条命令、同一套 flag 编译**，一致性就从「需要验证的性质」 +变成了**结构性事实**。代价是规则接口单独编译，只能 import `std` 与内置 `mcpp` 模块 +—— 一个规则包按构造就是叶子，这个限制写进文档而不是藏起来。 + +### 10.6 验证 + +- 单测 56/56 +- e2e 19/21；`07_static_library`（本机 binutils payload 的 `ar` 跑不起来）与 + `09_path_dependency`（`ninja missing dep BMI`）在**已发布的 2026.8.4.1 上同样 + 失败** ⇒ 环境性，非回归（判定回归前先跑已发布二进制做对照） +- 新增 3 个 e2e：**187**（端到端 / 成本门 / 默认关闭 / 错名列出可用 target / + override 生效）、**188**（三种 role + 输入变则重生成 + 无关重建不重跑 + + 失败的 check 让构建失败 + 畸形 action 被拒）、**189**（规则包导入生效 / 编辑规则 + 触发 build.mcpp 重跑 / 缺 lib root 的诊断） From beb25cd5f4271476c350ac675305806784d2e6e5 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:23:53 +0800 Subject: [PATCH 05/16] =?UTF-8?q?test:=20=E8=A6=86=E7=9B=96=20Form=20B=20?= =?UTF-8?q?=E6=8F=8F=E8=BF=B0=E7=AC=A6=E7=9A=84=E4=B8=A4=E5=A4=84=E8=A7=A3?= =?UTF-8?q?=E6=9E=90=E7=BC=BA=E5=8F=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #355 需要 xpkg (.lua) 解析器读两个它此前**根本没读**的键,而两处都是静默的 ——没读的键就只是「这个特性不存在」: - `targets..required_features`:描述符因此无法表达让可选 host 工具变得 可负担的成本门(compat.protobuf 的 protoc 会带进 libprotoc 的 ~157 个 TU, 只要运行时的消费者绝不能编译它)。没有这个门,target 要么总是构建、 要么根本拿不到。 - `deps` 的值此前只能是版本**字符串**,于是 Form B 描述符压根没有语法去向 自己的依赖请求工具。 之前的 e2e 只走了 Form A(path 依赖)路径,这两处没有任何覆盖。第三个用例钉住 「未知 dep 键要被记录而不是吞掉」——半配置的依赖且毫无信号,正是既有的 per-feature 键记录机制要防的那种失败。 --- tests/unit/test_xpkg_host_tools.cpp | 126 ++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 tests/unit/test_xpkg_host_tools.cpp diff --git a/tests/unit/test_xpkg_host_tools.cpp b/tests/unit/test_xpkg_host_tools.cpp new file mode 100644 index 00000000..6bb8b32e --- /dev/null +++ b/tests/unit/test_xpkg_host_tools.cpp @@ -0,0 +1,126 @@ +#include + +import std; +import mcpp.manifest; +import mcpp.manifest.xpkg; +import mcpp.platform; +import mcpp.platform.axis; + +// #355 needed two things from the Form B (xpkg .lua) descriptor parser that it +// simply did not read, and neither failed loudly: +// +// * `targets..required_features` was dropped, so a descriptor could not +// express the cost gate that makes an optional host tool affordable — +// compat.protobuf's `protoc` pulls in libprotoc's ~157 extra TUs, which the +// consumers who only want the runtime must never compile. Without the gate +// the target is either always built or never available. +// * `deps` values could only be a version STRING, so a Form B descriptor had +// no syntax at all for requesting a tool from one of its own dependencies. +// +// Both were silent: an unread key is just an absent feature. These tests are +// what makes them loud. + +namespace { + +mcpp::manifest::Manifest parse_or_fail(std::string_view lua) { + auto m = mcpp::manifest::synthesize_from_xpkg_lua( + lua, "compat.demo", "1.0.0", mcpp::platform::HostPlatform::current()); + EXPECT_TRUE(m.has_value()) << (m ? "" : m.error().message); + return m.value_or(mcpp::manifest::Manifest{}); +} + +const mcpp::manifest::Target* find_target(const mcpp::manifest::Manifest& m, + std::string_view name) { + for (auto const& t : m.targets) + if (t.name == name) return &t; + return nullptr; +} + +} // namespace + +TEST(XpkgHostTools, TargetsCarryRequiredFeatures) { + auto m = parse_or_fail(R"LUA( +package = { + spec = "1", name = "demo", namespace = "compat", type = "package", + mcpp = { + sources = { "*/src/**.cc" }, + targets = { + ["demo"] = { kind = "lib" }, + ["protoc"] = { kind = "bin", main = "src/compiler/main.cc", + required_features = { "protoc", "upb" } }, + }, + }, +} +)LUA"); + + auto const* lib = find_target(m, "demo"); + ASSERT_NE(lib, nullptr); + EXPECT_EQ(lib->kind, mcpp::manifest::Target::Library); + EXPECT_TRUE(lib->requiredFeatures.empty()); + + auto const* tool = find_target(m, "protoc"); + ASSERT_NE(tool, nullptr); + EXPECT_EQ(tool->kind, mcpp::manifest::Target::Binary); + EXPECT_EQ(tool->main, "src/compiler/main.cc"); + EXPECT_EQ(tool->requiredFeatures, + (std::vector{"protoc", "upb"})); +} + +TEST(XpkgHostTools, DepsAcceptBothAStringAndATable) { + // The string form is the long-standing one and must keep working + // unchanged; the table form is what lets a descriptor request a tool. + auto m = parse_or_fail(R"LUA( +package = { + spec = "1", name = "demo", namespace = "compat", type = "package", + mcpp = { + sources = { "*/src/**.cc" }, + deps = { + ["compat.zlib"] = "1.3.2", + ["compat.protobuf"] = { version = "35.1", tools = { "protoc" } }, + }, + }, +} +)LUA"); + + const mcpp::manifest::DependencySpec* zlib = nullptr; + const mcpp::manifest::DependencySpec* pb = nullptr; + for (auto const& [k, spec] : m.dependencies) { + if (k.find("zlib") != std::string::npos) zlib = &spec; + if (k.find("protobuf") != std::string::npos) pb = &spec; + } + + ASSERT_NE(zlib, nullptr); + EXPECT_EQ(zlib->version, "1.3.2"); + EXPECT_TRUE(zlib->tools.empty()); + + ASSERT_NE(pb, nullptr); + EXPECT_EQ(pb->version, "35.1"); + EXPECT_EQ(pb->tools, (std::vector{"protoc"})); +} + +TEST(XpkgHostTools, UnknownDepKeyIsRecordedRatherThanSwallowed) { + // A descriptor author writing an unsupported key must be told. Silently + // ignoring it leaves the dependency half-configured with no signal — the + // exact failure the per-feature key recording already exists to prevent. + auto m = parse_or_fail(R"LUA( +package = { + spec = "1", name = "demo", namespace = "compat", type = "package", + mcpp = { + sources = { "*/src/**.cc" }, + deps = { + ["compat.protobuf"] = { version = "35.1", no_such_key = "x" }, + }, + }, +} +)LUA"); + + bool recorded = false; + for (auto const& k : m.xpkgUnknownKeys) + if (k.find("no_such_key") != std::string::npos) recorded = true; + EXPECT_TRUE(recorded) << "unknown dep key was swallowed"; + + // ...and the keys it DOES understand still take effect. + for (auto const& [k, spec] : m.dependencies) + if (k.find("protobuf") != std::string::npos) + EXPECT_EQ(spec.version, "35.1"); +} From 8ed92f67b9b62d46b6311a2fe5583bc0a2c7f8e6 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:25:36 +0800 Subject: [PATCH 06/16] =?UTF-8?q?docs(zh):=20=E5=90=8C=E6=AD=A5=20host=20?= =?UTF-8?q?=E5=B7=A5=E5=85=B7=E3=80=81=E6=9E=84=E5=BB=BA=E5=9B=BE=E8=8A=82?= =?UTF-8?q?=E7=82=B9=E3=80=81=E8=A7=84=E5=88=99=E5=8C=85=E4=B8=89=E8=8A=82?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/zh/05 §2.14 与 docs/zh/07 的 dep_bin / action 两节 —— 与英文版一一对应。 --- docs/zh/05-mcpp-toml.md | 75 +++++++++++++++++++++++++++++++++++++ docs/zh/07-build-mcpp.md | 81 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 156 insertions(+) diff --git a/docs/zh/05-mcpp-toml.md b/docs/zh/05-mcpp-toml.md index 0a3b7041..3c1ebe58 100644 --- a/docs/zh/05-mcpp-toml.md +++ b/docs/zh/05-mcpp-toml.md @@ -724,6 +724,81 @@ OPENBLAS_NUM_THREADS = "1" host 工具(`make`/`cmake`/`protoc`…)、按项目固定工具版本、或设构建期环境变量——无需手改 `.xlings.json`。`[toolchain]`(§2.7)仍是编译器的便捷简写;`[xlings.workspace]` 是其通用形式。 +### 2.14 依赖产出的 host 工具(mcpp 2026.8.5.1+) + +一个包能构建出消费者在**构建期**需要的二进制 —— `protoc`、`grpc_cpp_plugin`、 +`flatc`、`moc`、转译器。在依赖上声明: + +```toml +[dependencies] +protobuf = { version = "35.1", tools = ["protoc"] } +grpc = { version = "1.83.0", tools = ["grpc_cpp_plugin"] } +``` + +每个名字必须是该包的一个 `kind = "bin"` target。mcpp 会**为构建机器**构建它, +并把绝对路径以 `MCPP_DEP__BIN_` 交给 `build.mcpp` —— 用 +`mcpp::dep_bin("protobuf", "protoc")` 读取(见 [07 — build.mcpp](07-build-mcpp.md))。 + +四条值得知道的性质: + +- **永远是 host 二进制。** 即使 `mcpp build --target `,工具依然为**本机** + 构建 —— 代码生成器必须在这里跑。它是一次独立的、面向 host 的子构建:工具包 + 自己的 `[toolchain]`、自己的依赖解析生效,不需要与你的构建一致。之所以安全, + 是因为可执行文件与你的代码**零 ABI 接触**。 +- **单一版本轴。** 工具的版本**就是**依赖的版本,所以「protoc 与其运行时不匹配」 + 这种情况**不可表达**。(把工具单独打包正是会出这个问题,而且它在**运行期**才咬人, + 不是编译期。) +- **默认关闭。** 没人要就什么都不构建,成本由消费者付。包用 `[features]` + + `required_features` 给昂贵的部分加门(protobuf 的 `protoc` 需要 libprotoc 的 + ~157 个额外 TU,只用运行时的人绝不该编译它)。 +- **全局缓存**,按 包版本 × host 工具链 × feature × 自身依赖闭包 键控 —— 每台机器 + 构建一次,而不是每个工程一次。 + +#### `[tools.overrides]` —— 用你已经有的二进制 + +```toml +[tools.overrides] +"compat.protobuf:protoc" = "/usr/bin/protoc" +``` + +或者不改 manifest(CI、发行版打包): + +```bash +MCPP_TOOL_PROTOBUF_PROTOC=/usr/bin/protoc mcpp build +``` + +命中 override 会**完全跳过构建**。每个同类系统都提供这条逃生舱(vcpkg 的 +`VCPKG_HOST_TRIPLET`、CMake 的 `LLVM_NATIVE_TOOL_DIR`、Qt 的 `QT_HOST_PATH`), +理由一样:一个在本机构建不出来的工具**不能是死路**。它**刻意不进** cache key —— +逃生舱不是可复现输入。 + +#### `host-module = true` —— 可复用的构建规则以包分发 + +一条规则(比如「对这些 `.proto` 跑 protoc」)应该**写一次**,而不是复制进每个 +消费者的 `build.mcpp`。把它做成普通的 mcpp 库包再 import: + +```toml +[dependencies] +"mcpp.rules.protobuf" = { version = "0.1.0", host-module = true } +``` + +```cpp +// build.mcpp +import mcpp; +import mcpp.rules.protobuf; +int main() { mcpp::rules::protobuf::generate(/* … */); } +``` + +mcpp 会把该包的 lib 根模块**为 host 编译,且与 `build.mcpp` 在同一条命令里** —— +这正是 BMI 能用的前提:一个模块接口只对「在 standard / dialect / 编译器身份上与 +它一致」的编译可导入。 + +于是规则**有版本、能测试、能通过你已有的包管理器分发**,而且是用 **C++** 写的 +—— 不引入第二门语言,这正是 `build.mcpp` 存在的理由。 + +*限制:* 规则接口是单独编译的,因此可以 import `std` 与内置 `mcpp` 模块, +但不能 import 第三个包。规则包按构造是叶子。 + ## 附录 A. Schema 所有权原则(新字段准入标准) > **语法封闭,词汇开放**:谁拥有解析语义谁定义键;谁拥有领域知识谁定义值。 diff --git a/docs/zh/07-build-mcpp.md b/docs/zh/07-build-mcpp.md index 05899a34..6b5e63b6 100644 --- a/docs/zh/07-build-mcpp.md +++ b/docs/zh/07-build-mcpp.md @@ -90,6 +90,87 @@ int main() { | `mcpp::source(p)` | `mcpp:source=` | | `mcpp::include_dir(d)` / `mcpp::include_dir_after(d)` | `mcpp:include-dir=` / `mcpp:include-dir-after=` | | `mcpp::rerun_if_changed(p)` / `mcpp::rerun_if_env_changed(v)` | 对应的 `rerun-*` 指令 | +| `mcpp::dep_bin(pkg, tool)` *(2026.8.5.1+)* | 读 `MCPP_DEP__BIN_` —— 依赖构建出的 **host 工具**的绝对路径(见下) | +| `mcpp::action{…}.submit()` *(2026.8.5.1+)* | `mcpp:action=` —— **声明一个构建图节点**,而不是在这里把活干了(见下) | + +### 依赖产出的 host 工具(2026.8.5.1+) + +在 `mcpp.toml` 里声明需求,然后调用它: + +```toml +[dependencies] +protobuf = { version = "35.1", tools = ["protoc"] } +``` + +```cpp +// build.mcpp +import mcpp; +int main() { + const char* protoc = mcpp::dep_bin("protobuf", "protoc"); + // … 调用它,然后声明它产出了什么 … +} +``` + +mcpp 会**为构建机器**构建那个 `kind = "bin"` target(即使在 `--target` 下), +全局缓存,并把路径交给你。这个请求写在 `mcpp.toml` 而不是这里,理由和依赖本身 +一样:向依赖图索取一个额外产物是**图级别**的请求,而图必须保持可静态分析。 +完整契约(含 `[tools.overrides]`)见 [05 §2.14](05-mcpp-toml.md)。 + +### 声明工作而不是干活:`mcpp::action`(2026.8.5.1+) + +在**这里**直接把源码写出来是省事的路,超过一定规模就是错的:它每次 prepare 跑 +一遍、全量、串行,失败还只报「build.mcpp exited 1」。**声明**这份工作,它就成为 +构建图里的一条边 —— 增量、并行,失败能归因到具体那条边。 + +```cpp +import mcpp; +int main() { + const std::string out = std::string(mcpp::out_dir()) + "/foo.pb.cc"; + mcpp::action a; + a.id = "protoc:foo"; + a.role = "source"; // "source" | "check" | "artifact" + a.arg(mcpp::dep_bin("protobuf", "protoc")) + .arg("--cpp_out=...").arg("proto/foo.proto") + .input("proto/foo.proto") + .output(out.c_str()) + .submit(); +} +``` + +三种 role,一个原语 —— `role` 只决定这条边的输出接到哪: + +| `role` | 输出 | 顺序 | 典型 | +|---|---|---|---| +| `source` | 进编译集 | 编译边消费它们 | protoc、转译器 | +| `check` | 一个 stamp 文件 | **与编译并行**(`blocking = true` 才前置) | clang-tidy、格式/ABI 检查 | +| `artifact` | 一个新文件 | 它的**输入**是链接产物,所以在链接之后跑 | 签名、打包、size budget | + +全程不涉及任何 phase 机制:顺序由 ninja 自己的文件依赖决定 —— 这也是为什么 +`artifact` 不会像朴素的「post 构建钩子」那样把自己重复施加一遍。 + +**必须写出输出文件名。** mcpp 在 prepare 期就定死源码集、fingerprint 与模块图, +所以名字未知的产物无法构建。内容可以晚到,名字不行。畸形 action 是**硬错误**, +绝不静默跳过。 + +生成**模块接口**时,把它的接口也声明出来: + +```cpp +a.output(gen.c_str()).provides("my.generated").imports("std").submit(); +``` + +mcpp 会播下一个带着该声明的占位文件,使 prepare 期的扫描与你的生成器将要产出的 +内容一致 —— 与 `[modules].scan_overrides` 同一条「声明 + 验证」的取舍,build 期由 +编译器自己的 P1689 输出复核。 + +命令是 **argv 而不是 shell 字符串**(不假设存在 shell —— Windows 没有能依赖的那个), +插值只有封闭的一组: + +| 变量 | 含义 | +|---|---| +| `${mcpp.out_dir}` | 构建输出目录 | +| `${mcpp.bin_dir}` | 产出的二进制所在目录 | +| `${mcpp.compile_db}` | `compile_commands.json` 的路径(clang-tidy 的 `-p` 要的就是它) | +| `${mcpp.target_file:}` | target `` 构建出的文件 | 上面的裸 stdout 协议仍是底层基底;`import mcpp;` 是其上的类型化层。 From 143c612182634b279a9071907022f8b5bcaefb19 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:29:53 +0800 Subject: [PATCH 07/16] =?UTF-8?q?test(e2e):=20=E9=92=89=E4=BD=8F=E3=80=8C?= =?UTF-8?q?=E6=94=B9=20action=20=E7=9A=84=E5=91=BD=E4=BB=A4=E3=80=8D?= =?UTF-8?q?=E4=B9=9F=E4=BC=9A=E7=94=9F=E6=95=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 与「改 action 的输入」是两条不同的路径,必须都成立,否则改了生成器的调用方式会被 静默忽略: - 改**输入** → ninja 直接看到,重跑那条边 - 改 **build.mcpp** → ninja 跟踪的东西一样没变;是 build.mcpp 自己重跑(源码在它 的缓存键里)、重新声明出一条命令不同的 action,ninja 再因为 rule 的 command 变了而重跑 只测前者会让后者的回归完全隐形。 --- tests/e2e/188_build_actions.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/tests/e2e/188_build_actions.sh b/tests/e2e/188_build_actions.sh index 6538d17e..e2a16a59 100755 --- a/tests/e2e/188_build_actions.sh +++ b/tests/e2e/188_build_actions.sh @@ -123,6 +123,19 @@ touch src/main.cpp grep -q "GENERATE" b3.log && { cat b3.log; echo "FAIL: the action re-ran although its inputs were unchanged"; exit 1; } +# ── 2b. changing the action's COMMAND also takes effect ──────────────────── +# Distinct from 2: there the action's declared INPUT changed and ninja noticed. +# Here nothing ninja tracks changed — only build.mcpp, which re-runs (its own +# source is part of its cache key), re-declares the action with a different +# command, and ninja re-runs the edge because the rule's command changed. Both +# halves have to work or an edited generator invocation is silently ignored. +sed -i 's|"/data/value.txt"|"/data/other.txt"|g' build.mcpp +echo "31" > data/other.txt +"$MCPP" build > b2b.log 2>&1 || { cat b2b.log; echo "FAIL: rebuild after editing build.mcpp failed"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^VALUE=' | tail -1)" +[[ "$out" == "VALUE=31" ]] || { + cat b2b.log; echo "FAIL: an edited action command did not take effect: $out"; exit 1; } + # ── 3. a failing check fails the build ───────────────────────────────────── touch FAIL_THE_CHECK rm -f "$OUTDIR/lint.stamp" From 00847b85dde547db207cb2ab787d4bcec379a7a0 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:31:57 +0800 Subject: [PATCH 08/16] =?UTF-8?q?fix(tool-store):=20upstream=20key=20?= =?UTF-8?q?=E8=B5=B0=E4=BC=A0=E9=80=92=E9=97=AD=E5=8C=85,=E4=B8=8E?= =?UTF-8?q?=E6=B3=A8=E9=87=8A=E5=A3=B0=E7=A7=B0=E7=9A=84=E4=B8=80=E8=87=B4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 注释写的是「recursive (Merkle)」,实现只遍历了直接边 —— 注释**多声称了一个它没有 的性质**,这比没有注释更糟。 两个选择:改注释、或让代码兑现。选后者,因为传递闭包确实更对:对索引包来说直接边 够用(冻结的版本改不了自己的依赖),但 **path 依赖可以** —— 往下两层改一处,工具的 直接依赖列表纹丝不动,于是 store 里留下一个陈旧的二进制。那是**静默的错误产物**, 本项目为这一族问题付过不止一次学费,而闭包遍历几乎不要钱。 --- src/build/prepare.cppm | 29 ++++++++++++++++++++++++----- src/build/tool_store.cppm | 13 +++++++++---- 2 files changed, 33 insertions(+), 9 deletions(-) diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index a3dd1f81..89ffb881 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -4038,11 +4038,30 @@ prepare_build(bool print_fingerprint, key.profile = "release"; key.features = closure; std::ranges::sort(key.features); - for (auto const& edge : dependencyEdges) { - if (edge.consumerPackageIndex != depIdx) continue; - auto const& up = packages[edge.dependencyPackageIndex]; - key.upstreamKeys.push_back(std::format("{}@{}", - up.manifest.package.name, up.manifest.package.version)); + // The tool package's TRANSITIVE dependency closure, not just + // its direct edges. Direct-only would be enough for index + // packages (a frozen version cannot change its own deps), + // but a path dependency can: bump something two levels down + // and the tool's direct list is unchanged, so a stale binary + // stays in the store. That is a silently wrong artifact — + // the failure mode this project has paid for more than once + // — and the closure walk costs nothing. + { + std::set seen{depIdx}; + std::vector queue{depIdx}; + while (!queue.empty()) { + auto cur = queue.back(); + queue.pop_back(); + for (auto const& edge : dependencyEdges) { + if (edge.consumerPackageIndex != cur) continue; + auto up = edge.dependencyPackageIndex; + if (!seen.insert(up).second) continue; + queue.push_back(up); + key.upstreamKeys.push_back(std::format("{}@{}", + packages[up].manifest.package.name, + packages[up].manifest.package.version)); + } + } } std::ranges::sort(key.upstreamKeys); diff --git a/src/build/tool_store.cppm b/src/build/tool_store.cppm index cbc1da3b..5f15b4ba 100644 --- a/src/build/tool_store.cppm +++ b/src/build/tool_store.cppm @@ -84,10 +84,15 @@ struct Key { std::string compilerIdentity; // the HOST toolchain that will build it std::string profile; std::vector features; // resolved closure, sorted - // The cache key of each direct dependency of the TOOL package, recursively - // (Merkle). Without it, bumping one of protobuf's own dependencies would - // leave a stale protoc in the store — a silently wrong artifact, which is - // the failure mode this project has paid for more than once. + // The tool package's TRANSITIVE dependency closure, as sorted + // `@` entries. Without it, bumping something the tool + // depends on leaves a stale binary in the store — a silently wrong + // artifact, the failure mode this project has paid for more than once. + // + // Transitive rather than direct-only: for an index package a frozen + // version cannot change its own dependencies, so direct edges would do — + // but a PATH dependency can, and then a change two levels down leaves the + // tool's direct list untouched. std::vector upstreamKeys; }; From c026ea37adc2e726eb2a060a9aed170ef744cb06 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:37:21 +0800 Subject: [PATCH 09/16] =?UTF-8?q?refactor:=20=E8=BE=B9=E5=9B=BE=E6=9F=A5?= =?UTF-8?q?=E8=AF=A2=E6=94=B6=E6=95=9B=E6=88=90=20mcpp.build.dep=5Fgraph,?= =?UTF-8?q?=E8=80=8C=E4=B8=8D=E6=98=AF=E5=86=8D=E6=89=8B=E5=86=99=E4=B8=80?= =?UTF-8?q?=E9=81=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 自审时发现的问题,而且正是本 PR 通篇在反对的那个模式:我给 tool store 的 upstream key **手写了一遍图遍历**,而 prepare.cppm 里**已经有**一个走同一张边图的递归遍历 (`self(self, …)`,构建缓存的 per-package key),带 memo 和环检测 —— 我那份是它的 一个更弱的副本。 核实后:`dependencyEdges` 有 14 处读者,其中大多数问的是同两个问题 ——「X 直接依赖 谁」和「X 的传递闭包」。两个 build.mcpp 调用点(发 `MCPP_DEP__DIR` 的那两处) 已经漂成了近乎逐字重复的两份,而 #355 正要再加第三个变体。 新增 `src/build/dep_graph.cppm`: - `direct_dependencies` / `transitive_dependencies`(按边类型模板化 —— `DependencyEdge` 是 prepare_build 里的局部结构体,把它搬出来的代价远大于这次要 还的债) - `name_spellings`:一个依赖可被寻址的两种拼写(canonical 与去 namespace 的尾段)。 不是图查询,但放这里理由相同:三处各自展开过同一个 rfind 分割,第四处正要出现。 **刻意不迁移**构建缓存那条递归 fold:它不是闭包查询 —— 它对每个节点算一个值(该包 完整的 cache key)、把环当**硬错误**而不是跳过、还顺带穿了一个 taint 标记。把它塞进 通用遍历要么丢掉这些性质,要么让抽象一路长到只描述一个调用者。**回答不同问题的两次 遍历不是重复。** 验证:单测 57/57;125/145(正是覆盖 MCPP_DEP_*_DIR 的两个)与 111/186/187/188/189 全通过。 --- src/build/dep_graph.cppm | 98 ++++++++++++++++++++++++++++++++++++++++ src/build/prepare.cppm | 96 ++++++++++++++++----------------------- 2 files changed, 138 insertions(+), 56 deletions(-) create mode 100644 src/build/dep_graph.cppm diff --git a/src/build/dep_graph.cppm b/src/build/dep_graph.cppm new file mode 100644 index 00000000..452476a4 --- /dev/null +++ b/src/build/dep_graph.cppm @@ -0,0 +1,98 @@ +// mcpp.build.dep_graph — queries over the resolved consumer→dependency edge +// graph. +// +// WHY THIS EXISTS +// +// prepare.cppm records one authoritative edge graph during resolution, and by +// now fourteen places read it. Most ask one of exactly two questions — +// "what does package X depend on directly?" and "what is X's transitive +// closure?" — and each had hand-written the loop. Two of them (the two +// build.mcpp call sites emitting MCPP_DEP__DIR) were near-identical +// copies, and #355 added a third variant plus a hand-rolled BFS. +// +// That is the shape this codebase keeps paying for (#233/#240/#242/#344): the +// same decision derived in N places does not fail when you add the N+1th, it +// fails later, somewhere else. Feature activation already learned it the hard +// way — activation and resolution each walked their own idea of "the edges" +// and silently disagreed about transitive requests (#242/#243). +// +// Templated on the edge type rather than owning it: `DependencyEdge` is a +// local struct inside prepare_build, and moving it out would be a much larger +// change than the one this pays for. An edge only has to expose +// `consumerPackageIndex` and `dependencyPackageIndex`. +// +// WHAT IS DELIBERATELY *NOT* HERE +// +// The build cache's per-package key walk (prepare.cppm, `self(self, …)`) is +// NOT a closure query and is not migrated. It is a memoized fold that computes +// a value per node (that package's full cache key), treats a cycle as a hard +// ERROR rather than something to skip, and threads a taint flag alongside. +// Folding it into a generic traversal would either lose those properties or +// force the abstraction to grow until it described exactly one caller. Two +// walks that answer genuinely different questions are not duplication. + +export module mcpp.build.dep_graph; + +import std; + +export namespace mcpp::build::dep_graph { + +// Package indices this consumer depends on DIRECTLY, in edge-record order, +// deduplicated. Order is preserved because several callers surface it to the +// user (dependency dirs, diagnostics) and a stable order keeps output +// diffable. +template +std::vector +direct_dependencies(const std::vector& edges, std::size_t consumer) { + std::vector out; + for (auto const& e : edges) { + if (e.consumerPackageIndex != consumer) continue; + if (std::find(out.begin(), out.end(), e.dependencyPackageIndex) == out.end()) + out.push_back(e.dependencyPackageIndex); + } + return out; +} + +// Every package reachable from `from`, excluding `from` itself. Sorted and +// deduplicated, so a caller folding it into a cache key gets a stable answer +// without re-sorting. +// +// A cycle is TOLERATED here (the visited set terminates it) rather than +// reported. This is a reachability question, and the callers that must reject +// a cycle — the build-cache key walk — detect it where they can say which +// package the cycle runs through, which is the only form of that message worth +// printing. +template +std::vector +transitive_dependencies(const std::vector& edges, std::size_t from) { + std::set seen; + std::vector stack{from}; + while (!stack.empty()) { + auto cur = stack.back(); + stack.pop_back(); + for (auto const& e : edges) { + if (e.consumerPackageIndex != cur) continue; + if (!seen.insert(e.dependencyPackageIndex).second) continue; + stack.push_back(e.dependencyPackageIndex); + } + } + seen.erase(from); + return {seen.begin(), seen.end()}; +} + +// The two spellings a dependency is addressable by: its canonical package name +// and, when it is namespaced, the namespace-stripped tail. +// +// Not a graph query, but it lives here for the same reason: three call sites +// had each open-coded the `rfind('.')` split, and a fourth was about to. A +// consumer may write `compat.zlib` or `zlib`, and every place that surfaces a +// dependency by name has to accept both. +inline std::vector name_spellings(const std::string& canonical) { + std::vector out{canonical}; + if (auto dot = canonical.rfind('.'); + dot != std::string::npos && dot + 1 < canonical.size()) + out.push_back(canonical.substr(dot + 1)); + return out; +} + +} // namespace mcpp::build::dep_graph diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 89ffb881..3cfa2a29 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -35,6 +35,7 @@ import mcpp.build.cache_key; import mcpp.build.build_program; import mcpp.build.directives; // directive table: mark / fold_private_tail import mcpp.build.tool_store; // #355 host tools: store layout + key + overrides +import mcpp.build.dep_graph; // queries over the resolved edge graph import mcpp.build.backend; // BuildOptions for the tool sub-build import mcpp.build.ninja; // make_ninja_backend — driving that sub-build import mcpp.lockfile; @@ -2613,6 +2614,7 @@ prepare_build(bool print_fingerprint, std::vector requestedTools; }; std::vector dependencyEdges; + namespace dg = mcpp::build::dep_graph; // #355: consumer package index → (env var, absolute path) for each host // tool that consumer requested. Filled by the provisioning pass below; // read by BOTH build.mcpp call sites (the dependency loop and the root), @@ -2681,6 +2683,22 @@ prepare_build(bool print_fingerprint, mcpp::build::directives::fold_private_tail(pkg.privateBuild, ran, t); }; + // mcpp#241: the (name → dir) pairs a package's build.mcpp receives as + // MCPP_DEP__DIR. ONE owner: the dependency loop and the root call + // site had drifted into two near-identical copies of this, and #355 was + // about to add a third. Each dependency is emitted under BOTH its + // canonical name and its namespace-stripped tail, so + // `mcpp::dep_dir("compat.zlib")` and `mcpp::dep_dir("zlib")` both resolve + // regardless of which spelling the author used in `deps`. + auto fillDepDirs = [&](mcpp::build::BuildProgramEnv& e, std::size_t consumer) { + for (auto d : dg::direct_dependencies(dependencyEdges, consumer)) { + auto const& depPkg = packages[d]; + for (auto const& spelling : + dg::name_spellings(depPkg.manifest.package.name)) + e.depDirs.emplace_back(spelling, depPkg.root); + } + }; + // A declared build-graph node's Source outputs must be visible to the // scan, so they are materialized as placeholders and joined to the source // set here — the same two lists `generated=` feeds, for the same reason @@ -3931,12 +3949,16 @@ prepare_build(bool print_fingerprint, // construction rather than by luck. for (auto const& [depName, spec] : m->dependencies) { if (!spec.hostModule) continue; - for (auto const& edge : dependencyEdges) { - if (edge.consumerPackageIndex != 0) continue; - auto const& depPkg = packages[edge.dependencyPackageIndex]; + for (auto d : dg::direct_dependencies(dependencyEdges, 0)) { + auto const& depPkg = packages[d]; auto const& canon = depPkg.manifest.package.name; - if (canon != depName && !depName.ends_with(canon) - && !canon.ends_with(depName)) continue; + // Match on either spelling, the same way `deps` keys and + // MCPP_DEP__DIR do — a consumer may have written + // `compat.zlib` or `zlib`. + bool hit = false; + for (auto const& s : dg::name_spellings(canon)) + if (depName == s || depName.ends_with("." + s)) hit = true; + if (!hit) continue; auto rel = mcpp::manifest::resolve_lib_root_path(depPkg.manifest); hostModulesByConsumer[0].emplace_back(canon, depPkg.root / rel); break; @@ -4043,26 +4065,11 @@ prepare_build(bool print_fingerprint, // packages (a frozen version cannot change its own deps), // but a path dependency can: bump something two levels down // and the tool's direct list is unchanged, so a stale binary - // stays in the store. That is a silently wrong artifact — - // the failure mode this project has paid for more than once - // — and the closure walk costs nothing. - { - std::set seen{depIdx}; - std::vector queue{depIdx}; - while (!queue.empty()) { - auto cur = queue.back(); - queue.pop_back(); - for (auto const& edge : dependencyEdges) { - if (edge.consumerPackageIndex != cur) continue; - auto up = edge.dependencyPackageIndex; - if (!seen.insert(up).second) continue; - queue.push_back(up); - key.upstreamKeys.push_back(std::format("{}@{}", - packages[up].manifest.package.name, - packages[up].manifest.package.version)); - } - } - } + // stays in the store — a silently wrong artifact. + for (auto up : dg::transitive_dependencies(dependencyEdges, depIdx)) + key.upstreamKeys.push_back(std::format("{}@{}", + packages[up].manifest.package.name, + packages[up].manifest.package.version)); std::ranges::sort(key.upstreamKeys); const auto cacheRoot = mcpp::home::cache_root(); @@ -4211,25 +4218,12 @@ prepare_build(bool print_fingerprint, bpEnv.artifactsDir = workRoot / "target" / ".build-mcpp" / "deps" / (dirSafe(pkg.manifest.package.name) + "@" + pkg.manifest.package.version); bpEnv.genBase = bpEnv.artifactsDir / "out"; - // mcpp#241: expose this package's resolved dependencies (verdir / - // payload root) as MCPP_DEP__DIR. Uses the authoritative - // consumer→dep edge graph (no name-guessing); covers feature- - // activated deps too (mergeActiveFeatureDeps folded them into - // `dependencies` before the edges were recorded). A dep is emitted - // under BOTH its canonical package name AND its namespace-stripped - // short name, so `mcpp::dep_dir("compat.zlib")` and - // `mcpp::dep_dir("zlib")` both resolve regardless of which spelling - // the author used in `deps`. (The ROOT project's build.mcpp gets - // the same treatment at its own call site right after this loop.) - for (auto const& edge : dependencyEdges) { - if (edge.consumerPackageIndex != i) continue; - auto const& depPkg = packages[edge.dependencyPackageIndex]; - const auto& canon = depPkg.manifest.package.name; - bpEnv.depDirs.emplace_back(canon, depPkg.root); - if (auto dot = canon.rfind('.'); dot != std::string::npos - && dot + 1 < canon.size()) - bpEnv.depDirs.emplace_back(canon.substr(dot + 1), depPkg.root); - } + // mcpp#241: this package's resolved dependencies as + // MCPP_DEP__DIR, from the authoritative edge graph (no + // name-guessing); covers feature-activated deps too + // (mergeActiveFeatureDeps folded them in before the edges were + // recorded). Shared owner — see fillDepDirs. + fillDepDirs(bpEnv, i); // #355: the host tools THIS package requested (resolved above). if (auto tit = toolEnvByConsumer.find(i); tit != toolEnvByConsumer.end()) bpEnv.toolPaths = tit->second; @@ -4361,18 +4355,8 @@ prepare_build(bool print_fingerprint, // contract hash — and therefore the build.mcpp cache — is unchanged // across the move for feature-identical builds. bpEnv.features = feature_closure(*m, parse_feature_request(overrides.features)); - // mcpp#241 (root): the root's resolved direct deps, from the same - // authoritative edge graph as the dep loop (consumer index 0 = root), - // emitted under canonical AND namespace-stripped names. - for (auto const& edge : dependencyEdges) { - if (edge.consumerPackageIndex != 0) continue; - auto const& depPkg = packages[edge.dependencyPackageIndex]; - const auto& canon = depPkg.manifest.package.name; - bpEnv.depDirs.emplace_back(canon, depPkg.root); - if (auto dot = canon.rfind('.'); dot != std::string::npos - && dot + 1 < canon.size()) - bpEnv.depDirs.emplace_back(canon.substr(dot + 1), depPkg.root); - } + // mcpp#241 (root): consumer index 0, same owner as the dep loop. + fillDepDirs(bpEnv, 0); // #355: the host tools the ROOT package requested (consumer index 0). if (auto tit = toolEnvByConsumer.find(0u); tit != toolEnvByConsumer.end()) bpEnv.toolPaths = tit->second; From fa5e9142feb91e154a52b998eeeae9632ac25f02 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:51:21 +0800 Subject: [PATCH 10/16] =?UTF-8?q?fix:=20=E5=90=88=E5=85=A5=E5=89=8D?= =?UTF-8?q?=E6=B7=B1=E5=BA=A6=E8=87=AA=E5=AE=A1=E5=8F=91=E7=8E=B0=E7=9A=84?= =?UTF-8?q?=E5=9B=9B=E4=B8=AA=E9=97=AE=E9=A2=98(=E5=90=AB=E4=B8=80?= =?UTF-8?q?=E5=A4=84=E5=86=85=E5=AD=98=E5=AE=89=E5=85=A8=E3=80=81=E4=B8=80?= =?UTF-8?q?=E5=A4=84=20workspace=20=E5=9B=9E=E5=BD=92)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## 1. 内存安全:内置 mcpp 模块的 action 缓冲区越界 `action::add()` 把上界**硬编码成 4096**,却被 `provides_[1024]` / `imports_[1024]` 调用 —— 最多越界写约 3KB。这段代码会被编进**每一个**用户的 build.mcpp,是本次改动 里最严重的一处。 修法:上界改成**参数**(`sizeof buf` 在调用点取)。一个「不挨着它所约束的数组」的 上界,正是会越界的那种形状。同时: - `command_` 原本 8192 却被 4096 的上界截半,现在按实际容量走 - 缓冲区放大到真实生成器调用的量级(protoc 带一堆 -I 的命令行很长) - 溢出不再静默截断:置 `overflow` 标记,引擎给出**点名容量限制**的诊断,而不是让 作者去找一个其实没错、只是太长的「拼写错误」 ## 2. workspace 回归:成员的 target/ 与 mcpp.lock 落到了 workspace 根 `workRoot` 在 `root` **尚未定稿**时就取了值 —— workspace 那段会把 `root` 改成选中的 成员(`root = memberDir`),于是成员的 `target/`、`mcpp.lock`、`.mcpp/`、 `compile_commands.json` 全落到 workspace 根。 CI 抓到了(macOS 与 Linux 的 `35_workspace` 报 "hello binary not found", `120_ws_root_indices` 报 "expected x.widget2 lock entry")。修法:把 `workRoot` 的 推导移到 workspace 段**之后**,并在原处留注释说明为什么不能在那里取。 教训记下来:我本地的回归面**太窄** —— 只跑了 build.mcpp 相关的用例,而这次改动动的是 「mcpp 往哪写」,workspace 才是它最敏感的形态。 ## 3. 并发:两个工程会共用同一个工具子构建的 scratch 目录 tool store 是**全局**的,所以两个工程可能同时要同一个工具。原实现共用 `/build`,于是它们并发写同一棵 ninja 树,先完成的那个还会 `remove_all` 把另一个的目录端掉。改成按**消费方**哈希隔离;被共享的是发布出来的 二进制,不是 scratch。哈希而非随机,这样重跑能复用自己的 scratch。 ## 4. JSON 有效性:除 \n 外的控制字符没转义 `\t` / `\r` 会通过 Windows 路径和日志文本进来,不转义就直接不是 JSON 了。 ## 文档 超时那条原文写成了跨平台生效 —— **Windows 上不生效**(进程启动器没有 kill-by-handle 的路径),与 `mcpp test --timeout` 是同一条限制。明说,而不是含糊过去。 ## 性能(实测,非断言) 在这台机器上**测不出可分辨的差异**:无 build.mcpp 的工程强制 prepare,15 次中位 OLD 156ms / NEW 140ms(min 62/96,max 181/194)—— 分布高度重叠,两个方向都出现过。 结构上也符合:新增路径全是 O(小),且未被使用时全部惰性(工具请求为空、 `plan.actions` 为空、hostModules 为空则一行都不多跑)。 ## 覆盖面的诚实说明 四个新 e2e 都声明 `# requires: gcc`,而 Windows runner **不提供** gcc 能力 —— 所以 action / host 工具 / 规则包这三条路在 Windows 上**未经验证**。它们在未使用时 完全惰性,不会影响既有 Windows 行为,但这不等于验证过。 验证:单测 57/57;`35_workspace` / `120_ws_root_indices` / `90_workspace_test` 三个 先前红的全绿;186–189 与 43/76/30/51/50/171/04/02 全通过。 --- docs/07-build-mcpp.md | 10 +++++--- docs/zh/07-build-mcpp.md | 9 ++++--- src/build/directives.cppm | 16 ++++++++++++ src/build/hostprogram.cppm | 50 ++++++++++++++++++++++++++++---------- src/build/prepare.cppm | 45 +++++++++++++++++++++++++--------- 5 files changed, 99 insertions(+), 31 deletions(-) diff --git a/docs/07-build-mcpp.md b/docs/07-build-mcpp.md index 48dd8186..eb90a4d9 100644 --- a/docs/07-build-mcpp.md +++ b/docs/07-build-mcpp.md @@ -344,9 +344,13 @@ When nothing changed you'll see `build.mcpp up to date (cached)`; otherwise - **CWD is the project root**, so relative paths (`src/generated.cpp`) land where you expect. - A non-zero exit from `build.mcpp` aborts the build and prints its output. -- **The run is bounded** (mcpp 2026.8.5.1+): a build program gets **600 s** by - default, after which mcpp kills it and fails the build naming the package. - Override with `MCPP_BUILD_PROGRAM_TIMEOUT=` (`0` = no limit). The +- **The run is bounded** (mcpp 2026.8.5.1+, **POSIX only**): a build program + gets **600 s** by default, after which mcpp kills it and fails the build + naming the package. Override with `MCPP_BUILD_PROGRAM_TIMEOUT=` + (`0` = no limit). **On Windows the bound is not enforced** — the process + launcher has no kill-by-handle path yet (`mcpp.platform.process`), so a + build program that hangs there still hangs the build. Same limitation as + `mcpp test --timeout`; stated rather than papered over. The **compile** is deliberately *not* bounded — the same asymmetry `mcpp test` uses: a long compile is usually legitimate (a first-run `std` module build is minutes) and killing it produces a baffling failure, while a long-running diff --git a/docs/zh/07-build-mcpp.md b/docs/zh/07-build-mcpp.md index 6b5e63b6..e12c254d 100644 --- a/docs/zh/07-build-mcpp.md +++ b/docs/zh/07-build-mcpp.md @@ -308,8 +308,11 @@ mcpp **不会**每次构建都重跑 `build.mcpp`。它会缓存程序产出的 [05 - mcpp.toml 工程文件指南](05-mcpp-toml.md)。 - **当前工作目录是工程根目录**,因此相对路径(`src/generated.cpp`)会落在你预期的位置。 - `build.mcpp` 非零退出会中止构建并打印其输出。 -- **运行有时间上限**(mcpp 2026.8.5.1+):构建程序默认有 **600 秒**,超时后 mcpp 杀掉它 - 并让构建失败,错误里会点名是哪个包。用 `MCPP_BUILD_PROGRAM_TIMEOUT=<秒>` 覆盖 - (`0` = 不限)。**编译**这一步刻意**不设**上限——与 `mcpp test` 同一条不对称纪律: +- **运行有时间上限**(mcpp 2026.8.5.1+,**仅 POSIX**):构建程序默认有 **600 秒**, + 超时后 mcpp 杀掉它并让构建失败,错误里会点名是哪个包。用 + `MCPP_BUILD_PROGRAM_TIMEOUT=<秒>` 覆盖(`0` = 不限)。**Windows 上这个上限不生效** + —— 进程启动器还没有 kill-by-handle 的路径(`mcpp.platform.process`),所以在那里 + 卡死的构建程序仍会把构建挂住。与 `mcpp test --timeout` 是同一条限制;明说,而不是 + 含糊过去。**编译**这一步刻意**不设**上限——与 `mcpp test` 同一条不对称纪律: 编译跑得久通常是正当的(首次构建 `std` 模块就是分钟级),杀掉它只会产生莫名其妙的 失败;而构建**程序**跑得久通常是卡住了,不设上限就会让整个构建挂死且毫无诊断。 diff --git a/src/build/directives.cppm b/src/build/directives.cppm index 4798f048..769925a8 100644 --- a/src/build/directives.cppm +++ b/src/build/directives.cppm @@ -517,6 +517,22 @@ std::optional decode_action(std::string_view payloa std::string action_error(const Directives& d) { for (auto const& payload : d.at(Slot::Actions)) { + // The typed API sets this when an argv did not fit its fixed buffer. + // Diagnosed separately because "malformed action" would send the + // author looking for a typo in something that was actually correct + // and merely too long. + if (payload.find("\"overflow\":true") != std::string::npos) { + return std::format( + "build.mcpp declared an action whose arguments did not fit.\n" + " The typed `mcpp::action` builder uses fixed buffers " + "(the bundled module has to stay\n" + " buildable before a std module exists, so it cannot use " + "std::string).\n" + " Shorten the command — e.g. pass a response file, or a " + "directory instead of\n" + " enumerating its files.\n" + " payload: {}", payload); + } if (decode_action(payload)) continue; // A malformed action is a hard error, never a skip: an action that // silently does not exist produces a build missing generated sources, diff --git a/src/build/hostprogram.cppm b/src/build/hostprogram.cppm index aeec6e00..135b0be6 100644 --- a/src/build/hostprogram.cppm +++ b/src/build/hostprogram.cppm @@ -64,19 +64,23 @@ struct action { const char* role = "source"; // "source" | "check" | "artifact" const char* description = ""; bool blocking = false; // check only: gate compilation on it - action& input(const char* p) { add(inputs_, p); return *this; } - action& output(const char* p) { add(outputs_, p); return *this; } - action& arg(const char* a) { add(command_, a); return *this; } + action& input(const char* p) { add(inputs_, sizeof inputs_, p); return *this; } + action& output(const char* p) { add(outputs_, sizeof outputs_, p); return *this; } + action& arg(const char* a) { add(command_, sizeof command_, a); return *this; } // Declare what a generated MODULE INTERFACE provides/imports. Same // "declare instead of discover" trade [modules].scan_overrides makes, and // what lets a generated .cppm exist as a graph node at all. - action& provides(const char* n) { add(provides_, n); return *this; } - action& imports(const char* n) { add(imports_, n); return *this; } + action& provides(const char* n) { add(provides_, sizeof provides_, n); return *this; } + action& imports(const char* n) { add(imports_, sizeof imports_, n); return *this; } void submit() const { std::printf("mcpp:action={\"id\":"); esc(id); std::printf(",\"role\":"); esc(role); std::printf(",\"description\":"); esc(description); std::printf(",\"blocking\":%s", blocking ? "true" : "false"); + // A truncated argv would otherwise be INVALID rather than obviously + // wrong — the engine turns this marker into a diagnostic that names + // the limit, instead of a generic "malformed action". + if (overflow_) std::printf(",\"overflow\":true"); std::printf(",\"inputs\":[%s]", inputs_); std::printf(",\"outputs\":[%s]", outputs_); std::printf(",\"command\":[%s]", command_); @@ -85,26 +89,42 @@ struct action { std::printf("}\n"); } private: - char inputs_[4096]{}, outputs_[4096]{}, command_[8192]{}, provides_[1024]{}, imports_[1024]{}; + // Fixed buffers because this module must stay buildable BEFORE a std BMI + // exists (it is what a build.mcpp imports, and it may be compiled first) — + // so no std::string. Sizes chosen for real generator invocations: a protoc + // command line with many -I paths runs long. + char inputs_[8192]{}, outputs_[8192]{}, command_[16384]{}, + provides_[2048]{}, imports_[2048]{}; + mutable bool overflow_ = false; static void esc(const char* s) { std::putchar('"'); for (const char* p = s; *p; ++p) { - if (*p == '"' || *p == '\\') std::putchar('\\'); - if (*p == '\n') { std::printf("\\n"); continue; } - std::putchar(*p); + unsigned char c = (unsigned char)*p; + if (c == '"' || c == '\\') { std::putchar('\\'); std::putchar(c); continue; } + // Any control character has to be escaped or the payload is not + // JSON at all. \n was handled before; \t and \r reach this code + // through ordinary Windows paths and log text. + if (c < 0x20) { std::printf("\\u%04x", c); continue; } + std::putchar(c); } std::putchar('"'); } - static void add(char* buf, const char* s) { + // Capacity is a PARAMETER. The previous revision hardcoded 4096 while the + // smallest buffer here was 1024 — a bound living somewhere other than next + // to the array it bounds is exactly the shape that overflows. + bool add(char* buf, unsigned long cap, const char* s) { unsigned long o = 0; while (buf[o]) ++o; + if (o + 4 >= cap) { overflow_ = true; return false; } if (o) buf[o++] = ','; buf[o++] = '"'; - for (const char* p = s; *p && o + 3 < 4096; ++p) { + for (const char* p = s; *p; ++p) { + if (o + 3 >= cap) { buf[o] = 0; overflow_ = true; return false; } if (*p == '"' || *p == '\\') buf[o++] = '\\'; buf[o++] = *p; } buf[o++] = '"'; buf[o] = 0; + return true; } }; inline void rerun_if_changed(const char* path) { std::printf("mcpp:rerun-if-changed=%s\n", path); } @@ -350,8 +370,12 @@ build_host_module(const fs::path& bdir, const fs::path& compiler, "(src/.cppm or [lib] path).", logicalName, interfacePath.string())); } - // A filesystem-safe stem: a module name contains dots, which are fine in a - // path but make `foo.rules.o` read as an extension chain. + // A filesystem-safe stem. Partition separators and any path separator that + // sneaks into a logical name would otherwise create directories that do + // not exist. Dots are left ALONE on purpose: `a.b.rules.o` is a legal + // filename, GCC's own gcm.cache uses the dotted module name verbatim, and + // rewriting them would make the object name disagree with the BMI name for + // no gain. std::string stem(logicalName); for (auto& c : stem) if (c == ':' || c == '/' || c == '\\') c = '-'; diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 3cfa2a29..9206f201 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -876,15 +876,11 @@ prepare_build(bool print_fingerprint, if (!root) { return std::unexpected("no mcpp.toml found in current directory or any parent"); } - // Where mcpp writes. Defaults to the project root, so every existing - // invocation is byte-for-byte unchanged; the tool-provisioning pass points - // it at the tool store instead (BuildOverrides::work_dir). - const std::filesystem::path workRoot = - overrides.work_dir.empty() ? *root : overrides.work_dir; - { - std::error_code wdEc; - std::filesystem::create_directories(workRoot, wdEc); - } + // NOTE: `workRoot` is deliberately NOT derived here. `root` is not final + // yet — the workspace block below reassigns it to the selected member + // (`root = memberDir`), and anchoring the write root to the pre-switch + // value puts a member's target/, mcpp.lock and .mcpp/ at the WORKSPACE + // root. See the derivation right after that block. auto m = mcpp::manifest::load(*root / "mcpp.toml"); if (!m) return std::unexpected(m.error().format()); @@ -988,6 +984,18 @@ prepare_build(bool print_fingerprint, } } + // Where mcpp WRITES — derived here because `root` is only final now: the + // workspace block above may have moved it to the selected member. Defaults + // to the project root, so every existing invocation is byte-for-byte + // unchanged; the tool-provisioning pass points it at the tool store + // instead (BuildOverrides::work_dir). + const std::filesystem::path workRoot = + overrides.work_dir.empty() ? *root : overrides.work_dir; + { + std::error_code wdEc; + std::filesystem::create_directories(workRoot, wdEc); + } + // Inject synthetic targets (e.g. test binaries from `mcpp test`). for (auto& t : extraTargets) m->targets.push_back(t); @@ -4093,7 +4101,20 @@ prepare_build(bool print_fingerprint, sub.project_root = depPkg.root; // Never the package root: it is shared across projects and // may be read-only. This is the reason work_dir exists. - sub.work_dir = entry / "build"; + // + // Scratch is keyed on the CONSUMING project, not shared: + // the store is GLOBAL, so two projects can want the same + // tool at once. A single `/build` would have them + // writing one ninja tree concurrently, and whichever + // finished first would `remove_all` it out from under the + // other. The published binary is what gets shared; the + // scratch is not. + // + // Hashed rather than random so a re-run reuses its own + // scratch (ninja stays incremental if the publish step + // never got to delete it). + sub.work_dir = entry / std::format("build-{}", + mcpp::toolchain::hash_string(workRoot.string())); sub.target_triple = ""; // HOST — the whole point sub.profile = "release"; sub.cache_mode = overrides.cache_mode; @@ -4181,8 +4202,8 @@ prepare_build(bool print_fingerprint, mcpp::build::tool_store::write_entry(entry, key); // The sub-build tree is large (protoc is several hundred // objects) and the key covers every input, so a hit never - // needs it again. - std::filesystem::remove_all(entry / "build", cpEc); + // needs it again. Removes only THIS consumer's scratch. + std::filesystem::remove_all(sub.work_dir, cpEc); record(binOut); } } From f651ed8efcb5e6d239d2b8486fa9d62da8d31ba2 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 14:58:34 +0800 Subject: [PATCH 11/16] =?UTF-8?q?fix(actions):=20ninja=20=E8=BD=AC?= =?UTF-8?q?=E4=B9=89=E4=B8=8E=E6=9C=AA=E8=A7=A3=E6=9E=90=E7=9B=AE=E6=A0=87?= =?UTF-8?q?=20=E2=80=94=E2=80=94=20=E8=87=AA=E5=AE=A1=E7=AC=AC=E4=BA=8C?= =?UTF-8?q?=E8=BD=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## 1. action 命令里的 `$` 会被 ninja 吃掉 命令 token 只做了 shell 引号,没做 ninja 转义。**引号救不了它**:ninja 在 shell 被 调用之前就会展开 `$foo`,所以带字面 `$` 的 token(含 `$` 的路径、 `-Wl,-rpath,$ORIGIN`、一段 awk 程序)会被当成变量引用。 改成本文件其余地方一直用的那个配对(与 `include_dir_token` 同序):**先 ninja 转义, 再 shell 引号**。对普通 token 无影响 —— `escape_ninja_chars` 只碰空格 / `$` / `:`, 而 `shell_quote_arg` 对不含元字符的串逐字节原样返回。 这里每个元素**按构造就是一个 argv token**(类型化 builder 一次追加一个),这正是 逐 token 引号成立的前提 —— #331 表明手工拼出来的 flag blob **不**满足这个前提。 顺带把 `escape_ninja_chars` 从匿名 namespace 移出并导出:它此前是内部的,而 ninja_backend 需要它。让它继续内部化就意味着 ninja 转义规则第四份手写副本,而那 正是它们会漂的原因。 ## 2. `${mcpp.target_file:拼错}` 静默变成空串 未解析的引擎变量原本会被替换成空字符串,于是生成一条路径为空的边,ninja 在离 错误很远的地方报出来。现在是**硬错误**,并列出本次构建里存在的 target,还提示 「被 required_features 门住的 target 在那些 feature 未激活时不存在」。 ## 测试 188 补两条:字面 `$` 能原样到达工具;未知 target 引用报错并**点名**。 两条放在自己的最小工程里 —— `app` 的 main.cpp 故意依赖生成出来的符号,在它上面 换掉 build.mcpp 会让**链接**失败,那说明不了这两条在测什么(我第一版就是这么写的)。 验证:单测 57/57;13 个 e2e 全绿,含先前红的 35_workspace / 120_ws_root_indices。 --- src/build/flags.cppm | 25 ++++++++++---- src/build/ninja_backend.cppm | 16 ++++++++- src/build/prepare.cppm | 19 +++++++++++ tests/e2e/188_build_actions.sh | 61 ++++++++++++++++++++++++++++++++++ 4 files changed, 114 insertions(+), 7 deletions(-) diff --git a/src/build/flags.cppm b/src/build/flags.cppm index 7b92c688..56229969 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -109,6 +109,14 @@ std::string atomic_link_flag(const std::vector& linkDirs, // escaped as `\"`) — cmd.exe/CreateProcess argv convention. std::string shell_quote_arg(std::string_view arg); +// Ninja's own escaping for a value that will sit on a `command = ` line: +// ` `, `$` and `:` get a leading `$`. Exported because it is needed WITH +// shell_quote_arg, not instead of it — quoting stops the SHELL from splitting +// a token, but ninja expands `$foo` before the shell is ever invoked, so a +// token carrying a literal `$` needs both. Callers apply ninja escaping first, +// then shell quoting (see include_dir_token). +std::string escape_ninja_chars(std::string_view s); + // One include-directory token, fully prepared for a ninja command line: // dialect prefix, ninja `$` escaping, and shell quoting — in that order. // @@ -146,16 +154,15 @@ std::string include_token(const mcpp::toolchain::CommandDialect& d, namespace mcpp::build { -namespace { - -std::filesystem::path staged_std_bmi_path(const BuildPlan& plan) { - return mcpp::toolchain::staged_std_bmi_path(plan.toolchain, plan.outputDir); -} - // Escape a string for embedding in ninja rule strings. Takes the text, not a // path: round-tripping through std::filesystem::path would re-normalize the // separators on Windows, which silently undoes a caller that deliberately // chose generic_string() for a response-file token (#261). +// +// Deliberately OUTSIDE the anonymous namespace below: it is declared in this +// module's export block so ninja_backend can pair it with shell_quote_arg for +// action command tokens. Leaving it internal would mean a fourth hand-written +// copy of ninja's escaping rules, which is how they drift. std::string escape_ninja_chars(std::string_view s) { std::string out; out.reserve(s.size()); @@ -167,6 +174,12 @@ std::string escape_ninja_chars(std::string_view s) { return out; } +namespace { + +std::filesystem::path staged_std_bmi_path(const BuildPlan& plan) { + return mcpp::toolchain::staged_std_bmi_path(plan.toolchain, plan.outputDir); +} + // Escape a path for embedding in ninja rule strings (native separators). std::string escape_path(const std::filesystem::path& p) { return escape_ninja_chars(p.string()); diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index a45ed3af..d9937b06 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -1314,7 +1314,21 @@ std::string emit_ninja_string(const BuildPlan& plan) { std::string cmd; for (auto const& tok : a.command) { if (!cmd.empty()) cmd += ' '; - cmd += shell_quote_arg(tok); + // BOTH escapes, in the order the rest of this file uses + // (include_dir_token does the same): ninja first, shell second. + // Shell-quoting alone is not enough — a literal `$` in a token + // (a path containing one, or an argument like `-Wl,-rpath,$ORIGIN`) + // is a VARIABLE REFERENCE to ninja, and quoting does not stop + // ninja from expanding it before the shell ever sees it. + // + // Safe for ordinary tokens: escape_ninja_chars only touches + // ` `, `$` and `:`, and shell_quote_arg returns anything without a + // metacharacter byte-for-byte, so a plain `--cpp_out=gen` is + // unchanged. Each element here is exactly one argv token by + // construction (the typed builder appends them one at a time), + // which is the assumption per-token quoting needs and which #331 + // showed is NOT true of hand-assembled flag blobs. + cmd += shell_quote_arg(escape_ninja_chars(tok)); } append(std::format("rule mcpp_action_{}\n", i)); append(std::format(" command = {}\n", cmd)); diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 9206f201..b490062a 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -4644,6 +4644,11 @@ prepare_build(bool print_fingerprint, // what makes an action portable (Windows has no shell to assume) and // cacheable (nothing can smuggle in ambient state). { + // An engine variable that resolves to nothing must be an ERROR, not an + // empty string: `${mcpp.target_file:tpyo}` would otherwise silently + // become an edge with a blank path, and ninja reports that far away + // from the typo that caused it. + std::set unresolvedTargets; auto substitute = [&](std::string s) { auto rep = [&](std::string_view what, const std::string& with) { for (std::size_t p; (p = s.find(what)) != std::string::npos; ) @@ -4668,6 +4673,7 @@ prepare_build(bool print_fingerprint, for (auto const& lu : ctx.plan.linkUnits) if (lu.targetName == name) resolved = lu.output.generic_string(); + if (resolved.empty()) unresolvedTargets.insert(name); s.replace(p, close - p + 1, resolved); } return s; @@ -4683,6 +4689,19 @@ prepare_build(bool print_fingerprint, collect(*m); for (std::size_t i = 1; i < packages.size(); ++i) collect(packages[i].manifest); + if (!unresolvedTargets.empty()) { + std::string bad, known; + for (auto const& n : unresolvedTargets) bad += (bad.empty() ? "" : ", ") + n; + for (auto const& lu : ctx.plan.linkUnits) + known += (known.empty() ? "" : ", ") + lu.targetName; + return std::unexpected(std::format( + "build.mcpp action references unknown target(s) via " + "${{mcpp.target_file:...}}: {}\n" + " targets in this build: [{}]\n" + " (a target gated by required_features is absent unless those " + "features are active)", + bad, known.empty() ? std::string("none") : known)); + } } ctx.plan.stdCompatBmiPath = stdCompatBmiPath; ctx.plan.stdCompatObjectPath = stdCompatObjectPath; diff --git a/tests/e2e/188_build_actions.sh b/tests/e2e/188_build_actions.sh index e2a16a59..5c72e369 100755 --- a/tests/e2e/188_build_actions.sh +++ b/tests/e2e/188_build_actions.sh @@ -144,6 +144,67 @@ if "$MCPP" build > b4.log 2>&1; then fi rm -f FAIL_THE_CHECK +# ── 3b/3c: their own minimal project ─────────────────────────────────────── +# Separate from `app`, whose main.cpp deliberately depends on the generated +# symbol — swapping its build.mcpp out would fail the LINK and say nothing +# about what these two are actually testing. +mkdir -p "$TMP/edge/src" +cd "$TMP/edge" +cat > mcpp.toml <<'EOF' +[package] +name = "edge" +version = "0.1.0" +EOF +printf 'int main() {}\n' > src/main.cpp + +# ── 3b. a literal `$` in a command survives to the tool ──────────────────── +# Shell-quoting alone does not save it: ninja expands `$foo` BEFORE the shell +# runs, so a token carrying a `$` (a path containing one, `-Wl,-rpath,$ORIGIN`, +# an awk program) needs ninja escaping too. +cat > dollar.sh <<'EOF' +#!/usr/bin/env bash +# $1 must arrive containing a literal dollar sign +case "$1" in *'$'*) : > "$2";; *) echo "lost the dollar: [$1]" >&2; exit 1;; esac +EOF +chmod +x dollar.sh +cat > build.mcpp <<'EOF' +#include +#include +import mcpp; +int main() { + const std::string root = mcpp::manifest_dir(); + mcpp::action a; + a.id = "dollar"; a.role = "check"; + a.arg((root + "/dollar.sh").c_str()).arg("-Wl,-rpath,$ORIGIN") + .arg("${mcpp.out_dir}/dollar.stamp") + .output("${mcpp.out_dir}/dollar.stamp") + .submit(); +} +EOF +rm -rf target +"$MCPP" build > b3b.log 2>&1 || { + cat b3b.log; echo "FAIL: a literal \$ in an action command did not survive"; exit 1; } + +# ── 3c. an unknown target reference is an error, not an empty path ───────── +cat > build.mcpp <<'EOF' +#include +import mcpp; +int main() { + mcpp::action a; + a.id = "bad-ref"; a.role = "artifact"; + a.arg("/bin/true").arg("${mcpp.target_file:no_such_target}") + .input("${mcpp.target_file:no_such_target}") + .output("${mcpp.out_dir}/x.out") + .submit(); +} +EOF +rm -rf target +if "$MCPP" build > b3c.log 2>&1; then + cat b3c.log; echo "FAIL: an unknown target reference was accepted"; exit 1 +fi +grep -q "no_such_target" b3c.log || { + cat b3c.log; echo "FAIL: error does not name the unknown target"; exit 1; } + # ── 4. a malformed action is refused, not skipped ────────────────────────── cat > build.mcpp <<'EOF' #include From 39b8cb4eaa272297e5ede56683b148a93b616cf8 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 15:01:40 +0800 Subject: [PATCH 12/16] =?UTF-8?q?test(e2e):=20=E8=A6=86=E7=9B=96=E5=B8=A6?= =?UTF-8?q?=20namespace=20=E7=9A=84=E5=8C=85=E7=9A=84=E4=B8=A4=E7=A7=8D?= =?UTF-8?q?=E5=B7=A5=E5=85=B7=E5=AF=BB=E5=9D=80=E6=8B=BC=E5=86=99?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 187 里的 toolpkg 没有 namespace,于是它的 canonical 名与去 namespace 的短名是同一个 串 —— 只发出一个环境变量,两种拼写那条路**从未被走到**。 补一个 `myns.tp` 的用例:消费者写 `myns.tp` 或 `tp` 都必须解析到同一个工具,与 mcpp::dep_dir() 两种拼写都接受保持一致。 --- tests/e2e/187_dep_host_tool.sh | 63 ++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/tests/e2e/187_dep_host_tool.sh b/tests/e2e/187_dep_host_tool.sh index 4436e5ab..9ebb2368 100755 --- a/tests/e2e/187_dep_host_tool.sh +++ b/tests/e2e/187_dep_host_tool.sh @@ -158,4 +158,67 @@ grep -q "override" b5.log || { cat b5.log; echo "FAIL: the override was not repo out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" [[ "$out" == "ANSWER=7" ]] || { echo "FAIL: the override was not actually used: $out"; exit 1; } +# ── a NAMESPACED package is addressable by both spellings ────────────────── +# `toolpkg` above has no namespace, so its canonical and short names are the +# same string and only one env var is emitted — the two-spelling path is never +# exercised. A consumer may write either `myns.tp` or `tp`, exactly as +# mcpp::dep_dir() accepts both, and the tool lookup has to match. +cd "$TMP" +mkdir -p ns/src +cat > ns/mcpp.toml <<'EOF' +[package] +name = "myns.tp" +version = "0.1.0" + +[build] +sources = ["src/lib.cpp"] + +[targets.gen] +kind = "bin" +main = "src/gen.cpp" +EOF +printf 'int tp_lib(){return 1;}\n' > ns/src/lib.cpp +cat > ns/src/gen.cpp <<'EOF' +#include +int main(int c, char** v) { + if (c < 2) return 2; + FILE* f = std::fopen(v[1], "w"); + if (!f) return 3; + std::fprintf(f, "int gv() { return 5; }\n"); + std::fclose(f); + return 0; +} +EOF +mkdir -p nsapp/src +cat > nsapp/mcpp.toml <<'EOF' +[package] +name = "nsapp" +version = "0.1.0" + +[dependencies] +"myns.tp" = { path = "../ns", tools = ["gen"] } +EOF +printf '#include \nint gv();\nint main(){std::printf("G=%%d\\n",gv());}\n' > nsapp/src/main.cpp +cat > nsapp/build.mcpp <<'EOF' +#include +#include +#include +import mcpp; +int main() { + // BOTH spellings must resolve to the same tool. + const char* full = mcpp::dep_bin("myns.tp", "gen"); + const char* brief = mcpp::dep_bin("tp", "gen"); + if (!*full) { std::fprintf(stderr, "canonical spelling did not resolve\n"); return 1; } + if (!*brief) { std::fprintf(stderr, "short spelling did not resolve\n"); return 1; } + std::string out = std::string(mcpp::out_dir()) + "/g.cpp"; + std::string cmd = std::string("\"") + brief + "\" \"" + out + "\""; + if (std::system(cmd.c_str()) != 0) return 1; + mcpp::generated(out.c_str()); +} +EOF +cd nsapp +"$MCPP" build > b6.log 2>&1 || { cat b6.log; echo "FAIL: namespaced tool lookup failed"; exit 1; } +out="$("$MCPP" run 2>&1 | grep '^G=' | tail -1)" +[[ "$out" == "G=5" ]] || { echo "FAIL: namespaced tool did not generate: $out"; exit 1; } + echo "OK" From 651e0e103e7e67127a04e4314955d4b71039f146 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 15:13:30 +0800 Subject: [PATCH 13/16] =?UTF-8?q?test:=20=E8=AE=A9=20directive=20=E7=9A=84?= =?UTF-8?q?=E8=B7=AF=E5=BE=84=E6=96=AD=E8=A8=80=E8=B7=A8=E5=B9=B3=E5=8F=B0?= =?UTF-8?q?=E6=88=90=E7=AB=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Windows CI 报 BuildDirectives.TransformsAreAppliedOnceAtParseTime 失败 —— 是**我这个 测试**的 bug,不是产品 bug: - 它把期望写成了字面 POSIX 串("/pkg/inc"),而 lexically_normal() 在 Windows 上会 转成反斜杠 - 更糟的是 "/abs/inc" 在 Windows 上**根本不是绝对路径**(没有 root name),于是 「绝对路径原样保留」那半条断言在那里测的是相反的东西 改成用同一套 std::filesystem 路径运算构造期望,并用 current_path() 派生出一个在**当前 平台上确实绝对**的输入。这样断言表达的是**行为**(「相对路径按包根解析」「绝对路径原样 保留」),而不是某一个平台的分隔符。 --- tests/unit/test_build_directives.cpp | 58 ++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 17 deletions(-) diff --git a/tests/unit/test_build_directives.cpp b/tests/unit/test_build_directives.cpp index 6bd26451..f8f642b0 100644 --- a/tests/unit/test_build_directives.cpp +++ b/tests/unit/test_build_directives.cpp @@ -20,10 +20,27 @@ const mcpp::toolchain::CommandDialect& gnu() { return mcpp::toolchain::gnu_dialect(); } -dirs::Directives parse(std::string_view text, - const std::filesystem::path& root = "/pkg") { +// A root that is genuinely absolute on the running platform. A literal "/pkg" +// is NOT absolute on Windows (no root name), so hardcoding POSIX strings makes +// these tests assert the wrong thing there rather than the right thing +// everywhere. +const std::filesystem::path& test_root() { + static const std::filesystem::path r = + (std::filesystem::current_path() / "pkg").lexically_normal(); + return r; +} + +// What the implementation should produce for a package-relative path — +// expressed through the same std::filesystem arithmetic, so the expectation is +// about the BEHAVIOUR ("relative resolves against the root") rather than about +// one platform's separator. +std::string under_root(std::string_view rel) { + return (test_root() / rel).lexically_normal().string(); +} + +dirs::Directives parse(std::string_view text) { dirs::Directives d; - dirs::accept_output(d, gnu(), root, text); + dirs::accept_output(d, gnu(), test_root(), text); return d; } @@ -96,26 +113,33 @@ TEST(BuildDirectives, NonDirectiveLinesAreIgnored) { } TEST(BuildDirectives, TransformsAreAppliedOnceAtParseTime) { - auto d = parse("mcpp:cxxflag=-Wall\n" - "mcpp:link-lib=z\n" - "mcpp:link-search=vendor/lib\n" - "mcpp:cfg=HAVE_X\n" - "mcpp:include-dir=inc\n" - "mcpp:include-dir-after=/abs/inc\n"); + // An input that is already absolute ON THIS PLATFORM, so the "taken as-is" + // half of the assertion tests what it claims to. + const auto alreadyAbs = + (std::filesystem::current_path() / "abs" / "inc").lexically_normal(); + + auto d = parse(std::format("mcpp:cxxflag=-Wall\n" + "mcpp:link-lib=z\n" + "mcpp:link-search=vendor/lib\n" + "mcpp:cfg=HAVE_X\n" + "mcpp:include-dir=inc\n" + "mcpp:include-dir-after={}\n", + alreadyAbs.string())); EXPECT_EQ(d.at(dirs::Slot::CxxFlags), (std::vector{"-Wall"})); // link-lib and link-search share the ldflags slot, in emission order. EXPECT_EQ(d.at(dirs::Slot::LdFlags), - (std::vector{"-lz", "-L/pkg/vendor/lib"})); + (std::vector{"-lz", "-L" + under_root("vendor/lib")})); EXPECT_EQ(d.at(dirs::Slot::Defines), (std::vector{"-DHAVE_X"})); // Relative resolves against the package root; absolute is taken as-is. - EXPECT_EQ(d.at(dirs::Slot::IncludeDirs), (std::vector{"/pkg/inc"})); + EXPECT_EQ(d.at(dirs::Slot::IncludeDirs), + (std::vector{under_root("inc")})); EXPECT_EQ(d.at(dirs::Slot::IncludeDirsAfter), - (std::vector{"/abs/inc"})); + (std::vector{alreadyAbs.string()})); } TEST(BuildDirectives, DialectDecidesTheSpelling) { dirs::Directives d; - dirs::accept_output(d, mcpp::toolchain::msvc_dialect(), "/pkg", + dirs::accept_output(d, mcpp::toolchain::msvc_dialect(), test_root(), "mcpp:link-lib=z\nmcpp:cfg=HAVE_X\n"); EXPECT_EQ(d.at(dirs::Slot::LdFlags), (std::vector{"z.lib"})); EXPECT_EQ(d.at(dirs::Slot::Defines), (std::vector{"/DHAVE_X"})); @@ -263,10 +287,10 @@ TEST(BuildDirectives, ApplyRoutesEachSlotToItsManifestChannel) { m.modules.sources.end()) << s; } EXPECT_NE(std::find(bc.includeDirs.begin(), bc.includeDirs.end(), - std::filesystem::path("/pkg/inc")), + std::filesystem::path(under_root("inc"))), bc.includeDirs.end()); EXPECT_NE(std::find(bc.includeDirsAfter.begin(), bc.includeDirsAfter.end(), - std::filesystem::path("/pkg/after")), + std::filesystem::path(under_root("after"))), bc.includeDirsAfter.end()); } @@ -309,7 +333,7 @@ TEST(BuildDirectives, FoldMovesOnlyTheTailAndOnlyPrivateChannels) { EXPECT_EQ(priv.cxxflags, (std::vector{"-Wall", "-DHAVE_X"})); EXPECT_EQ(priv.cflags, (std::vector{"-DHAVE_X"})); EXPECT_EQ(priv.includeDirs, - (std::vector{"/pkg/inc"})); + (std::vector{under_root("inc")})); // Link flags are NOT private — they reach the final link through their own // path, and folding them here would double-apply them. EXPECT_TRUE(priv.includeDirsAfter.empty()); @@ -325,7 +349,7 @@ TEST(BuildDirectives, FoldIsIdempotentOnIncludeDirs) { dirs::fold_private_tail(priv, m, before); dirs::fold_private_tail(priv, m, before); EXPECT_EQ(priv.includeDirs, - (std::vector{"/pkg/inc"})); + (std::vector{under_root("inc")})); } // ── Run bound ────────────────────────────────────────────────────────────── From 1737a17366f12c3b13b638982c2c8101e5ed05f3 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 15:32:59 +0800 Subject: [PATCH 14/16] =?UTF-8?q?fix:=20=E7=9C=9F=E5=AE=9E=E6=A1=88?= =?UTF-8?q?=E4=BE=8B(grpc-m/protobuf)=E6=9A=B4=E9=9C=B2=E7=9A=84=E4=B8=89?= =?UTF-8?q?=E4=B8=AA=20bug=20=E2=80=94=E2=80=94=20=E5=90=88=E6=88=90=20e2e?= =?UTF-8?q?=20=E4=B8=80=E4=B8=AA=E9=83=BD=E6=B2=A1=E6=8A=93=E5=88=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 拿 #355 的原始动机案例做端到端验证:让 mcpp 从 compat.protobuf 的源码构建出 protoc,再用它给一个消费者生成 .pb.cc。结果是**跑通了**(见下),但过程里连撞三个 bug —— 全部是我四个合成 e2e 覆盖不到的形态,因为它们用的都是 path 依赖(Form A)。 ## 1. Form B(compat 描述符)包根本不能当工具提供方 工具子构建走 `prepare_build`,而它从 `/mcpp.toml` 读 manifest。**compat 形态的 包没有 mcpp.toml** —— 它的 manifest 是解析期从 `.lua` 描述符合成出来的。于是子构建 第一步就死在 `cannot open .../mcpp.toml`。 这不是小众情况:索引里绝大多数是 Form B,**protobuf 就是**。也就是说 gRPC 那条链 (protoc 来自 compat.protobuf)整个走不通。 修法:`BuildOverrides::preloaded_manifest` —— 由调用方把它**已经合成好**的 manifest 交给子构建。这比重新推导也更正确:重新推导可能得到与父构建**不同**的 manifest (L1 cfg 合并、feature 激活出的依赖那时已经折进去了)。 必须传**未经 feature 激活的那份**(`dep_manifests[i-1]`,而不是 `packages[i].manifest` ——后者是被 `apply()` 改过的副本):子构建会激活自己的 feature 集,从已激活的副本出发 会把同一批 feature 源码折进去两次。 > 顺带一个 GCC modules 约束:`preloaded_manifest` 一开始写成 > `std::optional`,而 `BuildOverrides` 是**导出**结构体 —— GCC 直接写不出 > module cluster(`failed to read compiled module cluster 529: Bad file data`,在 > mcpp.build.execute 导入它时炸)。换成 `shared_ptr` 让导出布局保持 > trivial 即可,顺带也省掉每次工具构建拷贝一份 manifest。 ## 2. `targets..main` 不展开 `*/` 包装 glob Form B 包的源码在版本目录下的**包装目录**里,所以描述符写 `*/src/foo.cc` —— `*` 代表 tarball 顶层文件夹,描述符无从知道它叫什么。`[build] sources` 一直会展开 这种 glob,但 `main` 是按字面路径处理的,于是 ninja 拿到一个带 `*` 的路径,报 `missing and no known rule to make it`。 **#355 之前没有任何路径能走到这里**(依赖的 bin target 从不被构建),所以它一直没被 发现。现在在 manifest 定稿后、任何人读 `t.main` 之前解析掉;匹配不到恰好一个就报错。 ## 3. action 的**全部**输出都被当成翻译单元 最自然的生成器形态就是 protoc:同时产出 `foo.pb.cc` **和** `foo.pb.h`。原实现把每个 `role=source` 的输出都塞进编译集,于是两者算出同一个对象路径,撞上 `object path collision after uniqueness pass`。 头文件必须**由这条边产出**(别的 TU 要 include 它),但**绝不能被编译**。新增 `is_compilable_output()` 按扩展名过滤;非源码输出照旧声明给 ninja,只是不进编译集。 ## 验证结果:protoc 真的能被 mcpp 从源码构建出来 设计文档里标注的那条**未核实风险**(「libprotoc 能否被 mcpp 无 CMake 构建」)现在有 答案了 —— **能**: - 上游 `src/file_lists.cmake` 的 `libprotoc_srcs` 有 138 项,与 libprotobuf **零重叠** - 源码树里**没有** `.h.in` / `.cmake.in`,不需要 configure 步骤 - 全部编过,链接一次失败:缺 `upb_*` 符号 —— libprotoc 的 upb 生成器需要 upb 运行时, 那正是 compat.protobuf **已有**的 `upb` feature。把 target 的 `required_features` 写成 `{ "protoc", "upb" }` 即可 —— **这正是成本门机制该起的作用**, 不需要引擎改动 - 端到端跑通:`protoc` 建出 → `dep_bin` 拿到路径 → action 调用它生成 `demo.pb.cc`/`.pb.h` → 编译链接 → 程序输出 `NAME=mcpp` - store 命中实测:第二次 `rm -rf target` 后构建 **1.41s**,不重建 protoc ## 测试 188 补一条 companion 输出(`.cc` + `.h` 同时声明)的用例 —— 就是撞出 bug 3 的形状。 单测 57/57;11 个 e2e 全绿(`09_path_dependency` 在**已发布的 2026.8.4.1 上同样失败**, 环境性)。 --- src/build/directives.cppm | 22 +++++++++++ src/build/prepare.cppm | 71 +++++++++++++++++++++++++++++++++- tests/e2e/188_build_actions.sh | 46 ++++++++++++++++++++++ 3 files changed, 138 insertions(+), 1 deletion(-) diff --git a/src/build/directives.cppm b/src/build/directives.cppm index 769925a8..e03892ce 100644 --- a/src/build/directives.cppm +++ b/src/build/directives.cppm @@ -258,6 +258,18 @@ std::string action_error(const Directives& d); void prepare_actions(std::vector& actions, const std::filesystem::path& pkgRoot); +// Does this action output belong in the COMPILE set? +// +// A `source` action routinely emits companion files that must exist but must +// not be compiled — protoc writes `foo.pb.cc` AND `foo.pb.h`, and the header +// is an include, not a translation unit. Adopting everything gave both the +// same object path and tripped the uniqueness assertion +// ("object path collision after uniqueness pass"). +// +// The non-source outputs are still declared to ninja, so the edge still +// produces them and anything that includes them still waits for the generator. +bool is_compilable_output(const std::filesystem::path& p); + // ── Private-scope fold (was prepare.cppm's DirectiveMark / fold pair) ────── // // Lives here because "which compile-visible channels a PackagePrivate @@ -547,6 +559,16 @@ std::string action_error(const Directives& d) { return {}; } +bool is_compilable_output(const fs::path& p) { + auto ext = p.extension().string(); + // The same set the plan treats as translation units, plus .cppm/.ixx for a + // generated module interface. + return ext == ".cpp" || ext == ".cc" || ext == ".cxx" || ext == ".c" + || ext == ".m" || ext == ".mm" + || ext == ".S" || ext == ".s" || ext == ".asm" + || ext == ".cppm" || ext == ".ixx"; +} + void prepare_actions(std::vector& actions, const fs::path& pkgRoot) { for (auto& a : actions) { diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index b490062a..18645253 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -791,6 +791,24 @@ export struct BuildOverrides { int tool_depth = 0; // The request chain, for that diagnostic. "root → grpc:grpc_cpp_plugin → …" std::string tool_chain; + // Use THIS manifest instead of reading `/mcpp.toml`. + // + // Required for a `compat`-style registry package (Form B), which ships no + // mcpp.toml at all — its manifest is synthesized from the `.lua` + // descriptor during resolution. Without this the tool sub-build could only + // ever handle packages that carry their own manifest (Form A), which + // excludes most of the index, protobuf among them. + // + // Must be the PRISTINE manifest, before feature activation: the sub-build + // activates its own feature set, and starting from an already-activated + // copy would fold the same feature sources in twice. + // A shared_ptr rather than an optional: BuildOverrides is an + // EXPORTED struct, and embedding a large value type in the module + // interface made GCC fail to write the cluster at all + // ('failed to read compiled module cluster ...: Bad file data' when + // mcpp.build.execute imported it). A pointer keeps the exported layout + // trivial, and it also avoids copying the manifest per tool build. + std::shared_ptr preloaded_manifest; std::string target_triple; // empty = host triple, fall through to [toolchain] bool force_static = false; // --static (or implied by musl target) std::string package_filter; // -p : only build this workspace member @@ -882,7 +900,19 @@ prepare_build(bool print_fingerprint, // value puts a member's target/, mcpp.lock and .mcpp/ at the WORKSPACE // root. See the derivation right after that block. - auto m = mcpp::manifest::load(*root / "mcpp.toml"); + // A registry package in `compat` form (Form B) ships NO mcpp.toml — its + // manifest is synthesized from the `.lua` descriptor by the resolver. So a + // nested build of such a package cannot re-read one off disk, and the + // caller hands over the manifest it already synthesized instead. + // + // Passing it in rather than re-deriving it is also the more correct of the + // two: re-deriving could produce a DIFFERENT manifest than the one the + // parent resolved against (the L1 cfg merge and feature-activated deps + // have already been folded in by then). + auto m = overrides.preloaded_manifest + ? std::expected( + *overrides.preloaded_manifest) + : mcpp::manifest::load(*root / "mcpp.toml"); if (!m) return std::unexpected(m.error().format()); // ─── Workspace handling ──────────────────────────────────────────── @@ -996,6 +1026,30 @@ prepare_build(bool print_fingerprint, std::filesystem::create_directories(workRoot, wdEc); } + // A `compat`-form (Form B) package's sources live under a wrap directory + // inside the version dir, which is why its descriptor writes globs like + // `*/src/foo.cc` — the `*` stands for the tarball's top-level folder, + // whose name the descriptor cannot know. `[build] sources` has always + // expanded those; `targets..main` did NOT, so a bin target in such a + // package handed ninja a literal `*` and died with + // `missing and no known rule to make it`. + // + // Nothing could reach that path before #355 (a dependency's bin targets + // were never built), which is why it went unnoticed. Resolve it here, once + // the manifest is final and before anything reads `t.main`. + for (auto& t : m->targets) { + if (t.main.empty() || t.main.find('*') == std::string::npos) continue; + auto hits = mcpp::modgraph::expand_glob(*root, t.main); + if (hits.size() == 1) { + t.main = std::filesystem::relative(hits.front(), *root).generic_string(); + } else { + return std::unexpected(std::format( + "target '{}': `main = \"{}\"` matched {} files; it must name " + "exactly one entry source", + t.name, t.main, hits.size())); + } + } + // Inject synthetic targets (e.g. test binaries from `mcpp test`). for (auto& t : extraTargets) m->targets.push_back(t); @@ -2729,6 +2783,9 @@ prepare_build(bool print_fingerprint, if (a.role != mcpp::manifest::BuildAction::Role::Source) continue; for (auto const& o : a.outputs) { if (o.find("${mcpp.") != std::string::npos) continue; + // Companion outputs (protoc's .pb.h next to its .pb.cc) are + // produced by the edge but are NOT translation units. + if (!mcpp::build::directives::is_compilable_output(o)) continue; mm.buildConfig.sources.push_back(o); mm.modules.sources.push_back(o); } @@ -4119,6 +4176,18 @@ prepare_build(bool print_fingerprint, sub.profile = "release"; sub.cache_mode = overrides.cache_mode; sub.tool_depth = overrides.tool_depth + 1; + // The PRISTINE manifest the resolver produced for this + // package — `packages[depIdx].manifest` is a copy that + // feature activation has already mutated, and re-activating + // on top of it would fold the same feature sources in + // twice. A `compat` (Form B) package has no mcpp.toml on + // disk at all, so without this the sub-build could not read + // a manifest for it in the first place. + if (depIdx >= 1 && depIdx - 1 < dep_manifests.size() + && dep_manifests[depIdx - 1]) + sub.preloaded_manifest = + std::make_shared( + *dep_manifests[depIdx - 1]); sub.tool_chain = overrides.tool_chain.empty() ? std::format("root → {}:{}", depName, toolName) : std::format("{} → {}:{}", overrides.tool_chain, depName, diff --git a/tests/e2e/188_build_actions.sh b/tests/e2e/188_build_actions.sh index 5c72e369..4080c8a3 100755 --- a/tests/e2e/188_build_actions.sh +++ b/tests/e2e/188_build_actions.sh @@ -144,6 +144,52 @@ if "$MCPP" build > b4.log 2>&1; then fi rm -f FAIL_THE_CHECK +# ── 2c. a companion output that is NOT a translation unit ────────────────── +# The single most natural generator shape: protoc emits foo.pb.cc AND foo.pb.h. +# Adopting every declared output into the compile set gave both the same object +# path and tripped "object path collision after uniqueness pass". The header +# must still be PRODUCED by the edge (things include it) but never compiled. +cat > genpair.sh <<'EOF' +#!/usr/bin/env bash +# $1 = .cc to write, $2 = .h to write +printf '#include "%s"\nint paired() { return 13; }\n' "$(basename "$2")" > "$1" +printf 'int paired();\n' > "$2" +EOF +chmod +x genpair.sh +cat >> src/main.cpp <<'EOF' +EOF +cat > build.mcpp <<'EOF' +#include +#include +import mcpp; +int main() { + const std::string root = mcpp::manifest_dir(); + const std::string out = mcpp::out_dir(); + mcpp::action a; + a.id = "pair"; a.role = "source"; + a.arg((root + "/genpair.sh").c_str()) + .arg((out + "/p.cc").c_str()) + .arg((out + "/p.h").c_str()) + .output((out + "/p.cc").c_str()) + .output((out + "/p.h").c_str()) // companion: produced, NOT compiled + .submit(); + mcpp::include_dir(out.c_str()); + // Keep the earlier generated source in the build so main.cpp still links. + mcpp::action g; + g.id = "generate"; g.role = "source"; + g.arg((root + "/gen.sh").c_str()).arg((root + "/data/other.txt").c_str()) + .arg((out + "/gen.cpp").c_str()) + .input((root + "/data/other.txt").c_str()) + .output((out + "/gen.cpp").c_str()) + .submit(); +} +EOF +rm -rf target +"$MCPP" build > b2c.log 2>&1 || { + cat b2c.log; echo "FAIL: a companion (non-source) action output broke the build"; exit 1; } +grep -q "object path collision" b2c.log && { + cat b2c.log; echo "FAIL: the header was adopted as a translation unit"; exit 1; } + # ── 3b/3c: their own minimal project ─────────────────────────────────────── # Separate from `app`, whose main.cpp deliberately depends on the generated # symbol — swapping its build.mcpp out would fail the LINK and say nothing From 5d6e40868f6cac4d23197cb76fe2e60f77b174dc Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 15:35:03 +0800 Subject: [PATCH 15/16] =?UTF-8?q?docs(355):=20=E8=AE=B0=E5=BD=95=E7=9C=9F?= =?UTF-8?q?=E5=AE=9E=E6=A1=88=E4=BE=8B=E9=AA=8C=E8=AF=81=E7=BB=93=E6=9E=9C?= =?UTF-8?q?=20=E2=80=94=E2=80=94=20libprotoc=20=E5=8F=AF=E6=9E=84=E5=BB=BA?= =?UTF-8?q?,=E9=A3=8E=E9=99=A9=E8=A7=A3=E9=99=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 设计文档 §11 标注的未核实风险(libprotoc 能否被 mcpp 无 CMake 构建)现在有答案: **能**。138 个 TU 全部编过,源码树无 .h.in/.cmake.in,唯一缺口是 upb 运行时, 而那是 compat.protobuf 已有的 feature —— 用 required_features 就能表达,零引擎改动。 ⇒ Phase 2 的 prebuilt provider 确认为纯优化,不是必需路径。 另记三个只有真实案例(Form B 包)才暴露的 bug,与一个 GCC modules 约束 (导出结构体里不要放大的值类型)。 --- ...5-issue355-dependency-host-tools-design.md | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md b/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md index e1b0242e..7cf2c944 100644 --- a/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md +++ b/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md @@ -808,3 +808,63 @@ ninja 期执行、增量、并行；工具从本设计的 tool store 取。收 - CMake cross：； LLVM `LLVM_NATIVE_TOOL_DIR` - Zig build system： + +--- + +## 13. 实施与真实案例验证（2026.8.5.1） + +### 13.1 §11 第 4 步的「未核实风险」——已核实，结论是**能** + +设计里写着：`compat.protobuf` 描述符明文「does NOT build libprotoc」，而 +**libprotoc 能否被 mcpp 无 CMake 构建出来尚未核实**；若不能，Phase 2 的 +prebuilt provider 就从「优化」升级为 protoc 的唯一可行路径。 + +拿真实的 mcpp-index + protobuf 35.1 源码实测： + +- 上游 `src/file_lists.cmake` 的 `libprotoc_srcs` = **138 项**，与 libprotobuf 的 + 源码集**零重叠**（`importer.cc` / `parser.cc` 早已在 libprotobuf 里） +- 源码树里**没有** `.h.in` / `.cmake.in` —— 不需要任何 configure 步骤 +- 138 个 TU 全部编过；第一次链接失败，缺 `upb_*` 符号 —— libprotoc 的 upb 生成器 + 需要 upb 运行时，而那正是 compat.protobuf **已有**的 `upb` feature +- 把 target 写成 `required_features = { "protoc", "upb" }` 后**链接通过** + —— 这正是成本门机制该起的作用，**零引擎改动** + +端到端：`protoc` 从源码建出 → `mcpp::dep_bin("protobuf","protoc")` 拿到路径 → +`action` 调用它生成 `demo.pb.cc` / `demo.pb.h` → 编译链接 → 程序输出 `NAME=mcpp`。 +store 命中实测：第二次 `rm -rf target` 后构建 **1.41s**，不重建 protoc。 + +**⇒ Phase 2（prebuilt-asset provider）确认为纯优化，不是必需路径。** + +### 13.2 真实案例暴露的三个 bug（合成 e2e 全部漏掉） + +四个新 e2e 都用 path 依赖，也就是 **Form A**（包自带 mcpp.toml）。真实索引里绝大 +多数是 **Form B**（compat 描述符，manifest 由 `.lua` 合成），protobuf 就是。这个 +差异一次性暴露了三个 bug： + +| # | 问题 | 后果 | +|---|---|---| +| 1 | 子构建从 `/mcpp.toml` 读 manifest，**Form B 包没有这个文件** | Form B 包**完全不能**当工具提供方 —— gRPC 那条链整个走不通 | +| 2 | `targets..main` **不展开** `*/` 包装 glob（`sources` 一直会展开） | Form B 包的任何 bin target 都拿不到入口源码 | +| 3 | `role=source` 的**全部**输出都被当成翻译单元 | protoc 的 `.pb.h` 与 `.pb.cc` 撞同一个对象路径 | + +修法分别是：`BuildOverrides::preloaded_manifest`（由调用方交出**未经 feature 激活** +的那份 manifest —— `packages[i].manifest` 是被 `apply()` 改过的副本，从它出发会把 +同一批 feature 源码折两次）、在 manifest 定稿后解析 `main` 的 glob、 +`is_compilable_output()` 按扩展名过滤。 + +**方法论**：合成测试测的是「我想到的形状」，真实案例测的是「现实的形状」。这三个 +bug 没有一个能靠再多写几个 path-依赖 e2e 发现。 + +### 13.3 一个 GCC modules 约束 + +`preloaded_manifest` 初版写成 `std::optional`，而 `BuildOverrides` 是 +**导出**结构体 —— GCC 写不出 module cluster： + +``` +mcpp.build.prepare: error: failed to read compiled module cluster 529: Bad file data +src/build/execute.cppm:52:45: fatal error: failed to load pendings for 'std::pair' +``` + +`rm -rf target` 无效（不是陈旧 BMI）。换成 `shared_ptr` 让导出布局 +保持 trivial 后即通过，顺带省掉每次工具构建的一次 manifest 拷贝。 +**导出结构体里不要放大的值类型。** From 2f2f7a69622a46de42723925227279bcc2fce266 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Wed, 5 Aug 2026 15:35:27 +0800 Subject: [PATCH 16/16] =?UTF-8?q?docs(355):=20=E7=8A=B6=E6=80=81=E6=94=B9?= =?UTF-8?q?=E4=B8=BA=E5=B7=B2=E5=AE=9E=E6=96=BD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../docs/2026-08-05-issue355-dependency-host-tools-design.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md b/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md index 7cf2c944..4f593336 100644 --- a/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md +++ b/.agents/docs/2026-08-05-issue355-dependency-host-tools-design.md @@ -1,6 +1,7 @@ # issue #355：依赖产出的 host 工具（codegen 工具链缺口） -> 状态：**设计待 review，未实施**（2026-08-05 追加 §12 行业调研，并据此调整了两处，见 §12.4） +> 状态：**已实施（2026.8.5.1）**。§12 行业调研与据此调整的两处见 §12.4; +> 实施与**真实案例验证**（含 §11 那条风险的结论）见 §13。 > 关联：#355（本条）、#241（`MCPP_DEP__DIR`）、#274（显式 ninja 目标）、#344（cache 地址） > 涉及（预估）：`src/build/prepare.cppm`、`src/build/build_program.cppm`、 > `src/build/hostprogram.cppm`、`src/pm/dep_spec.cppm`、`src/manifest/toml.cppm`、