Skip to content

Commit ca1d40e

Browse files
committed
feat: 依赖产出的 host 工具、构建图节点、规则包(架构文档步 2–6)
实现 `.agents/docs/2026-08-05-build-mcpp-extensibility-architecture.md` 的步 2–6, 接在同一文档步 0+1 之后。核心是把 build.mcpp 里「**配置**」与「**施工**」这两件被 混在一起的事分开:程序继续回答「这次构建长什么样」,而「把这批输入变成那批输出」 交给构建图。 ## 步 2 — 依赖产出的 host 工具(#355) 一个包能构建出消费者构建期需要的二进制(protoc / grpc_cpp_plugin / flatc / 转译器), 但消费者拿不到:`dep_dir()` 给的是源码树,依赖的 `kind="bin"` target 从不被构建。 **为什么不能是主图里的节点**:时序。build.mcpp 跑在 prepare 内,BuildPlan 还不存在、 build.ninja 更在其后 —— 主图产出的东西对需要它的程序永远来得太晚;叠上交叉编译连 架构都不对。所以走**嵌套的、面向 host 的子构建** + 全局 store(Cargo build-dependencies / Bazel exec configuration / vcpkg host:true / Conan tool_requires 的形状)。 **为什么便宜**:工具是可执行文件,与主构建零 ABI 接触 ⇒ 子构建可用工具包自己的 toolchain / profile / 依赖解析,不必与消费者一致。 单一版本轴(工具版本=依赖版本,protoc 与运行时错配结构上不可表达)、默认关闭、 成本门复用已有 features+required_features、全局 store 按 版本×host 工具链×feature× 依赖闭包 缓存。逃生舱 `[tools.overrides]` / `MCPP_TOOL_*` 直接指现成二进制并**跳过 构建** —— 每个同类系统都有这一条,且**刻意不进 cache key**。 前提是**工作目录外置**:一次 build 往工程根写 5 处,而「注册表包根共享/可能只读/ 绝不写入」是 build_program.cppm 自 G2 起的明文不变量。五处一起搬 —— 只搬一部分比 一处不搬更糟。 ## 步 3+4 — `mcpp:action=` 构建图节点 一个原语三种接线(role 只决定输出接到哪):source 进编译集、check 产 stamp 且**默认 与编译并行**、artifact 的输入是链接产物。顺序全由 ninja 的文件依赖决定,不需要 phase 机制 —— 这也是 artifact 不会像朴素 post 钩子那样把自己重复施加一遍的原因。 **必须写出输出文件名**(INV-D):prepare 期就定死源码集/fingerprint/模块图。畸形 action 是硬错误。命令是 argv 而非 shell 字符串,插值只有封闭的四个。 ## 步 5 — `host-module = true` 规则包 规则以普通 mcpp 库包分发,消费者 `import mcpp.rules.x;`。有版本、能测试、能发布, 用 **C++** 写 —— 不引入第二门语言(xmake Lua rule / Bazel Starlark)。 **关键**:规则模块与 build.mcpp **同一条命令、同一套 flag** 编译。不是优化 —— BMI 只 对与它在 standard/dialect/编译器身份上一致的编译可用,两次独立解析的构建没有理由 一致,而不一致表现为 `module X CRC mismatch` 而非清楚的错误。 ## 步 6 — 生成的 .cppm 核实结论:真正的阻塞点不是 topoOrder 的**顺序**(那只是名字普查 + 发射次序),而是 未扫描的文件**根本没有 graph.units 条目**。解法用代码库已有的答案 ——「声明而非发现」: action 的 `.provides()/.imports()` 让 mcpp 播下带该声明的占位文件,prepare 期扫描与 生成器将产出的内容一致,build 期由编译器自己的 P1689 复核。 ## 顺带修 `.xlings.json` 自举 pin 指向索引已不再提供的 2026.8.3.2 —— `ci-aarch64-fresh-install` 自 2026-08-03 在 main 上就是红的。自举 pin 平时不该动,但它有一条硬约束是必须命名 可安装的版本,这条被打破时正是该 bump 的场合。 ## 验证 - 单测 56/56 - e2e 19/21 通过;失败的 `07_static_library`(本机 binutils payload 的 ar 跑不起来)与 `09_path_dependency`(ninja missing dep BMI)在**已发布的 2026.8.4.1 上同样失败** ⇒ 环境性,非回归 - 新增 3 个 e2e:187(host 工具:端到端 / 成本门 / 默认关闭 / 错名报可用列表 / override)、 188(三种 role + 增量 + 失败的 check 让构建失败 + 畸形 action 被拒)、 189(规则包:导入生效 / 编辑规则触发重跑 / 缺 lib root 的诊断)
1 parent 3cafdb5 commit ca1d40e

18 files changed

Lines changed: 1913 additions & 16 deletions

CHANGELOG.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,40 @@
3030

3131
**编译**这一步刻意不设上限——与 `mcpp test` 同一条不对称纪律(run 有限 / build 无限):编译跑得久通常是正当的(首次构建 `std` 模块就是分钟级),杀掉它只会产生莫名其妙的失败;构建**程序**跑得久通常是卡住了。`capture_exec_deadline` 顺带补上了 `cwd` 形参——没有它,加超时会**静默改变**构建程序相对写入的落点。
3232

33+
### 新增
34+
35+
- **依赖产出的 host 工具:`tools = ["protoc"]`(#355)。** 一个包能构建出消费者在**构建期**需要的二进制(protoc、grpc_cpp_plugin、flatc、moc、转译器),但消费者此前完全拿不到它 —— `mcpp::dep_dir()` 给的是**源码树**,而依赖的 `kind = "bin"` target **从不被构建**(`plan.cppm` 只遍历 root 的 targets;唯一的例外 `kind = "shared"` 是按 `--target` 构建的,当 host 工具用不了)。
36+
37+
**为什么它不能是主图里的一个节点**:时序。`build.mcpp` 跑在 prepare 内,那时 BuildPlan 还不存在、build.ninja 更在其后 —— 主图产出的东西对需要它的程序来说**永远来得太晚**。再叠上交叉编译,它连架构都不对。所以工具由**嵌套的、面向 host 的子构建**产出,落进全局 store。这正是 Cargo `[build-dependencies]` / Bazel exec configuration / vcpkg `"host": true` / Conan `tool_requires` 的形状。
38+
39+
**为什么它便宜**:工具是**可执行文件**,与主构建零 ABI 接触。子构建因此可以用工具包自己的 `[toolchain]`、自己的 profile、自己的依赖解析,不必与消费者一致 —— 对照 `kind = "lib"` 依赖,这几条**必须**一致。
40+
41+
**单一版本轴**:工具的版本就是依赖的版本,所以「protoc 与 protobuf 运行时错配」这类**运行期**才炸的问题结构上不可表达。默认关闭(成本由消费者付),成本门复用已有的 `[features]` + `required_features`。全局 store 按 包版本 × host 工具链 × feature × 自身依赖闭包 缓存。
42+
43+
**逃生舱** `[tools.overrides]` / `MCPP_TOOL_<PKG>_<TOOL>`:直接指一个现成二进制,**完全跳过构建**。每个同类系统都提供这一条(vcpkg `VCPKG_HOST_TRIPLET`、CMake `LLVM_NATIVE_TOOL_DIR`、Qt `QT_HOST_PATH`),理由一样 —— 源码在本机构建不出来的工具不能是死路。它**刻意不进 cache key**:逃生舱不是可复现输入。
44+
45+
- **`mcpp:action=`:声明构建图节点,而不是在 build.mcpp 里干活。** 在程序里直接写生成逻辑,是每次 prepare 跑一遍、全量、串行,失败报「build.mcpp exited 1」。**声明**成节点后它是图里的一条边 —— 增量、并行、失败可归因到具体那条边。
46+
47+
**一个原语,三种接线**(`role` 只决定输出接到哪,不是三套机制):`source` 进编译集(protoc、转译器)、`check` 产出 stamp 且**默认与编译并行**(clang-tidy、格式/ABI 检查;`blocking` 可改成前置)、`artifact`**输入**是链接产物(签名、打包、size budget)。顺序完全由 ninja 的文件依赖决定,不需要任何 phase 机制 —— 这也是为什么 `artifact` 不会像朴素的「post 钩子」那样把自己重复施加一遍。
48+
49+
**必须写出输出文件名**:mcpp 在 prepare 期就定死源码集、fingerprint 与模块图,名字未知的产物无法进图。内容可以晚到,名字不行。畸形 action 是**硬错误**而非静默跳过。生成**模块接口**时用 `.provides()/.imports()` 声明,mcpp 会按该声明播下占位文件让 prepare 期的扫描与生成器将要产出的内容一致 —— 与 `[modules].scan_overrides` 同一条「声明+验证」的取舍,build 期由编译器自己的 P1689 复核。
50+
51+
命令是 **argv 而非 shell 字符串**(不假设存在 shell),插值只有封闭的四个:`${mcpp.out_dir}` / `${mcpp.bin_dir}` / `${mcpp.compile_db}` / `${mcpp.target_file:<name>}`
52+
53+
- **`host-module = true`:可复用的构建规则以普通包分发。** 「跑 protoc」这类规则应该写一次,而不是每个消费者的 build.mcpp 复制一遍。把它做成普通 mcpp 库包,消费者 `import mcpp.rules.protobuf;` 即可。规则因此**有版本、能测试、能发布**,走的是已有的包管理机制,而且是用 **C++** 写的 —— 不引入第二门语言(xmake 用 Lua rule、Bazel 用 Starlark),这正是 build.mcpp 存在的理由。
54+
55+
实现上的关键:规则模块与 build.mcpp **在同一条命令里、用同一套 flag** 编译。这不是优化 —— BMI 只对「在 standard / dialect / 编译器身份上与它一致」的编译可用,分成两次独立解析的构建则毫无理由一致,而不一致的表现是 `module X CRC mismatch` 而不是一条清楚的错误。
56+
57+
- **工作目录可外置(`BuildOverrides::work_dir`)。** 一次 `mcpp build` 会往工程根写 5 处(`target/``mcpp.lock``compile_commands.json``.mcpp/``target/.build-mcpp`),而「注册表包根是共享的、可能只读、绝不写入」是 `build_program.cppm` 自 G2 起的明文不变量 —— 在此之前没有任何东西能对 build.mcpp 的临时目录之外兑现它。把「源码在哪」与「往哪写」拆开,是 host 工具子构建**能够存在**的前提。五处一起搬:只搬一部分比一处都不搬更糟,那等于照样写进共享目录、只是更不显眼。
58+
59+
### 改进
60+
3361
- **内带 xlings 升级到 `2026.8.5.1`**(自 `2026.8.4.1`)。13 个 pin 点由 `check_version_pins.sh` 机器校验并全部更新。
3462

63+
### 修复
64+
65+
- **自举 pin 指向了索引已不再提供的版本。** `ci-aarch64-fresh-install` 自 2026-08-03 起在 main 上就是红的:`version '2026.8.3.2' not found for 'mcpp'``.xlings.json` 的自举 pin 是**自举起点**、故意滞后、不随发布走(docs/09 §4),平时不该动 —— 但它有一条硬约束是**必须命名一个可安装的版本**,这条被打破时正是该 bump 的场合(而不是「发版顺手 bump」那种误用)。改到 `2026.8.4.1`
66+
3567
## [2026.8.4.1] — 2026-08-04
3668

3769
### 修复

docs/05-mcpp-toml.md

Lines changed: 85 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ required_features = ["gui"] # only built when feature `gui` is
123123
|---|---|
124124
| `defines` | Preprocessor macros (`name` or `name=value`); desugar to `-D<x>` on both the C and C++ entry compile. |
125125
| `cxxflags` / `cflags` | Extra compile flags for this target. Do **not** put `-std=...` here — use `[package].standard`. |
126-
| `required_features` | The target is emitted only when **every** listed feature is active in the build; otherwise it is silently skipped. A gate only — it does not activate features (use `--features` / `[features].default`). |
126+
| `required_features` | The target is emitted only when **every** listed feature is active in the build; otherwise it is silently skipped. A gate only — it does not activate features (use `--features` / `[features].default`). **One exception, and it is not a second rule:** when this target is requested as a host tool (`tools = [...]`, §2.14), the target is what was *asked for*, so its `required_features` become the sub-build's *inputs*. Same field, one meaning — the resolution just runs in the opposite direction. |
127127

128128
> **Scope (important):** `defines` / `cxxflags` / `cflags` on a target apply **only to that
129129
> target's exclusive entry source** (its `main`) — never to shared module/impl objects, which
@@ -964,6 +964,90 @@ build needs (`make`/`cmake`/`protoc`/…), pin tool versions per project, or set
964964
build-time env vars — without hand-editing `.xlings.json`. `[toolchain]` (§2.7) remains
965965
the ergonomic shorthand for the compiler; `[xlings.workspace]` is the general form.
966966

967+
### 2.14 Host tools from a dependency (mcpp 2026.8.5.1+)
968+
969+
A package can build a binary its consumers need *at build time*`protoc`, a
970+
`grpc_cpp_plugin`, `flatc`, `moc`, a transpiler. Ask for it on the dependency:
971+
972+
```toml
973+
[dependencies]
974+
protobuf = { version = "35.1", tools = ["protoc"] }
975+
grpc = { version = "1.83.0", tools = ["grpc_cpp_plugin"] }
976+
```
977+
978+
Each name must be a `kind = "bin"` target of that package. mcpp builds it **for
979+
the build machine** and hands `build.mcpp` its absolute path as
980+
`MCPP_DEP_<PKG>_BIN_<TOOL>` — read it with `mcpp::dep_bin("protobuf", "protoc")`
981+
(see [07 — build.mcpp](07-build-mcpp.md)).
982+
983+
Four properties worth knowing:
984+
985+
- **Always a host binary.** Under `mcpp build --target <triple>` the tool is
986+
still built for *this* machine, because a code generator has to run here. It
987+
is a separate, host-targeted sub-build — the tool package's own `[toolchain]`
988+
and its own dependency resolution apply, and none of it has to agree with
989+
your build. That is safe precisely because an executable has no ABI contact
990+
with your code.
991+
- **One version axis.** The tool's version *is* the dependency's version, so
992+
a `protoc` that does not match its runtime is not expressible. (This is the
993+
problem with packaging the tool separately, and it is the failure mode that
994+
bites at run time rather than compile time.)
995+
- **Default off.** Nothing is built unless someone asks; the cost is the
996+
consumer's to pay. A package gates the expensive part with
997+
`[features]` + `required_features` (protobuf's `protoc` needs libprotoc's
998+
~157 extra TUs, which the runtime's users must not compile).
999+
- **Cached globally**, keyed on package version × host toolchain × features ×
1000+
its own dependency closure — built once per machine, not once per project.
1001+
1002+
#### `[tools.overrides]` — use a binary you already have
1003+
1004+
```toml
1005+
[tools.overrides]
1006+
"compat.protobuf:protoc" = "/usr/bin/protoc"
1007+
```
1008+
1009+
or, without editing the manifest (CI, distro packaging):
1010+
1011+
```bash
1012+
MCPP_TOOL_PROTOBUF_PROTOC=/usr/bin/protoc mcpp build
1013+
```
1014+
1015+
An override **skips the build entirely**. Every comparable system provides this
1016+
escape hatch (vcpkg's `VCPKG_HOST_TRIPLET`, CMake's `LLVM_NATIVE_TOOL_DIR`,
1017+
Qt's `QT_HOST_PATH`), and for the same reason: a tool that cannot be built from
1018+
source on this machine must not be a dead end. It is deliberately **not** part
1019+
of the cache key — an override is an escape hatch, not a reproducible input.
1020+
1021+
#### `host-module = true` — reusable build rules as packages
1022+
1023+
A rule (say "run protoc over these `.proto` files") should be written once, not
1024+
copy-pasted into every consumer's `build.mcpp`. Ship it as an ordinary mcpp
1025+
library package and import it:
1026+
1027+
```toml
1028+
[dependencies]
1029+
"mcpp.rules.protobuf" = { version = "0.1.0", host-module = true }
1030+
```
1031+
1032+
```cpp
1033+
// build.mcpp
1034+
import mcpp;
1035+
import mcpp.rules.protobuf;
1036+
int main() { mcpp::rules::protobuf::generate(/**/); }
1037+
```
1038+
1039+
mcpp compiles that package's lib-root module **for the host, in the same
1040+
command as `build.mcpp`** — which is what makes the BMI usable at all, since a
1041+
module interface is only importable by a compile that agrees with it on
1042+
standard, dialect and compiler identity.
1043+
1044+
Rules are therefore versioned, testable and distributable through the package
1045+
manager you already have, written in **C++** — no second language, which is the
1046+
whole point of `build.mcpp` existing.
1047+
1048+
*Limit:* the rule interface is compiled alone, so it may import `std` and the
1049+
bundled `mcpp` module, but not a third package. A rule package is a leaf.
1050+
9671051
## Appendix A. Schema Ownership Principle (admission criteria for new fields)
9681052
9691053
> **Closed syntax, open vocabulary**: whoever owns the parsing semantics defines the keys; whoever owns the domain knowledge defines the values.

docs/07-build-mcpp.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,95 @@ int main() {
9797
| `mcpp::source(p)` | `mcpp:source=` |
9898
| `mcpp::include_dir(d)` / `mcpp::include_dir_after(d)` | `mcpp:include-dir=` / `mcpp:include-dir-after=` |
9999
| `mcpp::rerun_if_changed(p)` / `mcpp::rerun_if_env_changed(v)` | the matching `rerun-*` directives |
100+
| `mcpp::dep_bin(pkg, tool)` *(2026.8.5.1+)* | reads `MCPP_DEP_<PKG>_BIN_<TOOL>` — the absolute path of a **host tool** built by a dependency (see below) |
101+
| `mcpp::action{…}.submit()` *(2026.8.5.1+)* | `mcpp:action=` — declares a **build-graph node** instead of doing the work here (see below) |
102+
103+
### Host tools from a dependency (2026.8.5.1+)
104+
105+
Declare the need in `mcpp.toml`, then call it:
106+
107+
```toml
108+
[dependencies]
109+
protobuf = { version = "35.1", tools = ["protoc"] }
110+
```
111+
112+
```cpp
113+
// build.mcpp
114+
import mcpp;
115+
int main() {
116+
const char* protoc = mcpp::dep_bin("protobuf", "protoc");
117+
// … invoke it, then declare what it produced …
118+
}
119+
```
120+
121+
mcpp builds that `kind = "bin"` target **for the build machine** (even under
122+
`--target`), caches it globally, and hands you the path. The request lives in
123+
`mcpp.toml` rather than here for the same reason a dependency does: asking the
124+
graph for an extra artifact is a graph-level request, and the graph stays
125+
statically analysable. See [05 §2.14](05-mcpp-toml.md) for the full contract,
126+
including `[tools.overrides]`.
127+
128+
### Declaring work instead of doing it: `mcpp::action` (2026.8.5.1+)
129+
130+
Generating a source by writing it *here* is the easy path and the wrong one
131+
past a certain size: it happens once per prepare, for the whole set, serially,
132+
and a failure is reported as "build.mcpp exited 1". **Declare** the work and it
133+
becomes an edge in the build graph — incremental, parallel, and attributable to
134+
the edge that failed.
135+
136+
```cpp
137+
import mcpp;
138+
int main() {
139+
const std::string out = std::string(mcpp::out_dir()) + "/foo.pb.cc";
140+
mcpp::action a;
141+
a.id = "protoc:foo";
142+
a.role = "source"; // "source" | "check" | "artifact"
143+
a.arg(mcpp::dep_bin("protobuf", "protoc"))
144+
.arg("--cpp_out=...").arg("proto/foo.proto")
145+
.input("proto/foo.proto")
146+
.output(out.c_str())
147+
.submit();
148+
}
149+
```
150+
151+
Three roles, one primitive — `role` only decides where the edge's outputs
152+
attach:
153+
154+
| `role` | Outputs | Ordering | Typical |
155+
|---|---|---|---|
156+
| `source` | join the compile set | the compile edge consumes them | protoc, a transpiler |
157+
| `check` | a stamp file | runs **alongside** compilation (set `blocking = true` to gate it) | clang-tidy, a format or ABI check |
158+
| `artifact` | a new file | its *inputs* are link outputs, so it runs after the link | codesign, packaging, size budgets |
159+
160+
No phase machinery is involved: ninja's own file dependencies do the
161+
sequencing, which is also why an `artifact` action cannot double-apply itself
162+
the way a naive "post-build hook" would.
163+
164+
**You must name the output files.** mcpp fixes the source set, the fingerprint
165+
and the module graph during prepare, so an output whose *name* is unknown
166+
cannot be built. Content may arrive later; names may not. A malformed action is
167+
a hard error, never a silent skip.
168+
169+
For a generated **module interface**, declare its interface too:
170+
171+
```cpp
172+
a.output(gen.c_str()).provides("my.generated").imports("std").submit();
173+
```
174+
175+
mcpp seeds a placeholder carrying exactly that declaration so the prepare-time
176+
scan agrees with what your generator will emit — the same assertion-plus-
177+
verification trade `[modules].scan_overrides` makes, and the compiler's own
178+
P1689 output checks it at build time.
179+
180+
Commands are an **argv, not a shell string** (no shell is assumed — Windows has
181+
none to rely on), and the only interpolations are a closed set:
182+
183+
| Variable | Value |
184+
|---|---|
185+
| `${mcpp.out_dir}` | the build output directory |
186+
| `${mcpp.bin_dir}` | where produced binaries land |
187+
| `${mcpp.compile_db}` | path to `compile_commands.json` (what clang-tidy's `-p` wants) |
188+
| `${mcpp.target_file:<name>}` | the built file of target `<name>` |
100189

101190
The raw stdout protocol above remains the low-level substrate; `import mcpp;`
102191
is the typed layer over it.

0 commit comments

Comments
 (0)