Skip to content

Commit 655e58d

Browse files
SunrisepeakSunrisepeak
authored
refactor(toolchain): one producer for host-compile flags; build.mcpp gains MSVC modules (2026.8.2.2) (#334)
* chore: bump to 2026.8.2.2 + host-compile design/plan docs * refactor(toolchain): one producer for host-compile flags Three places assembled the same host-compile flags independently: flags.cppm for the main build, stdmod.cppm for the std module, and build_program.cppm for build.mcpp. The resolvers (linkmodel, clang driver model) were already shared; the ASSEMBLY was not, because the seam only produced strings and the one consumer that needs argv — build.mcpp, which execs directly with no shell — could not use it and hand-wrote the whole thing a second time. That split is where PR#332's batch came from. Every bug in it was the same sentence: flags.cppm knew about quoting / the macOS deployment target / the MSVC dialect, and the other two did not. A 0.0.9x fix had already corrected the same file once for the same reason (musl->static re-derived), and stdmod.cppm carried a comment instructing the reader to keep it in sync with flags.cppm by hand — the failure mode this repo wrote check_version_pins.sh to stop. Tokens are now the source of truth and strings are rendered from them, not the reverse. mcpp.toolchain.hostflags composes the resolvers once; the three consumers render with ninja escaping, shell quoting, or not at all. The knobs on HostFlagOptions are documented divergences, not switches to preserve accidents: the host helper keeps trusting clang's cfg on macOS/Windows (the macOS link needs libc++abi/unwind handling the main build's needs_explicit_libcxx path owns), the std module states -stdlib=libc++ explicitly (flags.cppm cannot — the same string feeds C compiles), and binutils -B / runtime dirs belong only to the one-shot host link. Pure refactor, verified as such: the global cxxflags/cflags/ldflags lines in build.ninja AND stdmod's std_build_commands are byte-identical before and after, under both gcc@16.1.0 and llvm@22.1.8, each from a clean target and dep cache. Byte-identity is not cosmetic here — std_build_commands feeds the std cache directory name, so a reordered flag would invalidate every user's std BMIs for no gain. hostflags lives in its own module rather than inside build_program.cppm's anonymous namespace: PR#332 established that adding a function there can miscompile a neighbouring one under clang 22.1.8 + C++20 modules + -O2. * feat(build.mcpp): named modules wherever mcpp can build a host program `import mcpp;` / `import std;` in a build.mcpp used to fail under native MSVC with "not yet supported", and the stated reason was that mcpp had not wired up cl.exe's .ifc pipeline. That reason was wrong: the main build has compiled named modules with cl.exe for a while — e2e 99 produces real .ifc artifacts on every Windows CI run, and ninja_backend has emitted /interface /TP /ifcOutput and /scanDependencies all along. What was missing was only that build.mcpp hand-rolled its own compile path and never read those tables. build_mcpp_module now dispatches on BmiTraits + CommandDialect — the same rows the main build uses — so cl.exe needed no new pipeline, just the row that was already there. The gate is gone. bmi_reference_tokens covers the shape difference the traits do not have to care about: `-fmodule-file=std=<p>` is one argv word, `/reference std=<p>` is two, and only an argv consumer notices. The traits keep storing the string form for the ninja channel. The capability-parity guard in test_hostflags.cpp is the part meant to outlast this change: it walks every CompilerId and fails if a family is missing a dialect row, a module row, or host flags. Stating "build.mcpp has no capability list of its own" in prose is exactly how the MSVC gate outlived its own justification. * fix(toolchain): keep -stdlib=libc++ where the std module has always put it Placing it after the libc++ -isystem flags instead of right after -nostdinc++ changed the std module's command string, and that string is part of the std cache identity (std_build_commands feeds the cache directory name) — every user's std BMIs would have been invalidated for a reordering that buys nothing. Caught only because the first byte-equality check scanned the whole shared cache directory and compared stale entries alongside fresh ones; the check now wipes the std cache first so it compares like with like. Also moves the bundled mcpp module's compile out of build_program.cppm's anonymous namespace into mcpp.build.hostprogram. Growing that namespace reproduced PR#332's clang miscompilation exactly — Segmentation fault: 11 on every macOS build.mcpp e2e, in contract_env, which this change never touched. * test(hostflags): pin the rendered flag strings Byte-equality with what earlier releases emitted is a compatibility surface here, not an implementation detail: stdmod folds its compile command into std_build_commands and the std cache directory name is derived from the metadata containing it, so a reordered flag invalidates every user's std BMIs. Verifying that by inspecting the cache after a build turned out to be unreliable — the cache is shared and accumulates entries from every toolchain used since, so the first check compared stale entries alongside fresh ones and passed while the string had in fact moved. These tests assert the spelling directly from a synthetic model, deterministically, and keep asserting it. * fix(build.mcpp): state the deployment target even when trusting clang's cfg; drop -x for cl Two failures the local runs could not see, both the same shape — a branch that was dead while the MSVC gate existed, or a platform this machine is not. host_compile_tokens returned early on the trust-cfg path, which on macOS IS the build.mcpp path: no clang flags and no deployment target, so the std BMI (built for 14.0) was rejected by a TU compiled without it. Trusting the cfg means contributing no include paths or stdlib selection — not contributing nothing. The deployment target is emitted regardless now, which is what the hand-written host_base_flags did deliberately ("FIRST and unconditionally") before this refactor absorbed it. The `-x none` before the module objects was unconditional and harmless only while cl.exe could not reach it. Removing the module gate made the dead branch live, and cl answered `D9002: ignoring unknown option '-x'` — after compiling the .ifc and reaching the link, which is further than build.mcpp has ever gone under MSVC. * fix(toolchain): per-file C++ force for cl, so object inputs stay objects cl.exe rejected the std and mcpp module objects with "C2018: character 'U+10' is not permitted here" — it was compiling them as C++ source. `/TP` is not the counterpart of `-x c++`: GNU's is positional and lasts until `-x none`, cl's applies to EVERY input on the line, so the objects that follow are fed to the C++ frontend. The dialect now carries the per-FILE form (`/Tp<file>`) alongside the positional one, and the build.mcpp compile uses whichever the driver has. That is the same structural difference the row already records for libFlag (prefix vs suffix): the two drivers are not spelling the same concept differently, they have different concepts. * fix(build.mcpp): reference the std BMI through the token helper too cl answered "C2230: could not find module 'std'" because the std reference was still built by string concatenation, producing one argv element with a space inside it. bmi_reference_tokens existed for exactly this and was only applied to the bundled mcpp module. Same shape for the third time — a table entry written for the ninja STRING channel concatenated into an argv element — so it now has a test that covers every family and both spellings, plus one for the language-force tokens. The compiler's own diagnostics name neither the flag nor the reason, which is why this kept costing a CI round each time.
1 parent 7169332 commit 655e58d

15 files changed

Lines changed: 1429 additions & 352 deletions

‎.agents/docs/2026-08-02-host-compile-implementation-plan.md‎

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
# 宿主编译单一生产者 — 实施计划
2+
3+
> 设计:`.agents/docs/2026-08-02-host-compile-single-producer-design.md`
4+
> 基线:main @ 7169332(2026.8.2.1 已发布)· 本批次发 **2026.8.2.2**
5+
6+
**Goal:** 让「宿主编译 flag」只有一个生产者,token 是真源、字符串由 token 渲染;
7+
`build.mcpp` 因此自动获得主构建的全部能力(方言、模块、引号、deployment target),
8+
MSVC 下的模块门可以删除。
9+
10+
---
11+
12+
## Global Constraints
13+
14+
- **版本号只改两处**:`mcpp.toml:3`、`src/toolchain/fingerprint.cppm:21` → `2026.8.2.2`。
15+
`.xlings.json` 是 bootstrap pin,**发布并上架后**才单独 bump。
16+
- **第 1 阶段是纯重构,必须零行为变化**:归一化 `build.ninja` diff 为空,
17+
且 `stdmod` 的 `std_build_commands` 逐字节相同 —— 否则所有用户的 std BMI 缓存
18+
失效重编(缓存目录名派生自含该命令的元数据,`stdmod.cppm:102,135`)。
19+
- **新模块独立**:生产者放 `src/toolchain/hostflags.cppm`,**不往
20+
`build_program.cppm` 的匿名 namespace 加函数**(设计 §6.2)。
21+
- **差异必须显式**:三处装配今天有真实分歧(见下表),共享后要成为**具名选项**,
22+
不是被抹平,也不是保留成无理由的开关。
23+
- **每轮验证双工具链 + 清缓存**:`rm -rf target && mcpp cache clean`,
24+
gcc@16.1.0 与 llvm@22.1.8 各跑一遍;先在基线 commit 上做同法对照。
25+
26+
### 今天的三处分歧(必须保留并显式化)
27+
28+
| 关注点 | `flags.cppm` | `build_program.cppm` | `stdmod.cppm` |
29+
|---|---|---|---|
30+
| clang 有 cfg 时是否绕过 | **全平台绕过** | **只在 Linux 绕过**;macOS/Windows 信 cfg 并返回空(`:214-221` 有理由:macOS 链接要 `needs_explicit_libcxx` 那条路负责 libc++abi/unwind) | 全平台绕过(`:232` 说 `--no-default-config` 在所有平台安全) |
31+
| binutils `-B` | 有(非 musl/mingw 且 libstdc++) | 有(非 musl) | 无 |
32+
| `tc.linkRuntimeDirs` | 走 `depRuntimeLibraryDirs` 另一条路 | `-L` +(仅 ELF)`-rpath` | 无 |
33+
| deployment target | 有 | 有(#332 才补) | 有 |
34+
35+
⇒ 选项面:`clangCfgBypass{Always,LinuxOnly}`、`binutilsB{bool}`、
36+
`runtimeDirs{bool}`。**每个选项在头文件里必须写清"谁需要它、为什么"**,否则
37+
就是把三处推导变成一处推导 + 三个魔法开关。
38+
39+
---
40+
41+
## Task 0:分支与版本号
42+
43+
- [ ] 建分支 `feat/host-compile-single-producer`
44+
- [ ] `mcpp.toml:3` 与 `fingerprint.cppm:21` → `2026.8.2.2`;`.xlings.json` 不动
45+
- [ ] `bash .github/tools/check_version_pins.sh` 通过
46+
- [ ] 提交
47+
48+
## Task 1:基线快照(零 diff 的对照物)
49+
50+
在**动任何代码之前**先取基线,否则无法证明零 diff。
51+
52+
- [ ] **Step 1:** 对 gcc 与 llvm 各生成一次 `build.ninja` 并归一化保存
53+
54+
```bash
55+
snap() { # $1 = 标签
56+
rm -rf target; mcpp cache clean >/dev/null
57+
mcpp build >/dev/null
58+
find target -name build.ninja | head -1 | xargs cat \
59+
| sed -E 's#/[^ ]*/target/[0-9a-f]{16}#<FP>#g' > /tmp/ninja-$1.txt
60+
}
61+
snap base-gcc
62+
sed -i 's|^default = "gcc@16.1.0"$|default = "llvm@22.1.8"|' mcpp.toml
63+
snap base-llvm
64+
git checkout -- mcpp.toml
65+
```
66+
67+
- [ ] **Step 2:** 保存 `std_build_commands` 基线
68+
69+
```bash
70+
find ~/.mcpp/build-cache -name '*.json' -path '*std*' \
71+
| xargs -I{} sh -c 'python3 -c "import json,sys;d=json.load(open(sys.argv[1]));print(sys.argv[1]);print(d.get(\"std_build_commands\"))" {}' \
72+
> /tmp/stdcmd-base.txt
73+
```
74+
75+
## Task 2:token 优先的 seam(`linkmodel.cppm`)
76+
77+
**Files:** `src/toolchain/linkmodel.cppm`
78+
79+
**Produces:**
80+
- `std::vector<std::string> LinkModel::compile_tokens(const PathEscape&) const`
81+
- `std::vector<std::string> ClangDriverModel::compile_tokens(const PathEscape&) const`
82+
- 现有 `compile_flags()` 改为 `render_tokens(compile_tokens(esc))`
83+
84+
- [ ] **Step 1:** 加 `render_tokens`(每个 token 前置一个空格后拼接)
85+
86+
```cpp
87+
// token 是真源,字符串是渲染结果 —— 不是反过来。历史上只有字符串形态,
88+
// 于是需要 argv 的消费者(build_program)只能自己重写一遍装配;
89+
// #332 的 split_ws 是同一个形状的产物。
90+
inline std::string render_tokens(const std::vector<std::string>& tokens) {
91+
std::string out;
92+
for (auto const& t : tokens) { out += ' '; out += t; }
93+
return out;
94+
}
95+
```
96+
97+
- [ ] **Step 2:** `LinkModel::compile_tokens` —— 把现有 `compile_flags` 逐句改成
98+
`push_back`,顺序不变:`--sysroot=<p>`、每个 `-isystem<p>` / `-idirafter<p>`
99+
- [ ] **Step 3:** `ClangDriverModel::compile_tokens` —— `--no-default-config`、
100+
`-nostdinc++`(**两个独立 token**)、每个 `-isystem<p>`
101+
- [ ] **Step 4:** 两个 `compile_flags` 改为 `return render_tokens(compile_tokens(esc));`
102+
- [ ] **Step 5:** 单测:对同一 `LinkModel` / `ClangDriverModel`,
103+
`render_tokens(compile_tokens(esc)) == 旧实现的字符串`(把旧实现内联进测试当预言)
104+
- [ ] **Step 6:** 提交
105+
106+
## Task 3:`hostflags.cppm` 生产者
107+
108+
**Files:** 新建 `src/toolchain/hostflags.cppm`
109+
110+
**Produces:**
111+
112+
```cpp
113+
export module mcpp.toolchain.hostflags;
114+
115+
struct HostFlagOptions {
116+
// clang 自带 cfg 时是否绕过它。主构建全平台绕过(可复现、不依赖
117+
// 安装期生成物);build.mcpp 的宿主 helper 只在 Linux 绕过 —— macOS 上
118+
// 链接 libc++abi/unwind 由主构建的 needs_explicit_libcxx 负责,helper
119+
// 自己重复一遍会产生 undefined __cxa_*(build_program.cppm 原注释)。
120+
enum class CfgBypass { Always, LinuxOnly } cfgBypass = CfgBypass::Always;
121+
// binutils -B:GCC/libstdc++ payload 才需要(musl 与 mingw 自带 as/ld)。
122+
bool binutilsPrefix = true;
123+
// tc.linkRuntimeDirs 的 -L(+ELF 上的 rpath):让产物能加载私有运行库。
124+
bool runtimeLibDirs = false;
125+
std::string macosDeploymentTarget; // 已解析值;空 = 不发
126+
};
127+
128+
// 宿主编译 flag,token 形态。三个消费者的唯一生产者。
129+
std::vector<std::string> host_compile_tokens(const Toolchain& tc,
130+
const HostFlagOptions& opt,
131+
const PathEscape& esc);
132+
```
133+
134+
- [ ] **Step 1:** 实现,顺序**严格照抄 `flags.cppm:319-352` 的现有顺序**
135+
(dm → deployment → lm),因为顺序影响渲染出的字符串
136+
- [ ] **Step 2:** 单测 `tests/unit/test_hostflags.cpp`:
137+
- 每个 `CompilerId` 都返回自洽结果(方言可取、无空 token)
138+
- `CfgBypass::LinuxOnly` 在非 Linux 上对 clang 返回空
139+
- deployment target 只在 macOS 且非空时出现
140+
- [ ] **Step 3:** 提交
141+
142+
## Task 4:三个消费者接入(**零 diff 验收**)
143+
144+
- [ ] **Step 1:** `build_program.cppm::host_base_flags` → 调
145+
`host_compile_tokens(tc, {CfgBypass::LinuxOnly, !isMusl, true, dt}, plainEsc)`,
146+
函数体删空。注意 `plainEsc` 是恒等转义(argv 不需要 ninja/shell 转义)。
147+
- [ ] **Step 2:** `stdmod.cppm:238-253` → `render_tokens(host_compile_tokens(tc,
148+
{CfgBypass::Always, false, false, dt}, shellEsc))`
149+
- [ ] **Step 3:** `flags.cppm:319-337` 的**编译侧** → `render_tokens(
150+
host_compile_tokens(tc, {CfgBypass::Always, …, false, dt}, ninjaEsc))`;
151+
链接侧(`link_toolchain_flags` / `f.sysroot` / `llvmRootForStdlib`)保持不动
152+
- [ ] **Step 4:** **零 diff 验收**
153+
154+
```bash
155+
snap after-gcc; diff /tmp/ninja-base-gcc.txt /tmp/ninja-after-gcc.txt # 必须空
156+
snap after-llvm; diff /tmp/ninja-base-llvm.txt /tmp/ninja-after-llvm.txt # 必须空
157+
# std_build_commands 逐字节
158+
diff /tmp/stdcmd-base.txt /tmp/stdcmd-after.txt # 必须空
159+
```
160+
161+
非空即说明抽错了 —— 回到 Task 2/3 找顺序或分支差异,**不要**改基线迁就。
162+
163+
- [ ] **Step 5:** 单测 + e2e(89/92/110/111/112/124/125/143/144/145/164/168/179/181),
164+
gcc 与 llvm 各一遍
165+
- [ ] **Step 6:** 提交
166+
167+
## Task 5:`host_program_argv` + 删 MSVC 模块门
168+
169+
**Files:** `src/toolchain/hostflags.cppm`(或新建 `hostprogram.cppm`)、
170+
`src/build/build_program.cppm`
171+
172+
- [ ] **Step 1:** 把 build.mcpp 的 argv 组装(方言、`forceCxxLangArgv`、输出前缀、
173+
静态运行时、模块 BMI 处理)收进生产者侧,`build_program.cppm` 只负责
174+
「读源文件 → 判定 imports → 调用 → 执行 → 解析指令」
175+
- [ ] **Step 2:** 模块处理按 `bmi_traits(tc)` 分派,含 MSVC 的
176+
`/interface /TP /ifcOutput` + `/reference <name>=<ifc>`
177+
—— 与主构建同一张表,不新写
178+
- [ ] **Step 3:** **删除** `build_program.cppm` 里
179+
`import mcpp; / import std; not yet supported under MSVC` 那道门
180+
- [ ] **Step 4:** `tests/e2e/180_msvc_build_mcpp.sh` 第三段:
181+
从「断言拒绝」改为「断言 `import std;` + `import mcpp;` 可用」
182+
- [ ] **Step 5:** e2e 181 参数化补 MSVC 一列(`# requires: windows msvc`,
183+
单独脚本 183 或在 180 内)
184+
- [ ] **Step 6:** 提交
185+
186+
## Task 6:能力矩阵守卫
187+
188+
- [ ] **Step 1:** `tests/unit/test_hostflags.cpp` 增一条:遍历所有已知
189+
`CompilerId`,断言 `host_compile_tokens` + `dialect_for` + `bmi_traits`
190+
三者都能给出完整答案(方言 id 非空、`forceCxxLangArgv` 非空、
191+
`outputExePrefix` 非空、bmi 的 `bmiDir`/`bmiExt` 非空)。
192+
**新增工具链族忘了接 = 测试失败**,而不是运行期 `not yet supported`。
193+
- [ ] **Step 2:** 提交
194+
195+
## Task 7:文档
196+
197+
- [ ] `docs/07-build-mcpp.md` / `docs/zh/07-build-mcpp.md`:删掉
198+
「MSVC 下不支持 import」的说明,改为陈述已支持
199+
- [ ] 设计文档 §5 能力矩阵勾掉 MSVC 一列
200+
- [ ] 提交
201+
202+
## Task 8:PR → CI 全绿 → 合入
203+
204+
- [ ] 本机全量:`mcpp test`、`tests/e2e/run_all.sh`、`check_version_pins.sh`
205+
- [ ] 开 PR,盯 CI(**重点看 macOS/Windows**:零 diff 只在本机证过两条工具链,
206+
Windows 的 MSVC 腿只有 CI 能验)
207+
- [ ] `gh pr merge --squash --admin --delete-branch`
208+
209+
## Task 9:Release + 生态闭环
210+
211+
- [ ] 触发 release.yml,四平台产物
212+
- [ ] 镜像 xlings-res 双端;失败则**本地 gtc 补传**
213+
(token 在 `~/.config/gitcode-tool/config.json`,用 repo 的 gtc + `/usr/bin/python3`)
214+
- [ ] **两端独立 GET + sha256 核验**(不信 workflow 绿灯)
215+
- [ ] xim-pkgindex bump PR(Sunrisepeak 账号合),sha256 逐个对照
216+
- [ ] 隔离 workspace 真装 `2026.8.2.2`,跑 e2e 179/181/89/112
217+
- [ ] **包上架后**才 bump `.xlings.json` bootstrap pin,单独 PR
218+
(索引 artifact 传播滞后会让这个 PR 假红:`not found` ≠ 回归,等传播后重跑)
219+
220+
---
221+
222+
## 实施记录(只有 CI 能发现的)
223+
224+
| 发现 | 教训 |
225+
|---|---|
226+
| **扩大 `build_program.cppm` 的匿名 ns 同样触发 clang 误编译** | 把 `build_mcpp_module` 在原地改写加大 → macOS 全部 build.mcpp e2e 段错误,和 PR#332 一模一样。约束不是「别加新函数」,是**「别再往那个 ns 加代码」**。修法=整块搬到 `src/build/hostprogram.cppm` |
227+
| **「扫缓存比对字节等价」是假验证** | std 缓存共享且累积,两次快照都含**陈旧条目**,diff 恒为空 —— 我因此放过了一次真实的字符串改动(`-stdlib=libc++` 位置)。**把主张写成测试**:用合成的 `ClangDriverModel`/`ToolchainLinkModel` 直接断言渲染出的字面串 |
228+
| **`-stdlib=libc++` 的位置是兼容面** | 它进 `std_build_commands` → 进 metadata → **决定 std 缓存目录名**。挪一个 flag = 让每个用户的 std BMI 全量失效 |
229+
| **「信任 cfg」不等于「什么都不发」** | 生产者在 trust-cfg 分支提前 `return`,把 deployment target 也跳过了;而 macOS 上 build.mcpp 走的正是这条分支 → std BMI 配置不匹配。旧的手写实现把它放在**最前、无条件**,注释还专门写了 "FIRST and unconditionally" —— 重构时要读懂那句话为什么在 |
230+
| **删掉能力门 = 死代码变活代码** | `-x none` 常年无条件发出,只因 MSVC 到不了那里才无害。门一删,cl 立刻 `D9002: ignoring unknown option '-x'`。**删门时要把门后所有「反正到不了」的分支重新审一遍** |
231+
232+
## Self-Review
233+
234+
**设计覆盖**:§4.1 token 生产者 → Task 2/3;§4.2 三种渲染 → Task 4;
235+
§4.3 build.mcpp 一等消费者 → Task 5;§5 能力矩阵 → Task 5/6;
236+
§6.1 字节等价 → Task 1 + Task 4 Step 4;§6.2 独立模块 → Task 3 约束;
237+
§6.4 双工具链验证 → Global Constraints。
238+
239+
**已知风险**:`compute_flags` 服务每一次构建,是全仓库 blast radius 最大的函数
240+
之一。缓解就是零 diff 硬约束 —— 它把"重构对不对"变成一个可机器判定的问题。

0 commit comments

Comments
 (0)