Skip to content

Commit b79eb43

Browse files
committed
fix(build): link through a response file on every platform, not just Windows
Discovered by running the real mcpp-index workspace against the #344 branch: `opencv-module`, `opencv-module-dnn` and `opencv-module-unifont` all died with ninja: fatal: posix_spawn: Argument list too long naming no edge, no file and no cause. #344 lengthened dependency object paths (they now carry a per-package directory) and that was enough to cross a ceiling nothing was watching. The rule emitter said rsp was needed only where commands spawn through CreateProcess, and that "POSIX driver-style keeps the inline form: ARG_MAX is ample". Both halves are wrong. ninja runs `sh -c "<whole command>"` on POSIX, so the command is a SINGLE argv entry and the limit is MAX_ARG_STRLEN — 32 pages, 128 KiB — not the 2 MiB ARG_MAX anyone would think to check. And it was never ample: measured on opencv-module, the inline link line was already 56 840 bytes before this branch, 43% of the ceiling. With #344's paths it reached 161 687. So: rsp always, every platform, every dialect. clang/gcc drivers, link.exe, GNU ar and llvm-ar all accept @rspfile, so this is one rule shape instead of two, and the response file next to the output reads better than a 160 KB command line anyway. A build system may not have a maximum project size that it discovers by crashing, and "how long is this command" must not be something anyone carries in their head when choosing an object path. The unit test asserted the falsified premise for POSIX ("must NOT use rspfile"); it now asserts the invariant on every platform, plus that `$in` appears exactly once per rule — as rspfile_content, never inlined into the command. Note this was invisible to all 18 CI jobs: none of them builds an opencv-class package. CI has no large-link coverage at all.
1 parent d1d02cb commit b79eb43

3 files changed

Lines changed: 53 additions & 22 deletions

File tree

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@
1818

1919
#233(编译边撞名)、#240(链接输入未跟改名)与本条是同一台机器的三个产物:**布局由一次全局普查决定**。所以修的不是再补一处同步,而是把依赖包从普查里彻底拿掉。
2020

21+
- **链接/归档命令一律走 response file,不再有「项目大到一定程度就崩」的隐形上限。** 此前只有 Windows(CreateProcess 32 KiB)与 msvc 方言用 rspfile,POSIX 走内联 `$in`,理由写的是「ARG_MAX 很宽裕」。两半都错:ninja 在 POSIX 上是 `sh -c "<整条命令>"`,整条命令是**一个 argv 项**,撞的是 `MAX_ARG_STRLEN`(32 页 = 128 KiB)而不是 2 MiB 的 `ARG_MAX`;而且它从来就不宽裕 —— 实测 mcpp-index 的 `opencv-module`,内联链接行**本来就已经 56840 字节**,占那条无人看守的上限的 43%。
22+
23+
上面 #344 让依赖对象路径变长(多一层包目录),同一条边到了 161687 字节,于是 ninja 直接 `ninja: fatal: posix_spawn: Argument list too long` —— **不报是哪条边、哪个文件、什么原因**。构建系统不能有一个「靠崩溃才被发现的项目规模上限」,「这条命令有多长」也不应该是选对象路径时需要有人记在脑子里的事。clang/gcc driver、link.exe、GNU ar、llvm-ar 全都认 `@rspfile`,现在全平台一个规则形态。
24+
2125
### 改进
2226

2327
- **cache 条目与本次构建的布局分歧,现在降级为 miss 并明确报告,而不是让 ninja 崩在图加载阶段。** `is_cached` 此前校验的是条目**自述的**文件表,而消费方随后按**自己算的**地址去取 —— 两处独立推导,从不比对。命中判据现在校验「本次实际要读的那批产物」,任何不匹配都只是一次重编。同时新增一行 warning:一个系统性的分歧否则会表现为「cache 永远不命中」而毫无信号,这正是 v2026.7.30.2 之前那个假 `Cached` 骗了三个月的失败模态。

src/build/ninja_backend.cppm

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -749,15 +749,33 @@ std::string emit_ninja_string(const BuildPlan& plan) {
749749
// link/archive command; revisit the first-match replace if a dialect
750750
// ever grows another).
751751
//
752-
// rsp is used when the command spawns through CreateProcess (32 KiB
753-
// command-line ceiling): always for the separate-linker msvc dialect,
754-
// and on Windows for driver-style too (#247 — ffmpeg/opencv-class
755-
// packages link thousands of objects; clang/gcc drivers and GNU/llvm ar
756-
// all accept @rspfile). POSIX driver-style keeps the inline form
757-
// byte-identical: ARG_MAX is ample and the plain command is easier to
758-
// reproduce by hand.
752+
// rsp is used ALWAYS, on every platform and every dialect. The objects of
753+
// one link edge are unbounded — an ecosystem package like opencv or ffmpeg
754+
// contributes thousands — and every way of spawning a command has a ceiling:
755+
//
756+
// Windows CreateProcess, 32 KiB command line (#247)
757+
// POSIX ninja spawns `sh -c "<whole command>"`, so the command is a
758+
// SINGLE argv entry and hits MAX_ARG_STRLEN — 32 pages, 128 KiB
759+
// — long before the 2 MiB ARG_MAX anyone would think to check.
760+
//
761+
// This used to read "POSIX keeps the inline form; ARG_MAX is ample and the
762+
// plain command is easier to reproduce by hand". Both halves were wrong.
763+
// ARG_MAX is the wrong limit, and it was never ample: measured on
764+
// mcpp-index's opencv-module, the inline link line was already 56 840 bytes
765+
// — 43% of a ceiling nothing was watching. mcpp#344 lengthened dependency
766+
// object paths (they now carry a per-package directory) and the same edge
767+
// reached 161 687 bytes, at which point ninja dies with
768+
//
769+
// ninja: fatal: posix_spawn: Argument list too long
770+
//
771+
// naming no edge, no file and no cause. A build system may not have a
772+
// maximum project size that it discovers by crashing, and "how long is this
773+
// command" must not be a thing anyone has to keep in their head when
774+
// choosing an object path. clang/gcc drivers, link.exe, GNU ar and llvm-ar
775+
// all accept @rspfile, so there is one rule shape everywhere; the response
776+
// file sits next to the output and `cat`ing it beats reading a 160 KB line.
759777
{
760-
const bool useRsp = separateLinker || mcpp::platform::is_windows;
778+
constexpr bool useRsp = true;
761779
auto link_rule = [&](std::string_view name, std::string cmd,
762780
std::string_view desc) {
763781
append(std::format("rule {}\n", name));

tests/unit/test_ninja_backend.cpp

Lines changed: 23 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -570,12 +570,23 @@ TEST(NinjaBackend, RawMultiTokenFlagIsNotQuoted) {
570570
std::string::npos) << ninja;
571571
}
572572

573-
// mcpp#247: driver-style (gnu dialect) link/archive/shared rules must route
574-
// the object list through a response file on Windows — every command spawns
575-
// via CreateProcess (32 KiB command-line ceiling), and ffmpeg/opencv-class
576-
// source packages link thousands of objects, so an inlined $in overflows it.
577-
// POSIX keeps the inline form byte-identical (ARG_MAX is ample).
578-
TEST(NinjaBackend, DriverStyleLinkRulesUseRspfileOnWindowsOnly) {
573+
// mcpp#247 + #344: link/archive/shared route the object list through a response
574+
// file on EVERY platform. The number of objects on one link edge is unbounded —
575+
// an ffmpeg/opencv-class source package contributes thousands — and every way of
576+
// spawning a command has a ceiling:
577+
//
578+
// Windows CreateProcess, 32 KiB command line
579+
// POSIX ninja runs `sh -c "<whole command>"`, so the command is a SINGLE
580+
// argv entry and hits MAX_ARG_STRLEN (32 pages, 128 KiB) long before
581+
// the 2 MiB ARG_MAX anyone would think to check
582+
//
583+
// This test used to assert the OPPOSITE for POSIX ("ARG_MAX is ample"), pinning
584+
// an assumption that was both about the wrong limit and false: mcpp-index's
585+
// opencv-module link line was already 56 840 bytes inline, and #344's per-package
586+
// object directories took it to 161 687 — at which point ninja dies with
587+
// `posix_spawn: Argument list too long`, naming no edge and no cause. A ceiling
588+
// nothing watches is not a ceiling anyone can stay under, so there isn't one now.
589+
TEST(NinjaBackend, DriverStyleLinkRulesAlwaysUseRspfile) {
579590
auto plan = minimal_plan(); // GCC → gnu dialect → driver-style branch
580591

581592
auto ninja = emit_ninja_string(plan);
@@ -587,14 +598,12 @@ TEST(NinjaBackend, DriverStyleLinkRulesUseRspfileOnWindowsOnly) {
587598
auto end = ninja.find("\n\n", start);
588599
ASSERT_NE(end, std::string::npos) << ninja;
589600
auto body = ninja.substr(start, end - start);
590-
if constexpr (mcpp::platform::is_windows) {
591-
EXPECT_NE(body.find("@$out.rsp"), std::string::npos) << body;
592-
EXPECT_NE(body.find("rspfile = $out.rsp"), std::string::npos) << body;
593-
EXPECT_NE(body.find("rspfile_content = $in"), std::string::npos) << body;
594-
} else {
595-
EXPECT_EQ(body.find("rspfile"), std::string::npos) << body;
596-
EXPECT_NE(body.find("$in"), std::string::npos) << body;
597-
}
601+
EXPECT_NE(body.find("@$out.rsp"), std::string::npos) << body;
602+
EXPECT_NE(body.find("rspfile = $out.rsp"), std::string::npos) << body;
603+
EXPECT_NE(body.find("rspfile_content = $in"), std::string::npos) << body;
604+
// And the object list must no longer be inlined into the command: that
605+
// is the whole point, so `$in` may appear only as rspfile_content.
606+
EXPECT_EQ(count_occurrences(body, "$in"), 1u) << body;
598607
}
599608
}
600609

0 commit comments

Comments
 (0)