Skip to content

feat(protobuf): expose protoc as a host tool - #155

Open
Sunrisepeak wants to merge 2 commits into
mainfrom
feat/protobuf-protoc-tool
Open

feat(protobuf): expose protoc as a host tool#155
Sunrisepeak wants to merge 2 commits into
mainfrom
feat/protobuf-protoc-tool

Conversation

@Sunrisepeak

Copy link
Copy Markdown
Member

What

compat.protobuf now exposes protoc as a kind = "bin" target, so a consumer can write:

compat.protobuf = { version = "35.1", tools = ["protoc"] }

and get the compiler built for its own machine, out of the same package it links.

Why this shape

A protoc that disagrees with the linked libprotobuf fails at runtime, not at build time. Every other ecosystem carries machinery to paper over this — Conan has a protobuf/<host_version> placeholder, xmake deletes protoc outright when cross-compiling, and protobuf's own CMake has an open issue (#14576) where Protobuf_PROTOC_EXECUTABLE is ignored in CONFIG mode.

Here it needs no machinery, because there is only one version axis: the tool's version is the dependency's version. The mismatch is not expressible. mcpp build --target <triple> does not change it either — the tool is still built for the build machine, because a code generator has to run here.

Contents

pkgs/c/compat.protobuf.lua protoc target + protoc feature (138 TUs)
tests/examples/protobuf-protoc new member — generated code end to end
.github/workflows/validate.yml pin -> 2026.8.5.1 (floor)
index.toml min_mcpp unchanged; latest_mcpp -> 2026.8.5.1
docs/package-types.md + zh, both READMEs shape H. Host tool provider

The source list

138 entries transcribed from upstream's own libprotoc_srcs in src/file_lists.cmake — not hand-picked. Zero overlap with the libprotobuf set (importer.cc / parser.cc were already there), and no configure step is needed: the tree holds no .h.in / .cmake.in.

The target declares required_features = { "protoc", "upb" }. upb is not optional — libprotoc's upb generator links the upb runtime, and without it the link fails on missing upb_* symbols. Stating it in the descriptor means a consumer asking for the tool does not have to know that. Consumers who only link the runtime compile none of the 138.

The new member

It is the deliberate complement of tests/examples/protobuf, whose header comment says it "deliberately uses NO protoc-generated code". This one is generated code end to end: nested messages, an enum, a repeated message field, a map, a oneof, a well-known-type import, and reflection over the generated pool — serialized and parsed back, asserting on every one.

Verification

  • All 73 descriptors parse with 2026.8.5.1 (mcpp xpkg parse --json), 0 failures.
  • mcpp test -p protobuf-protoc passes locally (18.6s cold, 1.6s warm).
  • Editing the .proto re-runs only that edge (1.38s vs 0.02s no-op) — the codegen is a build-graph node, not work done during prepare.
  • The strongest one: the four files this self-built toolchain generates for grpc-m's helloworld.proto are byte-identical to the officially-generated stubs checked into that repo. Self-built protoc == official protoc 35.1.
  • lint scripts (check_mirror_urls, check_package_name, check_cross_package_refs) pass.

Two version decisions, made in opposite directions

CI pin -> 2026.8.5.1 (required). Confirmed empirically against 2026.8.3.3:

error: [dependencies].compat.protobuf.tools must be a string, inline dep table, or nested table

min_mcpp stays at 2026.8.3.3 (deliberate). The floor is a gate — an older client that fails it cannot open this index at all, which is exactly the failure mode of mcpp#349. Raising it is only justified when a descriptor genuinely stops being readable, and that has not happened: 2026.8.3.3 still parses the new compat.protobuf with an empty unknown_keys. What needs the newer mcpp is the consumer spelling tools = [...], which lives in a consumer's own mcpp.toml — including this repo's new member, hence the CI pin and nothing more. latest_mcpp is an advisory hint with no gate behaviour anywhere in the engine, so it tracks what CI validates against.

Notes for review

  • This PR touches index.toml, which the member-selection step classifies as an unclassified change — so CI runs the full workspace on all three platforms against the new pin. That is the intended validation for a pin bump.
  • The MCPP_BUILD_CACHE: local bypass for mcpp#344 is now droppable (that fix landed in 2026.8.3.4). It is kept, with its comment updated to record why: re-enabling the global package cache changes how every member builds and deserves its own PR, where a failure is unambiguous.
  • One sharp edge is documented rather than hidden: protoc does not embed the well-known types, so import "google/protobuf/timestamp.proto" needs an -I derived from mcpp::dep_dir("protobuf"). The new member's build.mcpp shows the robust form (probe for the directory holding descriptor.proto rather than hardcoding the tarball's wrap-directory name).

Follow-up

mcpplibs.grpc-plugin — the gRPC C++ codegen plugin, a package of its own because a code generator needs a .proto parser and a C++ emitter, not TLS, DNS and a regex engine. It lands after grpc-m tags a release; this PR is its prerequisite.

compat.protobuf shipped the libprotobuf runtime and nothing else, so a
consumer that needed generated code had to find a protoc somewhere and
guarantee by hand that it matched the runtime being linked. That mismatch is
a RUNTIME failure — the classic protobuf footgun — and it is the whole reason
Conan carries a `protobuf/<host_version>` placeholder and xmake deletes protoc
outright when cross-compiling.

mcpp 2026.8.5.1 makes it inexpressible: a `kind = "bin"` target in the
descriptor, asked for by the consumer as `tools = ["protoc"]`, is built for
the BUILD machine out of the same package the consumer links. The tool's
version IS the dependency's version, and `--target` does not change that
because a code generator has to run here.

- pkgs/c/compat.protobuf.lua: add the `protoc` target
  (main = "*/src/google/protobuf/compiler/main.cc") plus a `protoc` feature
  carrying the 138 entries of upstream's own `libprotoc_srcs` from
  src/file_lists.cmake — zero overlap with the libprotobuf source set, and no
  configure step (the tree holds no .h.in/.cmake.in). The target declares
  required_features = { "protoc", "upb" }: libprotoc's upb generator links the
  upb runtime, and without it the link fails on missing `upb_*` symbols, so
  the descriptor states the constraint instead of leaving consumers to
  discover it. Consumers who only link the runtime compile none of this.

- tests/examples/protobuf-protoc: the complement of tests/examples/protobuf.
  That member deliberately uses NO generated code; this one is generated code
  end to end — nested messages, enum, repeated field, map, oneof, a
  well-known-type import and reflection over the generated pool, serialized
  and parsed back. Verified locally: the four files this toolchain generates
  for grpc-m's helloworld.proto are BYTE-IDENTICAL to the officially generated
  stubs checked into that repo.

- CI pin -> 2026.8.5.1. This is a floor, not a routine bump: `tools = [...]`
  does not parse before it ("tools must be a string, inline dep table, or
  nested table"), confirmed against 2026.8.3.3.

- index.toml: min_mcpp deliberately UNCHANGED. It is a gate — an older client
  failing it cannot open the index at all (mcpp#349) — and this change does
  not make any descriptor unreadable: 2026.8.3.3 still parses compat.protobuf
  with an empty unknown_keys. Only the consumer spelling needs the newer mcpp,
  and that lives in a consumer's own mcpp.toml. latest_mcpp, an advisory hint
  with no gate behaviour, tracks what CI validates against.

- docs: shape H (host tool provider) in package-types.md + zh, README rows.
  Includes the one sharp edge: protoc does not embed the well-known types, so
  `import "google/protobuf/timestamp.proto"` needs an -I derived from
  mcpp::dep_dir("protobuf").
@Sunrisepeak

Copy link
Copy Markdown
Member Author

macOS 全量 workspace 已通过 —— 新成员实测链路

workspace (macos) 绿了,新成员在 llvm@20.1.7(与我本机不同的工具链)下走完了完整链路:

Workspace testing member 'tests/examples/protobuf-protoc' (41/59)
  Building host tool compat.protobuf:protoc from compat.protobuf v35.1
    (once per package version × host toolchain)
  build.mcpp running
  Compiling codegen (test)
    Running bin/codegen
member 'tests/examples/protobuf-protoc' (41/59) ok — 1 passed in 393.31s

一处已知的外观问题(不阻塞,但会被每个用户看到)

工具子构建里会打一条:

warning: src/protobuf.cppm: lib target without conventional lib root 'src/protobuf.cppm'
         (create the file or set [lib].path)

它在整个 59 成员的运行里只出现一次,只在这个成员下 —— 时间戳落在
「Building host tool」与「build.mcpp compiling」之间,即子构建内部

原因:子构建把 compat.protobuf 当作 root 包来建,于是 targets.protobuf = { kind = "lib" } 这条被按「root 的 lib target」校验,而 Form B 包并没有
src/protobuf.cppm。作为普通依赖被消费时不会走到这条校验,所以以前没人见过。

不阻塞本 PR,产物与测试都正确。但它是噪声,而且出现在一个主打功能的路径上 ——
凡是写 tools = ["protoc"] 的用户都会看到。合理的修法在引擎侧:工具子构建只关心
那个 kind = "bin" 目标,不应该把面向消费者的 lib-root 警告透出来。记在这里,后续单独处理。

CI 暴露了两件独立的事。

── 1. windows:工具子构建失败,所以不在那里声明这个目标 ──────────────────
`workspace (windows)` 里 59 个成员挂了 3 个,其中一个是本 PR 新加的
protobuf-protoc。**不是 protobuf 的问题,也不是 flags 的问题** —— 同一次运行里
tests/examples/protobuf、protobuf-upb、protobuf-gzip 全部通过,用的是同一份
abseil + protobuf 源码,只不过是作为普通依赖构建的。只有**工具子构建**会死:

    error: building host tool 'compat.protobuf:protoc' failed
    error: cannot read 'obj/compat_abseil/…/absl/time/internal/test_util.cc.ddi'
           …/cctz/src/time_zone_posix.cc.ddi、…/cctz/src/zone_info_source.cc.ddi

**不是路径长度。** MAX_PATH 是最顺手的猜测,而且是错的:这三个相对路径分别是
31 / 46 / 47 字符,而同一个子构建里
`absl/container/internal/hashtablez_sampler_force_weak_definition.cc`(67 字符)
编得好好的。子构建内层 ninja 的输出是被汇总过的,真正的 scan 报错没进日志,
**原因未知**,也没有 windows 机器可复现。

在一个建不出来的平台上声明这个目标,等于把一个没有解释的失败甩给用户。所以
windows 平台块里放一份只含 `protobuf` 的 `targets`(平台块的 targets 覆盖顶层,
compat.vulkan 早就在用这个机制),等有 windows 环境查清楚再放开。

成员也随之按 `[target.'cfg(os)']` 分平台:linux/macOS 要 `tools = ["protoc"]`,
windows 只要运行时;build.mcpp 在 windows 上直接返回,tests/codegen.cpp 编成一条
**显式 skip** —— 一个什么都没验证却绿着的测试比红的更糟。

── 2. 还原 index.toml:它让 CI 跑全量 workspace 并超时 ────────────────────
成员选择把 `index.toml` 归到「未分类改动」→ `MEMBERS=__ALL__` → 59 个成员全建。
linux leg 因此**恰好跑满 150 分钟 timeout 被取消**(09:04:17 → 11:34:34),
其间每一条测试都是通过的。macOS 83 分钟侥幸跑完。

而我改它只是把 `latest_mcpp` 从 2026.8.3.3 抬到 2026.8.5.1 —— 一个**引擎里零消费者**
的提示字段(已 grep 确认:只解析、从不使用)。为一个纯装饰的字段换来 2.5 小时全量
构建并超时,不值。还原之后选择器只挑 protobuf 相关成员。

`min_mcpp` 本来就不动,那个判断没变。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant