feat(protobuf): expose protoc as a host tool - #155
Open
Sunrisepeak wants to merge 2 commits into
Open
Conversation
compat.protobuf shipped the libprotobuf runtime and nothing else, so a
consumer that needed generated code had to find a protoc somewhere and
guarantee by hand that it matched the runtime being linked. That mismatch is
a RUNTIME failure — the classic protobuf footgun — and it is the whole reason
Conan carries a `protobuf/<host_version>` placeholder and xmake deletes protoc
outright when cross-compiling.
mcpp 2026.8.5.1 makes it inexpressible: a `kind = "bin"` target in the
descriptor, asked for by the consumer as `tools = ["protoc"]`, is built for
the BUILD machine out of the same package the consumer links. The tool's
version IS the dependency's version, and `--target` does not change that
because a code generator has to run here.
- pkgs/c/compat.protobuf.lua: add the `protoc` target
(main = "*/src/google/protobuf/compiler/main.cc") plus a `protoc` feature
carrying the 138 entries of upstream's own `libprotoc_srcs` from
src/file_lists.cmake — zero overlap with the libprotobuf source set, and no
configure step (the tree holds no .h.in/.cmake.in). The target declares
required_features = { "protoc", "upb" }: libprotoc's upb generator links the
upb runtime, and without it the link fails on missing `upb_*` symbols, so
the descriptor states the constraint instead of leaving consumers to
discover it. Consumers who only link the runtime compile none of this.
- tests/examples/protobuf-protoc: the complement of tests/examples/protobuf.
That member deliberately uses NO generated code; this one is generated code
end to end — nested messages, enum, repeated field, map, oneof, a
well-known-type import and reflection over the generated pool, serialized
and parsed back. Verified locally: the four files this toolchain generates
for grpc-m's helloworld.proto are BYTE-IDENTICAL to the officially generated
stubs checked into that repo.
- CI pin -> 2026.8.5.1. This is a floor, not a routine bump: `tools = [...]`
does not parse before it ("tools must be a string, inline dep table, or
nested table"), confirmed against 2026.8.3.3.
- index.toml: min_mcpp deliberately UNCHANGED. It is a gate — an older client
failing it cannot open the index at all (mcpp#349) — and this change does
not make any descriptor unreadable: 2026.8.3.3 still parses compat.protobuf
with an empty unknown_keys. Only the consumer spelling needs the newer mcpp,
and that lives in a consumer's own mcpp.toml. latest_mcpp, an advisory hint
with no gate behaviour, tracks what CI validates against.
- docs: shape H (host tool provider) in package-types.md + zh, README rows.
Includes the one sharp edge: protoc does not embed the well-known types, so
`import "google/protobuf/timestamp.proto"` needs an -I derived from
mcpp::dep_dir("protobuf").
Member
Author
macOS 全量 workspace 已通过 —— 新成员实测链路
一处已知的外观问题(不阻塞,但会被每个用户看到)工具子构建里会打一条: 它在整个 59 成员的运行里只出现一次,只在这个成员下 —— 时间戳落在 原因:子构建把 不阻塞本 PR,产物与测试都正确。但它是噪声,而且出现在一个主打功能的路径上 —— |
CI 暴露了两件独立的事。
── 1. windows:工具子构建失败,所以不在那里声明这个目标 ──────────────────
`workspace (windows)` 里 59 个成员挂了 3 个,其中一个是本 PR 新加的
protobuf-protoc。**不是 protobuf 的问题,也不是 flags 的问题** —— 同一次运行里
tests/examples/protobuf、protobuf-upb、protobuf-gzip 全部通过,用的是同一份
abseil + protobuf 源码,只不过是作为普通依赖构建的。只有**工具子构建**会死:
error: building host tool 'compat.protobuf:protoc' failed
error: cannot read 'obj/compat_abseil/…/absl/time/internal/test_util.cc.ddi'
…/cctz/src/time_zone_posix.cc.ddi、…/cctz/src/zone_info_source.cc.ddi
**不是路径长度。** MAX_PATH 是最顺手的猜测,而且是错的:这三个相对路径分别是
31 / 46 / 47 字符,而同一个子构建里
`absl/container/internal/hashtablez_sampler_force_weak_definition.cc`(67 字符)
编得好好的。子构建内层 ninja 的输出是被汇总过的,真正的 scan 报错没进日志,
**原因未知**,也没有 windows 机器可复现。
在一个建不出来的平台上声明这个目标,等于把一个没有解释的失败甩给用户。所以
windows 平台块里放一份只含 `protobuf` 的 `targets`(平台块的 targets 覆盖顶层,
compat.vulkan 早就在用这个机制),等有 windows 环境查清楚再放开。
成员也随之按 `[target.'cfg(os)']` 分平台:linux/macOS 要 `tools = ["protoc"]`,
windows 只要运行时;build.mcpp 在 windows 上直接返回,tests/codegen.cpp 编成一条
**显式 skip** —— 一个什么都没验证却绿着的测试比红的更糟。
── 2. 还原 index.toml:它让 CI 跑全量 workspace 并超时 ────────────────────
成员选择把 `index.toml` 归到「未分类改动」→ `MEMBERS=__ALL__` → 59 个成员全建。
linux leg 因此**恰好跑满 150 分钟 timeout 被取消**(09:04:17 → 11:34:34),
其间每一条测试都是通过的。macOS 83 分钟侥幸跑完。
而我改它只是把 `latest_mcpp` 从 2026.8.3.3 抬到 2026.8.5.1 —— 一个**引擎里零消费者**
的提示字段(已 grep 确认:只解析、从不使用)。为一个纯装饰的字段换来 2.5 小时全量
构建并超时,不值。还原之后选择器只挑 protobuf 相关成员。
`min_mcpp` 本来就不动,那个判断没变。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
compat.protobufnow exposesprotocas akind = "bin"target, so a consumer can write:and get the compiler built for its own machine, out of the same package it links.
Why this shape
A protoc that disagrees with the linked libprotobuf fails at runtime, not at build time. Every other ecosystem carries machinery to paper over this — Conan has a
protobuf/<host_version>placeholder, xmake deletes protoc outright when cross-compiling, and protobuf's own CMake has an open issue (#14576) whereProtobuf_PROTOC_EXECUTABLEis ignored in CONFIG mode.Here it needs no machinery, because there is only one version axis: the tool's version is the dependency's version. The mismatch is not expressible.
mcpp build --target <triple>does not change it either — the tool is still built for the build machine, because a code generator has to run here.Contents
pkgs/c/compat.protobuf.luaprotoctarget +protocfeature (138 TUs)tests/examples/protobuf-protoc.github/workflows/validate.ymlindex.tomlmin_mcppunchanged;latest_mcpp-> 2026.8.5.1docs/package-types.md+ zh, both READMEsThe source list
138 entries transcribed from upstream's own
libprotoc_srcsinsrc/file_lists.cmake— not hand-picked. Zero overlap with the libprotobuf set (importer.cc/parser.ccwere already there), and no configure step is needed: the tree holds no.h.in/.cmake.in.The target declares
required_features = { "protoc", "upb" }.upbis not optional — libprotoc's upb generator links the upb runtime, and without it the link fails on missingupb_*symbols. Stating it in the descriptor means a consumer asking for the tool does not have to know that. Consumers who only link the runtime compile none of the 138.The new member
It is the deliberate complement of
tests/examples/protobuf, whose header comment says it "deliberately uses NO protoc-generated code". This one is generated code end to end: nested messages, an enum, a repeated message field, a map, a oneof, a well-known-type import, and reflection over the generated pool — serialized and parsed back, asserting on every one.Verification
mcpp xpkg parse --json), 0 failures.mcpp test -p protobuf-protocpasses locally (18.6s cold, 1.6s warm)..protore-runs only that edge (1.38s vs 0.02s no-op) — the codegen is a build-graph node, not work done during prepare.grpc-m'shelloworld.protoare byte-identical to the officially-generated stubs checked into that repo. Self-built protoc == official protoc 35.1.check_mirror_urls,check_package_name,check_cross_package_refs) pass.Two version decisions, made in opposite directions
CI pin -> 2026.8.5.1 (required). Confirmed empirically against 2026.8.3.3:
min_mcppstays at 2026.8.3.3 (deliberate). The floor is a gate — an older client that fails it cannot open this index at all, which is exactly the failure mode of mcpp#349. Raising it is only justified when a descriptor genuinely stops being readable, and that has not happened: 2026.8.3.3 still parses the newcompat.protobufwith an emptyunknown_keys. What needs the newer mcpp is the consumer spellingtools = [...], which lives in a consumer's ownmcpp.toml— including this repo's new member, hence the CI pin and nothing more.latest_mcppis an advisory hint with no gate behaviour anywhere in the engine, so it tracks what CI validates against.Notes for review
index.toml, which the member-selection step classifies as an unclassified change — so CI runs the full workspace on all three platforms against the new pin. That is the intended validation for a pin bump.MCPP_BUILD_CACHE: localbypass for mcpp#344 is now droppable (that fix landed in 2026.8.3.4). It is kept, with its comment updated to record why: re-enabling the global package cache changes how every member builds and deserves its own PR, where a failure is unambiguous.import "google/protobuf/timestamp.proto"needs an-Iderived frommcpp::dep_dir("protobuf"). The new member'sbuild.mcppshows the robust form (probe for the directory holdingdescriptor.protorather than hardcoding the tarball's wrap-directory name).Follow-up
mcpplibs.grpc-plugin— the gRPC C++ codegen plugin, a package of its own because a code generator needs a.protoparser and a C++ emitter, not TLS, DNS and a regex engine. It lands after grpc-m tags a release; this PR is its prerequisite.