Conversation
Switch from building the protobuf submodule via init.sh/init.bat to installing protobuf through vcpkg. This removes the bundled protobuf build, simplifying the setup process and significantly reducing initial bootstrap time. Update prepare.bat to install vcpkg and protobuf (x64-windows-static), expose the vcpkg-built protoc on PATH, and export VCPKG_ROOT. Revise README.md instructions to document vcpkg (recommended), system package, Homebrew, and source-based protobuf installation paths. Adjust gitignore rules for the vcpkg manifest and install tree accordingly.
Kybxd
changed the title
- prepare.bat: switch to vcpkg manifest mode when PROTOBUF_VCPKG_VERSION is set. Classic-mode `--x-version` is silently a no-op, so the previous pin attempt would lie to users. Render a vcpkg.json under %LOCALAPPDATA%\loader\vcpkg-manifest\ with builtin-baseline + overrides, install via `vcpkg install --x-install-root=...`, and post-assert that the resolved port version matches the request. Pin the vcpkg checkout itself to VCPKG_BASELINE_COMMIT (mirroring testing-cpp.yml's VCPKG_COMMIT) so classic mode is also reproducible. Drop `--depth 1` so the commit pin is reachable. Export VCPKG_INSTALLED_DIR for downstream cmake. - prepare.bat: add a `where cl.exe` preflight at the top of Step 5 so an unactivated MSVC environment fails fast with an actionable message instead of cryptic vcpkg compiler-detection errors. - testing-cpp.yml: replace deprecated `version-string` with `version` in the rendered vcpkg.json. - testing-csharp.yml: document inline that `protobuf-version` is the protoc release tag (e.g. 33.4) and explain how it maps to the C++ libprotobuf SemVer used in testing-cpp.yml. - README.md: add a "Migrating from the bundled-protobuf layout" callout with the submodule-deinit recipe so existing checkouts know how to clean up. Replace the misleading classic-mode `--x-version` snippet with a working manifest-mode example. Document the extra cmake flags (-DVCPKG_INSTALLED_DIR / -DVCPKG_MANIFEST_INSTALL=OFF) required when PROTOBUF_VCPKG_VERSION is used. - .gitignore: ignore .claude/settings.local.json. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the brainstormed design for adding a Dev Container under .devcontainer/ to give contributors a one-command, reproducible multi-language toolchain (C++17 + Go 1.24 + .NET 8 + Node 20 + buf 1.67.0 + protobuf 6.33.4 via vcpkg, all pinned to CI's exact versions). Key decisions captured: - Single all-in-one Ubuntu 24.04 image (covers all four languages) - Dockerfile in repo, build on-demand (no ghcr.io publish in v1) - Multi-arch native via TARGETARCH (amd64 + arm64; no QEMU on Apple Silicon) - Pinnable protobuf version via LOADER_PROTOBUF_VERSION host env var, flowing through devcontainer.json build args into vcpkg manifest mode - Devcontainer is recommended path; prepare.bat / per-language manual setup stays as fallback for contributors who can't run Docker - CI keeps lukka/run-vcpkg directly (devcontainer is not used in CI) Implementation plan to follow via writing-plans. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12-task plan implementing the design at docs/superpowers/specs/2026-05-29-devcontainer-design.md. Each task is a single Dockerfile layer or config file with build/verify/commit steps and concrete expected outputs. Refs: docs/superpowers/specs/2026-05-29-devcontainer-design.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bootstrap the devcontainer image with Microsoft's multi-arch mcr.microsoft.com/devcontainers/cpp:1-ubuntu-24.04 base. Subsequent commits layer Go, buf, vcpkg/protobuf, .NET, and Node on top. Refs: docs/superpowers/specs/2026-05-29-devcontainer-design.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves TARGETARCH (amd64 or arm64) into per-arch values (Go tarball arch, buf release-asset arch, vcpkg triplet) and writes them to /opt/buildargs.env for downstream RUN layers to source. Unknown arches fail the build. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Install Go from the official multi-arch tarball into /usr/local/go. PATH is exposed via ENV (not /etc/profile.d) so non-interactive shells (postCreateCommand, downstream RUNs) see `go` without sourcing profile. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single-binary release into /usr/local/bin/buf. Pinned to the same version testing-cpp.yml / testing-csharp.yml use. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pin vcpkg to commit dc8d75c…df932 (lock-step with prepare.bat and testing-cpp.yml's VCPKG_COMMIT). Render a minimal vcpkg.json manifest with the protobuf override + builtin-baseline, install via manifest mode (the only mode where the version pin actually takes effect), and post-assert that the resolved port version starts with the requested PROTOBUF_VERSION. Default is 6.33.4; legacy v3 reachable via --build-arg PROTOBUF_VERSION=3.21.12. Symlink /opt/vcpkg/active → installed/<triplet> and /usr/local/bin/protoc → active/tools/protobuf/protoc so downstream ENV/PATH stays arch-independent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Microsoft apt repo for dotnet-sdk-8.0 (matches testing-csharp.yml's target). NodeSource setup_20.x for Node 20 LTS (covers the experimental _lab/ts/ workflow; not currently in CI). Both clean their apt caches to keep the layer small. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CMAKE_PREFIX_PATH=/opt/vcpkg/active lets the existing README cmake recipe (-DCMAKE_BUILD_TYPE=Debug, no toolchain file) resolve protobuf inside the container without any flag changes from the host workflow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the Dockerfile under build.args, mounts a named volume for the Go module cache, declares the VS Code extension set, and prints a one-line ready-banner via postCreateCommand. PROTOBUF_VERSION flows from the host LOADER_PROTOBUF_VERSION env var (default 6.33.4) so contributors can rebuild against the legacy v3 line via: LOADER_PROTOBUF_VERSION=3.21.12 code . Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-pager covering prerequisites (Docker Desktop / Engine + VS Code Dev Containers extension), how to open the container, the LOADER_PROTOBUF_VERSION knob, host-OS caveats (WSL2 workspace location, Apple Silicon native arm64), and the layered Dockerfile architecture. Points users at prepare.bat / per-language manual setup as the explicit fallback path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a new "Recommended: Dev Container (any host OS)" subsection at the top of Prerequisites pointing contributors at .devcontainer/. Add a "Skip this section if you're using the devcontainer" lead-in to the existing "Install protobuf" and "Windows: bootstrap" blocks so the manual paths are clearly the fallback, not the primary route. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributors with an existing host checkout can hit confusing build errors when stale .pb.cc / .pb.cs files from a previous protoc-3.x session linger in the workspace (gitignored, so git pull doesn't remove them). The container's modern protoc 33.4 emits a different file layout, and the leftovers shadow what's freshly generated. Add a Troubleshooting section with the rm -rf recipe to wipe and retry. A fresh clone doesn't hit this. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…TODO The Troubleshooting recipe added in 8a583ba targeted directories that don't actually hold stale generated files: src/tableau is empty (real C++ stale files live at src/protoconf/tableau/), and protoconf/tableau doesn't exist (C# protoc emits flat .cs files into protoconf/). Fix the rm -rf paths so the recipe actually unblocks the failure mode it documents. Also drop the now-contradictory "> TODO: [devcontainer]" placeholder from README.md line 7 — the devcontainer is implemented and recommended in the section immediately below it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nfile cache Docker creates named volumes as root:root by default, which leaves the Go module cache mounted at /home/vscode/go unwritable for the `vscode` user inside the devcontainer. Chown it to `vscode:vscode` in postCreateCommand so `go mod download` / `go build` work out of the box. Also ignore dotnet/runfile-discovery/, an auto-generated .NET SDK 10 file-based app discovery cache produced by the dotnet CLI / C# Dev Kit.
Update test/buf.yaml dependency pin and refresh test/buf.lock with the new commit / digest.
…probes
The previous version probed three hardcoded Program Files paths for
dockerd.exe. None of them match the path Microsoft's MobyOnWindowsRunner
installer chose on current actions/runner-images windows-2022 builds —
the upstream installer doesn't commit to a stable location.
Resolve dockerd via the registered `docker` Windows service's ImagePath
instead. That works regardless of where the runner image installed
Docker, and degrades to a useful diagnostic ("docker service is not
registered") if the runner doesn't have Docker at all.
Reported by the previous workflow run as:
dockerd.exe not found in any of:
C:\Program Files\docker\dockerd.exe;
C:\Program Files\Docker\Docker\resources\dockerd.exe;
C:\Program Files\Docker\dockerd.exe
Also fixed: under $ErrorActionPreference='Stop', the previous Write-Error
fallback aborted before the diagnostic Get-ChildItem could run, so
nothing useful was logged. Throw with explicit diagnostic capture
beforehand.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Windows-container ecosystem (HNS, WinNAT, NDIS filter drivers,
Hyper-V virtual switches) is too sensitive to host config to be
reliable for daily dev. Real failures hit during validation included:
- WinNAT half-state where Get-NetNat returned empty but
Get-NetNatExternalAddress still showed stale bindings, blocking
New-NetNat with "Windows System Error 52: duplicate name".
- Ghost vEthernet (nat) interfaces registered to the IP stack with no
associated Get-NetAdapter (Description=<no adapter>).
- Container outbound TCP fully blocked at the host NAT layer despite
ICMP working and per-interface forwarding being enabled. WFP audit
logging found zero drop events for the container's source IP, so
the issue was in the NAT translation path, not in any filter callout.
- HNS service restart broke dockerd's Windows engine API with persistent
500 Internal Server Error survivable across factory reset and reboot.
For Windows hosts the alternatives are:
1. Linux container under WSL2 - the documented recommendation.
Multi-arch, multi-OS, none of the issues above.
2. Bare-metal native dev via prepare.bat (C++ toolchain) plus winget
for Go / .NET / Node. Documented in the repo root README's new
"Windows (bare-metal)" section.
Removes:
- .devcontainer/windows/ (Dockerfile, devcontainer.json, README,
vcpkg-install.ps1, Import-LoaderVersions.ps1)
- .github/workflows/devcontainer-windows-smoke.yml
- .devcontainer/shared/postcreate-banner.ps1 (orphaned without the
Windows devcontainer)
Updates docs to reflect two devcontainer subfolders (linux/, macos/)
instead of three; macOS already follows this docs-only pattern, and
the new Windows path is parallel - winget one-liners + prepare.bat
plus a pointer at the Linux container under WSL2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Linux/Windows/macOS split was never useful: with the
Windows-container variant gone, linux/ and shared/ subdirs were just
ceremony for a single Dockerfile + a single shared file. macos/ was
docs-only and easier to inline into the repo root README's
prerequisites section, alongside the Windows bare-metal recipe.
File moves (all via git mv so history is preserved):
.devcontainer/linux/Dockerfile -> .devcontainer/Dockerfile
.devcontainer/linux/devcontainer.json -> .devcontainer/devcontainer.json
.devcontainer/linux/README.md -> .devcontainer/README.md
.devcontainer/shared/versions.env -> .devcontainer/versions.env
.devcontainer/shared/postcreate-banner.sh -> .devcontainer/postcreate-banner.sh
.github/workflows/devcontainer-linux-smoke.yml
-> .github/workflows/devcontainer-smoke.yml
Deletions:
.devcontainer/shared/README.md (parsing-rules section moved into
.devcontainer/README.md)
.devcontainer/macos/README.md (brew-install recipe inlined into
the repo root README)
Path updates:
- Dockerfile: COPY shared/foo -> COPY foo
- devcontainer.json: drop "context": "..", default ('.') is correct
- prepare.bat: %~dp0.devcontainer\shared\versions.env -> %~dp0.devcontainer\versions.env
- load-versions action: file=.devcontainer/shared/versions.env -> file=.devcontainer/versions.env
- devcontainer-smoke.yml: drop linux/ prefix, broaden path filter to .devcontainer/**
- CLAUDE.md, repo root README.md: paths and the macOS pointer
Net diff: +132 / -252 lines. Single source of truth, single Dockerfile,
single banner script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two complementary cache mechanisms, both exploiting that the dominant
build cost (~15-30 min) is vcpkg compiling protobuf from source under
the host's C++ toolchain.
1. Dockerfile: BuildKit cache mount on /vcpkg-binarycache.
`RUN --mount=type=cache,target=/vcpkg-binarycache,sharing=locked`
plus `ENV VCPKG_DEFAULT_BINARY_CACHE=/vcpkg-binarycache` directs
vcpkg to read/write its compiled-package zips into a directory whose
contents persist across `docker build` invocations on the same host
(BuildKit's own cache, not a Docker volume — volumes don't mount at
build time). After the first build, every subsequent build with the
same VCPKG_BASELINE_COMMIT + PROTOBUF_VERSION skips the compile and
uses cached zips, even when an unrelated upstream layer (e.g. a
BUF_VERSION bump) invalidates Docker's per-layer cache for the
vcpkg RUN.
2. Smoke workflow: docker/build-push-action with cache-{from,to}=gha.
Replaces the inline `docker buildx build` with the canonical
build-push-action, which exports BuildKit's full cache (including
the cache mount from #1) to GitHub's per-repo Actions cache via
mode=max. Cold first run is still ~25 min; every subsequent run
restores cached layers AND cached vcpkg binaries from GHA and
completes in ~3-5 min. Mirrors testing-cpp.yml's approach (which
actions/cache's $VCPKG_INSTALLED_DIR for the same reason) using the
docker-native equivalent.
Cache scope is keyed on `matrix.arch` so amd64 and arm64 caches don't
cross-contaminate (vcpkg binaries are arch-specific).
Adds `actions: write` permission to the workflow — required for
`cache-to: type=gha` to push BuildKit cache entries to the per-repo
Actions cache (~10 GB free quota; well under the cache size).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Single Python entrypoint for setup/generate/build/test/clean across Windows native, macOS, Linux, and devcontainer. Stdlib-only, Python 3.10+. CI workflows call it directly so dev and CI run identical commands. Key design points: - Windows MSVC env wrap: every cl.exe/vcpkg/cmake call runs in `cmd /c "call vcvarsall.bat x64 >/dev/null && <cmd>"` via a sentinel + shell=True, bypassing CreateProcess quote mangling. Shell PATH is never mutated. - vcpkg manifest mode: --protobuf-version renders vcpkg.json, runs `vcpkg install --x-install-root=...`, and routes buf-generate's protoc to the manifest's tools dir so codegen matches libprotobuf. - Per-host --race default: ON for Linux/macOS, OFF for Windows (cgo+C compiler dependency). Explicit --race opts in. - Devcontainer detection skips host setup; CI's lukka/run-vcpkg path is preserved via --no-vcpkg-install. Tests: test_make.py (next to make.py, Go-style) — 72 unit + dry-run snapshot tests. Runs in <5s. New testing-make.yml workflow runs them on ubuntu/macos/windows on every push. Verified end-to-end on Windows native: - python make.py setup --lang cpp (idempotent) - python make.py test --lang go (Go suite, 1.4s) - python make.py test --lang cpp (C++ 6.33.4, 13/13 pass) - python make.py test --lang cpp --protobuf-version 3.21.12 (legacy v3, 13/13 pass) Docs simplified: README, CLAUDE.md, .devcontainer/README.md all collapsed to make.py-first usage. Old verbose per-language recipes removed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
testing-go.yml: - Drop `go vet ./...` step. It ran from the repo root before the Test step, but vet needs the generated test/go-tableau-loader/protoconf/* packages that only `make.py test` produces. Easier than re-ordering: the Test step's `go test` already type-checks every package transitively, and devcontainer-smoke.yml's `make.py test --lang go --smoke` covers the plugin-only vet pass. testing-csharp.yml: - Drop the legacy-v3 protobuf matrix entry. C# consumes only generated *.cs files; libprotobuf is irrelevant at the C# level so there's no ABI compatibility to test against. Halves C# CI time. - Pin protoc to "33.4" directly (was matrix.config.protobuf-version). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two fixes for testing-{make,cpp}.yml regressions:
1. Platform.cmake_toolchain_args() gains force_vcpkg=True. When set,
emits -DCMAKE_TOOLCHAIN_FILE=...vcpkg.cmake + -DVCPKG_TARGET_TRIPLET
on every host, not just Windows. Used by _cpp_build_or_test when
--protobuf-version is set: manifest mode means we ARE using vcpkg
regardless of OS, so cmake's find_package(Protobuf) needs the
toolchain to resolve against vcpkg_installed/.
Fixes testing-cpp.yml ubuntu-latest legacy-v3 failure ("Could not
find a package configuration file provided by Protobuf").
2. _cpp_build_or_test no longer aborts in --dry-run when VCPKG_ROOT
is unset. It substitutes a placeholder so snapshot tests still
verify the printed command sequence on hosts (CI runners) without
vcpkg installed. Real (non-dry-run) execution still hard-errors.
Fixes testing-make.yml ubuntu/macos failures
(test_test_lang_cpp_protobuf_version_manifest_mode + sibling).
Adds two regression tests:
- test_test_lang_cpp_manifest_no_vcpkg_root_dry_run_ok (delenv VCPKG_ROOT)
- test_test_lang_cpp_manifest_forces_toolchain_on_linux (asserts -DCMAKE_TOOLCHAIN_FILE in output)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous value `dc8d75cfc3281b8e2a4ed8ee4163c891190df932` was the SHA of an *annotated tag object* (release/2026.04.27), not a commit. Symptoms: - github.com/microsoft/vcpkg/commit/<tag-SHA> 404s. - `git log` doesn't list it (tag objects aren't commits). - `git cat-file -t` reports "tag", not "commit". Functionally everything worked (git/vcpkg auto-peel tag SHAs to commits), but it's unbrowsable on GitHub and confuses anyone trying to verify the pin. Switched to the underlying commit SHA so the GitHub /commit/<sha> view resolves; the resolved port catalog is identical. Also fixes a pre-existing bug uncovered while validating the SHA change: classic-mode `make.py test --lang cpp` (no --protobuf-version) didn't remove a stale vcpkg.json left over from a previous manifest-mode run. cmake's vcpkg toolchain auto-detected the manifest and built the wrong libprotobuf into `build/vcpkg_installed/`, mismatching the `buf generate`-produced .pb.h files (`Cannot open include file: google/protobuf/runtime_version.h`). Now the cpp handler unlinks any leftover vcpkg.json before configure when not in manifest mode. Workflow YAMLs in this commit are pure formatter normalization (quote style + indent), no semantic change. Verified: 74/74 unit tests pass; full C++ build + 13/13 ctest cases green on Windows against the new SHA. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…where)
Until now `make.py setup --lang cpp` on macOS/Linux installed whatever
`brew` / `apt` / `dnf` shipped — meaning a fresh dev machine in 2027
might pick up protobuf 7.x while CI is still pinned to 6.33.4. Now
mirrors the devcontainer Dockerfile's strategy across every native host:
- Go: official tarball from go.dev to ~/.local/go/ (Linux/macOS).
- buf: GitHub release binary at BUF_VERSION.
- protobuf: vcpkg at VCPKG_BASELINE_COMMIT, classic mode by default.
Manifest mode (--protobuf-version) still works as before.
- cmake / ninja / build-essential: distro/brew package (version-tolerant).
- .NET: Homebrew dotnet@N (macOS), Microsoft apt repo (Linux).
- Node: Homebrew node@N (macOS), NodeSource (Linux).
Internal refactor:
- _setup_vcpkg_windows lifted to cross-platform _setup_vcpkg
(.bat vs .sh bootstrap, MSVC wrap on Windows only, exe vs binary).
- Platform.cmake_toolchain_args() emits toolchain flags whenever
vcpkg_root is known, regardless of OS (was Windows-only). Devcontainer
still returns [] (CMAKE_PREFIX_PATH preset).
- hydrate_platform_from_env reads ~/.loader-env.json on every OS now
(was Windows-only), so `make.py test` after `make.py setup` works
without VCPKG_ROOT being on shell PATH.
- _ensure_buf_linux generalized to _ensure_buf_unix (Linux/Darwin).
- New _ensure_go_tarball matches devcontainer's Go install.
- force_vcpkg parameter on cmake_toolchain_args is no longer needed
in practice (default behaviour already does the right thing) but
kept as an explicit override for callers.
8 new regression tests (TestCrossPlatformPinning + expanded
TestPlatform.cmake_toolchain_args*). 82/82 pass. Real Windows C++ build
+ 13/13 ctest cases still green against the refactored code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pinning protobuf via 'overrides' alone leaves builtin-baseline on a newer
vcpkg commit, which lets transitive deps (abseil/utf8-range/re2/...) drift
forward and breaks ABI (e.g. missing absl::if_constexpr). Pin the baseline
itself to the vcpkg commit whose versions/baseline.json had the target
protobuf — that snapshot is by construction self-consistent.
make.py:
- Add _resolve_vcpkg_baseline_for_protobuf: pickaxe-search vcpkg's git
history (git log -S '"X.Y.Z"' -- versions/baseline.json), validate
each candidate via baseline.json (default.protobuf.baseline == X.Y.Z)
to defeat false positives, cache result in ~/.loader-env.json.
- Add --vcpkg-baseline override; auto-resolve from $VCPKG_ROOT when
only --protobuf-version is given. Dry-run returns a placeholder so
snapshot tests stay deterministic.
- Drop 'overrides' from rendered vcpkg.json — baseline alone now carries
protobuf + transitive deps as one consistent set.
CI (.github/workflows/testing-cpp.yml):
- Move vcpkg-commit into matrix.config alongside protobuf-version so
each row pins its own self-consistent snapshot.
- Pass --vcpkg-baseline explicitly to make.py: lukka/run-vcpkg checks
out vcpkg shallowly, so the runtime resolver can't pickaxe locally.
- Render vcpkg.json baseline-only (no 'overrides') with the same SHA
we hand to lukka/run-vcpkg, keeping cache key and final build aligned.
lukka/run-vcpkg@v11 auto-injects VCPKG_BINARY_SOURCES=clear;x-gha,readwrite,
but the 'x-gha' provider only exists in vcpkg ≳ 2023. The legacy-v3 matrix
row pins a 2022-era vcpkg snapshot (6245ce44..., for protobuf 3.21.12) whose
vcpkg binary errors out with:
unknown binary provider type: ... on expression: clear;x-gha,readwrite
lukka/run-vcpkg honours a pre-set VCPKG_BINARY_SOURCES, so set it to 'clear'
on the step. We don't actually lose caching: actions/cache@v4 above already
keys vcpkg_installed/ on (os, triplet, vcpkg-commit, vcpkg.json hash) and
restores it before lukka/run-vcpkg runs.
The 2023-01-15 vcpkg snapshot pinned by legacy-v3 still references MSYS2
packages whose URLs and SHA512s have since been pruned from repo.msys2.org
(MSYS2 doesn't keep older versions). protobuf:x64-windows pulls them in via
vcpkg_acquire_msys → vcpkg_fixup_pkgconfig, so the install fails:
Failed to download file with error: 1
... vcpkg-scripts version: 6245ce44a0 2023-01-15 (3 years, 5 months ago)
Linux is unaffected because it uses the system pkg-config. The modern matrix
row still exercises Windows on a current vcpkg snapshot, so we keep legacy-v3
as a Linux-only smoke test for the protobuf-3 ABI.
Captures the design for two cleanups surfaced while testing make.py setup --lang all on a fresh ubuntu:24.04: drop the TS language axis (CLAUDE.md already calls it experimental, no CI references it) and tighten the devcontainer detection to /opt/vcpkg/active only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/.dockerenv is created by Docker for every container, so the previous `exists() or exists()` heuristic silently no-op'd `make.py setup` in any plain Docker image (e.g. ubuntu:24.04). Use only the marker the devcontainer's Dockerfile actually sets. Adds two TestPlatform cases that monkeypatch Path.exists to lock in both the negative (/.dockerenv alone -> False) and positive (/opt/vcpkg/active -> True) signals. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CLAUDE.md already documents TS as experimental and not in CI; no
workflow references it. Rather than maintain a broken pipeline
(_ensure_node_linux was a no-op on Linux, _lab/ts/package.json has
no build script), shrink the language axis to (go, cpp, csharp).
Removes:
- LANGS_ALL "ts" entry (argparse choices update transitively)
- Versions.node_version property
- _ensure_node_linux (silent no-op)
- per-platform Node/npm install branches in _setup_{linux,macos,windows}
- _ts_build_or_test and its _build_or_test dispatch
- _lang_dir, generate, cmd_clean, cmd_env "ts"/node/npm arms
- module docstring `npm test` mention and --lang ts examples
The _lab/ts/ scratchpad stays in the tree for manual experimentation
but is no longer driven by make.py.
test_make.py loses NODE_VERSION/node_version assertions and the
test_ts_lives_under_lab case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to dropping TS support in make.py — NODE_VERSION is no longer consumed by any tooling. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
make.py no longer drives any TS pipeline (see prior commit), so the devcontainer image no longer needs Node. Also trims comment headers that named Node. Triggers a one-time devcontainer rebuild for users with the cached image. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to dropping Node from the Dockerfile — banner script called `node --version` which would fail at container startup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
make.py no longer supports --lang ts; the TS scratchpad at _lab/ts remains in-tree but is unmaintained and not surfaced in the docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On Windows `Path("/opt/vcpkg/active")` renders as
`\opt\vcpkg\active` via str(), so the mock never matched the
forward-slash literal and `test_detect_recognizes_vcpkg_active_marker`
failed on windows-latest. `Path.as_posix()` returns the
forward-slash form on every platform.
Also fixes `test_detect_ignores_dockerenv_marker` which was passing
on Windows for the wrong reason (mock never returned True; assertion
expected False, so it accidentally passed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Regression from dropping the NodeSource curl: that script ran its own
`apt-get update` as a side effect, which refreshed the cache after the
preceding `dpkg -i packages-microsoft-prod.deb` added Microsoft's
apt source. Removing it left no update between adding the repo and
installing dotnet-sdk-${DOTNET_VERSION}, so apt couldn't find the
package and the build failed with exit 100.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
versions.env now declares one (protobuf, vcpkg-baseline) row per variant
(MODERN_*, LEGACY_V3_*) plus a DEFAULT_VARIANT pointer, replacing the
flat single-row PROTOBUF_VERSION + VCPKG_BASELINE_COMMIT keys. This:
- eliminates the drift between versions.env's modern vcpkg SHA
(was 56bb2411) and testing-cpp.yml's modern SHA (was 1f6bbba3) —
they now both pin 56bb2411 (tip of vcpkg's 2026.04.27 quarterly)
- brings the legacy-v3 pair (protobuf 3.21.12 / vcpkg 6245ce44)
into versions.env so it's discoverable outside the CI YAML
- adds a clean knob to switch the entire row at once via
LOADER_DEFAULT_VARIANT host env -> DEFAULT_VARIANT build-arg ->
Dockerfile case statement
Consumers all learn the resolver:
* make.py Versions class: new default_variant + variants() accessors;
existing protobuf_version / vcpkg_baseline_commit properties
transparently resolve through DEFAULT_VARIANT (with fallback to
unprefixed keys for backward compat)
* .devcontainer/Dockerfile: case statement after sourcing versions.env
re-exports PROTOBUF_VERSION / VCPKG_BASELINE_COMMIT for the rest
of the build script
* .devcontainer/devcontainer.json: new LOADER_DEFAULT_VARIANT host
env mapping (alongside the existing LOADER_PROTOBUF_VERSION
surgical override)
* .github/actions/load-versions: same case logic, exports the
resolved unprefixed names to GITHUB_ENV so existing workflow
references keep working
* .github/workflows/testing-cpp.yml: modern row's vcpkg-commit synced
to versions.env, comment now points readers at versions.env
test_make.py: +4 new TestVersions cases covering default_variant
fallback, variants() enumeration, dash-vs-underscore label parsing,
and unprefixed-key backward compat. All 87 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wenchy
approved these changes
Jun 5, 2026
… compat - embed/util.pc.h: introduce util::GetExtension() wrapper that works around the protobuf v3.15.0 GetExtension reflection bug by falling back to reparse-via-FieldDescriptor when GOOGLE_PROTOBUF_VERSION < 3015000. - embed/util.pc.cc, embed/load.pc.cc: switch the two extension lookups (tableau::field on field options, tableau::worksheet on message options) to use util::GetExtension() instead of calling .GetExtension() directly. - test/cpp-tableau-loader/src/protoconf/*: regenerate the mirrored sources to stay in sync with the embed templates. - make.py: vcpkg-based protobuf integration for cpp-loader testing.
- Add MapKeyFd helper: falls back to field(0) on protobuf < v3.12.0 where Descriptor::map_key() is unavailable (map-entry key is always field 0 per proto3 wire-format contract). - Rework GetExtension to work around a static-initialization-order quirk on runtimes < v3.15.0 by serializing and reparsing into a fresh OptionsT; unify the signature with a single return type across branches. - Drop redundant util:: qualifier on GetExtension call inside tableau::util::PatchMessage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace the
third_party/_submodules/protobufbuild pipeline with a vcpkg-based flow, and replace the Windows-onlyprepare.batwith a cross-platformmake.pyorchestrator. Same setup/build/test commands on Windows, macOS, Linux, and inside the devcontainer; same flow on CI.Key changes
Toolchain
third_party/_submodules/protobuf+init.{sh,bat}. Protobuf comes from vcpkg pinned toVCPKG_BASELINE_COMMITin.devcontainer/versions.env(single source of truth: Dockerfile,make.py, CI all read from it).test/cpp-tableau-loader/CMakeLists.txtusesfind_package(Protobuf CONFIG REQUIRED)only.make.py(new)setup/generate/build/test/clean/env.vcvarsall.batitself; shell PATH/INCLUDE/LIB never mutated.--protobuf-versionswitches to vcpkg manifest mode for legacy/version-specific builds — no re-runningsetuprequired.~/.loader-env.json.Tests for
make.pyitselftest_make.py(next tomake.py) — pytest unit tests + dry-run snapshot tests.testing-make.ymlworkflow runs them on ubuntu/macos/windows on every push (<2 min).CI
testing-cpp.yml: matrix overmodern(protobuf 6.33.4) ×legacy-v3(3.21.12), ubuntu × windows.lukka/run-vcpkgfor cached vcpkg. Build/test step ispython make.py test --lang cpp ….testing-go.yml:python make.py test --lang go --race --coverage.testing-csharp.yml: single modern protoc (libprotobuf is irrelevant for C#);python make.py test --lang csharp.devcontainer-smoke.yml: bind-mount +python3 make.py test --lang go --smoke(amd64 + arm64).Docs
make.py-first usage.prepare.batremoved.Test plan
python make.py test --lang cpp(modern 6.33.4) — 13/13 ctest cases pass on Windows native.python make.py test --lang cpp --protobuf-version 3.21.12— 13/13 pass (manifest mode).python make.py test --lang goand--lang csharp— full suites pass.pytest test_make.py— 82+ unit + dry-run tests, <5s.Migration
🤖 Generated with Claude Code