# Troubleshooting
A growing list of failure patterns seen building packages for pantry, with the diagnostic that pins them down and the fix that's worked. Build with `bk -L build` (Linux docker) to reproduce CI on a Mac.
## Quickest-win checklist
Before deep-diving, try in order — these explain ~60% of new-recipe failures:
1. **Read the upstream `INSTALL` / `BUILDING.md`** for required build tools you didn't declare.
2. **Read another packager's recipe** for the same project — `nixpkgs/pkgs/.../<name>/default.nix`, `Homebrew/homebrew-core/Formula/<name>.rb`, or `gitlab.archlinux.org/.../<name>/PKGBUILD`. Their patches and configure flags are the canonical answer to "what does the upstream really need".
3. **Run `bk build` then inspect the build dir** — `cd ./builds/<pkg>/<pkg>+<platform>/`, edit `dev.pkgx.build.sh`, and reproduce manually. The script is plain bash; you can `set -x` it and rerun pieces.
4. **Drop the build into the env** without running it: edit `dev.pkgx.build.sh` to keep only the `export` block, then `source dev.pkgx.build.sh`. You now have the CI env in your shell to poke around.
## c / c++
C and C++ are complex. Try ensuring the `cc` (c-compiler) used is `clang`. How varies by build system. Usually `CC=clang` / `CXX=clang++` is enough. CMake is weird — see [`cmake`](./cmake) for that.
### `error: 'std::set::contains' was not declared` (or similar C++17/20 features)
Project compiled by clang but is using a libstdc++ that's too old to expose the C++17/20 API. The cleanest fix is to make clang resolve libstdc++ from a known-good GCC bottle (this is the pattern nixpkgs uses via its `cc-wrapper`):
```yaml
env:
linux:
CXXFLAGS: $CXXFLAGS -isystem {{deps.gnu.org/gcc.prefix}}/include/c++/{{deps.gnu.org/gcc.version.major}} \
-isystem {{deps.gnu.org/gcc.prefix}}/include/c++/{{deps.gnu.org/gcc.version.major}}/x86_64-pkgx-linux-gnu
```
Forcing `-DCMAKE_CXX_STANDARD=20` rarely sticks because most projects pin via `target_compile_features(target cxx_std_17)`, which overrides the global.
### `GLIBCXX_3.4.31 not found` at run time
Build links libstdc++ from a newer gcc bottle than what's available at runtime. Options:
- **Statically link the C++ stdlib** in the final binary: `-static-libstdc++ -static-libgcc` on the link line. Trades binary size for ABI hermeticity. Best for self-contained tools (wasm-opt, ripgrep-style).
- **Declare `gnu.org/gcc` as a runtime dep** so the bottle's libstdc++ travels with the binary. Heavier on the user's pkgx prefix but correct for libraries.
### Linker can't find `-lrocksdb` / `-lbz2` / `-lsnappy` despite vendored builds
CGO-style projects (e.g. cubefs, tikv) compile vendored static libs under `build/lib`, then expect the Go build to find them via `CGO_LDFLAGS=-L${BuildDependsLibPath}`. The build script sets that env up at the start — make sure you're invoking it via `make <target>` rather than calling `go build` directly, otherwise `CGO_LDFLAGS` is empty and the link fails.
## Go binaries
### `error: cannot find section '.dynamic'` from `fix-patchelf`
Static Go binaries have no `.dynamic` section, so brewkit's default ELF-relocation pass fails. Disable it for the recipe:
```yaml
build:
skip: fix-patchelf
```
This is safe specifically because pure-Go binaries don't need RPATH rewriting — they have no dynamic dependencies to relocate. Recipes that link cgo libraries should *not* use this.
### `module not found` in test step but build worked
The test sandbox starts in a clean `$HOME` without your build artifacts. Use absolute paths or place files explicitly into `{{prefix}}` during the build step; the test step only sees `{{prefix}}`.
## Rust / cargo
### `cargo:warning=cannot find -lssl` on linux
openssl-sys looks at `OPENSSL_DIR` / `OPENSSL_LIB_DIR` / `OPENSSL_INCLUDE_DIR`. But projects with bundled grpcio-sys / librocksdb-sys / libtitan-sys also run their *own* CMake configure, which reads `OPENSSL_ROOT_DIR` instead. Set all four:
```yaml
env:
OPENSSL_DIR: '{{deps.openssl.org.prefix}}'
OPENSSL_LIB_DIR: '{{deps.openssl.org.prefix}}/lib'
OPENSSL_INCLUDE_DIR: '{{deps.openssl.org.prefix}}/include'
OPENSSL_ROOT_DIR: '{{deps.openssl.org.prefix}}'
PKG_CONFIG_PATH: '{{deps.openssl.org.prefix}}/lib/pkgconfig:$PKG_CONFIG_PATH'
```
### `error: SSL_get_peer_certificate undefined`
The project uses OpenSSL 1.1 APIs that were *renamed* in OpenSSL 3.0. pkgx's `openssl.org` recipe ships 3.x but configured `enable-deprecated`, so the old names still resolve. If you see this error, you're depending on a *different* openssl (system libs, a bundled one) — point your build at pkgx's.
### `error: linking with cc failed: rust-toolchain.toml pinned a channel I don't have`
The recipe needs `rust-lang.org/rustup` (not `rust-lang.org/rust`), and the build script should resolve the channel from `rust-toolchain.toml` and add the resolved toolchain's `bin/` to `$PATH`:
```yaml
script:
- run:
- ln -sf {{deps.rust-lang.org/rustup.prefix}}/bin/rustup rustup
- rustup default "$(grep '^channel' $SRCROOT/rust-toolchain.toml | sed 's/channel = "//;s/"//')"
- ln -sf $HOME/.rustup/toolchains/*/bin/* .
working-directory: $HOME/.cargo/bin
```
## CMake projects with very old `cmake_minimum_required`
CMake 4.x dropped policy compatibility below 3.5. Vendored projects pinned to 2.x emit:
```
CMake Error: Compatibility with CMake < 3.5 has been removed from CMake.
```
Fix without patching the upstream `CMakeLists.txt`:
```yaml
env:
CMAKE_POLICY_VERSION_MINIMUM: '3.5'
```
This is a CMake 4.x escape hatch — see also [`cmake`](./cmake).
## ELF / RPATH on Linux bottles
### `cannot create directory '/...': No such file or directory` during install
brewkit doesn't pre-create `{{prefix}}` before `build.script`. Your script must `mkdir -p "{{prefix}}"` itself before copying anything in (`install -D` handles this automatically).
### Bottle works in `~/.pkgx/...` but segfaults in `~/scratch/pkgx/...`
Loader is parsing an obsolete absolute path baked into the binary. Two common sources:
1. **`PT_INTERP` (the dynamic loader path) is absolute** and `$ORIGIN` substitution does not apply to it. brewkit's `fix-elf` currently rewrites RUNPATH but not PT_INTERP — if you bottle a libc, wrap the resulting binaries in shell scripts that re-exec with the bottle's interpreter.
2. **Linker scripts with baked-in absolute paths.** Glibc installs `libc.so` / `libm.so` etc as TEXT linker scripts (`GROUP ( /opt/.../lib/libc.so.6 ... )`) — those absolute paths break under relocation. Either strip the `+brewing` suffix post-install, or sed the prefix down to basenames and rely on `-L $LIBDIR -lc`.
### `fix-patchelf` writes an RPATH so long it crashes the loader
Seen during glibc bottling: brewkit chains 46 transitive dep prefixes into RPATH, including on `ld-linux-*.so.2` itself, which SIGSEGVs at startup. Use `build.skip: fix-patchelf` and handle relocation yourself (wrapper scripts pattern).
## Test step
### `... is not an explicit dependency!` warning followed by wrong-binary execution
brewkit's test sandbox auto-fetches commands invoked by bare name but not in `dependencies:`. The routing goes through `/opt/bin/pkgx`, which can resolve to a *different* binary than the one your build just installed (very common for cross-compilers and tools whose name collides with a different pantry package).
Fix: invoke own-bottle binaries by full path:
```yaml
env:
MY_CLANG: "{{prefix}}/bin/x86_64-w64-mingw32-clang"
test:
script:
- '"$MY_CLANG" --version'
```
### `file: command not found` (or `man`, `which`, ...) in the test sandbox
Build sandbox has more PATH than test sandbox. If your test needs a basic utility, declare it as a runtime or test dependency, or in-line the check.
## `versions:` block
### `versions: github:` mode silently ignores `match:`
Only `ignore:` and `strip:` are honored in `github:` mode. To filter releases with a regex, use the `url:` variant pointing to the project's tag/release directory listing, with `match:` as the filter:
```yaml
versions:
url: https://github.com/foo/bar/releases
match: /v[0-9]+\.[0-9]+\.[0-9]+$/
strip: /^v/
```
### `distributable: ~`
Yes, that literal null is valid and means "no source archive — fetch the payload in `build.script` yourself." Useful for binary-only packages (e.g. ziglang.org's prebuilt tarballs) where the canonical distribution is downloaded inside the script.
## Sub-packages and modularity
If the upstream project produces a heavy / optional component (Java SDK, gRPC bindings, GPU stack), do **not** silently drop it to make the bottle smaller. Either ship the full surface, or split into an explicit sub-package (e.g. `cubefs.io` + `cubefs.io/blobstore`, `openjdk.org` + `openjdk.org/v8`). Hidden trimming breaks user expectations and forces them back to system packages.
## Process: look at how the others did it
Before iterating ad nauseam on a build failure, check how nixpkgs / guix / homebrew / aur / debian / vcpkg handle the same package. Priority order, fastest signal first:
1. **nixpkgs** — `https://github.com/NixOS/nixpkgs/blob/master/pkgs/<category>/<pkg>/default.nix`. Most thorough wire-up; their stdenv handles the cc-wrapper/libstdc++ question we hit on many C++ projects.
2. **homebrew core** — `https://github.com/Homebrew/homebrew-core/blob/master/Formula/<pkg>.rb`. Brews are pinned to one specific macOS toolchain so they're less useful on Linux, but they document upstream's quirks (configure flags, env vars, patches).
3. **arch (AUR or official)** — `https://gitlab.archlinux.org/archlinux/packaging/packages/<pkg>` or `https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=<pkg>`. PKGBUILDs are very readable shell.
4. **guix** — `https://github.com/guix-mirror/guix/tree/master/gnu/packages/`. Like nix but smaller surface; hermetic build matches pkgx's problem space.
5. **vcpkg / conan** — `https://github.com/microsoft/vcpkg/blob/master/ports/<pkg>/` (CMake-driven; helpful for C++ toolchain).
6. **debian** — `https://salsa.debian.org/.../<pkg>/-/tree/debian/`; the `debian/rules` + `debian/patches/` directory often holds the canonical fix for upstream bugs.
Even when their answer doesn't apply directly (different stdenv assumptions), reading their recipe surfaces: the upstream bug thread, which configure flags / patches to apply, and which deps actually link in.
Wiki enrichment proposal
After contributing ~30 recipes over the last few weeks I've accumulated a fair amount of pantry-authoring tribal knowledge (cgo build chains, cc-wrapper-style libstdc++ resolution, CMake 4.x policy compat,
fix-patchelfquirks, brewkit test-sandbox PATH gotchas, etc.). The current wiki is mostly stubs (Troubleshooting.md= 3 lines,cmake.md= 1 line,Specifics.md= 1 line,Packaging-Guide.md= 8 lines), so I drafted enriched versions of those four pages.I can't PR a wiki repo directly through GitHub UI, so I'm filing this for review. Three options on how to land:
gollum-style wikis).Happy with whichever works. All four pages are below in collapsible blocks for readability.
Troubleshooting.md — failure patterns by language + toolchain (~180 lines)
cmake.md — CMake-specific gotchas (~90 lines)
Specifics.md — per-language notes: Go / Rust / Bazel / Maven / Node / Python, self-bootstrapping toolchains, sub-package pattern (~70 lines)
Packaging-Guide.md — extended (existing tips kept verbatim) + "check how other packagers solved it" + ship-the-full-upstream-surface guideline + when-to-fork-vN sub-recipes (~70 lines)
Sourcing
Most of the content comes from concrete debug sessions on my own recent PRs (binaryen, TiKV, cubefs, openjdk-v8, valgrind, the gcc-libc-wrapper work, etc.). Where I learned a pattern from another packager (nixpkgs' cc-wrapper, Arch's PKGBUILD for TiKV, etc.) I credit them. The "check how others did it" methodology has saved me hours and is worth promoting as a first-class part of the contributor flow.
Out of scope (suggestions for follow-up)
Things I noticed but didn't draft for this issue:
Home.mdis good as-is. Could link to the new Troubleshooting/cmake/Specifics from the sidebar.Examples.mdcould use one or two more "complex case" examples (vendored C++ deps, cgo+rocksdb, self-bootstrapping toolchain) — let me know if you want a follow-up issue for those.Building-for-Your-Language.mdis currently very thin; it could absorb the per-language content I put inSpecifics.mdinstead, if you prefer that split.Happy to iterate.