dbt-extractor: add build-dbt-extractor.yml for riscv64 wheels - #479
Merged
Conversation
dbt-extractor is a PyO3/maturin extension (Rust + the tree-sitter-jinja2 grammar) that dbt-core uses to extract refs/sources/configs from model files. Upstream publishes manylinux wheels for x86_64, i686, aarch64, armv7, s390x, ppc64le and ppc64, but not riscv64. Modelled on the linux/linux-cross jobs of upstream's release.yml, narrowed to riscv64 and run through cibuildwheel. pyo3 carries the `abi3-py39` feature, so maturin emits one cp39-abi3 wheel - the same single Linux wheel per arch upstream ships - built once on cp312 and reused (and tested) on cp313/cp314. musllinux is dropped: rustup.rs has no riscv64 musl toolchain. Upstream cuts releases without git tags, so the checkout is pinned to the 0.6.0 release commit; its tree is byte-identical to the PyPI sdist. The test step goes beyond upstream's `import dbt_extractor` smoke check and extracts from a model exercising refs, sources and configs.
cibuildwheel's default `build` frontend runs `python -m build /project` with cwd=/project. dbt-extractor keeps a dev helper named build.py at its repo root, which then shadows the `build` module; it imports tree_sitter at module level, so the build dies with ModuleNotFoundError before maturin is ever reached. Reproduced in quay.io/pypa/manylinux_2_39_aarch64: the `build` frontend fails there and `pip wheel` builds the cp39-abi3 wheel cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
.github/workflows/build-dbt-extractor.yml, building riscv64 wheels for dbt-extractor 0.6.0 — the PyO3/maturin extension (Rust + thetree-sitter-jinja2grammar) that dbt-core uses to pullref()/source()/config()values out of model files without rendering Jinja in Python.Upstream (dbt-labs/dbt-extractor) publishes manylinux wheels for x86_64, i686, aarch64, armv7, s390x, ppc64le and ppc64, plus musllinux, macOS and Windows — but nothing for riscv64, so
pip install dbt-extractorthere falls back to the sdist and needs a full Rust toolchain.Shape
Modelled on the
linux/linux-crossjobs of upstream's release.yml, narrowed to riscv64: a plain build-from-checkout run through cibuildwheel, with the Rust toolchain installed in-container viaCIBW_BEFORE_ALL_LINUX(same asbuild-fastuuid.yml/build-tiktoken.yml, since the project ships no[tool.cibuildwheel]table).One abi3 wheel, not a per-interpreter matrix.
Cargo.tomlbuilds pyo3 with theabi3-py39feature, so maturin tags the outputcp39-abi3whatever interpreter compiles it — exactly the single Linux wheel per arch that upstream publishes.CIBW_BUILDtherefore selectscp312/cp313/cp314: cibuildwheel compiles once for cp312, thenfind_compatible_wheelreuses that wheel for cp313 and cp314 while still running the test phase on each, so one wheel comes out with load-and-test coverage on three interpreters. cp314t is excluded — abi3 cannot target free-threaded builds, and upstream ships no free-threaded wheel on any arch.musllinux is dropped because rustup.rs has no riscv64 musl host toolchain (same reason
build-fastuuid.ymlandbuild-tiktoken.ymlskip it).The checkout is pinned to a commit, not a tag. dbt-labs cuts dbt-extractor releases without ever pushing a git tag (
repos/dbt-labs/dbt-extractor/git/refs/tagsis a 404), soDBT_EXTRACTOR_REFnames the 0.6.0 release commit89d4672("bump patch version, add changelog", #111, uploaded to PyPI 12 minutes later). I verified it is the release:Cargo.toml,Cargo.lock,pyproject.toml,LICENSE, all ofsrc/,tests/integration_tests.rsandCHANGELOG.mdare byte-identical between that commit's tarball and the publisheddbt_extractor-0.6.0.tar.gz. Theversioninput default stays the plain0.6.0so the nightly PyPI check keeps working.Crate versions do not float (gotcha 10):
Cargo.lockis committed at the release commit, sotree-sitter 0.20.10,tree-sitter-jinja2 v0.2.0,rayon,pyo3 0.24.1andthiserror 1.0.69all resolve to what upstream released against.Testing
Upstream's release workflow only does
python -c "import dbt_extractor"after building a wheel. The test command here extracts from a model that exercises all three result kinds and asserts the full structure:→
{'refs': [{'name': 'other_model'}], 'sources': {('my_src','my_tbl')}, 'configs': [('materialized','table'), ('tags',['a','b'])]}. That covers the parser, the type checker and the extractor end to end. A following step assertsdbt_extractor/dbt_extractor.abi3.soand the Apache-2.0LICENSEare actually inside the wheel, so a degraded build cannot go green.Local validation
actionlintis clean apart from the usuallabel "ubuntu-24.04-riscv" is unknown. The full recipe (rustup in-container, maturin build, wheel install, the exactCIBW_TEST_COMMANDrun throughsh -c) was rehearsed inquay.io/pypa/manylinux_2_39_aarch64— the same Rocky 10 image family — producingdbt_extractor-0.6.0-cp39-abi3-linux_aarch64.whland passing the smoke assertions, which confirms the abi3 tag, the extension name and the test command's quoting before spending a riscv64 cycle.Licensing
The wheel carries upstream's Apache-2.0
LICENSEindist-info/licenses/(asserted by the check step). The statically linked crates are MIT / Apache-2.0, matching how the repo's other Rust ports (fastuuid, tiktoken, hf-xet, orjson, litellm) ship.The
build.pytrapThe first CI attempt exposed something worth recording (folded into CLAUDE.md as gotcha 102): cibuildwheel's default
buildfrontend runspython -m build /project --wheelwith the container's cwd set to/project, andpython -mputs the cwd onsys.path[0]— so dbt-extractor's repo-rootbuild.py(a dev helper that regenerates the tree-sitter grammar, opening withfrom tree_sitter import Language, Parser) shadows thebuildmodule and the run dies withModuleNotFoundError: No module named 'tree_sitter'before maturin is ever reached. Upstream never sees it becausePyO3/maturin-actioncalls the backend directly.CIBW_BUILD_FRONTEND: pipfixes it — nothing shadowspip, and pip runs the PEP 517 hooks in a subprocess whosesys.path[0]is the wrapper's directory rather than the cwd. Reproduced and fixed in the aarch64 container before re-pushing.CI result
Green on run 33005309515: one
dbt_extractor-0.6.0-cp39-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl(458.8 kB, 7 minutes) built on cp312, thenFound previously built wheel … compatible with cp313-manylinux_riscv64 / cp314-manylinux_riscv64. Skipping build step…— the extraction assertions ran and passed on all three interpreters. The wheel-content check passed, andpublishdry-ran cleanly:Dry run (not on main branch — no upload will happen)/Would upload 1 file(s), plus[dry-run] Would write docs/packages/dbt-extractor.yaml.