Skip to content

dbt-extractor: add build-dbt-extractor.yml for riscv64 wheels - #479

Merged
luhenry merged 2 commits into
mainfrom
dbt-extractor
Aug 26, 2026
Merged

dbt-extractor: add build-dbt-extractor.yml for riscv64 wheels#479
luhenry merged 2 commits into
mainfrom
dbt-extractor

Conversation

@luhenry

@luhenry luhenry commented Aug 26, 2026

Copy link
Copy Markdown
Member

Adds .github/workflows/build-dbt-extractor.yml, building riscv64 wheels for dbt-extractor 0.6.0 — the PyO3/maturin extension (Rust + the tree-sitter-jinja2 grammar) that dbt-core uses to pull ref()/source()/config() values out of model files without rendering Jinja in Python.

Upstream (dbt-labs/dbt-extractor) publishes manylinux wheels for x86_64, i686, aarch64, armv7, s390x, ppc64le and ppc64, plus musllinux, macOS and Windows — but nothing for riscv64, so pip install dbt-extractor there falls back to the sdist and needs a full Rust toolchain.

Shape

Modelled on the linux / linux-cross jobs of upstream's release.yml, narrowed to riscv64: a plain build-from-checkout run through cibuildwheel, with the Rust toolchain installed in-container via CIBW_BEFORE_ALL_LINUX (same as build-fastuuid.yml / build-tiktoken.yml, since the project ships no [tool.cibuildwheel] table).

One abi3 wheel, not a per-interpreter matrix. Cargo.toml builds pyo3 with the abi3-py39 feature, so maturin tags the output cp39-abi3 whatever interpreter compiles it — exactly the single Linux wheel per arch that upstream publishes. CIBW_BUILD therefore selects cp312/cp313/cp314: cibuildwheel compiles once for cp312, then find_compatible_wheel reuses that wheel for cp313 and cp314 while still running the test phase on each, so one wheel comes out with load-and-test coverage on three interpreters. cp314t is excluded — abi3 cannot target free-threaded builds, and upstream ships no free-threaded wheel on any arch.

musllinux is dropped because rustup.rs has no riscv64 musl host toolchain (same reason build-fastuuid.yml and build-tiktoken.yml skip it).

The checkout is pinned to a commit, not a tag. dbt-labs cuts dbt-extractor releases without ever pushing a git tag (repos/dbt-labs/dbt-extractor/git/refs/tags is a 404), so DBT_EXTRACTOR_REF names the 0.6.0 release commit 89d4672 ("bump patch version, add changelog", #111, uploaded to PyPI 12 minutes later). I verified it is the release: Cargo.toml, Cargo.lock, pyproject.toml, LICENSE, all of src/, tests/integration_tests.rs and CHANGELOG.md are byte-identical between that commit's tarball and the published dbt_extractor-0.6.0.tar.gz. The version input default stays the plain 0.6.0 so the nightly PyPI check keeps working.

Crate versions do not float (gotcha 10): Cargo.lock is committed at the release commit, so tree-sitter 0.20.10, tree-sitter-jinja2 v0.2.0, rayon, pyo3 0.24.1 and thiserror 1.0.69 all resolve to what upstream released against.

Testing

Upstream's release workflow only does python -c "import dbt_extractor" after building a wheel. The test command here extracts from a model that exercises all three result kinds and asserts the full structure:

{{ config(materialized='table', tags=['a','b']) }} select * from {{ ref('other_model') }} join {{ source('my_src','my_tbl') }}

{'refs': [{'name': 'other_model'}], 'sources': {('my_src','my_tbl')}, 'configs': [('materialized','table'), ('tags',['a','b'])]}. That covers the parser, the type checker and the extractor end to end. A following step asserts dbt_extractor/dbt_extractor.abi3.so and the Apache-2.0 LICENSE are actually inside the wheel, so a degraded build cannot go green.

Local validation

actionlint is clean apart from the usual label "ubuntu-24.04-riscv" is unknown. The full recipe (rustup in-container, maturin build, wheel install, the exact CIBW_TEST_COMMAND run through sh -c) was rehearsed in quay.io/pypa/manylinux_2_39_aarch64 — the same Rocky 10 image family — producing dbt_extractor-0.6.0-cp39-abi3-linux_aarch64.whl and passing the smoke assertions, which confirms the abi3 tag, the extension name and the test command's quoting before spending a riscv64 cycle.

Licensing

The wheel carries upstream's Apache-2.0 LICENSE in dist-info/licenses/ (asserted by the check step). The statically linked crates are MIT / Apache-2.0, matching how the repo's other Rust ports (fastuuid, tiktoken, hf-xet, orjson, litellm) ship.

The build.py trap

The first CI attempt exposed something worth recording (folded into CLAUDE.md as gotcha 102): cibuildwheel's default build frontend runs python -m build /project --wheel with the container's cwd set to /project, and python -m puts the cwd on sys.path[0] — so dbt-extractor's repo-root build.py (a dev helper that regenerates the tree-sitter grammar, opening with from tree_sitter import Language, Parser) shadows the build module and the run dies with ModuleNotFoundError: No module named 'tree_sitter' before maturin is ever reached. Upstream never sees it because PyO3/maturin-action calls the backend directly. CIBW_BUILD_FRONTEND: pip fixes it — nothing shadows pip, and pip runs the PEP 517 hooks in a subprocess whose sys.path[0] is the wrapper's directory rather than the cwd. Reproduced and fixed in the aarch64 container before re-pushing.

CI result

Green on run 33005309515: one dbt_extractor-0.6.0-cp39-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl (458.8 kB, 7 minutes) built on cp312, then Found previously built wheel … compatible with cp313-manylinux_riscv64 / cp314-manylinux_riscv64. Skipping build step… — the extraction assertions ran and passed on all three interpreters. The wheel-content check passed, and publish dry-ran cleanly: Dry run (not on main branch — no upload will happen) / Would upload 1 file(s), plus [dry-run] Would write docs/packages/dbt-extractor.yaml.

dbt-extractor is a PyO3/maturin extension (Rust + the tree-sitter-jinja2
grammar) that dbt-core uses to extract refs/sources/configs from model
files. Upstream publishes manylinux wheels for x86_64, i686, aarch64,
armv7, s390x, ppc64le and ppc64, but not riscv64.

Modelled on the linux/linux-cross jobs of upstream's release.yml, narrowed
to riscv64 and run through cibuildwheel. pyo3 carries the `abi3-py39`
feature, so maturin emits one cp39-abi3 wheel - the same single Linux wheel
per arch upstream ships - built once on cp312 and reused (and tested) on
cp313/cp314. musllinux is dropped: rustup.rs has no riscv64 musl toolchain.

Upstream cuts releases without git tags, so the checkout is pinned to the
0.6.0 release commit; its tree is byte-identical to the PyPI sdist.

The test step goes beyond upstream's `import dbt_extractor` smoke check and
extracts from a model exercising refs, sources and configs.
cibuildwheel's default `build` frontend runs `python -m build /project` with
cwd=/project. dbt-extractor keeps a dev helper named build.py at its repo
root, which then shadows the `build` module; it imports tree_sitter at
module level, so the build dies with ModuleNotFoundError before maturin is
ever reached. Reproduced in quay.io/pypa/manylinux_2_39_aarch64: the `build`
frontend fails there and `pip wheel` builds the cp39-abi3 wheel cleanly.
@luhenry
luhenry merged commit 1963a32 into main Aug 26, 2026
6 checks passed
@luhenry
luhenry deleted the dbt-extractor branch August 26, 2026 19:52
@luhenry luhenry linked an issue Aug 26, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dbt-extractor riscv64 support

1 participant