Skip to content

tree-sitter-languages: add build-tree-sitter-languages.yml for riscv64 wheels - #467

Merged
luhenry merged 1 commit into
mainfrom
tree-sitter-languages
Aug 26, 2026
Merged

tree-sitter-languages: add build-tree-sitter-languages.yml for riscv64 wheels#467
luhenry merged 1 commit into
mainfrom
tree-sitter-languages

Conversation

@luhenry

@luhenry luhenry commented Aug 26, 2026

Copy link
Copy Markdown
Member

Adds .github/workflows/build-tree-sitter-languages.yml, building tree-sitter-languages 1.10.2 riscv64 wheels for cp312/cp313/cp314/cp314t. Upstream publishes real per-interpreter compiled wheels (cp37–cp312, ~8.5 MB each) for x86_64/i686/aarch64/musl/macOS/Windows but nothing for riscv64.

The workflow mirrors upstream's release.yml: cibuildwheel over a checkout of the tag, with before-build running build.py, which shallow-clones the 48 grammar repositories pinned in repos.txt and links their parser.c/scanner.cc into a single tree_sitter_languages/languages.so (~88 MB unstripped) that the Cython core extension dlopens through py-tree-sitter. Tests are upstream's own pytest --showlocals {package}/tests, which parses Python with the built parser and instantiates all 48 grammars.

Upstream pins only Cython, and three build inputs had to be pinned to build the tag at all today — all three failures are arch-independent and reproduce on x86/arm:

  • tree-sitter<0.22build.py calls Language.build_library and core.pyx calls Language(path, name); both were removed in tree-sitter 0.22, so an unpinned resolve (upstream's pip install -e .) gets 0.26 and neither the build nor the test suite can work. The pin covers the test phase too.
  • cython==3.3.0 — upstream's cython==3.0.8 predates free threading and fails to compile core.pyx under cp314t; 3.3.0 builds cleanly on all four interpreters (core.pyx is 15 lines of plain Python, so there is no codegen-drift exposure of the kind gotcha 23 describes).
  • setuptools<79 — setuptools 79.0.0 regressed distutils' C++ link command: the class default for linker_exe_cxx is ["c++", "-shared"], and _linker_params strips that whole prefix off linker_so_cxx, so -shared is dropped and build.py's link of languages.so dies with crt1.o: undefined reference to 'main'. build.py uses new_compiler() without customize_compiler(), so LDCXXSHARED never gets a look-in and an env var cannot work around it. Bisected in the manylinux image: 78.1.1 emits c++ -shared …, 79.0.1 emits c++ ….

patches/tree-sitter-languages/1.10.2/0001-ship-the-vendored-grammar-licences-in-the-wheel.patch closes a licensing gap: languages.so is 48 third-party grammars compiled in, each under its own MIT/Apache-2.0 terms, while the wheel ships only tree-sitter-languages' own LICENSE. The patch adds license_files=['LICENSE', 'vendor/*/LICENSE*'] to setup.py (the default glob only looks at the project root), which lands 46 licence files in dist-info/licenses/vendor/ — the two exceptions, tree-sitter-ql and tree-sitter-fixed-form-fortran, carry no licence file at the commits repos.txt pins. The test command asserts the count so the patch cannot silently stop applying.

Local validation before pushing: a full cibuildwheel --only cp312-manylinux_aarch64 run against quay.io/pypa/manylinux_2_39_aarch64 (same Rocky 10 image family) — clone, compile, wheel, auditwheel repair to manylinux_2_34_aarch64, licence assertion, 3 tests passed — plus the same build-and-test loop by hand in that image for cp313, cp314 and cp314t. cp314t works but re-enables the GIL on import (Cython does not declare Py_mod_gil for this module), which is a runtime property of upstream's source, not of the port.

…4 wheels

Builds tree-sitter-languages 1.10.2 riscv64 wheels for cp312/cp313/cp314/cp314t,
mirroring upstream's release.yml (cibuildwheel over a checkout, before-build runs
build.py to clone the 48 pinned grammar repos and link them into languages.so).

Upstream pins nothing but Cython, so three build inputs are pinned here: tree-sitter
below 0.22 (build.py needs Language.build_library and core.pyx needs
Language(path, name), both removed there), Cython 3.3.0 (upstream's 3.0.8 predates
free threading and cannot compile core.pyx under cp314t), and setuptools below 79
(79.0.0 dropped -shared from distutils' C++ link command, so languages.so links as
an executable and fails on a missing main).

Carries one patch adding the vendored grammars' licence files to the wheel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@luhenry
luhenry merged commit c1857b7 into main Aug 26, 2026
8 checks passed
@luhenry
luhenry deleted the tree-sitter-languages branch August 26, 2026 19:20
@luhenry luhenry linked an issue Aug 26, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tree-sitter-languages riscv64 support

1 participant