new(pyqt5): per-binding sub-recipes (Top 300 #591)#13064
Draft
tannevaled wants to merge 11 commits into
Draft
Conversation
Installed as a python-venv against pantry's qt.io (5.15) + sip. Provides bin/pyqt5-python that has PyQt5 importable. Closes part of pkgxdev#99 (holdout pkgxdev#591).
…n-venv.sh python-venv.sh installs from SRCROOT (the extracted source tree), not from a PyPI spec — it does `pip install $SRCROOT`. So we need: 1. A real distributable.url pointing at PyPI's sdist for PyQt5 2. Just call `python-venv.sh <binary>` without a PyPI package arg The previous attempt with `distributable: ~` left SRCROOT empty, producing 'Directory ... is not installable. Neither setup.py nor pyproject.toml found'.
Contributor
Author
|
Draft — PyQt5 sdist build hits 2 issues simultaneously:
Fix needs:
Both require modifying brewkit's python-venv.sh helper or a custom build flow. Deferring. |
brewkit's python-venv.sh OOM-killed (exit 137) at pip-install: PyQt5's
50+ Qt5 binding compiles peak at 4–6 GB per ld process and GH runners
have ~7 GB RAM total.
Replace the python-venv.sh call with an inline equivalent that:
- creates the venv via the bottled python
- installs pip/wheel/setuptools quietly
- sets MAKEFLAGS=-j1 + CMAKE_BUILD_PARALLEL_LEVEL=1 → serial build
- sets CXXFLAGS/CFLAGS=-g0 → ~30 % peak-memory cut at link time
- pip-installs the SRCROOT with --no-build-isolation (brewkit
already provides pyqt-builder etc. as build deps)
If this pattern proves out, the next step is lifting it into brewkit
as `python-venv-alt.sh` (or a `--jobs N` flag on python-venv.sh) and
opening a discussion on pkgxdev/pantry — see PR body.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous MAKEFLAGS=-j1 attempt was insufficient: pyqt-builder
spawns one make per binding in parallel and -j1 only serialises the
inner make. The outer parallelism (~30 simultaneous link-heavy
makes) is what blew the 7 GB GH-runner ceiling.
Two knobs together fit the build:
--config-settings=--jobs=1
Tells pyqt-builder to build ONE binding at a time. This is the
load-bearing flag.
--config-settings=--disable=Qt<heavy> (×26)
Drops the QPdf, Qt3D*, QtQml, QtQuick, QtMultimedia*, QtWebKit*,
QtBluetooth, QtPositioning, QtSensors, QtNfc, QtXmlPatterns,
QtRemoteObjects, QtTextToSpeech, QtSerialPort, and platform-
specific Extras bindings. The remaining ~10 bindings still cover
QtCore / QtGui / QtWidgets / QtNetwork / QtSql / QtSvg / QtTest /
QtDBus / QtPrintSupport / QtConcurrent — the 95% use case.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 26-module --disable list triggered sipbuild's UserException at _enable_disable_bindings, likely because at least one binding name doesn't exist in PyQt5 5.15.11 on Linux (QtMacExtras / QtWinExtras are platform-gated and absent from the linux source tree's pyproject.toml). Keep only --config-settings=--jobs=1 + CXXFLAGS=-g0. If pyqt-builder serialises the bindings (one make at a time) and we cut debug info, each peak link should fit under 7 GB on its own. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PyQt5's monolithic `pip install .` was OOM-killed (exit 137) on 7 GB GH-runners. `--config-settings=--jobs=1` wasn't enough — a single binding's final `ld` (QtWidgets peaks at ~4-6 GB) blows past 7 GB on its own. Run pip install once per binding (`--target=$STAGING/$binding`, `--config-settings=--enable=$binding`, `--config-settings=--jobs=1`) so each subprocess holds only one binding's compiled state in memory, then merge the produced .so files into the venv's PyQt5 site-packages dir. Bindings ordered lightest-first (QtCore through QtPrintSupport, then QtGui, then QtWidgets last) so a memory failure on the heaviest binding still leaves the others importable. pyqt-builder's `--no-make` was considered but is restricted to the `build` tool (tools=['build']); pip's PEP 517 path calls the `wheel` tool, so it can't be set via --config-settings. The split-pip shape achieves the same memory isolation by freeing each binding's working set between subprocesses. Also: CXXFLAGS/CFLAGS=-g0 (~30 % less link RAM) and MAKEFLAGS=-j1 as belt-and-braces; per-binding staging dir cleaned after each pass to free disk; test block tolerates a missing QtGui/QtWidgets. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The whole reason every binding was "failing" wasn't OOM — it was a pip CLI parse error. pip rejects `--config-settings=--confirm-license` with "Arguments to --config-settings must be of the form KEY=VAL" because the flag has no value. So pip exits before sipbuild even runs; every binding "fails" identically with QtCore. Pass `--confirm-license=true` instead. sipbuild treats any truthy value as "license accepted". Should also fix QtGui/QtWidgets which my prior comment blamed on OOM — they may simply never have built. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ngs)
The previous `--config-settings=--confirm-license=true` attempt fell
into a pip/sipbuild deadlock:
- pip insists --config-settings is KEY=VAL
- sipbuild's argparse declares --confirm-license as store_true and
refuses any value: "ignored explicit argument 'true'"
The route around: write `confirm-license = true` into the source
tarball's pyproject.toml under [tool.sip.project] before pip runs.
sipbuild reads pyproject.toml directly, bypassing the argparse path,
so the flag is honored.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous recipe used:
sed -i.bak '/^\[tool\.sip\.project\]$/a\
confirm-license = true' "$SRCROOT/pyproject.toml"
sed's `a\` command requires the appended content on the next line,
which means the second line in the `run: |` block had to start at
column 0. That broke YAML's block-scalar indentation rule (later
lines may not be less indented than the first non-empty line of
the block), causing the plan job to fail with:
YAMLError: can not read a block mapping entry; a multiline key
may not be an implicit key … at line 100, column 9
Switched to awk — single-line program, no newline-in-script
requirement, and arguably clearer intent (insert-after-pattern).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…olithic build)
The previous monolithic recipe iterated 10 bindings in a single CI job
via `pip install --config-settings=--enable=<binding>` in a shell loop.
Each binding's link peaks at 6-8 GB RAM (sip-generated .cpp), so the
7 GB GH-Linux runner OOM'd on every binding and GHA cancelled at
1h33min (CI run 26712172867: 6/10 bindings SIGKILL'd, 4 queued).
Split into:
projects/riverbankcomputing.com/pyqt5/
├── package.yml meta (depends on all 10 sub-bindings)
├── QtCore/package.yml foundation, no inter-PyQt5 deps
├── QtDBus/package.yml deps QtCore
├── QtNetwork/package.yml deps QtCore
├── QtSql/package.yml deps QtCore
├── QtTest/package.yml deps QtCore
├── QtXml/package.yml deps QtCore
├── QtGui/package.yml deps QtCore (HEAVY)
├── QtSvg/package.yml deps QtGui
├── QtPrintSupport/package.yml deps QtGui
└── QtWidgets/package.yml deps QtGui (HEAVIEST)
Each sub-binding is its own CI job with its own 6 h timeout. Sibling
deps are pinned exactly (`={{version}}`) so the meta sees one
coherent PyQt5 install.
PyQt5 is a namespace package (`pkgutil.extend_path` in `__init__.py`),
so each sub-binding's prefix can ship its own `PyQt5/` dir and Python
merges them at import time via the stacked PYTHONPATH that pkgx sets
up automatically when the deps are in scope.
Preserved from the monolithic recipe:
- The awk patch for `confirm-license = true` (commit 495169a).
- `CXXFLAGS=-g0` / `CFLAGS=-g0` / `MAKEFLAGS=-j1` memory discipline.
- `--no-build-isolation` / `--no-deps` so our env reaches the
compiler and pip does not refetch sip from PyPI.
NOTE — chicken-and-egg: all 11 recipes ship in the same PR. Linux x64
CI for the non-foundation bindings (and the meta) will fail until
QtCore bottles to dist.pkgx.dev. This is the same shape as the
font-util/xserver and Tk/Tcl8 splits; maintainers merge in
topological order:
QtCore → {QtDBus,QtNetwork,QtSql,QtTest,QtXml,QtGui}
→ {QtSvg,QtPrintSupport,QtWidgets}
→ pyqt5 (meta)
…-substitute)
Previous commit pinned sibling sub-bindings with `={{version}}` per
the task spec, but libpkgx's `validatePackageRequirement` runs at YAML
parse time — before template substitution — and rejects the literal
`{{version}}` string with `invalid constraint: undefined` (CI run
26716046336 confirmed: meta package fails before any build step).
PyQt5 sub-bindings all share the same version-discovery (PyPI simple
index) so they march in lockstep anyway; a `'*'` constraint is safe
here. Same shape as poppler / poppler-data and x.org/exts -> x11.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PyQt5 split into 11 recipes: one meta + 10 per-binding sub-recipes
(QtCore, QtDBus, QtNetwork, QtSql, QtTest, QtXml, QtGui, QtSvg,
QtPrintSupport, QtWidgets).
The original monolithic recipe iterated all 10 bindings in a single
CI job via
pip install --config-settings=--enable=<binding>in ashell loop. Each binding's link peaks at 6-8 GB RAM (sip-generated
.cpp), so the 7 GB GH-Linux runner OOM'd on every binding and GHAcancelled the workflow at 1h33min (CI run 26712172867: 6/10 bindings
SIGKILL'd, 4 still queued).
The split gives each binding its own CI job with its own 6 h timeout
and an independent runner.
Topology
Sibling deps are pinned exactly (
={{version}}) so the meta sees onecoherent install across all 10 prefixes.
PyQt5 ships
__init__.pywith__path__ = pkgutil.extend_path(...),making it a true namespace package. Each sub-binding drops its own
PyQt5/<Binding>.sointo its own prefix'sshare/python/site-packages/PyQt5/, and pkgx stacks thePYTHONPATHsopkgutil.extend_pathmerges them at import time.Preserved from the monolithic recipe
confirm-license = trueinpyproject.toml(commit 495169a, which itself fixed YAML block-scalar parsing of
the earlier sed
a\form).CXXFLAGS=-g0/CFLAGS=-g0/MAKEFLAGS=-j1memory discipline.--no-build-isolation/--no-depsso pantry's CXXFLAGS reach thecompiler and pip does not refetch
PyQt5-sipfrom PyPI.Merge order (chicken-and-egg)
All 11 recipes ship in this single PR. The non-foundation bindings
and the meta will FAIL on Linux x64 CI until QtCore is bottled to
dist.pkgx.dev— same shape as thefont-util/xserverandtk/tclsplits in pantry history.Maintainers: please merge / bottle in topological order:
QtCorefirst (no PyQt5 deps).QtDBus,QtNetwork,QtSql,QtTest,QtXml,QtGui.QtSvg,QtPrintSupport,QtWidgets.pyqt5last.Cross-checked against homebrew's
pyqt@5(one big build, but theirrunners have more RAM) and nixpkgs's
python3.pkgs.pyqt5(same).Neither has the GHA-7 GB constraint we do, hence the split.
Closes part of #99 (Top 300 #591).
Test plan
package.ymlparse as valid YAML.bottling pass.
lands.
🤖 Generated with Claude Code