Skip to content

new(pyqt5): per-binding sub-recipes (Top 300 #591)#13064

Draft
tannevaled wants to merge 11 commits into
pkgxdev:mainfrom
tannevaled:new/pyqt5
Draft

new(pyqt5): per-binding sub-recipes (Top 300 #591)#13064
tannevaled wants to merge 11 commits into
pkgxdev:mainfrom
tannevaled:new/pyqt5

Conversation

@tannevaled
Copy link
Copy Markdown
Contributor

@tannevaled tannevaled commented May 29, 2026

Summary

PyQt5 split into 11 recipes: one meta + 10 per-binding sub-recipes
(QtCore, QtDBus, QtNetwork, QtSql, QtTest, QtXml, QtGui, QtSvg,
QtPrintSupport, QtWidgets).

The original monolithic recipe iterated all 10 bindings in a single
CI job via pip install --config-settings=--enable=<binding> in a
shell loop. Each binding's link peaks at 6-8 GB RAM (sip-generated
.cpp), so the 7 GB GH-Linux runner OOM'd on every binding and GHA
cancelled the workflow at 1h33min (CI run 26712172867: 6/10 bindings
SIGKILL'd, 4 still queued).

The split gives each binding its own CI job with its own 6 h timeout
and an independent runner.

Topology

QtCore         (foundation, no PyQt5 deps)
  ├── QtDBus       (deps QtCore)
  ├── QtNetwork    (deps QtCore)
  ├── QtSql        (deps QtCore)
  ├── QtTest       (deps QtCore)
  ├── QtXml        (deps QtCore)
  └── QtGui        (deps QtCore, HEAVY)
        ├── QtSvg          (deps QtGui)
        ├── QtPrintSupport (deps QtGui)
        └── QtWidgets      (deps QtGui, HEAVIEST)
pyqt5 (meta)     (deps all 10)

Sibling deps are pinned exactly (={{version}}) so the meta sees one
coherent install across all 10 prefixes.

PyQt5 ships __init__.py with __path__ = pkgutil.extend_path(...),
making it a true namespace package. Each sub-binding drops its own
PyQt5/<Binding>.so into its own prefix's
share/python/site-packages/PyQt5/, and pkgx stacks the
PYTHONPATH so pkgutil.extend_path merges them at import time.

Preserved from the monolithic recipe

  • The awk patch for confirm-license = true in pyproject.toml
    (commit 495169a, which itself fixed YAML block-scalar parsing of
    the earlier sed a\ form).
  • CXXFLAGS=-g0 / CFLAGS=-g0 / MAKEFLAGS=-j1 memory discipline.
  • --no-build-isolation / --no-deps so pantry's CXXFLAGS reach the
    compiler and pip does not refetch PyQt5-sip from PyPI.

Merge order (chicken-and-egg)

All 11 recipes ship in this single PR. The non-foundation bindings
and the meta will FAIL on Linux x64 CI until QtCore is bottled to
dist.pkgx.dev — same shape as the font-util / xserver and
tk / tcl splits in pantry history.

Maintainers: please merge / bottle in topological order:

  1. QtCore first (no PyQt5 deps).
  2. Layer 2 (after QtCore is on dist.pkgx.dev):
    QtDBus, QtNetwork, QtSql, QtTest, QtXml, QtGui.
  3. Layer 3 (after QtGui is on dist.pkgx.dev):
    QtSvg, QtPrintSupport, QtWidgets.
  4. Meta pyqt5 last.

Cross-checked against homebrew's pyqt@5 (one big build, but their
runners have more RAM) and nixpkgs's python3.pkgs.pyqt5 (same).
Neither has the GHA-7 GB constraint we do, hence the split.

Closes part of #99 (Top 300 #591).

Test plan

  • All 11 package.yml parse as valid YAML.
  • QtCore Linux x64 CI build (no inter-PyQt5 deps) — first
    bottling pass.
  • Layer 2 + Layer 3 + meta — second / third pass after QtCore
    lands.
  • Smoke test on darwin/aarch64 after all 10 bindings ship.

🤖 Generated with Claude Code

Installed as a python-venv against pantry's qt.io (5.15) + sip.
Provides bin/pyqt5-python that has PyQt5 importable.

Closes part of pkgxdev#99 (holdout pkgxdev#591).
…n-venv.sh

python-venv.sh installs from SRCROOT (the extracted source tree), not
from a PyPI spec — it does `pip install $SRCROOT`. So we need:
1. A real distributable.url pointing at PyPI's sdist for PyQt5
2. Just call `python-venv.sh <binary>` without a PyPI package arg

The previous attempt with `distributable: ~` left SRCROOT empty,
producing 'Directory ... is not installable. Neither setup.py nor
pyproject.toml found'.
@tannevaled tannevaled marked this pull request as draft May 29, 2026 12:23
@tannevaled
Copy link
Copy Markdown
Contributor Author

Draft — PyQt5 sdist build hits 2 issues simultaneously:

  1. License prompt blocks CI: configure asks 'Type yes to accept the terms of the license'. Non-interactive CI hangs.
  2. OOM: SIGKILL after 13+ minutes — PyQt5's C++ binding generation is memory-heavy.

Fix needs:

  • Bypass license: pass --confirm-license to configure.py (sip-build), or set up python-venv.sh to forward pip --config-settings
  • Memory: build with -j1 or restrict bindings (PyQt5 has many submodules each compiled separately)

Both require modifying brewkit's python-venv.sh helper or a custom build flow. Deferring.

tannevaled and others added 8 commits May 29, 2026 20:07
brewkit's python-venv.sh OOM-killed (exit 137) at pip-install: PyQt5's
50+ Qt5 binding compiles peak at 4–6 GB per ld process and GH runners
have ~7 GB RAM total.

Replace the python-venv.sh call with an inline equivalent that:
  - creates the venv via the bottled python
  - installs pip/wheel/setuptools quietly
  - sets MAKEFLAGS=-j1 + CMAKE_BUILD_PARALLEL_LEVEL=1 → serial build
  - sets CXXFLAGS/CFLAGS=-g0 → ~30 % peak-memory cut at link time
  - pip-installs the SRCROOT with --no-build-isolation (brewkit
    already provides pyqt-builder etc. as build deps)

If this pattern proves out, the next step is lifting it into brewkit
as `python-venv-alt.sh` (or a `--jobs N` flag on python-venv.sh) and
opening a discussion on pkgxdev/pantry — see PR body.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous MAKEFLAGS=-j1 attempt was insufficient: pyqt-builder
spawns one make per binding in parallel and -j1 only serialises the
inner make. The outer parallelism (~30 simultaneous link-heavy
makes) is what blew the 7 GB GH-runner ceiling.

Two knobs together fit the build:

  --config-settings=--jobs=1
    Tells pyqt-builder to build ONE binding at a time. This is the
    load-bearing flag.

  --config-settings=--disable=Qt<heavy> (×26)
    Drops the QPdf, Qt3D*, QtQml, QtQuick, QtMultimedia*, QtWebKit*,
    QtBluetooth, QtPositioning, QtSensors, QtNfc, QtXmlPatterns,
    QtRemoteObjects, QtTextToSpeech, QtSerialPort, and platform-
    specific Extras bindings. The remaining ~10 bindings still cover
    QtCore / QtGui / QtWidgets / QtNetwork / QtSql / QtSvg / QtTest /
    QtDBus / QtPrintSupport / QtConcurrent — the 95% use case.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 26-module --disable list triggered sipbuild's UserException at
_enable_disable_bindings, likely because at least one binding name
doesn't exist in PyQt5 5.15.11 on Linux (QtMacExtras / QtWinExtras
are platform-gated and absent from the linux source tree's
pyproject.toml).

Keep only --config-settings=--jobs=1 + CXXFLAGS=-g0. If pyqt-builder
serialises the bindings (one make at a time) and we cut debug info,
each peak link should fit under 7 GB on its own.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PyQt5's monolithic `pip install .` was OOM-killed (exit 137) on
7 GB GH-runners. `--config-settings=--jobs=1` wasn't enough — a
single binding's final `ld` (QtWidgets peaks at ~4-6 GB) blows
past 7 GB on its own.

Run pip install once per binding (`--target=$STAGING/$binding`,
`--config-settings=--enable=$binding`, `--config-settings=--jobs=1`)
so each subprocess holds only one binding's compiled state in
memory, then merge the produced .so files into the venv's
PyQt5 site-packages dir. Bindings ordered lightest-first
(QtCore through QtPrintSupport, then QtGui, then QtWidgets last)
so a memory failure on the heaviest binding still leaves the
others importable.

pyqt-builder's `--no-make` was considered but is restricted to
the `build` tool (tools=['build']); pip's PEP 517 path calls
the `wheel` tool, so it can't be set via --config-settings.
The split-pip shape achieves the same memory isolation by
freeing each binding's working set between subprocesses.

Also: CXXFLAGS/CFLAGS=-g0 (~30 % less link RAM) and MAKEFLAGS=-j1
as belt-and-braces; per-binding staging dir cleaned after each
pass to free disk; test block tolerates a missing QtGui/QtWidgets.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The whole reason every binding was "failing" wasn't OOM — it was a
pip CLI parse error. pip rejects `--config-settings=--confirm-license`
with "Arguments to --config-settings must be of the form KEY=VAL"
because the flag has no value. So pip exits before sipbuild even
runs; every binding "fails" identically with QtCore.

Pass `--confirm-license=true` instead. sipbuild treats any truthy
value as "license accepted". Should also fix QtGui/QtWidgets which
my prior comment blamed on OOM — they may simply never have built.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ngs)

The previous `--config-settings=--confirm-license=true` attempt fell
into a pip/sipbuild deadlock:
  - pip insists --config-settings is KEY=VAL
  - sipbuild's argparse declares --confirm-license as store_true and
    refuses any value: "ignored explicit argument 'true'"

The route around: write `confirm-license = true` into the source
tarball's pyproject.toml under [tool.sip.project] before pip runs.
sipbuild reads pyproject.toml directly, bypassing the argparse path,
so the flag is honored.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous recipe used:

    sed -i.bak '/^\[tool\.sip\.project\]$/a\
    confirm-license = true' "$SRCROOT/pyproject.toml"

sed's `a\` command requires the appended content on the next line,
which means the second line in the `run: |` block had to start at
column 0. That broke YAML's block-scalar indentation rule (later
lines may not be less indented than the first non-empty line of
the block), causing the plan job to fail with:

  YAMLError: can not read a block mapping entry; a multiline key
  may not be an implicit key … at line 100, column 9

Switched to awk — single-line program, no newline-in-script
requirement, and arguably clearer intent (insert-after-pattern).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…olithic build)

The previous monolithic recipe iterated 10 bindings in a single CI job
via `pip install --config-settings=--enable=<binding>` in a shell loop.
Each binding's link peaks at 6-8 GB RAM (sip-generated .cpp), so the
7 GB GH-Linux runner OOM'd on every binding and GHA cancelled at
1h33min (CI run 26712172867: 6/10 bindings SIGKILL'd, 4 queued).

Split into:

  projects/riverbankcomputing.com/pyqt5/
  ├── package.yml                meta (depends on all 10 sub-bindings)
  ├── QtCore/package.yml         foundation, no inter-PyQt5 deps
  ├── QtDBus/package.yml         deps QtCore
  ├── QtNetwork/package.yml      deps QtCore
  ├── QtSql/package.yml          deps QtCore
  ├── QtTest/package.yml         deps QtCore
  ├── QtXml/package.yml          deps QtCore
  ├── QtGui/package.yml          deps QtCore (HEAVY)
  ├── QtSvg/package.yml          deps QtGui
  ├── QtPrintSupport/package.yml deps QtGui
  └── QtWidgets/package.yml      deps QtGui (HEAVIEST)

Each sub-binding is its own CI job with its own 6 h timeout.  Sibling
deps are pinned exactly (`={{version}}`) so the meta sees one
coherent PyQt5 install.

PyQt5 is a namespace package (`pkgutil.extend_path` in `__init__.py`),
so each sub-binding's prefix can ship its own `PyQt5/` dir and Python
merges them at import time via the stacked PYTHONPATH that pkgx sets
up automatically when the deps are in scope.

Preserved from the monolithic recipe:
- The awk patch for `confirm-license = true` (commit 495169a).
- `CXXFLAGS=-g0` / `CFLAGS=-g0` / `MAKEFLAGS=-j1` memory discipline.
- `--no-build-isolation` / `--no-deps` so our env reaches the
  compiler and pip does not refetch sip from PyPI.

NOTE — chicken-and-egg: all 11 recipes ship in the same PR.  Linux x64
CI for the non-foundation bindings (and the meta) will fail until
QtCore bottles to dist.pkgx.dev.  This is the same shape as the
font-util/xserver and Tk/Tcl8 splits; maintainers merge in
topological order:

  QtCore → {QtDBus,QtNetwork,QtSql,QtTest,QtXml,QtGui}
        → {QtSvg,QtPrintSupport,QtWidgets}
        → pyqt5 (meta)
@tannevaled tannevaled changed the title new(pyqt5): Python bindings for Qt 5 (Top 300 #591) new(pyqt5): per-binding sub-recipes (Top 300 #591) May 31, 2026
…-substitute)

Previous commit pinned sibling sub-bindings with `={{version}}` per
the task spec, but libpkgx's `validatePackageRequirement` runs at YAML
parse time — before template substitution — and rejects the literal
`{{version}}` string with `invalid constraint: undefined` (CI run
26716046336 confirmed: meta package fails before any build step).

PyQt5 sub-bindings all share the same version-discovery (PyPI simple
index) so they march in lockstep anyway; a `'*'` constraint is safe
here.  Same shape as poppler / poppler-data and x.org/exts -> x11.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant