Vmlinux gen fix (#97)

* widen bitfields whose declared width exceeds their base type

clang2py can emit a bitfield typed as a smaller ctype than its own
declared width (e.g. a 15-bit field typed ctypes.c_ubyte, which only
has 8 bits) - ctypes rejects these with 'ValueError: number of bits
invalid for bit field'. Surfaced by struct_vmbus_channel_offer_channel
on a kernel with Hyper-V support enabled (not present locally, but
present on GitHub Actions' Azure-hosted runners).

Generalizes the existing c_bool-specific workaround to all integer
ctypes, widening to the smallest standard type that fits.

* fix ruff-format violation in vmlinux-gen.py

* Add CI workflow to run the test suite (#96)

* add CI workflow to run the test suite

Runs make test on every push/PR. Kernel verifier tests need passwordless
sudo and a kernel with BTF/BPF enabled, which isn't guaranteed on every
runner, so the workflow probes for working sudo first and only attempts
them if it's available.

Also add ctypeslib2 to the test extra: make test now regenerates
vmlinux.py via tools/vmlinux-gen.py, which needs clang2py.

* fix CI: bpftool is a virtual package on Ubuntu, install linux-tools instead

* fix CI: locate bpftool binary and add it to PATH manually

linux-tools-generic's update-alternatives symlink for bpftool doesn't
fire on GitHub-hosted runners, since their kernel version has no
matching linux-tools-<version> package.

* pin clang bindings to 16.0.6 to match Ubuntu's default libclang

pip installs the latest 'clang' release by default, whose libclang API
surface is newer than Ubuntu 24.04's apt libclang-16, causing a
LibclangError about an undefined symbol. Older bindings against a
newer libclang stay compatible, so pin the bindings low rather than
the system library.

* DEBUG: dump struct_vmbus_channel_offer_channel on test failure

* remove debug step now that the bitfield issue is fixed

* fix CI: install LLVM 19 from apt.llvm.org, Ubuntu's default is too old

llvmlite>=0.49's ArgumentAttributes only recognizes the LLVM 19+
'captures(none)' spelling of the renamed 'nocapture' attribute.
Ubuntu noble's default llvm/clang packages are LLVM 18, whose llc
can't parse that attribute in the emitted .ll text.

* DEBUG: show real llc stderr on failure

* fix YAML syntax in debug step

* fix CI: install LLVM 22 (matching llvmlite's bundled version), not 19

Verified on CI: LLVM 19's llc still rejects 'captures(none)' as a
parse error ('expected ) at end of argument list'). llvmlite 0.49
bundles LLVM 22.1.0 internally; match that generation instead.

* remove debug step, LLVM 22 fix confirmed working on CI

* avoid running CI twice per push: scope push trigger to master

Branches live in this repo, not forks, so a push to a branch with an
open PR fired both push and pull_request for the same commit. push
now only fires for master (a post-merge check); pull_request already
covers every commit on a feature branch.

Author: varun-r-mallya

.github/workflows/test.yml

@@ -0,0 +1,92 @@
+# Runs the pytest suite (IR generation + LLC compilation). Kernel verifier
+# tests additionally need passwordless sudo and a kernel with BTF/BPF
+# enabled, which isn't guaranteed on every runner, so we probe for working
+# sudo first and only attempt them if it's there.
+#
+# `push` is scoped to master only: branches here live in this repo rather
+# than forks, so a push to a branch with an open PR would otherwise fire
+# both `push` and `pull_request` for the same commit, running everything
+# twice. `pull_request` covers feature branches; `push` still gives master
+# a post-merge check.
+
+name: Test
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [master]
+  pull_request:
+
+jobs:
+  test:
+    name: Test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y lsb-release wget software-properties-common gnupg linux-tools-common linux-tools-generic
+
+          # Ubuntu's default `llvm`/`clang` packages (LLVM 18 on noble) are
+          # too old to assemble the IR llvmlite>=0.49 emits: llvmlite's
+          # ArgumentAttributes only knows the 'captures(none)' spelling of
+          # the renamed 'nocapture' attribute, matching the LLVM 22.1.0 it
+          # bundles internally - and an llc from an older LLVM (verified:
+          # 19 still rejects it as a parse error) can't read that attribute
+          # in the .ll text. Install a matching-generation LLVM from
+          # apt.llvm.org instead of the distro default, and make its tools
+          # the ones found on PATH.
+          wget https://apt.llvm.org/llvm.sh
+          chmod +x llvm.sh
+          sudo ./llvm.sh 22 all
+          sudo ln -sf /usr/bin/clang-22 /usr/local/bin/clang
+          sudo ln -sf /usr/bin/llc-22 /usr/local/bin/llc
+          clang --version
+          llc --version
+
+          # bpftool isn't an installable package by itself on Ubuntu: it's a
+          # virtual package provided by linux-tools-common + a kernel-flavor
+          # linux-tools-<flavor> package. The runner's exact kernel version
+          # has no matching linux-tools-<version> package, so the
+          # update-alternatives symlink for `bpftool` doesn't get set up;
+          # find whatever binary the generic-flavor package installed and
+          # put it on PATH ourselves.
+          bpftool_bin=$(sudo find /usr/lib/linux-tools* -name bpftool -type f 2>/dev/null | head -1)
+          if [ -z "$bpftool_bin" ]; then
+            echo "::error::Could not find a bpftool binary after installing linux-tools-generic"
+            exit 1
+          fi
+          sudo ln -sf "$bpftool_bin" /usr/local/bin/bpftool
+          bpftool version
+
+      - name: Install uv
+        run: pip install uv
+
+      - name: Install project
+        run: uv pip install --system -e ".[test]"
+
+      - name: Run test suite
+        run: make test
+
+      - name: Check whether sudo is usable
+        id: sudo-check
+        run: |
+          if sudo -n true 2>/dev/null; then
+            echo "Passwordless sudo is available."
+            echo "available=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "No passwordless sudo on this runner; kernel verifier tests will be skipped."
+            echo "available=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Run kernel verifier tests
+        if: steps.sudo-check.outputs.available == 'true'
+        run: |
+          sudo -v
+          make test-verifier

pyproject.toml

@@ -44,6 +44,14 @@ docs = [
 test = [
     "pytest>=8.0",
     "pytest-cov>=5.0",
+    "ctypeslib2",
+    # Pinned rather than left to ctypeslib2's own (unpinned) dependency: pip
+    # installs the latest release by default, and its libclang API surface
+    # can be newer than the system libclang (e.g. Ubuntu 24.04 ships
+    # libclang-16), which fails with a LibclangError about an undefined
+    # symbol. Older bindings against a newer libclang stay compatible, so
+    # pin to an old-enough release instead of pinning apt's libclang.
+    "clang==16.0.6",
 ]
 
 [tool.setuptools.packages.find]

tools/vmlinux-gen.py

@@ -247,14 +247,57 @@ def step5_postprocess(self, input_file):
         # Replace ('_20', ctypes.c_char, 8) with ('_20', ctypes.c_uint8, 8)
         data = re.sub(r"(ctypes\.c_char)(\s*,\s*\d+\))", r"ctypes.c_uint8\2", data)
 
-        # below to replace those c_bool with bitfield greater than 8
-        def repl(m):
-            name, bits = m.groups()
-            return (
-                f"('{name}', ctypes.c_uint32, {bits})" if int(bits) > 8 else m.group(0)
-            )
+        # Some bitfields come out of clang2py with a declared width that
+        # exceeds their own base type's bit width (e.g. a 15-bit field typed
+        # as ctypes.c_ubyte, which only has 8 bits) - ctypes rejects these
+        # outright with "ValueError: number of bits invalid for bit field".
+        # Widen the base type to the smallest standard integer type that can
+        # actually hold the declared width.
+        bitfield_type_widths = {
+            "c_bool": 8,
+            "c_byte": 8,
+            "c_ubyte": 8,
+            "c_int8": 8,
+            "c_uint8": 8,
+            "c_short": 16,
+            "c_ushort": 16,
+            "c_int16": 16,
+            "c_uint16": 16,
+            "c_int": 32,
+            "c_uint": 32,
+            "c_int32": 32,
+            "c_uint32": 32,
+            "c_long": 64,
+            "c_ulong": 64,
+            "c_longlong": 64,
+            "c_ulonglong": 64,
+            "c_int64": 64,
+            "c_uint64": 64,
+        }
+        promoted_type_for_width = {
+            8: "c_uint8",
+            16: "c_uint16",
+            32: "c_uint32",
+            64: "c_uint64",
+        }
 
-        data = re.sub(r"\('([^']+)',\s*ctypes\.c_bool,\s*(\d+)\)", repl, data)
+        def widen_oversized_bitfields(m):
+            name, base_type, bits = m.group(1), m.group(2), int(m.group(3))
+            type_width = bitfield_type_widths.get(base_type)
+            if type_width is None or bits <= type_width:
+                return m.group(0)
+            for width in (8, 16, 32, 64):
+                if bits <= width:
+                    return (
+                        f"('{name}', ctypes.{promoted_type_for_width[width]}, {bits})"
+                    )
+            return m.group(0)
+
+        data = re.sub(
+            r"\('([^']+)',\s*ctypes\.([a-zA-Z0-9_]+),\s*(\d+)\)",
+            widen_oversized_bitfields,
+            data,
+        )
 
         # Remove ctypes. prefix from invalid entries
         invalid_ctypes = ["bpf_iter_state", "_cache_type", "fs_context_purpose"]