chunkers: buzhash64 blockwise hit test, clang scalar-loop pins, always_inline (kernel codegen fixes) - #10307
Merged
ThomasWaldmann merged 3 commits intoSep 2, 2026
Conversation
…e aarch64 default The blockwise kernel tested its 8 lanes with a loop accumulating `hit |= ((c ^ s[k]) & M[k]) == 0`. clang lowers that into a chain of conditional increments; gcc >= 12 at -O2 vectorises it and reloads the eight 8-byte stores of s[] as 16-byte vectors, which fails store-to-load forwarding on every block - the "blockwise 8x slower than scalar" seen on Zen CPUs in borgbackup#10160. Written as a short-circuit chain it compiles to eight test-and-branch pairs that leave the loop at the first hit. Measured at the kernel level (bit-identical cut points and hash state): - Apple M3 Pro, Apple clang 17 -O3: blockwise 2450 -> 2950 MB/s of scanned data, i.e. +24% over the shipped aarch64 default, the NEON kernel (2370) - Zen 4, gcc 14 -O2 (Debian flags): blockwise 766 -> 1707 MB/s (2.2x); the x86-64 default stays the sequential loop (2390) So the aarch64 default becomes blockwise: the NEON kernel already lost to the shipped blockwise kernel there (the file's own note said so while bz64_kernel_default() still picked neon), and it loses more now. It stays selectable via BORG_BUZHASH64_KERNEL=neon. The default tables in the tests are now per chunker family, and the env var docs say which kernel each family defaults to on aarch64. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…sh64 loops The sequential loops are the x86-64 default kernels and the reference for all others, and their speed is their loop-carried dependency chain. gcc emits the shortest one, clang does not: - fastcdc: gcc folds `fp = (fp << 1) + gear[b]` into one lea (table value + fp * 2), a 1-cycle chain; clang emits add + add-from-memory, 2 cycles. - buzhash64: gcc keeps `sum = rotl(sum, 1) ^ Trot[out] ^ T[in]` at rotate + one xor (2 cycles) by combining the table values off the chain; clang re-associates both xors into the chain, 3 cycles. Loading the table value(s) into a variable and pinning it with an empty asm statement makes clang emit gcc's form as well (verified on clang 17 and 22 targeting x86-64: 28 lea-with-index*2 instead of 7 in the object, the memory-operand adds gone). The pin is applied for clang only: gcc emits the wanted instructions by itself, and the asm statement makes its code placement flag-dependent - with Debian's python flags the pinned Gear loop ran at 1.07 cycles/byte on a Zen 4 like the unpinned one, with the `-O2 -fPIC -fno-strict-overflow` of a pip build at 2.0, with the very same loop instructions. That matters for every clang-built borg on x86-64 - Intel Macs, Homebrew, the macOS binaries - where the scalar loop is the default kernel and was running 2x (fastcdc) resp. 1.5x (buzhash64) below its bound. Measured on an Apple M3 Pro (clang 17, forced scalar kernel): buzhash64 1210 -> 1780 MB/s of scanned data; fastcdc is unchanged there (the update is one shifted add on aarch64 either way). Cut points and hash state are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gcc at -O2 - Debian's python build flags, so what `pip install` gets on Debian and Ubuntu - does not inline bz64_block_prefix() into the blockwise and NEON scan loops on aarch64 (nor bz64_block_delta() into the AVX2/AVX-512 loops on x86-64): one call per 8-byte block, and the 64 bytes of results travel through the stack, which the NEON kernel then reloads as two 16-byte vectors that cannot be store-forwarded. clang and gcc -O3 inline them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10307 +/- ##
==========================================
+ Coverage 87.56% 87.68% +0.11%
==========================================
Files 103 103
Lines 18686 18712 +26
Branches 2875 2880 +5
==========================================
+ Hits 16362 16407 +45
+ Misses 1622 1603 -19
Partials 702 702 ☔ View full report in Codecov by Harness. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small, bit-identical codegen fixes for the fastcdc / buzhash64 scan kernels, found while auditing the kernels against their dependency-chain and issue bounds (llvm-mca plus measurements on an Apple M3 Pro and a Zen 4 with perf counters). One commit per change.
hit |= ...accumulate loop is lowered by clang into a chain of conditional increments and by gcc >= 12 at-O2into vectorised stores + wide reloads that fail store-to-load forwarding on every block (the "blockwise 8x slower" pathology from borg2: benchmarking needed #10160). As a||chain it compiles to eight test-and-branch pairs. M3 Pro: 2450 -> 2950 MB/s scanned, +24% over the shipped NEON default (2370); Zen 4 gcc-O2: blockwise 766 -> 1707 MB/s (x86-64 keeps the sequential loop as default, 2390). aarch64 default becomes blockwise; NEON stays selectable. borg-level on the M3 Pro: buzhash64 2193 -> 2835 MB/s.lea (base,index,2)instead of 7 in the object). Applied for clang only: gcc emits the wanted code by itself, and with the pin gcc's code placement became flag-sensitive (1.07 vs 2.0 c/b on Zen 4 with identical loop instructions, depending on-fPIC/-fno-strict-overflow). Matters for clang-built borg on x86-64 (Intel Macs, Homebrew, the macOS binaries), where the scalar loop is the default kernel. M3 Pro forced-scalar buzhash64: 1210 -> 1780 MB/s.-O2(Debian's python flags) does not inlinebz64_block_prefixon aarch64 (norbz64_block_deltaon x86-64): one call per 8-byte block plus a stack round trip that the NEON kernel reloads as non-forwardable 16-byte loads.Verification: chunker test suite on macOS (Apple clang 17) and on Debian 13 / Zen 4 (gcc 14); every kernel bit-identical to the scalar reference in a C harness on PRNG and adversarial data; paired kernel-level A/B on both machines for the pins (no change on gcc / aarch64, as intended).
🤖 Generated with Claude Code