fastcdc: short-circuit candidate test in the blockwise kernel - #10310
Merged
ThomasWaldmann merged 4 commits intoSep 2, 2026
Conversation
…e aarch64 default The blockwise kernel tested its 8 lanes with a loop accumulating `hit |= ((c ^ s[k]) & M[k]) == 0`. clang lowers that into a chain of conditional increments; gcc >= 12 at -O2 vectorises it and reloads the eight 8-byte stores of s[] as 16-byte vectors, which fails store-to-load forwarding on every block - the "blockwise 8x slower than scalar" seen on Zen CPUs in borgbackup#10160. Written as a short-circuit chain it compiles to eight test-and-branch pairs that leave the loop at the first hit. Measured at the kernel level (bit-identical cut points and hash state): - Apple M3 Pro, Apple clang 17 -O3: blockwise 2450 -> 2950 MB/s of scanned data, i.e. +24% over the shipped aarch64 default, the NEON kernel (2370) - Zen 4, gcc 14 -O2 (Debian flags): blockwise 766 -> 1707 MB/s (2.2x); the x86-64 default stays the sequential loop (2390) So the aarch64 default becomes blockwise: the NEON kernel already lost to the shipped blockwise kernel there (the file's own note said so while bz64_kernel_default() still picked neon), and it loses more now. It stays selectable via BORG_BUZHASH64_KERNEL=neon. The default tables in the tests are now per chunker family, and the env var docs say which kernel each family defaults to on aarch64. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…sh64 loops The sequential loops are the x86-64 default kernels and the reference for all others, and their speed is their loop-carried dependency chain. gcc emits the shortest one, clang does not: - fastcdc: gcc folds `fp = (fp << 1) + gear[b]` into one lea (table value + fp * 2), a 1-cycle chain; clang emits add + add-from-memory, 2 cycles. - buzhash64: gcc keeps `sum = rotl(sum, 1) ^ Trot[out] ^ T[in]` at rotate + one xor (2 cycles) by combining the table values off the chain; clang re-associates both xors into the chain, 3 cycles. Loading the table value(s) into a variable and pinning it with an empty asm statement makes clang emit gcc's form as well (verified on clang 17 and 22 targeting x86-64: 28 lea-with-index*2 instead of 7 in the object, the memory-operand adds gone). The pin is applied for clang only: gcc emits the wanted instructions by itself, and the asm statement makes its code placement flag-dependent - with Debian's python flags the pinned Gear loop ran at 1.07 cycles/byte on a Zen 4 like the unpinned one, with the `-O2 -fPIC -fno-strict-overflow` of a pip build at 2.0, with the very same loop instructions. That matters for every clang-built borg on x86-64 - Intel Macs, Homebrew, the macOS binaries - where the scalar loop is the default kernel and was running 2x (fastcdc) resp. 1.5x (buzhash64) below its bound. Measured on an Apple M3 Pro (clang 17, forced scalar kernel): buzhash64 1210 -> 1780 MB/s of scanned data; fastcdc is unchanged there (the update is one shifted add on aarch64 either way). Cut points and hash state are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gcc at -O2 - Debian's python build flags, so what `pip install` gets on Debian and Ubuntu - does not inline bz64_block_prefix() into the blockwise and NEON scan loops on aarch64 (nor bz64_block_delta() into the AVX2/AVX-512 loops on x86-64): one call per 8-byte block, and the 64 bytes of results travel through the stack, which the NEON kernel then reloads as two 16-byte vectors that cannot be store-forwarded. clang and gcc -O3 inline them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Same change as for buzhash64: the loop accumulating `cand |= ...` over the 8 lanes is vectorised by gcc >= 12 at -O2, which reloads the eight 8-byte stores of s[] as 16-byte vectors and fails store-to-load forwarding on every block - the "blockwise 8x slower than scalar" seen on Zen CPUs in borgbackup#10160. As a short-circuit chain it compiles to eight test-and-branch pairs that leave the loop at the first candidate. Measured at the kernel level (bit-identical cut points and hash state): - Zen 4, gcc 14 -O2 (Debian flags): 6.04 -> 1.92 cycles per byte, 834 -> 2607 MB/s of scanned data (the x86-64 default stays the sequential loop at 1.07) - Apple M3 Pro, Apple clang 17 -O3: 1.91 -> 1.09 cycles per byte, 2081 -> 3640 MB/s (the aarch64 default stays NEON at 0.93) So the portable kernel - the default on every platform that is neither x86-64 nor aarch64, and what BORG_FASTCDC_KERNEL=blockwise selects - is now within 15-20% of the fastest kernel instead of 2-3x behind. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10310 +/- ##
==========================================
+ Coverage 87.56% 87.69% +0.13%
==========================================
Files 103 103
Lines 18686 18712 +26
Branches 2875 2880 +5
==========================================
+ Hits 16362 16410 +48
+ Misses 1622 1600 -22
Partials 702 702 ☔ View full report in Codecov by Harness. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #10307 (its three commits are included; only the last commit is new). Merge #10307 first, this one then rebases to a single commit.
The fastcdc blockwise kernel's candidate test was a loop accumulating
cand |= ...over the 8 lanes. gcc >= 12 at-O2vectorises it, reloading the eight 8-byte stores ofs[]as 16-byte vectors, which fails store-to-load forwarding on every block - the "blockwise 8x slower than scalar" seen on Zen CPUs in #10160. As a short-circuit||chain it compiles to eight test-and-branch pairs that leave the loop at the first candidate. Same change as #10307 makes for buzhash64.Kernel-level, bit-identical (cut points and hash state, PRNG plus adversarial data):
-O2(Debian flags): 6.04 -> 1.92 cycles/byte, 834 -> 2607 MB/s of scanned data;borg benchmark cpuwithBORG_FASTCDC_KERNEL=blockwise: 821 -> 2550 MB/s-O3: 1.91 -> 1.09 cycles/byte, 2081 -> 3640 MB/sThe defaults are unchanged (scalar on x86-64 at 1.07 c/b, NEON on aarch64 at 0.93 c/b); the portable kernel is the default everywhere else and is now within 15-20% of the fastest kernel instead of 2-3x behind.
I also tried the "named scalars instead of the s[]/M[] arrays" rewrite for buzhash64 from the audit: +35% on Zen 4 with gcc, but -20% to -50% on Apple clang (where blockwise is now the default), so it is not included.
🤖 Generated with Claude Code