From 08d172021910d6c2878c4943740f1837a38d9d75 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Wed, 2 Sep 2026 01:50:12 +0200 Subject: [PATCH 1/3] buzhash64: short-circuit hit test in the blockwise kernel, make it the aarch64 default The blockwise kernel tested its 8 lanes with a loop accumulating `hit |= ((c ^ s[k]) & M[k]) == 0`. clang lowers that into a chain of conditional increments; gcc >= 12 at -O2 vectorises it and reloads the eight 8-byte stores of s[] as 16-byte vectors, which fails store-to-load forwarding on every block - the "blockwise 8x slower than scalar" seen on Zen CPUs in #10160. Written as a short-circuit chain it compiles to eight test-and-branch pairs that leave the loop at the first hit. Measured at the kernel level (bit-identical cut points and hash state): - Apple M3 Pro, Apple clang 17 -O3: blockwise 2450 -> 2950 MB/s of scanned data, i.e. +24% over the shipped aarch64 default, the NEON kernel (2370) - Zen 4, gcc 14 -O2 (Debian flags): blockwise 766 -> 1707 MB/s (2.2x); the x86-64 default stays the sequential loop (2390) So the aarch64 default becomes blockwise: the NEON kernel already lost to the shipped blockwise kernel there (the file's own note said so while bz64_kernel_default() still picked neon), and it loses more now. It stays selectable via BORG_BUZHASH64_KERNEL=neon. The default tables in the tests are now per chunker family, and the env var docs say which kernel each family defaults to on aarch64. Co-Authored-By: Claude Fable 5.1 --- docs/changes.rst | 2 + docs/usage/general/environment.rst.inc | 10 ++-- src/borg/archiver/help_cmd.py | 10 ++-- src/borg/chunkers/buzhash64_impl.c | 60 +++++++++++---------- src/borg/testsuite/chunkers/fastcdc_test.py | 7 ++- 5 files changed, 51 insertions(+), 38 deletions(-) diff --git a/docs/changes.rst b/docs/changes.rst index 8af6211e2f..5eca75cc50 100644 --- a/docs/changes.rst +++ b/docs/changes.rst @@ -216,6 +216,8 @@ Other changes: - setuptools-scm: drop tag.strict, #10193 - diff: show timestamp changes with full nanosecond precision, #9147 - benchmark cpu: also benchmark zstd,-4 +- buzhash64: faster blockwise kernel (the hit test as a short-circuit chain) and make + it the aarch64 default, +24% over the NEON kernel on an Apple M3 Pro - archive: resolve the item metadata stream chunk ids lazily, big win for repo-list on remote repos, #10204 - lock exceptions: tell who holds the lock, #2261 diff --git a/docs/usage/general/environment.rst.inc b/docs/usage/general/environment.rst.inc index 3cd3912a67..d2390f8c66 100644 --- a/docs/usage/general/environment.rst.inc +++ b/docs/usage/general/environment.rst.inc @@ -246,10 +246,12 @@ General: BORG_FASTCDC_KERNEL / BORG_BUZHASH64_KERNEL Select the scan kernel the ``fastcdc`` / ``buzhash64`` chunker uses. Accepted values are ``avx512``, ``avx2``, ``neon``, ``blockwise`` and ``scalar``. - The default is whichever benchmarked fastest for the architecture: ``neon`` on - aarch64, and ``scalar`` (the plain sequential loop) on x86-64, where the compiler - folds the rolling hash update into a single instruction and thereby beats the vector - kernels. Other architectures get ``blockwise``, the portable multi-lane C kernel. + The default is whichever benchmarked fastest for the architecture: ``scalar`` (the + plain sequential loop) on x86-64, where the compiler folds the rolling hash update + into one or two instructions and thereby beats the vector kernels; on aarch64 + ``neon`` for ``fastcdc`` and ``blockwise`` for ``buzhash64``, whose NEON kernel + loses to the portable multi-lane C kernel there. Other architectures get + ``blockwise``. All kernels chunk identically - same cut points, same chunk ids - and differ only in speed, so this is safe to change at any time, also for an existing repository. Which kernel is fastest is not predictable from the instruction set: it depends on the diff --git a/src/borg/archiver/help_cmd.py b/src/borg/archiver/help_cmd.py index 8063e37457..0b4c766d08 100644 --- a/src/borg/archiver/help_cmd.py +++ b/src/borg/archiver/help_cmd.py @@ -840,10 +840,12 @@ class HelpMixIn: BORG_FASTCDC_KERNEL / BORG_BUZHASH64_KERNEL Select the scan kernel the ``fastcdc`` / ``buzhash64`` chunker uses. Accepted values are ``avx512``, ``avx2``, ``neon``, ``blockwise`` and ``scalar``. - The default is whichever benchmarked fastest for the architecture: ``neon`` on - aarch64, and ``scalar`` (the plain sequential loop) on x86-64, where the compiler - folds the rolling hash update into a single instruction and thereby beats the vector - kernels. Other architectures get ``blockwise``, the portable multi-lane C kernel. + The default is whichever benchmarked fastest for the architecture: ``scalar`` (the + plain sequential loop) on x86-64, where the compiler folds the rolling hash update + into one or two instructions and thereby beats the vector kernels; on aarch64 + ``neon`` for ``fastcdc`` and ``blockwise`` for ``buzhash64``, whose NEON kernel + loses to the portable multi-lane C kernel there. Other architectures get + ``blockwise``. All kernels chunk identically - same cut points, same chunk ids - and differ only in speed, so this is safe to change at any time, also for an existing repository. Which kernel is fastest is not predictable from the instruction set: it depends on the diff --git a/src/borg/chunkers/buzhash64_impl.c b/src/borg/chunkers/buzhash64_impl.c index 1d82c46053..06eaafb077 100644 --- a/src/borg/chunkers/buzhash64_impl.c +++ b/src/borg/chunkers/buzhash64_impl.c @@ -21,9 +21,10 @@ * Trot[b] = ROTL(T[b], window_size % 64) is precomputed by the caller, which * also removes one rotate per byte from the sequential path. * - * Kernel dispatch: AVX-512 or AVX2 on x86-64 (runtime-detected), blockwise - * scalar everywhere else, including aarch64 by default - its NEON kernel is - * selectable by name but not auto-selected, see the note above it. + * Which kernel runs is decided per platform by bz64_kernel_default(): the + * sequential loop on x86-64, blockwise everywhere else, including aarch64, + * where the NEON kernel is selectable by name but not auto-selected (see the + * note above it); BORG_BUZHASH64_KERNEL overrides that. * All kernels return bit-identical results. */ #include @@ -82,7 +83,7 @@ static inline void bz64_block_prefix(const uint64_t *T, const uint64_t *Trot, s[7] = s8; } -/* --- blockwise scalar (the default kernel) --------------------------------- */ +/* --- blockwise scalar (the default kernel, except on x86-64) --------------- */ static size_t bz64_scan_blockwise(const uint64_t *T, const uint64_t *Trot, const uint8_t *pr, const uint8_t *pa, @@ -97,10 +98,16 @@ static size_t bz64_scan_blockwise(const uint64_t *T, const uint64_t *Trot, while (j + 8 <= n && (sum & mask) != 0) { bz64_block_prefix(T, Trot, pr + j, pa + j, s); uint64_t c = BZ_ROTL(sum, 8); - uint64_t hit = 0; - for (int k = 0; k < 8; k++) - hit |= (((c ^ s[k]) & M[k]) == 0); - if (hit) { + /* The hit test is a short-circuit chain on purpose: it compiles to + * eight test-and-branch pairs that leave the loop as soon as a lane + * hits. Written as a loop accumulating `hit |= ...`, clang lowers it + * into a chain of conditional increments (M3 Pro: 20% slower) and + * gcc >= 12 at -O2 vectorises it, reloading the eight 8-byte stores + * of s[] as 16-byte vectors - which fails store-to-load forwarding + * on every block (Zen 4: 2.2x slower). */ + if (((c ^ s[0]) & M[0]) == 0 || ((c ^ s[1]) & M[1]) == 0 || ((c ^ s[2]) & M[2]) == 0 || + ((c ^ s[3]) & M[3]) == 0 || ((c ^ s[4]) & M[4]) == 0 || ((c ^ s[5]) & M[5]) == 0 || + ((c ^ s[6]) & M[6]) == 0 || ((c ^ s[7]) & M[7]) == 0) { size_t r = bz64_scan_seq(T, Trot, pr + j, pa + j, 8, &sum, mask); /* exact re-scan */ *sum_io = sum; return j + r; @@ -116,28 +123,26 @@ static size_t bz64_scan_blockwise(const uint64_t *T, const uint64_t *Trot, /* --- NEON (aarch64) ------------------------------------------------------ * - * Nothing selects this by default (the default is the sequential kernel - * everywhere); BORG_BUZHASH64_KERNEL=neon selects it. On an Apple M3 Pro it - * loses to the blockwise kernel - 2590 vs 2360 MB/s, the same ~9% gap at - * every mask size from 17 to 23 bits - so it is not the one to reach for - * there. + * Nothing selects this by default; BORG_BUZHASH64_KERNEL=neon does. On an + * Apple M3 Pro it loses to the blockwise kernel: 2370 vs 2950 MB/s of + * scanned data at the kernel level (Apple clang 17, -O3, same at every mask + * size from 19 to 23 bits). * * Why it loses there: the 16 table lookups per block have to happen in * general registers (NEON has no gather), so the vector form only ADDS the * move to the SIMD side plus a cross-lane reduce (umaxv) before the loop - * branch can resolve - all it saves is the 8-lane test, three cheap ops per - * lane, which Apple's very wide scalar ALUs retire at more than one lane per - * cycle anyway. fastcdc keeps NEON as its default because its per-lane test + * branch can resolve - all it saves is the 8-lane test, which as a + * short-circuit chain of test-and-branch pairs is cheap on a wide core and + * exits early. fastcdc keeps NEON as its default because its per-lane test * work is larger (add plus per-lane shifted masks), enough to pay for the * trip; there it wins by 2x. * * It is kept because that reasoning is about core width, and the measurement * comes from the widest scalar ARM core there is. Neoverse (Graviton, * Ampere), Cortex-A7x and friends are 3-4 wide on the scalar side with - * comparatively healthy NEON, which is exactly where this should get - * competitive - on this machine's much narrower E-cores the 9% gap already - * collapses into measurement noise. If you have such hardware, compare - * BORG_BUZHASH64_KERNEL=neon against =blockwise and please report. */ + * comparatively healthy NEON, which is where this might get competitive. If + * you have such hardware, compare BORG_BUZHASH64_KERNEL=neon against + * =blockwise and please report. */ #if defined(__aarch64__) #define BZ_KIND "neon" @@ -409,16 +414,15 @@ int bz64_kernel_select(const char *name, int *out_id) int bz64_kernel_default(void) { -#if defined(__aarch64__) - /* Same picture as for fastcdc: NEON is baseline on aarch64 and wins, so - * this only falls back if the build lacks it. */ - int kid; - if (bz64_kernel_select("neon", &kid) == BZ_KSEL_OK) - return kid; - return BZ_K_BLOCKWISE; -#elif defined(__x86_64__) || defined(_M_X64) +#if defined(__x86_64__) || defined(_M_X64) + /* The compiler keeps the sequential loop's update at rol + xor, a 2-cycle + * chain that the block kernels' 16 table lookups per 8 bytes do not beat + * (Zen 4, gcc -O2: scalar 2.1 cycles/byte, blockwise 2.95, avx2/avx512 + * 3.9 - see the double-buffer note above). */ return BZ_K_SCALAR; #else + /* Everywhere else, including aarch64: unlike fastcdc's, this NEON kernel + * loses to the blockwise one (see the note above it), so blockwise it is. */ return BZ_K_BLOCKWISE; #endif } diff --git a/src/borg/testsuite/chunkers/fastcdc_test.py b/src/borg/testsuite/chunkers/fastcdc_test.py index 7352e2a47f..d99d5a6424 100644 --- a/src/borg/testsuite/chunkers/fastcdc_test.py +++ b/src/borg/testsuite/chunkers/fastcdc_test.py @@ -226,7 +226,8 @@ def test_fastcdc_portable_kernel_available(kernel, monkeypatch): # first: the first entry this build and CPU can actually run is the default. # Mirrors fc_kernel_default() / bz64_kernel_default() (fastcdc_impl.c, # buzhash64_impl.c) and phte_kernel_default() (phte_core.h). -ROLLING_HASH_DEFAULTS = {"x86_64": ["scalar"], "aarch64": ["neon", "blockwise"]} +FASTCDC_DEFAULTS = {"x86_64": ["scalar"], "aarch64": ["neon", "blockwise"]} +BUZHASH64_DEFAULTS = {"x86_64": ["scalar"], "aarch64": ["blockwise"]} AES_DEFAULTS = {"x86_64": ["vaes", "aes-ni", "evp"], "aarch64": ["aes-arm64", "evp"]} @@ -255,8 +256,10 @@ def expected_default_kernel(envvar, make, key, monkeypatch): machine = default_kernel_arch() if envvar == "BORG_AES_CHUNKER_KERNEL": preference = AES_DEFAULTS.get(machine, ["evp"]) + elif envvar == "BORG_BUZHASH64_KERNEL": + preference = BUZHASH64_DEFAULTS.get(machine, ["blockwise"]) else: - preference = ROLLING_HASH_DEFAULTS.get(machine, ["blockwise"]) + preference = FASTCDC_DEFAULTS.get(machine, ["blockwise"]) for kernel in preference: monkeypatch.setenv(envvar, kernel) try: From f8f59a2a447deaf89f0810ba5f331f750359f797 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Wed, 2 Sep 2026 02:04:00 +0200 Subject: [PATCH 2/3] chunkers: pin the table value in clang's sequential fastcdc and buzhash64 loops The sequential loops are the x86-64 default kernels and the reference for all others, and their speed is their loop-carried dependency chain. gcc emits the shortest one, clang does not: - fastcdc: gcc folds `fp = (fp << 1) + gear[b]` into one lea (table value + fp * 2), a 1-cycle chain; clang emits add + add-from-memory, 2 cycles. - buzhash64: gcc keeps `sum = rotl(sum, 1) ^ Trot[out] ^ T[in]` at rotate + one xor (2 cycles) by combining the table values off the chain; clang re-associates both xors into the chain, 3 cycles. Loading the table value(s) into a variable and pinning it with an empty asm statement makes clang emit gcc's form as well (verified on clang 17 and 22 targeting x86-64: 28 lea-with-index*2 instead of 7 in the object, the memory-operand adds gone). The pin is applied for clang only: gcc emits the wanted instructions by itself, and the asm statement makes its code placement flag-dependent - with Debian's python flags the pinned Gear loop ran at 1.07 cycles/byte on a Zen 4 like the unpinned one, with the `-O2 -fPIC -fno-strict-overflow` of a pip build at 2.0, with the very same loop instructions. That matters for every clang-built borg on x86-64 - Intel Macs, Homebrew, the macOS binaries - where the scalar loop is the default kernel and was running 2x (fastcdc) resp. 1.5x (buzhash64) below its bound. Measured on an Apple M3 Pro (clang 17, forced scalar kernel): buzhash64 1210 -> 1780 MB/s of scanned data; fastcdc is unchanged there (the update is one shifted add on aarch64 either way). Cut points and hash state are unchanged. Co-Authored-By: Claude Fable 5.1 --- src/borg/chunkers/buzhash64_impl.c | 19 ++++++++++++++++++- src/borg/chunkers/fastcdc_impl.c | 19 ++++++++++++++++++- 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/src/borg/chunkers/buzhash64_impl.c b/src/borg/chunkers/buzhash64_impl.c index 06eaafb077..b842d0cd8f 100644 --- a/src/borg/chunkers/buzhash64_impl.c +++ b/src/borg/chunkers/buzhash64_impl.c @@ -34,6 +34,16 @@ #define BZ_ROTL(x, k) (((x) << ((k) & 63)) | ((x) >> ((64 - (k)) & 63))) +/* Pin a value: the compiler must materialize it here and cannot re-associate + * expressions across it (an empty asm statement that "modifies" it). Only + * clang needs it (see bz64_scan_seq); gcc emits the wanted form by itself and + * is left alone, an asm statement can move its code around. */ +#if defined(__clang__) +#define BZ_PIN(v) __asm__("" : "+r"(v)) +#else +#define BZ_PIN(v) ((void)0) +#endif + /* --- sequential reference loop (also: block re-scan and tail) ----------- */ static size_t bz64_scan_seq(const uint64_t *T, const uint64_t *Trot, @@ -43,7 +53,14 @@ static size_t bz64_scan_seq(const uint64_t *T, const uint64_t *Trot, uint64_t sum = *sum_io; size_t j = 0; while (j < n && (sum & mask) != 0) { - sum = BZ_ROTL(sum, 1) ^ Trot[pr[j]] ^ T[pa[j]]; + /* The two table values are combined off the chain and the result is + * pinned, so that the loop-carried dependency is exactly rotate + one + * xor (2 cycles per byte). Unpinned, clang re-associates the xors into + * the chain and makes it 3 cycles per byte - M3 Pro: 1210 vs 1780 MB/s + * of scanned data; gcc emits the 2-cycle form either way. */ + uint64_t d = Trot[pr[j]] ^ T[pa[j]]; + BZ_PIN(d); + sum = BZ_ROTL(sum, 1) ^ d; j++; } *sum_io = sum; diff --git a/src/borg/chunkers/fastcdc_impl.c b/src/borg/chunkers/fastcdc_impl.c index a93eef1616..16931778c3 100644 --- a/src/borg/chunkers/fastcdc_impl.c +++ b/src/borg/chunkers/fastcdc_impl.c @@ -30,13 +30,30 @@ #include "fastcdc_impl.h" +/* Pin a value: the compiler must materialize it here and cannot re-associate + * expressions across it (an empty asm statement that "modifies" it). Only + * clang needs it (see fc_scan_seq); gcc emits the wanted form by itself and + * is left alone, an asm statement can move its code around. */ +#if defined(__clang__) +#define FC_PIN(v) __asm__("" : "+r"(v)) +#else +#define FC_PIN(v) ((void)0) +#endif + /* --- sequential reference loop (also: block recheck and tail) ----------- */ static int64_t fc_scan_seq(const uint64_t *gear, const uint8_t *p, size_t n, uint64_t *fp_io, uint64_t mask) { uint64_t fp = *fp_io; for (size_t i = 0; i < n; i++) { - fp = (fp << 1) + gear[p[i]]; + /* The table value is pinned so that the update stays one instruction + * on the loop-carried chain: on x86-64 a lea (table value + fp * 2), + * 1 cycle per byte, which gcc emits anyway but clang only with the + * pin (unpinned it emits add + add-from-memory, 2 cycles per byte); + * on aarch64 it is one shifted add either way. */ + uint64_t g = gear[p[i]]; + FC_PIN(g); + fp = (fp << 1) + g; if ((fp & mask) == 0) { *fp_io = fp; return (int64_t)i; From d7a9cf8c40f02dc3e1169e1f61806cbae59bf879 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Wed, 2 Sep 2026 02:04:04 +0200 Subject: [PATCH 3/3] buzhash64: always inline the per-block table lookup helpers gcc at -O2 - Debian's python build flags, so what `pip install` gets on Debian and Ubuntu - does not inline bz64_block_prefix() into the blockwise and NEON scan loops on aarch64 (nor bz64_block_delta() into the AVX2/AVX-512 loops on x86-64): one call per 8-byte block, and the 64 bytes of results travel through the stack, which the NEON kernel then reloads as two 16-byte vectors that cannot be store-forwarded. clang and gcc -O3 inline them. Co-Authored-By: Claude Fable 5.1 --- src/borg/chunkers/buzhash64_impl.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/borg/chunkers/buzhash64_impl.c b/src/borg/chunkers/buzhash64_impl.c index b842d0cd8f..d9ffba6884 100644 --- a/src/borg/chunkers/buzhash64_impl.c +++ b/src/borg/chunkers/buzhash64_impl.c @@ -44,6 +44,15 @@ #define BZ_PIN(v) ((void)0) #endif +/* The per-block table lookup helpers must be inlined into the scan loops: + * gcc at -O2 (Debian's python flags) leaves them as calls otherwise, one per + * 8-byte block, and passes their 64-byte result through the stack. */ +#if defined(__GNUC__) +#define BZ_ALWAYS_INLINE static inline __attribute__((always_inline)) +#else +#define BZ_ALWAYS_INLINE static inline +#endif + /* --- sequential reference loop (also: block re-scan and tail) ----------- */ static size_t bz64_scan_seq(const uint64_t *T, const uint64_t *Trot, @@ -70,7 +79,7 @@ static size_t bz64_scan_seq(const uint64_t *T, const uint64_t *Trot, /* Load the block's 8 out/in byte pairs (via two 8-byte data loads) and * compute the aligned-domain prefix XORs s[0..7] with a depth-3 tree. * Endianness-independent: bytes are extracted by shifting. */ -static inline void bz64_block_prefix(const uint64_t *T, const uint64_t *Trot, +BZ_ALWAYS_INLINE void bz64_block_prefix(const uint64_t *T, const uint64_t *Trot, const uint8_t *pr, const uint8_t *pa, uint64_t s[8]) { uint64_t wr, wa; @@ -212,7 +221,7 @@ static size_t bz64_scan_simd(const uint64_t *T, const uint64_t *Trot, /* Load the block's 8 per-byte deltas D_t = Trot[out_t] ^ T[in_t] (via two * 8-byte data loads), without the rotations and the prefix XOR: the vector * kernels do those in the vector domain. Endianness-independent. */ -static inline void bz64_block_delta(const uint64_t *T, const uint64_t *Trot, +BZ_ALWAYS_INLINE void bz64_block_delta(const uint64_t *T, const uint64_t *Trot, const uint8_t *pr, const uint8_t *pa, uint64_t d[8]) { uint64_t wr, wa;