Skip to content

AES chunkers: unroll the x86-64 aes-ni and vaes scan paths, branch-free lane test - #10309

Merged
ThomasWaldmann merged 2 commits into
borgbackup:masterfrom
ThomasWaldmann:aes-chunkers-x86-unroll
Sep 2, 2026
Merged

AES chunkers: unroll the x86-64 aes-ni and vaes scan paths, branch-free lane test#10309
ThomasWaldmann merged 2 commits into
borgbackup:masterfrom
ThomasWaldmann:aes-chunkers-x86-unroll

Conversation

@ThomasWaldmann

Copy link
Copy Markdown
Member

Two bit-identical speedups for the x86-64 hardware paths of the toeplitz-aes / rabin-aes / goldilocks-aes chunkers, one commit each, from the chunker kernel audit (perf counters on a Zen 4, gcc 14, Debian's -O2 python flags):

  1. aes-ni path (every x86-64 CPU without VAES): the AES rounds and the 8-way lane test were loops. gcc -O2 keeps them rolled (12 instructions per round, two branches per tested position), clang serialises the early-exit test loop into 8 dependent AES chains. Now the rounds are spelled out by macro and the 8 lanes are tested branch-free in the vector domain (unpcklqdq + mask + pcmpeqq + or-reduce; SSE4.1 is now part of the availability check, every AES-NI CPU has it). borg benchmark cpu with BORG_AES_CHUNKER_KERNEL=aes-ni: toeplitz-aes 735 -> 960 MB/s, rabin-aes 720 -> 870, goldilocks-aes 472 -> 505.
  2. vaes path (the x86-64 default where available): the 15 stride-2 roll steps and the 9 AES rounds over 8 zmm vectors were loops too (gcc -O2: an index chain plus 72 zmm register copies per group). Unrolled by macro: toeplitz-aes 871 -> 1000 MB/s, rabin-aes 879 -> ~1010, goldilocks-aes 489 -> ~555.

The aarch64 path is untouched. Verified with the chunker test suite (which checks that evp, aes-ni and vaes produce identical cut points) on the Zen 4 box and on macOS, and with the audit's C harness (bit-identity on PRNG and adversarial data, perf-cycle measurements).

Not included: the software-pipelined vaes variant from the audit (a further -10% for toeplitz/goldilocks but codegen-sensitive) and the arm64 instruction diet (+7-8%, needs inline asm).

🤖 Generated with Claude Code

@codecov

codecov Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.70%. Comparing base (26fcb6d) to head (9d831da).
⚠️ Report is 18 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #10309   +/-   ##
=======================================
  Coverage   87.69%   87.70%           
=======================================
  Files         103      103           
  Lines       18712    18717    +5     
  Branches     2880     2881    +1     
=======================================
+ Hits        16410    16415    +5     
  Misses       1600     1600           
  Partials      702      702           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann
ThomasWaldmann force-pushed the aes-chunkers-x86-unroll branch from e1a1c56 to aedb123 Compare September 2, 2026 09:29
ThomasWaldmann and others added 2 commits September 2, 2026 11:30
The 128-bit hardware path (aes-ni, i.e. every x86-64 CPU without VAES)
looped over the 9 AES rounds and over the 8 ciphertexts to test. gcc at
-O2 - Debian's python flags - keeps both loops rolled: 12 instructions per
round instead of 8, and two branches per tested position; clang serialises
the early-exit test loop into 8 dependent 40-cycle AES chains. Both cost
about a fifth of the path's time on a Zen 4.

Spell the rounds out by macro and test all 8 lanes at once in the vector
domain: pack the low ciphertext qwords pairwise, mask, compare with zero
(pcmpeqq, SSE4.1 - every CPU with AES-NI has it), or-reduce, and only a
hit enters the lane search. phte_hw_available() now checks SSE4.1 too.

Zen 4, gcc 14 -O2, borg benchmark cpu with BORG_AES_CHUNKER_KERNEL=aes-ni:
toeplitz-aes 735 -> 960 MB/s, rabin-aes 720 -> 870, goldilocks-aes
472 -> 505. Cut points are unchanged (the hw/vaes/evp equivalence tests
pass).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gcc at -O2 - Debian's python flags - keeps the VAES/AVX-512 path's two
loops rolled: the 15 stride-2 roll steps run with an index chain and the
9 AES rounds over 8 zmm vectors come with 72 zmm register copies per
group. Spelled out by macro (like the aes-ni path now), the group loop is
straight-line code.

Zen 4, gcc 14 -O2, borg benchmark cpu with BORG_AES_CHUNKER_KERNEL=vaes:
toeplitz-aes 871 -> 1000 MB/s, rabin-aes 879 -> ~1010, goldilocks-aes
489 -> ~555. At the kernel level toeplitz-aes goes from 7.06 to 6.40
cycles per byte. Cut points are unchanged (the hw/vaes/evp equivalence
tests pass).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ThomasWaldmann
ThomasWaldmann force-pushed the aes-chunkers-x86-unroll branch from aedb123 to 9d831da Compare September 2, 2026 09:31
@ThomasWaldmann
ThomasWaldmann merged commit c2ce9e9 into borgbackup:master Sep 2, 2026
25 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the aes-chunkers-x86-unroll branch September 2, 2026 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant