AES chunkers: unroll the x86-64 aes-ni and vaes scan paths, branch-free lane test - #10309
Merged
ThomasWaldmann merged 2 commits intoSep 2, 2026
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10309 +/- ##
=======================================
Coverage 87.69% 87.70%
=======================================
Files 103 103
Lines 18712 18717 +5
Branches 2880 2881 +1
=======================================
+ Hits 16410 16415 +5
Misses 1600 1600
Partials 702 702 ☔ View full report in Codecov by Harness. |
ThomasWaldmann
force-pushed
the
aes-chunkers-x86-unroll
branch
from
September 2, 2026 09:29
e1a1c56 to
aedb123
Compare
The 128-bit hardware path (aes-ni, i.e. every x86-64 CPU without VAES) looped over the 9 AES rounds and over the 8 ciphertexts to test. gcc at -O2 - Debian's python flags - keeps both loops rolled: 12 instructions per round instead of 8, and two branches per tested position; clang serialises the early-exit test loop into 8 dependent 40-cycle AES chains. Both cost about a fifth of the path's time on a Zen 4. Spell the rounds out by macro and test all 8 lanes at once in the vector domain: pack the low ciphertext qwords pairwise, mask, compare with zero (pcmpeqq, SSE4.1 - every CPU with AES-NI has it), or-reduce, and only a hit enters the lane search. phte_hw_available() now checks SSE4.1 too. Zen 4, gcc 14 -O2, borg benchmark cpu with BORG_AES_CHUNKER_KERNEL=aes-ni: toeplitz-aes 735 -> 960 MB/s, rabin-aes 720 -> 870, goldilocks-aes 472 -> 505. Cut points are unchanged (the hw/vaes/evp equivalence tests pass). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gcc at -O2 - Debian's python flags - keeps the VAES/AVX-512 path's two loops rolled: the 15 stride-2 roll steps run with an index chain and the 9 AES rounds over 8 zmm vectors come with 72 zmm register copies per group. Spelled out by macro (like the aes-ni path now), the group loop is straight-line code. Zen 4, gcc 14 -O2, borg benchmark cpu with BORG_AES_CHUNKER_KERNEL=vaes: toeplitz-aes 871 -> 1000 MB/s, rabin-aes 879 -> ~1010, goldilocks-aes 489 -> ~555. At the kernel level toeplitz-aes goes from 7.06 to 6.40 cycles per byte. Cut points are unchanged (the hw/vaes/evp equivalence tests pass). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ThomasWaldmann
force-pushed
the
aes-chunkers-x86-unroll
branch
from
September 2, 2026 09:31
aedb123 to
9d831da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two bit-identical speedups for the x86-64 hardware paths of the toeplitz-aes / rabin-aes / goldilocks-aes chunkers, one commit each, from the chunker kernel audit (perf counters on a Zen 4, gcc 14, Debian's
-O2python flags):-O2keeps them rolled (12 instructions per round, two branches per tested position), clang serialises the early-exit test loop into 8 dependent AES chains. Now the rounds are spelled out by macro and the 8 lanes are tested branch-free in the vector domain (unpcklqdq+ mask +pcmpeqq+ or-reduce; SSE4.1 is now part of the availability check, every AES-NI CPU has it).borg benchmark cpuwithBORG_AES_CHUNKER_KERNEL=aes-ni: toeplitz-aes 735 -> 960 MB/s, rabin-aes 720 -> 870, goldilocks-aes 472 -> 505.-O2: an index chain plus 72 zmm register copies per group). Unrolled by macro: toeplitz-aes 871 -> 1000 MB/s, rabin-aes 879 -> ~1010, goldilocks-aes 489 -> ~555.The aarch64 path is untouched. Verified with the chunker test suite (which checks that evp, aes-ni and vaes produce identical cut points) on the Zen 4 box and on macOS, and with the audit's C harness (bit-identity on PRNG and adversarial data, perf-cycle measurements).
Not included: the software-pipelined vaes variant from the audit (a further -10% for toeplitz/goldilocks but codegen-sensitive) and the arm64 instruction diet (+7-8%, needs inline asm).
🤖 Generated with Claude Code