Skip to content

Fix 8-bit fixed right shifts and rotations of signed integers - #1421

Open
Arthur031221 wants to merge 1 commit into
xtensor-stack:masterfrom
Arthur031221:fix-8bit-rshift-signed-rotations
Open

Arthur031221 wants to merge 1 commit into
xtensor-stack:masterfrom
Arthur031221:fix-8bit-rshift-signed-rotations

Conversation

@Arthur031221

Copy link
Copy Markdown

Anyone calling bitwise_rshift<N> on batch<uint8_t> or batch<int8_t>, or rotl/rotr on a signed batch or scalar, got wrong lane values.

On batch<uint8_t> with -msse2 or -mavx2, bitwise_rshift<1> of a batch filled with 0x81 returned 0x00 instead of 0x40 in every lane, and bitwise_rshift<7> returned 3 instead of 1 on SSE2. The mask applied after the 16-bit shift was (1 << N) - 1, the mask for a left shift, instead of 0xFF >> N. On AVX2 bitwise_rshift<0> of a uint8_t batch returned zeros. The signed 8-bit version on SSE2 (and the AVX512VL 128-bit kernels that inherit it) also returned wrong values for some lanes (I checked every 8-bit input against the scalar result).

rotl and rotr were written as (x << n) | (x >> (bits - n)) on the signed type, so the right shift is arithmetic and a negative value gets sign bits instead of the bits shifted out. For int32_t batches on SSE2, rotl(INT_MIN, 1) returned -1 instead of 1. Native AVX512 rotate instructions were already right for 32 and 64 bit lanes, but the 8 and 16 bit lanes and the AVX/AVX2/SSE paths all went through that formula. rotl<0> and rotr<0> did not compile because of the static_assert on bits - count.

The fix:

  • use 0xFF >> N as the byte mask in the SSE2 and AVX2 unsigned kernels,
  • for signed 8-bit on SSE2, shift, mask, then sign-extend with (x ^ m) - m,
  • compute rotations on the unsigned type in the common kernels and in xsimd_scalar.hpp, and let a count of 0 return the input.

The two fixes are in one change because the compile-time rotl<N>/rotr<N> of an 8-bit batch are built on bitwise_rshift<N>, so the 8-bit rotation tests only pass with both. The scalar rotl/rotr now reduce the count modulo the bit width, which replaces the undefined shift by the full width that a count of 0 used to produce.

Tests: test_xsimd_api.cpp gets one case that compares bitwise_rshift<0..7> of every 8-bit value against the scalar result, and one that rotates the sign bit through rotl/rotr (runtime and compile-time counts, batch and scalar types). The existing rotation tests only used the value 12, and compared against the same formula the implementation used.

Results on an AVX-512 machine with g++, test_xsimd_api, built three times (-march=native, -mavx2, -msse2): 165 test cases pass in each. With the source changes reverted and the tests kept, 11, 12 and 12 test cases fail respectively. test_batch_int, test_bit and test_batch_manip still pass with -mavx2. Only x86 was run; NEON has its own fixed shift kernels and the other backends reach the common kernels, which I checked by reading only.

The unsigned 8-bit bitwise_rshift<N> on SSE2 and AVX2 masked with the
left shift mask (1 << N) - 1 instead of 0xFF >> N, the signed 8-bit
version on SSE2 built a wrong sign mask, and the AVX2 version returned
zero for N == 0. Rotations were written with arithmetic right shifts, so
rotl/rotr of a negative value filled the vacated bits with the sign.
Rotate through the unsigned type in the common kernels and in the scalar
overloads, and accept a rotation count of 0.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant