[WIP][POC][BenchmarkingOnly] PFOR interleaved and fastlanes numbers - #51296
Draft
prtkgaur wants to merge 111 commits into
Draft
[WIP][POC][BenchmarkingOnly] PFOR interleaved and fastlanes numbers#51296prtkgaur wants to merge 111 commits into
prtkgaur wants to merge 111 commits into
Conversation
Implements the PFOR (Patched Frame of Reference) integer compression algorithm as a standalone utility library in arrow/util/pfor/. Includes: - Cost model for optimal bit width selection (histogram-based) - Vector-level encode/decode with FOR + bit-packing + exceptions - Page-level wrapper with header, offset array, and multi-vector layout - Comprehensive unit tests covering edge cases and round-trips
Adds PFOR = 11 to the Encoding enum and wires it into the parquet read/write pipeline: - PforEncoder<DType> in encoder.cc (buffers values, calls PforWrapper::Encode) - PforDecoder<DType> in decoder.cc (decodes all values on first access) - PFOR case in column_reader.cc InitializeDataDecoder - Encoding string mapping in types.cc Supports INT32 and INT64 column types.
Benchmarks encode/decode throughput for int32/int64 across 10 data distributions inspired by Snowflake's NumericComprBenchmark: constant, sequential, small range, high-base-small-range (timestamps), with outliers (exception path), random, TPC-DS date/store/item/quantity keys. Each distribution runs at 1K/10K/100K/1M elements. Reports bytes/s, items/s, and compression ratio.
Load() now returns Result<PforVectorInfo> after the Status/Result refactoring. Use ASSERT_OK_AND_ASSIGN to properly unwrap the result in tests.
Make LoadHeader fallible: move the header-size check from Decode into LoadHeader, return Result<PforHeader>, and update Decode to use ARROW_ASSIGN_OR_RAISE. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Replace std::memcpy / raw byte writes in PforWrapper::StoreHeader, LoadHeader, and the offset-array read/write paths with util::SafeLoadAs and util::SafeStore. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Reject invalid packing_mode, value_byte_width mismatch, log_vector_size out of [kMin, kMax] range, and negative num_elements when loading the PFOR page header. Removes the redundant packing_mode and value_byte_width checks from Decode now that they live in LoadHeader. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…sites Replace size_t with int64_t for max_size/comp_size to match the PforWrapper API signature, and qualify pfor::PforWrapper as ::arrow::util::pfor::PforWrapper to avoid ADL ambiguity.
Aligns with Arrow buffer conventions (Buffer::data() returns uint8_t*). Removes the reinterpret_cast<char*> at the parquet encoder/decoder call sites and switches std::vector<char> compressed buffers to std::vector<uint8_t> in the unit test and benchmark. Also fixes a pre-existing size_t / int64_t* mismatch in pfor_benchmark.cc that surfaced once the buffer pointer type was tightened. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…th validation Per Google C++ style, replace the PforVectorInfo struct with a class that has private trailing-underscore members and getter/setter accessors. Replace std::memcpy calls in Store/Load and the exception patch loop in DecodeVector with util::SafeLoadAs / util::SafeStore. Add bit_width range validation inside Load() so callers don't have to repeat the check. Updates all access sites in pfor.cc and pfor_test.cc to go through the new accessors. Caches num_exceptions() in a local in DecodeVector so the #pragma GCC unroll can still see a constant loop bound. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Per Google C++ style, both types become classes with private trailing-underscore members and const getters, mutable getters, and setters. Updates all access sites in pfor.cc (EncodeVector, LoadView, SerializedVectorSize, SerializeVector) and pfor_test.cc to go through the new accessors. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…, use ctor in EncodeVector - Move the num_exceptions < 0 check from DecodeVector into PforVectorInfo::Load alongside the bit_width range check, so all loaded-data invariants are enforced at the same layer. - Use PforVectorInfo's parameterized constructor in EncodeVector instead of three separate setter calls on a default-constructed instance.
Commit 00b6318 introduced ARROW_DCHECK(bit_util::IsPowerOf2(vector_size)) in PforWrapper<T>::Encode, but vector_size is int32_t and bit_util has overloads only for int64_t and uint64_t -- the call is ambiguous and the file no longer compiles. Cast to int64_t to disambiguate. CeilDiv calls in the same file already promote to int64_t implicitly via its int64_t-only signature.
Portable C++ port of FastLanes (Afroozeh & Boncz, VLDB '23) for int32_t
columnar data. No SIMD intrinsics in the kernels — the inner lane loop
is structured (contiguous loads from packed[w*kLanes + lane], contiguous
stores to transposed[r*kLanes + lane]) so the compiler auto-vectorizes
to 4-wide NEON / 8-wide AVX2 / 16-wide AVX512 without source changes.
Layout: lane-interleaved 1024-bit format per the paper. 1024 values
pack as w u32 rows of 32 u32 lanes. FL_ORDER (8x16 -> 16x8 sub-block
transpose + 3-bit-reversal sub-block reorder) is applied OUTSIDE the
kernel: FastLanesForCodec::Encode gathers input[fromTransposed32(t)]
before packing; Decode produces output in transposed order (no scatter,
output[t] == input[fromTransposed32(t)] + min within each 1024-block).
FastLanesForCodec adds Frame-of-Reference on top:
- 2048-value chunks (2 FastLanes blocks per chunk)
- Per-chunk 5-byte header: [min(4B int32 LE)] [bit_width(1B)]
- Subtract min before packing; add back on decode
- bit_width=0 path stores no payload (constant chunk)
Files:
cpp/src/arrow/util/fastlanes/fastlanes_kernels.h
- PackBlock<W>(in, out) / UnpackBlock<W>(packed, out)
- W=32 fast path: std::memcpy
- fromTransposed32 helper
cpp/src/arrow/util/fastlanes/fastlanes_for.{h,cc}
- FastLanesForCodec::{Encode,Decode}
cpp/src/arrow/util/fastlanes/fastlanes_for_test.cc
- 5 round-trip tests (narrow range, single value, full int32 range,
multiple chunks, boundary values) — all passing
CMakeLists.txt wires the test as arrow-fastlanes-for-test.
Wires the new FastLanesForCodec into the existing pfor_comparison_benchmark harness alongside PFOR, DeltaBitPack, ZSTD, LZ4, RleBitPack, and Bss codecs. New BM_FastLanesEncode / BM_FastLanesDecode functions follow the same Gen32 + ::Apply(CustomArgs) shape; REGISTER_DATASET macro picks them up for every ClickBench dataset. Notes on the comparison: - FastLanes decoder produces output in TRANSPOSED order (output[chunk*2048 + block*1024 + t] == input[chunk*2048 + block*1024 + fromTransposed32(t)] + min). PFOR/DeltaBitPack produce flat output. The benchmark measures decoder throughput head-to-head; consumers of FastLanes output must be permutation-aware (which is the FastLanes paper's intended architecture). - num_values is rounded down to a multiple of 2048 (FastLanes chunk size) inside BM_FastLanesEncode / BM_FastLanesDecode for compatibility with the existing 102400-value test sizes. Also guards add_executable(parquet-pfor-comparison-benchmark) with if(ARROW_BUILD_BENCHMARKS) so non-benchmark configurations don't fail the cmake configure step. Bench numbers on aarch64 (102400 int32, 3-run median): EventDate decode: FastLanes 20us vs PFOR 36us vs Delta 122us EventTime decode: FastLanes 24us vs PFOR 56us vs Delta 140us GoodEvent decode: FastLanes 20us vs PFOR 34us vs Delta 119us Compression ratios match or slightly beat PFOR on every dataset tested.
FastLanesForCodec::DecodeFlat unpacks into a transposed scratch buffer
per chunk and then scatters via fromTransposed32 to produce output in
original input order — output[i] == input[i] for the encoded input.
This is the FL_ORDER inverse of the gather step in Encode.
Adds:
- DecodeFlat method + round-trip test (DecodeFlatIsIdentity) covering
4 chunks of random data. All 6 round-trip tests still pass.
- BM_FastLanesDecodeFlat in pfor_comparison_benchmark, registered in
the per-dataset macro for apples-to-apples vs PFOR / DeltaBitPack
(both of which produce flat output).
Bench (102400 int32, 3-run median, aarch64):
Dataset FL Decode FL DecodeFlat PFOR Decode Delta Decode
EventDate 20 us 108 us 38 us 123 us
EventTime 23 us 113 us 57 us 140 us
GoodEvent 20 us 107 us 35 us 119 us
The transposed-kernel decode beats every other codec by 1.5-7x. The
flat-output decode pays an ~85 us scatter cost per 100K values that
makes it slower than PFOR but still faster than DeltaBitPack. The gap
is exactly the FL_ORDER scatter — the reason FastLanes' intended
architecture keeps data in transposed order through the query.
The 8x16 -> 16x8 within-sub-block transpose is mutual-inverse with the 16x8 -> 8x16 transpose, NOT self-inverse. The previous docstring on fromTransposed32 said "Self-inverse: fromTransposed32 is also toTransposed32" — that was wrong. fromTransposed32(fromTransposed32(t)) does not equal t in general; e.g. fromTransposed32(1) = 16, fromTransposed32(16) = 2. Add the actual toTransposed32 (forward-direction mapping) and fix the docstring. Callers that need to invert a gather computed with fromTransposed32 (i.e. read out[i] = transposed["the t whose fromTransposed32(t) = i"]) must use toTransposed32(i).
Adds an additive packing-mode option to PFOR. Existing vectors round-trip
unchanged (default PackingMode::BitPack); new vectors can opt in to the
FastLanes lane-interleaved bit-packing layout via the per-vector flag.
On-disk format change (backwards-compatible):
- The 1-byte bit_width field of PforVectorInfo now packs two values:
bits 0..5 = the actual bit width (range 0..32 fits in 6 bits)
bit 7 = packing-mode flag (0 = BitPack, 1 = FastLanes)
bit 6 = reserved
- Legacy encoders only wrote the bit width, leaving high bits clear,
so they decode as PackingMode::BitPack via the new Load.
- PFOR header (page-level) is unchanged.
API:
- New enum class arrow::util::pfor::PackingMode { BitPack, FastLanes }.
- PforVectorInfo gains a packing_mode field and getter/setter.
- PforCompression<T>::EncodeVector takes an optional PackingMode (default
BitPack). FastLanes mode is only honored when num_elements equals the
FastLanes block size (1024) and T is 32-bit; otherwise it falls back
to BitPack per-vector (so tails and 64-bit values continue to work).
- PforCompression<T>::DecodeVector reads the per-vector flag and
dispatches between arrow::internal::unpack and the FastLanes kernel.
- PforWrapper<T>::Encode takes an optional PackingMode threaded down to
EncodeVector.
Decode-side perf (fused gather + FOR-add + SafeCopy):
The FL_ORDER inverse needs toTransposed32(i) — note: NOT
fromTransposed32(i), the two are mutual inverses, not self-inverse.
The scalar gather over can't be SIMD-vectorized, so
PFOR+FastLanes decode is ~2-3x slower than PFOR+BitPack end-to-end
despite the kernel itself being competitive. The win is only available
when the downstream consumer can work with data in FastLanes transposed
order (i.e. relax the flat-output contract).
Tests: 5 new tests in PforPackingModeTest cover round-trip identity for
both modes, the partial-tail fallback to BitPack, mixed-mode round-trip
through PforWrapper, and the bit_width=0 (constant vector) path. All 30
PFOR tests pass.
Benchmark: BM_PforFastLanesEncode / BM_PforFastLanesDecode added to
pfor_comparison_benchmark.cc, registered per dataset alongside the
existing 8 codec variants.
For FastLanes-encoded vectors the decoder previously always paid a
1024-element scalar FL_ORDER gather to produce flat output. That gather
is what made pfor+fastlanes 2-3x slower than pfor+bitpack overall, even
though the FastLanes unpack kernel itself is competitive.
The FastLanes paper's intended decode path is to NOT do that scatter at
all: keep the data in FastLanes stream order and let downstream
operators be permutation-aware (apply fromTransposed32 lazily, when
they need original index). This commit exposes that path.
API:
- New enum class arrow::util::pfor::OutputOrder { Flat, Transposed }.
- PforCompression<T>::DecodeVector and PforWrapper<T>::Decode take an
optional OutputOrder (default Flat, backwards-compatible).
- OutputOrder::Transposed only affects FastLanes-encoded vectors.
BitPack vectors have no permutation to skip, so they always produce
flat output regardless of the argument (mixed pages with a BitPack
tail end up flat in the tail, transposed in the full blocks).
Decoder paths in DecodeVector when packing_mode == FastLanes:
- Flat (existing): unpack -> scratch transposed[] -> fused
values[i] = SafeCopy(transposed[toTransposed32(i)] + FOR)
The toTransposed32 gather is scalar, breaks auto-vec.
- Transposed (new): unpack -> scratch transposed[] -> sequential
values[t] = SafeCopy(transposed[t] + FOR)
Pure sequential read/write, auto-vectorizes cleanly. Exceptions are
patched at toTransposed32(pos) so the stored-flat positions land in
the right transposed slots.
Tests: 4 new tests in PforOutputOrderTest cover (a) transposed output
satisfies the FL_ORDER relation, (b) manual inversion of the
permutation reconstructs the input, (c) BitPack vectors ignore the
Transposed request, (d) wrapper-level transposed decode across many
vectors. All 34 PFOR tests pass.
Benchmark: BM_PforFastLanesDecodeTransposed added, registered per
dataset. On 18 ClickBench-style datasets (102400 int32 each):
pfor+bitpack 33-57 us
pfor+fastlanes (flat) 98-108 us (0.34-0.53x — slower)
pfor+fastlanes (transp) 20-23 us (1.6-2.5x faster than bitpack)
The transposed path beats every other codec measured in the comparison
benchmark on every dataset.
Every bit width round-trips, framed and unframed, and the framed cases go through the unpack that folds the frame into its own store. The widths are built from a delta mask rather than a range so the cost model cannot land on a narrower width and leave the loop testing one kernel thirty-one times; the test reads back the width the encoder settled on and checks it. One test exists only to keep the others honest. Because the two layouts write the same number of bytes, a mode request that quietly fell back to sequential on both the encode and the decode side would pass every round trip above. So one test asserts the payloads really are different bytes at widths 1 through 31 -- and identical at 32, where the two layouts coincide and the kernel is a memcpy.
The layout is not in the Parquet specification, so a file written with it can only be read by an implementation that knows it. It is therefore off unless a writer asks: WriterProperties::Builder::enable_pfor_interleaved_bit_packing. A reader that does not know the layout rejects the page rather than misreading it, because the mode is recorded in a header byte the reader already validates. The flag is a request. It reaches only 4-byte columns, and within those only the full vectors of a page, so a file whose columns are not all eligible can still be written with one setting: everything the layout cannot cover is written the way PFOR has always written it. MakeEncoder grows a defaulted parameter rather than taking the properties, which would put the whole of the writer's configuration behind a factory that needs one bit of it.
Because both layouts write the same number of bytes, a flag that never reached the encoder would still round-trip a table. So the test writes the same table with the flag and without it and compares the two files: same length, different bytes is what shows the flag arrived. The int64 case asserts the opposite -- the two files are identical, byte for byte -- because the layout cannot apply there and the request is dropped rather than refused.
The benchmark only ever encoded with the default packing mode, so nothing in the tree measured the layout this branch adds. Thread the mode through the two benchmark bodies and register a paired int32 arm for each of the ten distributions. The encode arms report CompRatio%, which comes out identical to the sequential arm's for every distribution: the two layouts write the same number of bits, so the layout is a decode-speed choice and not a size trade-off. There is no int64 arm because only 32-bit values reach the interleaved kernels.
Every existing size is an exact multiple of the 1024-value vector, so the tail path that falls back to sequential packing was never timed. The Parquet writer caps a data page at max_rows_per_page = 20,000 rows, which for int32 always binds before the 1 MB size target, so 20,000 values is the destination a reader actually decodes into. Add it. It also has a partial last vector (19 x 1024 + 544), which is what makes it worth measuring separately from 10240 and 102400.
DecodeVector left UnpackOptions::max_read_bytes at its -1 default, which tells the bit-unpacker it may not read a byte past the vector's own packed payload. The vector kernels load a fixed-size window per step, wider than a step consumes at most bit widths, so under that bound the last step is refused and the tail of the vector falls to the scalar epilog -- 32 of 1024 values at bit width 3. The span DecodeVector is handed runs from this vector to the end of the page, so the true bound is already in hand; pass it. Whole-page int32 decode gains a median 1.13x over a 29-column corpus and 1.9x on the columns that pack to 3 bits, and nothing at widths 1, 2, 4, 8, 16 and 31, whose kernels strand no values to begin with. The gain is flat from a 78 KiB data page to a 4 MB destination, which is what a per-vector cost looks like.
The property told a writer the layout "decodes faster", which the measurements do not support without a qualifier: on whole-page int32 decode it is about 1.08x at -O3, a tie at the -O2 an Arrow Release build compiles with, and slower than the sequential layout once the destination reaches a few MB. Its kernels are plain loops that depend on the autovectorizer, which is where the level sensitivity comes from. The byte-count half of the claim stands -- the two layouts pack the same number of bytes, which InterleavedCostsNoSpace pins.
lane_delta.h gives lane l the container rows l, 32+l, ..., so its in-lane predecessor is 32 positions back in file order and its differences are wider than DELTA_BINARY_PACKED's. The paper assigns lane l the contiguous run [32l, 32l+32) instead, which makes the in-lane predecessor the immediately preceding value and the stored differences exactly the format's, at the price of 32 entry points per block and a transpose back to file order. transposed_delta.h implements that, with the bases stored either raw or delta-encoded across lanes, and the benchmark adds four decode arms so the base stream and the transpose can be priced separately from the container.
The paper's lane assignment finishes a block in transposed order, and a Parquet decoder's contract is positional, so it owes a 32x32 permutation. Run as a separate pass that permutation reads 1.14x even in NEON, which made the ordering look like a trade against the container. It is not one. A lane's prefix sum does not depend on any other lane, so four lanes' chains run in one register and four rows of results can be transposed and stored while they are still in registers. Fused that way the permutation costs 0.995x against an arm that skips it and memcpys the block out -- the same speed -- because it also spares the block the second 4 KB traversal the memcpy arm pays. The base stream is now the whole remaining gap, so make its bit reader branchless: one unaligned 64-bit load per base covers any width up to 32 at any bit offset, in place of a data-dependent shift loop. The size bound gains 8 bytes of slack for the last load's over-read. Conforming decode over 33 columns, -O2: 11.13 GB/s with a separate transpose, 11.81 fused, 12.28 fused and branchless, against 12.17 for the non-conforming arm and 13.41 for the +32 stride.
The layout had only ever been called as a kernel, from a benchmark that handed it a payload and a value count. That skips the page header, the decoder's dispatch and the level handling, so dividing a kernel rate by a full decoder's rate overstates what a reader would gain. Give it page framing and register it as an encoding, so the same encoder and decoder interfaces every other encoding is reached through also reach this one. The framing adds a little-endian value count ahead of the payload: the payload's own header does not carry one, decode needs it to locate every section, and a page's level count cannot stand in because it counts nulls too. INT32 only, because the packing kernel works on 32-bit lanes. The payload still stores its packed words and per-block minimums in host byte order, which a format specification would have to pin down before a file written this way could be read anywhere else.
The existing arm measures the kernel. Add one that goes through the registered decoder, so it pays for page framing, for SetData and for reading the value count out of the page, and add the matching encode arm. Both are then comparable with the DELTA_BINARY_PACKED encoder and decoder in the same binary, which is the comparison a margin should be quoted over: decoder against decoder rather than kernel against decoder.
The encoding is registered for INT32, so the fuzzer reaches it with pages it has mutated. Four kinds of damage now raise instead of reading past the buffer: a page shorter than the value count it declares, a page too short to hold a count at all, a declared count larger than the page header's level count allows, and a block claiming a bit width no 32-bit lane can hold.
The 32 entry points of a block are bit-packed, and reading them with one unaligned 64-bit load per value reads up to 7 bytes past the stream. That was covered by slack in the kernel's own size bound, but a page is sized to its contents: when the last block's differences are all equal its payload is empty, the entry points end the page, and the load runs off the buffer. Staging the stream -- at most 128 bytes, once per 1024 values -- keeps the loop free of both a bounds test and short loads, and costs 3% of decode. Splitting the loop instead, so that only the last values take a short load, costs 8%: the split needs a division to find where to stop, and the fast half loses its compile-time trip count. A sanitizer harness reports an 8-byte read one byte past a 478-byte page against the old code and nothing against this one, and a sweep of all 33 widths against a reference unpack agrees value for value.
The kernel encodes a whole array and is told its length; a page has to say how long it is, and a decoder has to be able to tell a truncated or mutated page from a valid one before it reads a byte of payload. The framing is a 4-byte value count followed by the kernel's output, and validation walks the blocks: each block's payload width and entry-point width are bounded, and every block's contribution to the size is added up and checked against the page. Decode reads in place when the payload is 4-byte aligned, which it is for a page written by this encoder, and copies only when a caller hands it something else.
Both lane assignments break the format's single 1024-long dependency chain into 32 independent ones, and both therefore decode with one vector add per row. They differ in what a lane's predecessor is, and that decides the size: striding 32 positions apart stores differences the format never would, which costs 2.1-3.4x the bytes on the three correlated columns of the benchmark corpus. Holding a contiguous run of 32 values per lane stores exactly the format's differences, and lands within 1.5% of DELTA_BINARY_PACKED's size across the corpus. One encoding cannot mean both, so the encoding writes the second and the stride stays a kernel-level comparison. Measured against the DELTA_BINARY_PACKED decoder in the same binary, over 33 columns at -O2: decode 2.9x faster on 33 of 33, encode 2.6x on 33 of 33. The page framing costs under 2% of decode and no bytes. The corruption test moves with the layout: block widths now open the payload rather than sitting 128 bytes in, and the entry-point width lives inside a block, so there are two positions to corrupt instead of one. Two tests cover what the arrangement adds -- a page whose last block carries no payload, so the entry points end it, and entry points far enough apart to need the full width.
Compares the portable sequential bit-unpacker (new, arm 1), the in-tree interleaved kernel (arm 2), and Arrow's dispatched unpacker (arm 3, called through its real header against the built library) on instructions/value and IPC per bit width, via self-process perf_event_open counters with correctness gates on every arm. Arm 1 and arm 3 are fixed, prebuilt inputs; only arm 2 recompiles between the driver's -O2 and -O3 builds. Not wired into CMake: perf_event_open is Linux-only and needs counter access the build system can't guarantee everywhere. Build commands are documented in driver.cc's header comment.
InterleavedBitPackingLayout and InterleavedRequestIgnoredForInt64 built WriterProperties directly and never called enable_pfor_encoding(), so both threw the preview-feature guard instead of exercising the layout.
|
Thanks for opening a pull request! This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format. If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or After updating the title, you can mark the pull request as ready for review. See also: |
github-actions
Bot
added
Component: Parquet
Component: C++
awaiting review
prtkgaur
changed the title
Sweeps ARROW_USER_SIMD_LEVEL across SSE4_2/AVX2/AVX512 on a single native build of parquet-pfor-comparison-benchmark, with identical repetitions at every width, and combines the three runs into one tarball. Meant to be handed to someone with x86 hardware so the sequential-vs-transposed decode ratio can be measured at each register width from one benchmark invocation.
Arrow's shipped PFOR decoder, plain PFOR through the FastLanes interleaved container in file order, and the same container with the paper's lane assignment now all run against the existing TPC-DS, ClickBench, TPC-H, NYC taxi, and synthetic-shape column corpus in one benchmark binary, so the layout question is measured on data instead of on a synthetic per-bit-width sweep. The lane-assignment arm exists to price the gather it forces, not to recommend it -- plain PFOR has no dependency chain for that ordering to help.
The decode arm this benchmark used as the sequential baseline runs the PFOR encoder with its default options, which let the planner difference a vector whenever its cost model prefers that. On a sorted or correlated column it does, and the resulting decode also walks a serial prefix sum. Ratios taken against a layout arm that never deltas were therefore pricing two decisions at once: on the delta-shaped columns the shipped decoder spends 3.3x-4.3x of its own plain-mode time on the prefix sum alone, which is large enough to dominate anything the layout contributes. The encoder already exposes both knobs the comparison needs -- a flag to decline delta and an advisory packing mode, one of whose values is the lane-interleaved container. So add two arms that hold delta off and vary only the layout, both going through the production encoder and decoder rather than a benchmark-local reimplementation. Their payloads come out byte-identical on all 43 int32 columns, which is the check that the layout is the only difference between them. The sweep script picks up both arms and its header now says which pair to quote for the layout question.
bpacking_simd_avx512.cc is the one bit-unpacking translation unit never
migrated off the legacy generated kernels in
bpacking_simd512_generated_internal.h. Those build their SIMD input register
from an initializer list of individual scalar loads and carry no
ARROW_FORCE_INLINE, so each step is an out-of-line call. The width-12 uint32
body is 128 instructions per 32 values, of which six are actual unpack math
(4x vpsrlvd, 2x vpandd); 32 are element-by-element register assembly via
vmovd/vpinsrd and 55 are scalar bit-splicing in GPRs. It references xmm 44
times and zmm five times.
Measured on Granite Rapids over 102400 values via exported
unpack_bias<uint32_t>, forcing the target with ARROW_USER_SIMD_LEVEL so only
the kernel source varies (geomean widths 1..31, GiB/s):
scalar 9.74
sse4_2 41.38
avx2 40.97
avx512 6.54 0.16x of avx2, 0.67x of scalar
Being slower than the scalar kernel is the sanity check that says bug rather
than tradeoff. Width 32 is the control the bug cannot reach (memcpy path): all
four targets agree at 46.5-46.8.
Because ARROW_RUNTIME_SIMD_LEVEL defaults to MAX, every AVX-512 machine
preferred this kernel. aarch64 has no AVX-512 and was never affected, which
accounts for PFOR decoding at ~30 GB/s on ARM against ~6 GiB/s on x86.
With the cap, default dispatch goes from 6.60 to 50.74 GiB/s at width 12 and
is bit-identical to the scalar kernel across all 32 widths and both entry
points.
This is a shared-library dispatch policy change and wants an upstream
decision, so it is kept as its own commit. Re-enable once the 512-bit kernels
issue real vector loads. Recorded in the comment so nobody repeats it: naively
pointing the AVX-512 TU at the Kernel<> machinery used by
bpacking_simd_{128,256}.cc compiles and is bit-exact but measures 1.17 GiB/s,
5.6x worse again, because most widths land on is_oversized() -> NoOpKernel and
fall through to the naive path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FL_ORDER places a contiguous run of 32 values in each SIMD lane so a delta
chain splits into 32 independent 32-long chains. Parquet cannot hand that
order to a reader -- definition/repetition level association, cross-column row
alignment, row-range predicate pushdown and sub-range reads all depend on
value order, and the first of those breaks silently -- so file order has to be
restored on every decode. This makes that restoration cheap.
The naive implementation unpacks into a 4 KiB stack grid and runs a separate
Transpose32x32 over it, materializing the grid purely to read it straight
back: 128 extra 32-byte stores and 128 extra loads per block on top of the 128
stores the output needs. Folding the permutation into the kernel's store
addressing instead is a trap and is documented as such -- out[lane * 32 + row]
puts the 32 lanes of a row 128 bytes apart, turning each contiguous 32-byte
store into eight 4-byte scatters, 1024 stores per block instead of 128.
What works is doing the transpose in registers on values that never reach
memory in grid form. UnpackBlockFlToFileOrder fills the same 8x8 register
block Transpose32x32Avx2 already used, directly from the unpack, and runs the
same unpacklo/unpackhi/permute2x128 ladder. Per-block load and store counts
are then identical to the file-order kernel's; the only thing FL_ORDER pays
over file order is the shuffle ladder. lb is the outer loop deliberately: a
fixed lb writes one contiguous 1 KiB run, whereas rb outer would stride 128
bytes across the whole block.
The same fusion is applied to the delta path, where unpack, prefix sum and
permute now happen in one pass. Measured over 43 real columns at 400 KiB
(GiB/s), the order-restoration ladder:
kSeparate unpack -> grid -> prefix sum -> transpose out 21.6
kFused prefix sum and permute in one pass, reads grid 21.57
kFusedUnpack no grid at all 31.30
Notably the order-agnostic arm that skips repair entirely but still writes and
re-reads a grid measures 16.05 -- slower than doing the full permutation
without one. The memory round trip costs more than the permutation.
Against Arrow's DELTA_BINARY_PACKED at 3.77 GiB/s that is 8.31x. Holding the
layout fixed and measuring what delta costs over frame-only on the same
columns: sequential 11.24x (one 1024-long chain), interleaved file order
2.33x, FL_ORDER 1.55x.
For plain PFOR, where there is no chain to break, FL_ORDER has nothing on the
benefit side and still owes the transpose; the transpose costs 43% at L1, 10%
at L2 and 3% at DRAM. kFlOrderRaw is kept as a harness self-check: it runs the
same PackBlock/UnpackBlock as kFileOrder against a grid merely filled
differently at encode time, so the two must decode at equal speed, and they
agree within 0.2%.
pfor_comparison_benchmark.cc gains the arms these numbers come from.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scripts that produced the numbers in the two preceding commits, so the
measurements are reproducible rather than asserted.
run_width_matrix.sh, run_width_matrix_v2.sh, time_width_matrix_v2.sh
sweep bit widths 1..32 across register-width targets (-march=nehalem /
haswell / skylake-avx512). This is the matrix that showed two of three
width tiers selecting a kernel worse than scalar: at SSE4.2 the "SIMD"
unpacker measures 3.31 GiB/s against its own scalar kernel's 13.18.
run_zmm_points.sh
forces ARROW_USER_SIMD_LEVEL per target on one binary, which is how the
AVX-512 dispatch numbers in "Cap the bit-unpack SIMD dispatch at 256
bits" were isolated to kernel source rather than to the call site.
build_transpose_ab.sh, run_transpose_ab.sh, ab_compare.sh
A/B the order-restoration ladder (kSeparate / kFused / kFusedUnpack)
within one binary, arms alternating inside each repetition so cache
warmth and output address cannot separate them.
wait_quiet.sh
blocks until the box is quiet enough to time on; run_transpose_ab.sh
depends on it. Checks three conditions rather than loadavg alone,
because loadavg is a lagging 1-minute average and a benchmark that has
just started shows a low load while already owning a core.
Build directories are deliberately left untracked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ARM handoff instructions referenced build.sh, fl5_corpus and seq_granularity, none of which were ever committed -- they lived outside any git repository, so the instructions could not be executed. This adds the sources. build.sh now locates the checkout from its own path, so only ARROW_BUILD has to be set, and it preflights the three headers it needs instead of failing with a wall of compiler errors. seq_granularity is the calibration probe, not an optional extra: calling Arrow's exported unpack_bias once per 1024-value block rather than once per buffer costs 1.27x at L1 and L2 on x86, a handicap the seq_simd arm pays and the header-inlined interleaved arms do not. Without dividing it out, intlv/seq_simd reads 1.43x at L2 where the layout-only effect is ~1.13x. The divisor is toolchain-specific and has to be measured locally. The README also records a correction to the working-set ladder. The point labelled DRAM is not DRAM: the reference machine is a Xeon 6975P-C with 2 MiB of L2 per core and 480 MiB of shared L3, so a 32 MiB working set is L3-resident and that row measures L3 bandwidth. Additional points, and a concurrency sweep, are still to come. Binaries are gitignored; x86 reference outputs are checked in so the aarch64 run has something to compare against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
Please remove this line and the above text before creating your pull request.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)