Test build perf v2 by mjp41 · Pull Request #835 · microsoft/snmalloc

mjp41 · 2026-03-28T22:24:31Z

No description provided.

SchrodingerZhu · 2026-03-28T22:26:14Z

We can also try modules if speed is a concern.

mjp41 · 2026-03-28T22:27:30Z

We can also try modules if speed is a concern.

Mostly there is a lot of redundant work in the build, and I wanted to experiment with improving that.

Pure file move. allocconfig.h contains only compile-time constants with dependencies only on bits.h and mitigations.h (both in ds_core/). Moving it enables sizeclassstatic.h (next step) to live in ds_core/ without pulling in ds/. Changes: - git mv ds/allocconfig.h -> ds_core/allocconfig.h - Add #include bits.h and mitigations.h to moved file - Update include paths in ds/ds.h, ds/mpmcstack.h - Add allocconfig.h to ds_core/ds_core.h umbrella

Create ds_core/sizeclassstatic.h with compile-time sizeclass functions extracted from mem/sizeclasstable.h: - smallsizeclass_t (size_t typedef) - size_to_sizeclass_const() - NUM_SMALL_SIZECLASSES - sizeclass_to_size_const() - is_small_sizeclass() Move sizeclasstable.h from mem/ to ds/ (depends on PAL, not allocator). Remove redundant sizeclasstable.h includes from mem/ files.

Convert smallsizeclass_t from a size_t typedef to a struct with: - Explicit construction from size_t (no implicit conversion FROM) - Implicit conversion to size_t (for array indexing, comparisons) - Pre/post increment operators Provides type safety: alloc(size_t) (byte size) cannot be confused with alloc(smallsizeclass_t) (sizeclass index). Update all iteration sites to use explicit construction.

Add alloc(smallsizeclass_t) overload to corealloc and globalalloc that skips the dynamic sizeclass lookup. Refactor small_alloc to accept (smallsizeclass_t, size_t) and keep the single-arg (size_t) version as a forwarding overload. Update alloc<size>() to use compile-time sizeclass when the size is a known small sizeclass. Add libc::malloc_small(smallsizeclass_t) and libc::malloc_small_zero(smallsizeclass_t) as non-template entry points.

These are general-purpose data structures, not allocator-specific. Moving them reduces the dependency surface of mem/.

Add __malloc_start_pointer() and __malloc_last_byte_pointer() to libc.h, alongside the existing __malloc_end_pointer(). These provide the external_pointer functionality through the libc API surface.

Add snmalloc_testlib.h and snmalloc_testlib.cc. CMake build_test_library() creates a static library per flavour linked to all tests. Add SNMALLOC_BUILD_TESTING=OFF to Bazel, src/test/*.cc to Bazel filegroup, and sanitizer flags to build_test_library.

Convert simple alloc/dealloc tests to use snmalloc_testlib.h instead of snmalloc.h: - func/first_operation - func/teardown - func/multi_atexit - func/multi_threadatexit Replace MAX_SMALL_SIZECLASS_BITS with max_small_sizeclass_bits(). Replace alloc<Zero> with alloc<ZeroMem::YesZero>. Add pal/pal.h include to testlib header for report_fatal_error.

Convert perf tests that use only the public API: - perf/startup - perf/contention - perf/large_alloc - perf/low_memory Replace DefaultPal::tick() with pal_tick(), Aal::pause() with pal_pause().

Convert tests that use only the public API to snmalloc_testlib.h: - func/bits, func/memory, func/memory_usage, func/pool - func/protect_fork, func/redblack, func/statistics - perf/external_pointer, perf/lotsofthreads, perf/post_teardown - perf/singlethread Replace DefaultPal/Aal calls with pal_*() wrappers. Replace external_pointer<> with libc __malloc_*_pointer(). Replace alloc<Zero/Uninit> with alloc<ZeroMem::YesZero/NoZero>. Fix include order in ds/ds.h for pool.h/pooled.h dependencies.

Convert tests that need allocator internals (sizeclasses, pagemap, etc.) to use snmalloc_core.h alongside snmalloc_testlib.h: - func/malloc (replace our_* with testlib_* prefix) - func/pagemap - func/release-rounding - func/sizeclass

Move report_fatal_error() and message() template definitions from pal/pal.h into ds_core/helpers.h. They now call error() and message_impl() which are forward-declared non-template functions in ds_core/defines.h, with implementations in pal/pal.h that delegate to DefaultPal. This breaks the dependency of ds_core/ code on pal/: any code that includes ds_core/helpers.h can call report_fatal_error() without needing PAL headers. The linker resolves error() and message_impl() from the translation unit that includes pal.h. Update testlib.h to include pal/pal_consts.h instead of pal/pal.h. Fix memory.cc to use ds_aal/ds_aal.h for address_cast. Fix protect_fork.cc to include ds_aal/prevent_fork.h explicitly. Revert low-memory.cc to snmalloc.h (needs PalNotificationObject).

The header doesn't use errno itself. Tests that need it can include it directly.

Use minimal includes for protect_fork (ds_core/helpers.h + ds_aal/prevent_fork.h). Fix missed testlib conversions for multi_atexit and multi_threadatexit.

Create ds_core/sizeclassconfig.h with the mitigation-independent sizeclass constants (INTERMEDIATE_BITS, MIN_ALLOC_STEP_SIZE, etc.). This file depends only on bits.h — no mitigations.h. Update sizeclassstatic.h to include sizeclassconfig.h instead of allocconfig.h. allocconfig.h now includes sizeclassconfig.h and keeps only the mitigation-dependent constants (MIN_OBJECT_COUNT, DEALLOC_BATCH_*, MAX_SLAB_SPAN_*, etc.). This makes the testlib header chain fully mitigation-independent: testlib.h -> sizeclassstatic.h -> sizeclassconfig.h -> bits.h

Move mitigations.h, allocconfig.h, and cheri.h from ds_core/ to a new mitigations/ directory. These files depend on SNMALLOC_CHECK_CLIENT compile-time flags and don't belong in ds_core/ (which should be mitigation-independent). New include hierarchy: ds_core/ → aal/ → ds_aal/ → mitigations/ → pal/ → ds/ → mem/ ds_core/ is now fully mitigation-independent. The testlib header (which includes only ds_core/ headers) produces identical output regardless of SNMALLOC_CHECK_CLIENT, enabling tests to be compiled once and linked against both fast and check testlib variants.

Testlib-only tests are mitigation-independent: compile into shared OBJECT libraries. Also fix MSVC C4701 warning in cleanup_unused and add unistd.h to protect_fork.

Introduce SNMALLOC_API macro for public API functions (dealloc, debug_teardown, libc::malloc, etc.) that need standalone definitions when compiled into a static library. Defaults to SNMALLOC_FAST_PATH_INLINE (normal header-only usage). testlib.cc defines SNMALLOC_API as NOINLINE on MSVC before including snmalloc.h, producing strong definitions that ARM64EC exit-thunks can reference. On GCC/Clang, __attribute__((used)) via the existing SNMALLOC_USED_FUNCTION annotation suffices.

Add a const void* overload of remaining_bytes alongside the existing address_t version. The testlib API uses const void* so callers don't need address_cast or ds_aal headers. Remove address_t from testlib.h. This makes memory.cc a testlib-only test (no ds_aal.h needed), adding it to the compile-once list.

Expose snmalloc's checked memcpy through the libc API surface: libc::memcpy<Checked, ReadsChecked>(dst, src, len) Checked controls destination bounds checking, ReadsChecked controls source bounds checking (defaults to same as Checked). Explicit instantiations provided in testlib for the common variants: <true,true>, <true,false>, <false,false>.

Replace DefaultPal::message() calls in START_TEST and INFO macros with snmalloc::message() which is now defined in ds_core/helpers.h. This removes the PAL dependency from test infrastructure, allowing tests that only need ds_core/ or ds_aal/ headers to avoid including pal/pal.h.

The seqset test is a unit test for ds_aal/seqset.h and only needs ds_aal and ds_core headers. With test/helpers.h no longer depending on DefaultPal, the pal.h include is unnecessary.

Fix header includes that were missing.

mjp41 · 2026-03-31T09:56:26Z

@SchrodingerZhu there is a lot of changes here. The individual commits are all pretty self-contained.

If you don't have time, just let me know, and I'll take a second pass on the commits myself.

SchrodingerZhu · 2026-03-31T20:06:15Z

      auto& key = freelist::Object::key_root;

-      for (size_t i = 0; i < NUM_SMALL_SIZECLASSES; i++)
+      for (smallsizeclass_t i; i < NUM_SMALL_SIZECLASSES; i++)


this looks a bit strange. maybe just use smallsizeclass_t i{} for explicit default initialization.

SchrodingerZhu · 2026-03-31T20:12:17Z

LGTM in general, the change duplicates the symbols on slow paths:

schrodingerzy@promaxgb10-edf0:~/Documents/snmalloc/build$ llvm-nm-21 libsnmallocshim.so --demangle  | grep __malloc_end_pointer
000000000001e9d8 t snmalloc::libc::__malloc_end_pointer(void*)
000000000001e9d8 T __malloc_end_pointer

Is this expected?

mjp41 · 2026-04-24T16:16:04Z

LGTM in general, the change duplicates the symbols on slow paths:

schrodingerzy@promaxgb10-edf0:~/Documents/snmalloc/build$ llvm-nm-21 libsnmallocshim.so --demangle  | grep __malloc_end_pointer
000000000001e9d8 t snmalloc::libc::__malloc_end_pointer(void*)
000000000001e9d8 T __malloc_end_pointer

Is this expected?

This is a good question. I have not been precise on what symbols should be exposed by each library.

What do you think is the right thing? Should the libsnmallocshim.so just provide a libc interface or also the snmalloc::libc interface?

mjp41 force-pushed the test-build-perf-v2 branch 4 times, most recently from e0efc18 to 8eb5e51 Compare March 30, 2026 13:21

mjp41 added 6 commits March 30, 2026 16:09

Move pool.h/pooled.h from mem/ to ds/

d3a3914

These are general-purpose data structures, not allocator-specific. Moving them reduces the dependency surface of mem/.

Add external_pointer libc functions

13ed7cd

Add __malloc_start_pointer() and __malloc_last_byte_pointer() to libc.h, alongside the existing __malloc_end_pointer(). These provide the external_pointer functionality through the libc API surface.

mjp41 force-pushed the test-build-perf-v2 branch 2 times, most recently from ff3b9f1 to b87a6a0 Compare March 30, 2026 19:53

mjp41 added 16 commits March 31, 2026 09:47

Create test library infrastructure

ac468b3

Add snmalloc_testlib.h and snmalloc_testlib.cc. CMake build_test_library() creates a static library per flavour linked to all tests. Add SNMALLOC_BUILD_TESTING=OFF to Bazel, src/test/*.cc to Bazel filegroup, and sanitizer flags to build_test_library.

Convert perf tests to use testlib (batch 9d partial)

6d68c71

Convert perf tests that use only the public API: - perf/startup - perf/contention - perf/large_alloc - perf/low_memory Replace DefaultPal::tick() with pal_tick(), Aal::pause() with pal_pause().

Convert tests needing snmalloc_core.h to use testlib

14ad0e6

Convert tests that need allocator internals (sizeclasses, pagemap, etc.) to use snmalloc_core.h alongside snmalloc_testlib.h: - func/malloc (replace our_* with testlib_* prefix) - func/pagemap - func/release-rounding - func/sizeclass

Remove unnecessary errno.h include from snmalloc_testlib.h

2f65c2c

The header doesn't use errno itself. Tests that need it can include it directly.

Convert protect_fork, multi_atexit, multi_threadatexit

01d650e

Use minimal includes for protect_fork (ds_core/helpers.h + ds_aal/prevent_fork.h). Fix missed testlib conversions for multi_atexit and multi_threadatexit.

Compile testlib-only tests once for both fast and check

2dba9af

Testlib-only tests are mitigation-independent: compile into shared OBJECT libraries. Also fix MSVC C4701 warning in cleanup_unused and add unistd.h to protect_fork.

Document mitigations/ layer in include hierarchy README

1a66f10

Remove pal.h include from seqset test

6e453f9

The seqset test is a unit test for ds_aal/seqset.h and only needs ds_aal and ds_core headers. With test/helpers.h no longer depending on DefaultPal, the pal.h include is unnecessary.

mjp41 force-pushed the test-build-perf-v2 branch from b87a6a0 to ab94ccb Compare March 31, 2026 08:48

Apply clangformat

98d7950

Fix header includes that were missing.

mjp41 force-pushed the test-build-perf-v2 branch from ab94ccb to 98d7950 Compare March 31, 2026 09:55

mjp41 requested a review from SchrodingerZhu March 31, 2026 09:55

SchrodingerZhu reviewed Mar 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test build perf v2#835

Test build perf v2#835
mjp41 wants to merge 24 commits intomicrosoft:mainfrom
mjp41:test-build-perf-v2

mjp41 commented Mar 28, 2026

Uh oh!

SchrodingerZhu commented Mar 28, 2026

Uh oh!

mjp41 commented Mar 28, 2026

Uh oh!

mjp41 commented Mar 31, 2026

Uh oh!

SchrodingerZhu Mar 31, 2026

Uh oh!

SchrodingerZhu commented Mar 31, 2026

Uh oh!

mjp41 commented Apr 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mjp41 commented Mar 28, 2026

Uh oh!

SchrodingerZhu commented Mar 28, 2026

Uh oh!

mjp41 commented Mar 28, 2026

Uh oh!

mjp41 commented Mar 31, 2026

Uh oh!

SchrodingerZhu Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

SchrodingerZhu commented Mar 31, 2026

Uh oh!

mjp41 commented Apr 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants