[CoreCLR] Experimental: dlopen()-based assembly store loading

[simonrozsival]

Summary

The CoreCLR host used to locate the assembly store by parsing the APK ZIP central directory, mmap-ing the wrapper .so, and walking its ELF section headers by hand to find the payload. Since the store is already a valid ELF shared library (#9154), this PR instead makes its payload a loadable section (SHF_ALLOC, in a PT_LOAD segment) exposed via an exported dynamic symbol, so the host just:

void *h     = dlopen("libassembly-store.so", RTLD_NOW | RTLD_LOCAL);
void *start = dlsym(h, "_assembly_store");   // -> AssemblyStoreHeader

…and lets the platform dynamic linker locate + map it out of the APK. The hash-index lookup (AssemblyStore::open_assembly) is unchanged.

This is CoreCLR-only. MonoVM is untouched and keeps the classic DSOWrapperGenerator + ZIP-scan store. The mechanism is hardcoded by runtime and cannot be overridden (_AndroidDlopenAssemblyStore = true for CoreCLR, false for MonoVM), because after this change the CoreCLR host has no ZIP fallback and MonoVM's host doesn't understand the loadable-symbol layout.

What's removed (CoreCLR)

• src/native/clr/startup/zip.cc in its entirety (the ZIP central-directory parser) + the xamarin-startup static library.
• Host::zip_scan_callback, AssemblyStore::map(fd/offset), and Util::get_wrapper_dso_payload_pointer_and_size usage (the ELF section-header walk).
• The DSOApkEntry {fd,offset} native-library fast path in MonodroidDl (it was populated by the same ZIP scan). Bundled native libraries now load via dlopen-by-name, which works because they're uncompressed + page-aligned in lib/{ABI}/ (extractNativeLibs=false). The now-dead DSOApkEntry struct, its dso_apk_entries[] emission in ApplicationConfigNativeAssemblyGeneratorCLR, and the stub definition are removed too (MonoVM keeps its own copy).

Net: 889 lines deleted, 197 added (−692 net). The arm64 CoreCLR host (libnet-android.release.so) shrinks 34,784 B (−2.73%) in a clean same-tree isolation (referenced code removed; arm −3.05%, x64 −3.34%); the actual shipped main-vs-PR APK host delta is −28,640 B (see the Size table below).

Why this hasn't been done before (history)

dlopen-based loading was never used, and this is not a regression from a nicer past design — ZIP-scan + mmap is the original and only design, three layers accreted over the years:

When	PR	What changed	Loading
2019	#2570	assemblies mmap-ed in place from the APK (never extracted)	ZIP-scan + mmap
2021	#6311	assembly store introduced (assemblies.blob, O(n)→O(1))	raw blob, ZIP-scan + mmap
2024	#8478	all assemblies RID-specific; store moved to lib/{ABI}/, renamed lib*.so (not real ELF)	ZIP-scan + mmap
2024	#9154	wrapped in a real ELF (objcopy --add-section, non-loadable, 16 KB-aligned) for the Android 16 KB page-size requirement	ZIP-scan + mmap + ELF section-header walk
2025	#9778, #9572	CoreCLR host re-implements the same approach	same

dlopen only became possible in #9154 (the store became real ELF), and even then the payload was deliberately kept non-loadable so the existing mmap loader could be reused — on the unverified assumption (see ApkSharedLibraries.md: "…without having to load the ELF image into memory") that mmap-the-section beats loading the ELF. This PR measures that assumption.

Benchmark results

Measured on a physical Samsung A16 (SM-A165F), net11.0-android MAUI sample-content app (dotnet new maui --sample-content), Release, arm64, CoreCLR, extractNativeLibs=false, 30 interleaved cold-starts (am start -W TotalTime).

Apples-to-apples (both built from the same local P7 SDK — isolates the store mechanism)

Three independent 30-interleaved rounds, main (classic ZIP-scan) vs this PR (dlopen). Rounds 1–2 reused a fixed dlopen APK; round 3 rebuilt latest fresh from the committed code (verified store exports _assembly_store, section payload is loadable, host has no DT_NEEDED):

Round	main mean	dlopen mean	Δ (dlopen − main)	paired test
1	2059.7 ms	2076.1 ms	+16.4 ms (+0.80%)	t=2.31, p≈0.03, CI [+1.9, +31.0]
2	2061.1 ms	2054.4 ms	−6.6 ms (−0.32%)	t=−1.00, p≈0.32, CI [−19.7, +6.4]
3 (fresh)	2056.0 ms	2032.3 ms	−23.6 ms (−1.15%)	t=−3.95, p≈0.0004, CI [−35.4, −11.9]

The three rounds span +16 → −24 ms (a ~40 ms between-session spread that exceeds any single round's effect) ⇒ the cross-session noise floor dominates. dlopen is startup-neutral — within noise, if anything trending faster, and never a regression.

Size (arm64, extractNativeLibs=false, all Stored/uncompressed in the APK; round-3 fresh build)

Component	main	dlopen	Δ
APK total	29,670,166	29,637,398	−32,768 B
libmonodroid.so (host)	1,264,464	1,235,824	−28,640 B (−2.27%)
libassembly-store.so	7,674,048	7,674,784	+736 B (negligible)

(The host shrinks because the ZIP parser + ELF section-header walk + DSOApkEntry path are gone. Clean referenced-code isolation on the same tree: arm64 −34,784 B, arm −3.05%, x64 −3.34%.)

Is there a real difference? No — it's noise

The dlopen+dlsym call itself was measured on-device against the real 7.6 MB store:

dlopen = 0.14–0.49 ms      dlsym = ~0.006 ms      (reads magic 0x41424158 "XABA")

That's ~0.02% of a ~2050 ms cold start — far below the ±20–40 ms cross-session variance. Deleting the ZIP scan + ELF walk is provably not a startup regression.

⚠️ An earlier naive comparison (stock global .NET 11 preview 5 vs this local preview 7 build) showed +72 ms — but that was confounded by the P5→P7 delta and official-vs-local-build differences, not the dlopen change. Building main with the same local P7 toolchain isolates it to the ~0 ms above. (The residual local-P7-vs-global-P5 gap is likely partly #11730, the LZ4→Zstd store-compression switch merged after P5.)

Can we optimize the dlopen further? No — there's nothing to skip

The store .so is inspected with readelf: 0 relocations, no DT_NEEDED, no DT_INIT/init-array, two PT_LOAD segments. The linker therefore does no relocation, dependency resolution, or constructor work — it just mmaps the (demand-paged) segments + minor bookkeeping. Concretely:

• android_dlopen_ext has no "skip relocations/init" flag — and there's nothing to skip anyway.
• RTLD_LAZY vs RTLD_NOW is moot (no code → no PLT relocations). We pass RTLD_LOCAL (we only dlsym our own handle, so there's no need for RTLD_GLOBAL's extra global-scope bookkeeping), though the difference is immaterial at <0.5 ms.
• The bionic apk!/lib/{abi}/lib*.so zip-path syntax (how extractNativeLibs=false is implemented) works, but measured slower on-device (~0.52–0.98 ms — the linker still opens the zip + does a CD lookup for the entry) and would re-introduce runtime APK-path / split-config handling. The by-soname dlopen is simpler and already at the floor.

Follow-ups (out of scope)

• Pairs naturally with [experiment] On-device decompressed AssemblyStore cache (CoreCLR) #11967 (on-device decompressed store cache): the decompressed cache can itself be the dlopen-able wrapped .so, so warm starts skip both Zstd decompression and any ZIP work; android_dlopen_ext(USE_LIBRARY_FD[_OFFSET]) is the right tool to load it by fd.
• The now-orphaned Util::get_wrapper_dso_payload_pointer_and_size definition can be deleted as pure dead-code cleanup.
• The runtime-config blob uses the same wrapper mechanism and could adopt the same symbol-based loading.

Validation

• CoreCLR app built with no flag auto-produces a store exporting _assembly_store (hardcode works) and cold-launches cleanly (Status: ok).
• Store is Stored (uncompressed) + 16 KB-aligned; dlsym returns a pointer to a valid AssemblyStoreHeader (magic XABA).
• The wrapper's payload lives in a section named payload (matching the MonoVM DSOWrapperGenerator and tools/assembly-store-reader-mk2), so the CoreClrTrimmableTypeMap* build tests that inspect the APK through the reader keep working.
• The whole read-only store pointer chain (configure_from_payload(const void*), data_start, image_data/debug_info_data/config_data, assemblies, descriptor, assembly_store_hashes) is const — no const_cast on the store path.

Reference PRs

• Embedded assemblies blob #6311, Make all assemblies RID-specific #8478, [native] Real shared libraries in APK /lib directories #9154, Move all of assembly store building code to a separate class #9382, CLR hosting bits #9778, CLR hosting #9572
• LZ4→Zstd store compression: Replace AssemblyStore compression algorithm LZ4 with Zstd #11730 · on-device store cache: [experiment] On-device decompressed AssemblyStore cache (CoreCLR) #11967
• AssemblyStores.md · ApkSharedLibraries.md · Android 16 KB pages