AOT semantic-hash desync (error[50101]) on shared-module function with a direct-require caller — suspect inline-decision divergence between AOT-gen and runtime compiles

[borisbat]

Symptom

test_aot.exe -use-aot dastest/dastest.das -- --use-aot --test tests/dasLLAMA/test_kquant.das (branch bbatkin/dasllama-flash-slices, PR #3444, rebased on current master) fails:

error[50101]: AOT link failed on _lambda_thismodule_706_17`function ...
semantic hash is b512ca360d7a6de3

Localized per skills/aot_hash_desync_debugging.md: Case B on a single dependency — dasllama_math_gen::kq_groupn_gen's OWN hash is recorded as 0xf50fe9ec3d4033f6 by the AOT generator (both in the test's generated cpp AND in the freshly regenerated shared-module lib cpp) while the runtime computes 0xaa077ae0553fa9d. With options no_aot on the test, the failure moves to the module function's own link — the shared-module lib carries the mismatched stub.

UPDATE — require-order whack-a-mole (strengthens the inline-instability theory)

The runtime hash of shared-module functions is a function of the requiring program's require order:

test_kquant require order	failing symbol
original (gguf, quant, math, math_default, math_gen, transformer)	dasllama_math_gen::kq_groupn_gen
math_gen FIRST (mirroring test_kgroup_repack, which passes)	kq_groupn_gen links fine → finalize XS<dasllama_gguf::GGUFKVLoc> fails instead
exact AOT-module-lib canonical order (math, math_default, math_gen, quant, gguf, transformer)	still finalize XS<dasllama_gguf::GGUFKVLoc>

So there is no require ordering of this program that hashes ALL shared modules the way the AOT lib recorded them — reordering just moves which module diverges. tests/dasLLAMA/test_kgroup_repack.das (direct-requires dasllama_math_gen, no gguf) passes; every umbrella-required test passes. The divergence is per-(program, module) and deterministic.

Ruled out

• Staleness: deterministic across full wipes of tests/dasLLAMA/_aot_generated/ and modules/dasLLAMA/dasllama/_aot_generated/ + test_aot rebuilds with /nodeReuse:false; a genuinely stale sibling (..._gemm_schema.das.cpp, day-old) was force-regenerated with no change.
• is_in_aot branching: no is_in_aot/set_aot anywhere under modules/dasLLAMA.
• CMake registration: tests/dasLLAMA/*.das is glob'd with CONFIGURE_DEPENDS; excluding the test's AOT cpp just converts mismatched-stub into missing-stub (same 50101).
• Program options: persistent_heap / stack are shared by a dozen passing dasLLAMA tests.
• Test-side dodges: options no_aot (shared-module simulate still AOT-links), require reordering (see table), removing the direct call edge (the failure is at simulate time, not call time).

Clues

• The generated C++ carries auto-inline argument renames (__fmt_rename_at_1294_1218), and a probe options auto_inline_functions = false in the test file moved the mismatch wholesale to other inline-touched functions — hashes are inline-decision-dependent program-wide, including shared modules.
• Working theory: the auto-inliner makes different decisions per program environment when compiling shared modules, so the same module source legitimately hashes differently than the AOT lib's canonical compile. src/ast/ast_inline.cpp changed on master recently; the branch's AOT suite has no earlier baseline (pre-rebase HEAD f379a89eb vs rebased 287a4a329 is the first bisect step).

Repro

cmake --build build --config Release --target test_aot -j 64 -- /nodeReuse:false
bin/Release/test_aot.exe -use-aot dastest/dastest.das -- --use-aot --test tests/dasLLAMA/test_kquant.das

Hunt rig

skills/aot_hash_desync_debugging.md step 3 (options log_nodes + log_nodes_aot_hash) on kq_groupn_gen / finalize GGUFKVLoc in the two require orders — the diff should show the differing inline splice subtrees. First step: bisect the desync across the master rebase window (ast_inline.cpp changes).

PR #3444's AOT lanes will red on this until fixed; the test-side workarounds all fail (see table), so this needs the compiler-side fix.

🤖 Generated with Claude Code

[borisbat]

PR #3444 merged with all CI lanes green, including both Windows lanes and every AOT-running lane — the 50101 does not reproduce in CI's from-scratch builds.

Severity downgrade: the repro is local-environment-only. Best current explanation: the local rig's AOT generation ran under a DLL-flavor daslang.exe while test_aot.exe links the static libDaScript — across a night of incremental rebuilds those two hosts can embed slightly different compiler states (the master rebase window changed src/ast/ast_inline.cpp), and their inliners then legitimately disagree on shared-module bodies. A clean build dir likely cures the local repro (unverified — next time it bites, verify clean-build-first before the deep hunt).

What remains genuinely interesting from the investigation: the runtime semantic hash of shared-module functions is require-order-dependent (the whack-a-mole table above) — meaning any coherent-build guarantee relies on the AOT lib and every requiring program converging on identical inline decisions. Fragile by construction; worth a look whether inline decisions should be excluded from (or normalized in) the semantic hash.

🤖 Generated with Claude Code