Semantic load path memory is bounded by estimate only; actual RSS growth during model open/indexing is unobserved

[ScriptedAlchemy]

Follow-up finding from #753 part 2 (observed incident: projection load reported loading for 16 minutes past load_deadline_ms: 30000, RSS climbing 11.4 GB -> 18.5 GB until a kernel OOM).

The state-truth defects around that incident are fixed on the PR #707 lane:

• the load deadline now fires during a cold open_session (typed LoadDeadlineExceeded, loader thread keeps the slot/reservation until the runtime returns, then releases), and
• lifecycle Indexing{completed_units,total_units} now surfaces as runtime Indexing progress instead of being flattened to Loading, so long "loading" phases that are actually indexing are no longer mislabeled.

What remains and is worth its own slice: the semantic load path's memory ceiling is enforced only against estimates, never against actual usage while work is in flight.

Attribution (code pointers, crates/tracedecay-semantic):

1. SessionPool reserves runtime.resident_bytes_reservation(authority) (the manifest resident_bytes_estimate) before open and re-checks session.resident_bytes_estimate() after open (session_pool.rs, cold-open path). Nothing observes actual RSS between those two points.
2. FastEmbedEmbeddingRuntime::open_session (fastembed_adapter.rs:1179) buffers the entire model + tokenizer members in memory (fastembed_model, fastembed_adapter.rs:1295) and then builds the ORT session via TextEmbedding::try_new_from_user_defined. ORT's graph parse/optimization/arena allocation during that constructor is unbounded by any TraceDecay ceiling; peak transient memory is a multiple of model bytes and invisible to the pool's accounting.
3. After the (new) deadline abandonment, the loader thread cannot cancel ORT mid-build; the memory stays genuinely held (and truthfully accounted as reserved) until ORT returns. A hung or thrashing ORT build therefore still holds its memory unboundedly in time, just no longer blocks the acquisition.

The 7.1 GB growth in the incident is far larger than one ~600 MB model graph build, which is consistent with the mislabeled-indexing mechanism (real projection/indexing work accumulating while the surface said loading) rather than a single ORT session build; with the progress mapping fixed, the next occurrence will be attributable directly from status output.

Suggested direction (not started): enforce an actual resident bound during load/indexing — either sample the loader's RSS against resident_byte_ceiling and fail typed, or isolate the ORT session build so it can be cancelled/killed at the deadline instead of merely abandoned. Per repo rules the ceilings themselves are symptoms guards and stay unchanged.

[ScriptedAlchemy]

Memory-cost slice landed (store-side accumulation)

Attributed and fixed the dominant unbounded-memory mechanism in the projection/indexing path. Measured with a new commit-loop scale bench (crates/tracedecay-usecases/benches/semantic_vector_commit_scale.rs) driving the real VectorGenerationStateMachineV1 at 120k chunks x 768 dims (351 MiB of raw floats), full-rebuild, 512-chunk batches, dev profile, VmHWM from /proc/self/status.

Attribution (measured, before)

Site	Mechanism	Measured cost @120k×768
commit_batch_records (graph adapter)	cloned the whole accumulated state machine twice per committed batch (before/after), one clone never read	RSS slope 3× corpus rows: 2254 MiB at batch 232 vs 572 MiB for the clone-free floor
publish_generation	staged.get(build_id).cloned() before the completeness check — a full corpus deep-copy on every per-batch publish probe, plus another at real publish	part of the same 3× slope; terminal spike
publish-time interning	intern_generation_vectors copied every float payload into the physical byte pool (Arc<[f32]>) — a pool with no graph-path reader	publish spike 572 → 1914 MiB (machine floor)
staged float retention	every committed ProjectedChunkVectorV1 kept its Vec<f32> resident in the machine although each batch is already durably in the graph before the in-memory commit settles	~570 MiB steady at 120k (≈1.1 GiB at the incident's 226k)
total adapter path		VmHWM 3044 MiB, 184 s wall

Extrapolated to the incident corpus (~226k chunks) the adapter path alone accounts for ~5.7 GiB of the observed 7.1 GiB growth; the rest is the graph engine's own entity+HNSW residency, the scheduler's cloned batch chunk texts, and the ORT session build (all unchanged in this slice, listed below).

Fix (landed)

• 183f18a9ac perf(semantic): bound projection commit memory to O(batch) in the store
  • split machine commit into validate_batch (all fallible checks while the batch floats are in hand) + infallible apply_batch; the adapter appends the durable batch between the halves, so transactionality needs no machine clone at all;
  • publish_generation validates completeness on borrows first (per-batch probe is allocation-free) and consumes the staged build instead of cloning; replay admission compares staged parts directly;
  • interning moved from publication to persist_sealed, its only consumer;
  • typed StagedVectorValueRetentionV1 — the graph adapter elides committed float payloads (row identity/digests retained; native rows are the byte authority), the in-memory reference model and sealed persistence stay Retained, and sealed persistence refuses elided state;
  • native batch encoding reads payloads from the prepared batch and the post-batch census from the validated decision — byte-identical output.
• 5a4f91875d test(semantic): the scale bench above; b644c66b22 carries the fmt pass.

Before/after (same bench, same scale)

Pattern	Before VmHWM	After VmHWM	Before wall	After wall
adapter path (production)	3044 MiB	236 MiB	184 s	24 s
retained reference model	1914 MiB	589 MiB	45 s	29 s

After: adapter-side working set is O(batch floats) + O(corpus ids/digests) (~225 MiB @120k, no terminal spike). Verified at tip 2ed5a604f0: usecases+semantic suites 1051 passed, semantic_vector_generation_prep_test 21 passed, code-index-runtime semantic 28 passed, graph-db vector suites 10 passed, lean (--no-default-features) and hotpath feature lanes check clean.

Still open on this issue

• No actual-RSS observation/enforcement against resident_byte_ceiling during load/indexing (the ceiling still compares estimates only).
• ORT session build remains uncancellable after deadline abandonment (memory held until the runtime returns).
• The evaluation/activation journey still materializes the whole embedded corpus before its paged commit (EvaluationPreparedPageIndexV1), and incremental builds hydrate full base-generation float payloads for lineage checks.

[ScriptedAlchemy]

The three remaining slices landed on the PR #707 branch (codex/tracedecay-total-redesign-plan-reopened):

Actual-RSS enforcement — 6a59435205. SessionPool::open_session_bounded now samples real process RSS (canonical VmRSS sampler moved to tracedecay-runtime-core, daemon gauge cut over in 03ddcac82f) every 100ms during a cold load, against a baseline captured at load start. Growth beyond the authority's resident-byte ceiling fails typed (SessionAcquireError::ResidentCeilingExceeded { observed_growth_bytes, ceiling_bytes }) and fires the load-interruption signal so the loader aborts instead of finishing into a discard. Tests: scripted-sampler breach fails typed while the load is mid-flight; a slow load under the ceiling completes; a Linux-only synthetic-corpus test allocates real memory through the kernel sampler and observes the typed breach.

ORT build cancellation — same commit. EmbeddingRuntime::open_session now takes the standard SemanticExecutionAuthority, and the FastEmbed adapter checks it at every stage boundary: before each artifact member read (tokenizer/config/weights) and immediately before the ORT session constructor. The pool fires that authority on deadline or RSS breach, so a cancelled load aborts at the next boundary and releases its slot, reservation, and resident-byte accounting promptly. The one caveat: the ORT constructor itself is uninterruptible — onnxruntime exposes no mid-build cancellation hook — so the residual uncancellable window is that single FFI call. True mid-constructor cancellation would need process isolation, which is deliberately deferred. Tests cover prompt abort at a stage boundary on deadline fire and cancellation before member reads.

Paged evaluation — a5ddd51264. Evaluation activation commits now flow through a borrowed lazy page index: commit_evaluation_prepared_generation borrows the prepared corpus and materializes one page at a time, so peak memory is O(page) rather than O(corpus), and callers no longer clone the prepared generation. On the incremental side, base generations installed for lineage checks now have their float payloads elided — reuse validation only needs identity digests, so incremental builds no longer hydrate the full base-generation float corpus. Tests assert paged and unpaged commits publish byte-identical content (generation id, manifest digest, vectors, tombstones) and that elided base payloads still pass reused-row lineage checks.

All touched functions carry feature-gated #[hotpath::measure] spans. Focused lanes green: 60 semantic session-pool/adapter tests, 110 semantic_runtime + 28 vector-generation tests in usecases, plus check --all-targets on runtime-core and the root crate. Leaving the issue open for the maintainer to close after review, given the documented ORT-constructor residual.

[ScriptedAlchemy]

Replacement lane follow-up is now on PR #707 via a no-force push:

• 6cd66c1e7 fix(semantic): settle cancelled loads under RSS budgets — measured cold-load growth is charged against already-retained session bytes, cancellation settlement is regression-tested, and touched semantic impls have feature-gated measure_all coverage.
• ba909b547 perf(usecases): instrument vector generation impls — adds feature-gated measure_all coverage across the paged evaluation/vector-generation transition path.
• 9659b2879 removes the all-features-only FastEmbed closure warning.

Verification on the rebased replacement tree (CARGO_BUILD_JOBS=8): cold-load suite 8/8; exact RSS-overlap settlement 1/1; exact pre-member-read cancellation 1/1; paged/unpaged equivalence 3/3; elided-base lineage 1/1; touched semantic/usecases default, hotpath, and --all-features checks all green.

Leaving #754 open. Actual RSS enforcement and paged evaluation are covered, and cancellation is prompt at every supported stage boundary, but ORT 2.0.0-rc.12 still exposes no cancellation hook during CreateSessionFromArray. Mid-constructor termination therefore remains the explicit residual acceptance item; claiming it complete would require unsupported in-process preemption or a process-isolation design that conflicts with the current product boundary.

[ScriptedAlchemy]

Completed on the PR #707 branch at 113959040 (no-force push).

The previously documented residual is now closed: FastEmbed 5.17.3 hid ORT rc.12's LoadCanceler, even though the pinned runtime exposes SessionOptionsSetLoadCancellationFlag. The pinned fork commit ScriptedAlchemy/fastembed-rs@ad2af710 publishes that handle immediately before CreateSessionFromArray; TraceDecay now monitors the existing typed interruption authority during the constructor and fires the ORT canceler on deadline or RSS verdict. The prior stage-boundary checks remain in place.

Salvage adjudication:

• /fast/tmp/wt-754-efficiency was clean and contained no issue-specific divergence; its tip was already an ancestor of the target.
• /fast/tmp/wt-754-replacement was also clean by takeover time (the reported dirty work had been committed). Its usable work was already on the target: 6a5943520 actual-RSS enforcement/stage cancellation, a5ddd5126 paged evaluation, and follow-ups 6cd66c1e7, ba909b547, 9659b2879. There was no uncertain dirty or divergent content to export.

Acceptance evidence after the final rebase:

• cold-load/RSS suite: 8 passed, including the Linux synthetic-corpus actual-RSS breach and cancellation settlement;
• ORT cancellation bridge: 1 passed with token fire forwarded within the prompt bound; composed semantic-fastembed build green against the real rc.12 handle;
• paged/unpaged publication identity: 1 passed;
• elided-base lineage: 1 passed;
• tracedecay-semantic and tracedecay-usecases: default, hotpath, and --all-features checks all green; runtime-core default check green.

The broader root default check remains independently red at the moving target due to a compiler query-depth overflow while laying out tracedecay-session-runtime/src/session_sync.rs:431; I did not raise the recursion limit or repeat the known-red failure under hotpath/all-features. This is outside #754 and does not affect the touched package matrix above.

Allocator verdict (comparison not rerun, per instruction): keep the default glibc allocator with the #762 publication-boundary MVCC GC + malloc_trim policy and installed MALLOC_ARENA_MAX=2. Recovered provenance is allocator-evaluation commit d532a84a4: sandbox A/B runs found jemalloc with background purge still retained 5–8 GiB after index+publish settled. Mimalloc was part of that concluded comparison, but a separate recoverable numeric split was not present in the predecessor lanes, so no stronger number is claimed. The default allocator remains unchanged.