fix(server): prefix-cache deepening (stop pinning the restore point + stop truncating at stray end-markers)

[jkyamog]

commit	root cause it fixes
ee398df7	restore point pinned at the deepest slot: the in-flight restore source is evicted, the new snapshot is cancelled by the snap_slot == cache_slot guard, so per-turn prefill grew unbounded (observed 1s -> 60s on the 3090s)
4e53532	find_all_boundaries() truncates the boundary walk at a stray end-of-message marker embedded in content, so no boundary is recorded past it and the inline snap has nothing to deepen to

They fail for different reasons: one cancels the snapshot (eviction), the other
leaves no candidate boundary to snapshot (truncation). Both must be fixed for
the restore point to keep sliding forward turn after turn.

4e53532 — don't truncate boundaries on stray end-markers in content

find_all_boundaries() walks the prompt for end-of-message markers. On an
end-marker not followed by a role-start within the 5-token probe window it
breaked, cutting the list off at the first stray. Content that legitimately
contains a stray end-marker (file/log dumps, chatml-in-content, tool output)
therefore lost every real message boundary after it. Fix: skip the stray marker
and keep scanning (capped at 8192 consecutive strays to bound pathological
input); clean prompts are byte-for-byte unchanged. prefix_cache.cpp +28 -2;
2 regression tests — test_find_boundaries_stray_end_msg_does_not_truncate
({5,15,19}) and test_find_boundaries_clean_prompt_unchanged ({6,10,14});
plus a one-line diagnostic on the prepare_inline_snap blocked path.

ee398df7 — slide the restore point past the deepest slot

With --prefix-cache-slots 4 and a long linearly-growing tool-heavy
conversation, all slots form one chain and the only leaf is the deepest slot =
the current restore source. select_inline_evict_victim() evicted that leaf and
prepare_inline_snap() re-assigned the snapshot to the same slot; the
prepare_generation_cache guard (snap_slot == cache_slot) then cancelled it,
pinning the restore point. Fix: thread the restore-source slot into victim
selection so it is never evicted (the snapshot lands in a different slot, so the
restore point slides forward each turn), with a shallowest-ancestor fallback
that never touches the protected tools pin or the restore source. 5 unit tests.

Verification

• In-image test_server_unit: 441/441 pass (new lucebox-hub:latest).
• Standalone harness on the real find_all_boundaries(): stray -> [5,15,19],
  clean -> [6,10,14], deepseek -> [8].
• Live 2x3090 service: a conversation with a stray end-marker in content now
  deepens the inline snap to prefix_len=684 (past the marker) and the next
  turn HITs it — lookup hit slot=3 prefix_len=684, cache_hit=true,

[cubic-dev-ai]

All reported issues were addressed across 4 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 4 files

_{You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/server/prefix_cache.h">

<violation number="1" location="server/src/server/prefix_cache.h:53">
P3: When `skip_index` is set and every leaf is protected, the implementation returns `-1` rather than the oldest protected leaf; qualify this fallback as applying only without a restore source.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[michele1967lux]

Independent validation on AMD / ROCm (gfx1201)

We hit this same livelock in a long ds-harness coding session and worked it
through from the source before finding this PR — arriving at essentially the
same root cause and the same fix (skip_index, threading the physical restore
slot, -1 when nothing is safe to evict). Since #665 got there first and is
more thoroughly verified, this is a validation report rather than a competing
patch.

One useful additional data point is that your verification is on RTX 3090s,
while ours is on AMD. The fix is not incidentally CUDA-specific.

Setup

• AMD Radeon AI PRO R9700 (gfx1201, 32 GB), ROCm 7.2.1, Ubuntu 24.04.4, kernel 6.17
• Qwen3.8-27B target (16 GB GGUF) + DFlash2 draft, --draft-block-size 16
• --prefix-cache-slots 4 --max-ctx 32768, 16 linear turns, tool schema attached
• Your patch applied onto 136ad9c (clean apply), built and run against the
  unpatched build of the same commit — same machine, same session, back to back

Results

	pre-fix	PR #665
restored prefix_len past saturation	frozen at 3072 for all 12 turns	3072 → 7680, advancing
inline snapshots committed / 16 turns	4	14
eviction attempts	12	3
victim chosen	deepest entry, 12/12	redundant ancestor (idx=1)
total prefill	65.2 s	29.0 s
latency at turn 16	8.60 s, climbing monotonically	2.45 s, flat

test_slide_* 4/4 and the pre-existing eviction-policy tests 7/7 pass on ROCm.

The protected pin behaves as intended: the log shows
prefix-aware evict: victim idx=1 ... kept oldest ancestor (len=2016) — the
system+tools head survives and a redundant ancestor is sacrificed instead.

For what it's worth, we independently wrote a fix with one difference: our
fallback picks the oldest ancestor whose cached descendants all still lie on
the restore chain, rather than the shallowest unprotected one. On this workload
the two are indistinguishable — identical prefix_len sequence turn by turn,
same victim, and total prefill within run-to-run noise (29.0 s vs 30.6 s). The
difference only shows up when a side branch is itself pinned, and it is a policy
choice about preserving future divergence points, not a correctness issue. Your
reasoning that the shallowest ancestor's cached prefix is already represented
by deeper snapshots on the same chain holds for this eviction decision.
Mentioning it only in case it is useful.

One reproducer caveat that cost us a cycle

Our first end-to-end reproducer passed against the buggy binary. Ten turns
with small prompts and no tools: only 3 snapshots ever committed against 4
slots, zero evictions — the cache never saturated, so the run never entered the
regime where the livelock lives. A multi-turn test can report a false green
here unless turn size and tool schema are tuned so --prefix-cache-slots is
genuinely filled well before the run ends.

We made both explicit parameters for that reason. Happy to contribute the
script separately if a regression guard of that shape is wanted — it asserts
that the restored prefix_len never regresses and advances at least once over
the turns past saturation. (That invariant is a property of strictly-extending
prompts, not of the server in general: a changed template, tool set, or a real
branch may legitimately shrink the reusable prefix.)

Thanks for the fix — it takes a real coding-agent session from unusable back to
usable on this hardware too.

[jkyamog]

@michele1967lux thanks for testing and reporting it. Good to know that I am not the only who suffered from this issue. I guess I really push the agents and encounter these issues.

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

_{You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

_{You’re at about 98% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>