From a56c77eac07351fe858f3a7028df9ceed4137689 Mon Sep 17 00:00:00 2001 From: Phil Merrell Date: Wed, 16 Sep 2026 09:47:29 -0600 Subject: [PATCH] docs(kaizen): queue the compaction-stack follow-ups for Friday review Quality-veto eval before the new defaults reach prod; the PR-5 flag decision (dev week after the hourly-tick fix, probe baselines recorded); the 28% neither cohort and content-class ordering as the next lever; housekeeping follow-ups; the compaction v2 go/no-go with the stack's own metrics; and the document-offload epic with analytics as a deliverable. Co-Authored-By: Claude Fable 5.1 --- docs/kaizen/review-queue.md | 43 +++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/kaizen/review-queue.md b/docs/kaizen/review-queue.md index b84ede98..cab4258f 100644 --- a/docs/kaizen/review-queue.md +++ b/docs/kaizen/review-queue.md @@ -5,6 +5,49 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Open +### [2026-09-16] Compaction stack (#1125 → #1128 → #1129 → #1131 → #1132) — run the quality-veto eval BEFORE the new defaults reach prod +- **Source**: Phil-initiated — `docs/specs/compaction-model-relative-thresholds.md` §5 (the gate the spec sets for itself) + `docs/specs/compaction-over-threshold-cache-spiral.md` §4.3 (the veto eval) + `docs/specs/agentcore-evaluations-spike-findings.md` (what the managed evaluation service already supplies). Stack built 2026-09-15/16; the 2026-09-15 replay of 20 heavy Sonnet 5 sessions priced 100k/25k at **$73.70 vs $102.64 actual** input-side. +- **Surface**: backend — `session/compaction_policy.py` (floor-seeking cut), `session/compaction_summary.py` (8k summary via Nova Micro), `session/turn_based_session_manager.py` (`apply_pending_compaction`); eval harness per the spiral spec §4.3 (constraint retention / revision continuity / reference lookup), fixed-threshold arm as control. +- **Effort × Impact**: M × H — the stack cuts deeper and earlier than v1 ever did and rewrites the summary with a cheap model; that is a bigger model-visible context change than anything the cost arc has shipped. Cost is measured (replay + PR-2/3 metrics); quality is not yet. +- **Subtracts**: no — it is the condition under which the stack's defaults may stay on in prod. Without it we are optimizing dollars against an asserted quality constant (the counterweight the roadmap has never had). +- **Status**: open — **blocks calling the feature done.** Run once the stack is merged to develop (dev auto-deploys): paired arms, same corpus, blinded holistic judge outside the managed path (spike §2), plus the AgentCore Evaluations judges for the three families. Record the result in the thresholds spec §5 and move `FLOOR_RATIO` / `CEILING_CAP_TOKENS` only on that evidence. Merge mechanics: **in order, parents first, never `--delete-branch` on a parent (kills the child PR), and inner stack PRs get zero CI — let each re-targeted PR run before merging the next** (memory: stacked-PR merge traps). + +### [2026-09-16] Decide PR-5 (selective 1h TTL on the static prefix, #1132, flag OFF) — a dev week AFTER the hourly system-prompt tick fix +- **Source**: measured — `backend/scripts/probe_static_prefix_ttl.py` in dev-ai, 2026-09-16, Haiku 4.5. **420 s gap:** Bedrock honors `ttl: "1h"` on the tools+system points — 1h arm second call read 5,924 / wrote 327 (message segment only) vs the 5m arm re-writing all 6,251; pair **12% cheaper**. **60 s gap:** both arms warm; 1h arm **+$0.005157**, exactly the 0.75×-base premium on the first write with nothing to recover. Both recorded in the thresholds spec §6 PR-5 and the probe docstring. +- **Surface**: backend — `core/model_config.py` (`static_prefix_cache_ttl`, `CacheConfig(system_prompt_ttl="1h", tools_ttl="1h")`), `apis/shared/costs/calculator.py` (1h writes billed at 2× base from the context-breakdown static size), the `staticPrefixTtl: "1h"` marker on `C#` rows; the hourly `%H:00` tick in `utils/timezone.py` (separate task chip, in progress). +- **Effort × Impact**: S × M — the code is done; the decision is a flag flip. Arithmetic: surcharge ≈ $0.023 per static write on a 28k prod prefix at Haiku rates ($0.042 Sonnet 5); saving ≈ $0.034 / $0.061 per return landing 5–60 min later; pays when such returns outnumber static writes by > ~0.7 : 1. The audit's 36% cold-re-write share says it might; the probe cannot tell. +- **Subtracts**: no — an experiment arm. Deliberately default-off against the flags-default-on house style (#954/#956: never adopt a caching default on inspection). +- **Status**: open — **do not flip until the hourly tick fix has landed** (it would bust the 1h entry every hour and make the week meaningless). Then `AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h` in dev for a week; compare cache-write dollars per session split on `staticPrefixTtl`, and the write:read ratio. Ship / Decline on that number. + +### [2026-09-16] The 28% "neither" cohort — content-class ordering inside the cut is the next compaction lever, if the metrics say the floor is still missed +- **Source**: 2026-09-15 prod cost audit (content-free): of 95 September sessions over 100k, **52 (55%) had a ≥4k-token tool result in the last 3 turns** (→ #1131 intake offload), **25 (26%) an attachment there** (→ document offload, its own item below), **27 (28%) neither** — long sessions (a3b884b1 46 turns, 4cb93e6e 45 turns) whose bulk is old history plus a 23–40k summary. +- **Surface**: backend — `compaction_policy.choose_checkpoint` (today: oldest tool-pair-safe cut under the floor, whole turns), `CompactionFloorUnreachable` / `CompactionRetainedTokens` on the `AgentCoreStack/Compaction` EMF record, `compactionEvents` on `C#` rows once #1130 lands. +- **Effort × Impact**: M × M +- **Subtracts**: no +- **Status**: open — **watch, don't build yet.** PR-1's cut + PR-2's 8k cap should reach this cohort. If `CompactionFloorUnreachable` stays high after a dev week, the lever is evicting old tool results before old prose inside the retained window (content-class ordering), never a deeper protected-tail cut. Also still unbuilt from the spec's §7.2 data points: **retained-estimate vs next turn's measured `contextBreakdown.messages`** (estimator calibration, one number per cut) and **down-thumb rate keyed to turns-since-last-cut** (the first outcome signal joined to compaction; kaizen has listed this counterweight before). + +### [2026-09-16] Small follow-ups from the compaction stack (fold into any PR, no new epic) +- **Source**: Phil-initiated — thresholds spec §3.6 / §7 "not in PR-N" notes. +- **Surface**: backend — `apis/app_api/admin/costs/diagnoses.py` (`OVER_COMPACTION_THRESHOLD` still compares against the fixed 100k default; should read the persisted `compaction.policy` map's `ceiling`), `core/model_config.py` (`context_window_limit` is a valid `BedrockConfig` key in 1.55 but needs the catalog window at agent construction — `ModelConfig` has no field for it; only affects Strands' own `estimate_utilization` warning today). +- **Effort × Impact**: S × L +- **Subtracts**: no +- **Status**: open — housekeeping; do when touching those files. + +### [2026-09-16] Compaction v2 (#835) go/no-go — re-evaluate once the model-relative stack has a week of data +- **Source**: `docs/specs/compaction-v2-versioned-prefix.md` §7 (gate criteria) vs the stack just built, which implements v2's I3 (hysteresis/floor) and I4 (paid-when-free) as policy on the v1 machinery. +- **Surface**: backend — `TurnBasedSessionManager` vs Strands `SummarizingConversationManager` + `compression/` helpers (the engine swap v2 proposes); upstream `main` is also growing a first-party context-manager/offloading stack (#4254 et al., noted 2026-09-11) that would collide with a hand-built engine. +- **Effort × Impact**: L × M (if go) — the policy layer maps onto v2 unchanged, so the remaining value of v2 is the frozen-segment engine (I1/I2/I6), i.e. removing custody of slice/summary/restore-derivation code. +- **Subtracts**: yes, if go — `_apply_compaction`'s slicing, `_retrieve_session_summaries`, `_prepend_summary_to_first_message`. +- **Status**: open — **defer until the stack's metrics exist.** v2's own go criteria (over-threshold sessions > ~3% of active or > ~15% of spend; or `partial_miss` waste from compaction-state mutation > ~$50/month post-fixes) are now measurable from `CompactionForced`, `CompactionApplied` and the ledger. Decide with numbers, and weigh against adopting upstream's context manager when it ships rather than v2's hand-built engine. + +### [2026-09-16] Document offload epic — attachments are 31% of spend and 26% of the over-100k tail; spawned as a task, with actionable analytics as a first-class deliverable +- **Source**: `docs/specs/document-context-offload.md` (Draft, no branch; 7 PRs, PRs 1–3 a unit) + the 2026-09-15 audit cohort split + `docs/specs/document-conversations-cost.md` (31%). The compaction stack deliberately does not touch attachments — a cut cannot fix them, a digest + page-range read can. +- **Surface**: backend — `document_read` tool gated on the session having an attachment (ids kept out of `INJECTED_TOOL_IDS`), `DocumentDigest` at upload on `FileMetadata`, `_strip_document_bytes` → digest + live handle on restore; analytics: `hasDocuments` / `documentTokens` on `MessageMetadata` (the spec's PR-7) plus whatever else makes the cost-effectiveness of the next change *decidable* — per-session attachment tokens vs. answer quality signal, `document_read` call counts and page volumes, digest-vs-full-document turn shares. +- **Effort × Impact**: L × H +- **Subtracts**: partially — retires the contentless restore placeholder (a correctness bug: a returning user's document is silently lost) and the 14% re-upload rate. +- **Status**: open — task chip spawned 2026-09-16 ("Explore document offloading with actionable cost analytics"). Ask of the exploration: sequence PRs 1–3 together as the correctness fix, pull the analytics (PR-7) forward so the cost work (PR-4) is measured from day one, and align the per-call fields with the compaction ledger (#1130) and `AgentCoreStack/Compaction` so one anatomy page explains a session's cost. + + ### [2026-09-11] ✅ ANSWERED: `bedrock_cache_points_supported()` STAYS — and it is wrong in the opposite direction from the one we suspected - **Source**: measured, not read — `backend/scripts/probe_bedrock_cache_point_support.py` (added by this entry) against the pinned strands-agents 1.55.0 and live `bedrock-runtime` in dev-ai/us-west-2, 2026-09-11. **Answers check (d)** of the [2026-09-11] "Retire the hand-placed system `cachePoint`" entry below, and **narrows (c)** of the [2026-09-11] `ClientSession` monkeypatch entry. Does **not** close either — (a), (b) and the `tools_ttl` mixed-TTL evaluation are untouched. - **Surface**: backend — `core/model_config.py` (`bedrock_cache_points_supported`, the cachePoint-budget comment), `core/agent_factory.py` (the hand-placed system block), `backend/scripts/probe_bedrock_cache_point_support.py`.