feat(cache): selective 1h TTL on the static prefix, behind a flag, priced honestly (PR-5) - #1132
Merged
Merged
Conversation
…rics (PR-2) The compaction summary was an unbounded join of AgentCore LTM ConversationSummary records (165k chars / ~40k tokens in the #833 incident): a summary that is 40% of the threshold guarantees compaction can never get back under it. Spiral-spec PR-2; thresholds spec §3.6 / §7.1. - compaction_summary.bound_summary(): hold the persisted summary at COMPACTION_SUMMARY_TOKEN_BUDGET (8,000 tokens, chars/4 — the same estimate the admin SUMMARY_OVER_BUDGET diagnosis uses). Within budget → unchanged. Over budget → one Nova Micro converse call (side-channel, never touches agent.messages) with a prompt that keeps standing instructions, decisions, current state of the work, open items and exact identifiers, and drops narration and superseded drafts. Model failure, a ceiling-hit generation or an overshoot → newest-first truncation (keep the newest records that fit; if none fit, the tail of the newest). Runs once at checkpoint advance — the turn that already pays a prefix re-write — and the result is persisted verbatim, so the byte-stability contract is unchanged. - Kill switch AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ENABLED=false skips the model and truncates; AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ID selects the model. - Provenance on the persisted compaction.policy map: summarySource (ltm|fallback), summaryOutcome, summaryTokensBefore/After, summaryTokenBudget. - One content-free EMF record per cut in AgentCoreStack/Compaction: CompactionCut, CompactionForced, CompactionInputTokens, CompactionRetainedTokens, CompactionSummaryTokens, CompactionSummaryOverBudget, with policySource/window/ceiling/floor/ summaryOutcome as queryable properties. Silenced by PROMPT_CACHE_OBSERVABILITY_ENABLED=false with the rest of the layer. - forced flag narrowed to "ran while disarmed" (same hunk as the PR-1 fix). Tests: newest-first truncation, within-budget passthrough, model compression, model failure / ceiling / overshoot fallbacks, kill switch, env loading, oversized LTM join bounded and persisted through update_after_turn, EMF record shape and kill switch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…te is free (PR-3) Compaction decided WHAT to cut (PR-1) and bounded the summary (PR-2), but a cut still landed at the next restore regardless of whether the prompt-cache prefix was warm, and never landed at all on a warm (cached) agent. Under Bedrock caching a cut costs one re-write of what survives, so the cheapest turn to pay it on is one that was going to re-write anyway. Thresholds spec §3.5. - update_after_turn PARKS the cut: pendingCheckpoint / pendingSummary / pendingHardCeiling / pendingSince on CompactionState. `checkpoint` stays the APPLIED value (what _apply_compaction slices at on restore). A second over-ceiling turn while a cut is parked is a no-op, never a deeper cut. - apply_pending_compaction(agent, prefix_key) runs at the head of every turn (stream coordinator, right after the turn lease is stamped), on cached and freshly restored agents alike. It promotes the pending cut and slices agent.messages IN PLACE (slice assignment, never rebinding — the #741 alias) when, in order: cache_expired (more than cache_ttl_seconds since the previous turn), prefix_changed (model|agent key differs from the persisted lastPrefixKey), or hard_ceiling (previous input reached the hard ceiling the cut was computed under). Otherwise it waits. - The in-place result is byte-identical to what _apply_compaction derives from stored history under the promoted state (pinned by test_live_apply_matches_a_cold_restore_of_the_same_state), so a cold restore after a live apply reads the same prefix. - _adopt_session_conversation copies _live_offset when it points a new agent at the live list — the list's coordinate system travels with it. - Each application persists the reason, cacheGapSeconds and pendingSince on compaction.policy, logs rewrite_scheduled vs rewrite_forced, and emits CompactionApplied / CompactionAppliedForced / CompactionCacheGapSeconds; the cut record gains CompactionDeferred. - Kill switch AGENTCORE_MEMORY_COMPACTION_DEFERRED_APPLY_ENABLED=false applies immediately (PR-1/2 behavior); legacy mode is always immediate. The shared test fixture pins the immediate path explicitly; deferral has its own suite. Not in this PR: context_window_limit on the Strands model config (needs the window at agent construction). Tests: parking, no-deeper-cut, kill switch, legacy; warm/same-prefix waits; cache-expired / prefix-changed / hard-ceiling apply; first-ever key is not a change; live apply == cold restore parity; pending beyond the live list is dropped; re-arm and cut again after apply; metrics with reason; state round-trip; offset sync on alias. Full backend suite: 8659 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The apply is the moment the bytes the model sees change, so it is the event the cost anatomy should mark. Same attribute-resolved helper as the PR-1 events; no-op until the ledger lands. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
promoted=1 distinguishes the pending-cut promotion from the restore slice's own applied event without a schema change; both are real byte changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The case that defeats the compaction floor is a huge tool result inside the
protected tail: the last N turns are kept whole, so no cut can bring the
session back under the ceiling (compaction_floor_unreachable). Cutting
inside the tail would drop the turn the user is working on; the right move
is to keep the reference and drop the bytes. Thresholds spec §3.6.
Built as intake offload rather than cut-time escalation: Strands 1.55's
vended ContextOffloader bounds an oversized tool result on AfterToolCallEvent,
before it is appended to the conversation, so the persisted message already
carries the bounded form and restore reproduces it byte-for-byte — nothing
in the prefix is ever mutated after the fact. Cut-time escalation would have
mutated protected turns and needed a restore-replay of every edit, and the
workspace tools it would have escalated into (workspace_files) are granted
to no prod role.
- core/tool_result_offload.py: BoundedToolResultOffloader (the vended plugin
plus a chars/4 pre-filter so its per-result CountTokens round trip only
runs for results near or over the gate, and a content-free
ToolResultOffloaded EMF record per offload). Storage: unified
strands.storage.S3Storage in the user-files bucket under
compaction-offload/{userId}/{sessionId}/ — one namespaced storage per
agent, so references are session-scoped by construction and an @-mention
agent resolves the same ones. evict_after_cycles=None: eviction from the
model path would turn a retrieval into a miss mid-turn. Gate 4,000 tokens
/ preview 1,000 (AGENTCORE_TOOL_RESULT_OFFLOAD_MAX_TOKENS / _PREVIEW_TOKENS);
AGENTCORE_TOOL_RESULT_OFFLOAD_ENABLED=false removes the plugin. Fail-open
everywhere: no bucket, construction failure, storage or CountTokens error
all keep the original result.
- ChatAgent._create_agent appends the plugin. It registers
retrieve_offloaded_content itself (pattern / line range / full) — one
stable spec in toolConfig, not an RBAC-gated tool, on the read_skill_file
precedent.
- CDK: 90-day lifecycle expiry on the compaction-offload/ prefix of the
user-files bucket (merged into its existing rules; shortest expiration
wins for the prefix). The runtime role already had Put/Get on the bucket.
- Cut record gains CompactionFloorUnreachable — the residual after intake
offload (attachments, sub-gate results).
- User attachments are deliberately not touched: the digest + page-range
read tool in document-context-offload.md is the right shape for those.
- Kaizen review-queue [2026-07-19] ContextOffloader spike closed as adopted.
Tests: flag / no-bucket / per-session prefix / config / preview clamp;
small result skips CountTokens; borderline result is measured; oversized
result is offloaded with preview + reference and a content-free metric;
storage and CountTokens failures keep the original; ChatAgent wiring with
and without a bucket. Full backend suite: 8670 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…iced honestly (PR-5) The 2026-07-27 model found a blanket 1h prompt-cache TTL a wash (the 2x write premium ate the saving) and a SELECTIVE one — 1h on the tools and system points, 5m on the messages point — the variant worth testing. The 2026-09-15 audit says 36% of cache-write dollars are cold re-writes after a >5 min pause, and every one of those re-writes the ~28k static prefix that never changed. Thresholds spec §3.6. Built as a measured experiment arm, default OFF — deliberately against the flags-default-on house style: a caching default adopted on inspection alone has already shipped wrong once (#954 measured 57% more expensive live before #956 reverted it), and this one is a bet on the gap distribution. - AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h → ModelConfig.to_bedrock_config emits CacheConfig(system_prompt_ttl="1h", tools_ttl="1h"). Upstream rewrites the hand-placed TTL-less system point ("honored as written") and gives the tools point its own; cache_config.ttl stays unset so the auto message point stays at 5m: tools(1h) → system(1h) → messages(5m), the non-increasing order Bedrock requires. Anything but the literal "1h" emits exactly today's bytes; non-Anthropic models and caching-off are unaffected. - The experiment's own cost rows are honest: Bedrock bills a 1h write at 2x base, not the catalog's 1.25x, and usage does not split writes by TTL. CostCalculator.calculate_message_cost(long_ttl_static_prefix_tokens=...) bills the unread remainder of the static segment at 2x and the rest of the write at the 5m rate; the coordinator sizes the static segment from the context-attribution breakdown (system + tools) and marks every such row staticPrefixTtl: "1h" so the anatomy can split arms. - scripts/probe_static_prefix_ttl.py is the gate: two arms against the same static prefix with a configurable gap; reports read/write per call, prices the pair at the model's rates, and flags a model that does not honor the 1h point. Enable in dev only after it and a week of cost rows say it pays. Tests: flag on → 1h on tools and system tails, bare message point, still exactly three points; every other value → today's bytes; non-Anthropic and caching-off unaffected; calculator: cold-everything, static-read, partially-read, write-smaller-than-static, no-write; coordinator static-size helper. Full backend suite: 8713 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The second call rebuilt the request with a new marker, which changed the system text and would have forced a cache miss in both arms — the probe would have reported the 1h arm as not honored regardless of the truth. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bedrock honors ttl=1h on the tools and system points for Haiku 4.5: after the 5m entry expired the 1h arm read 5,924 of 6,251 static tokens and re-wrote only the 327-token message segment; the 5m arm re-wrote all 6,251. Pair cost 12% lower at a 420s gap. The 60s gap and the dev-week measurement (after the hourly system-prompt tick fix) remain the gate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Both arms read fully inside five minutes; the 1h arm is a fixed 0.75x-base surcharge per static write ($0.005157 on a 6.3k prefix) with nothing to recover. Together with the 420s run this pins the arm's economics: pays when returns between five and sixty minutes outnumber static writes by more than ~0.7:1. The dev week decides that; the probe cannot. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
philmerrell
changed the base branch from
feature/compaction-offload-escalation
to
develop
September 16, 2026 14:46
philmerrell
added a commit
that referenced
this pull request
Sep 16, 2026
Resolves the one integration conflict with the compaction stack (#1125, #1128, #1129, #1131, #1132) in ``update_after_turn``. Both sides append to the same post-cut block: the stack added ``_emit_compaction_metrics`` and refactored to a local ``state`` alias (``state = self.compaction_state``, so the two save calls were already equivalent); this branch added the ledger's ``checkpoint`` event. Keep both, and route the event through the stack's ``_record_ledger_event`` seam instead of calling ``record_compaction_event`` directly, so the recorder stays resolved-by-attribute like every other cut decision. ``test_no_ledger_is_a_noop`` asserted the recorder was *absent* — true only while this branch was unmerged. It now simulates a ledger-less build via ``monkeypatch.delattr`` so it still guards the getattr seam. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1131 (PR-4) → #1129 → #1128 → #1125. Merge in order; this PR's diff is the one commit on top.
Why
Every cold re-write after a pause re-writes the ~28k-token tools + system prefix that never changed. The 2026-09-15 audit put cold re-writes after a >5 min pause at 36% of cache-write dollars. The 2026-07-27 model found a blanket 1h TTL a wash (the 2× write premium ate the saving) and the selective variant — 1h on the static points, 5m on the messages point — the one worth testing. Thresholds spec §3.6.
Default OFF, on purpose
This goes against the flags-default-on house style, and the reason is the repo's own history: #954 adopted a caching default on inspection alone and measured 57% more expensive live before #956 reverted it. A 1h write costs 2× base against 1.25× at 5m, so whether this pays depends on how often sessions return between five and sixty minutes. That is measured, not read off the source.
Gate to enable in dev:
backend/scripts/probe_static_prefix_ttl.py(two arms against the same static prefix,--gap-seconds 420and60), then a week of cost rows with the flag on in dev, split on the row marker below.What
AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h→ModelConfig.to_bedrock_configemitsCacheConfig(system_prompt_ttl="1h", tools_ttl="1h"). Upstream rewrites our hand-placed TTL-less system point ("honored as written") and gives the tools point its own;cache_config.ttlstays unset so the auto message point stays at 5m. Order tools(1h) → system(1h) → messages(5m) is the non-increasing order Bedrock requires. Anything but the literal1hemits exactly today's bytes; non-Anthropic models and caching-off are unaffected. Still exactly three cache points.cacheWritePricePerMtokis the 1.25× 5m rate, and usage does not split writes by TTL.CostCalculator.calculate_message_costnow takes the static segment size and bills its unread remainder at 2× (static that was read was not written) and the rest of the write at the 5m rate. The coordinator sizes the static segment from the context-attribution breakdown and marks every such cost rowstaticPrefixTtl: "1h".scripts/probe_static_prefix_ttl.py: the live gate. Prices each arm's pair at the model's rates and flags a model that does not honor the 1h point.Expected economics
+0.75× base on every static-prefix write; −1.15× base on every return inside the hour but past five minutes. It pays when the second event is more frequent than the first. Note the hourly system-prompt tick (being fixed separately) would bust the 1h entry every hour until it lands, which is another reason not to enable this yet.
Tests
Flag on: 1h on the tools and system tails, bare message point, still three points. Every other value: today's bytes. Non-Anthropic and caching-off unaffected. Calculator: cold-everything, static-read, partially-read, write-smaller-than-static, no-write. Coordinator static-size helper. Full backend suite: 8713 passed, 3 skipped.
🤖 Generated with Claude Code