Keep chat Redis data short-lived and schema-light

[sentry-junior]

Redis chat/runtime records have nested shapes and multi-day TTLs, so schema changes hang around after deploy. SQL remains the durable history authority. Redis should only hold bounded operational state that is safe to expire. OAuth tokens/sessions are out of scope.

Goals

1. Redis schema changes must not require multi-day compatibility code.
2. Durable conversation history and metadata have one authority in SQL.
3. Redis holds only bounded operational state that is safe to expire.
4. Terminal state is deleted quickly or kept only for a short diagnostic grace period.
5. Redis records avoid embedding shared nested plugin/API contracts when stable ids or an authoritative lookup suffice.

A mailbox or idempotency record can still be authoritative in Redis while work is in flight. The test is whether loss/expiry is an accepted operational outcome—not whether the data is "history."

Exit criteria

• No chat Redis record uses the chat-sdk 30d TTL.
• Every key family documents authority, loss behavior, active TTL, and terminal TTL/deletion.
• SQL-owned conversation metadata is not duplicated into Redis except during a named rollout.
• Redis schemas do not store full Source / Destination / actor contracts unless that store explicitly owns the bounded operational record.
• Dashboard/reporting reads SQL, not Redis indexes.
• Compatibility code has a release/date removal condition and an owning TODO.
• User-authored configuration and user-visible context do not silently disappear because a scratch TTL elapsed.
• Tests prove terminal cleanup and authoritative reconstruction without legacy Redis fields.

Cutover rule

1. Deploy readers that use the canonical authority (usually SQL).
2. Dual-write only while old readers may still be running.
3. Once canonical readers are deployed, remove the Redis write path and obsolete schema fields immediately. Do not wait for old Redis TTLs.
4. Put cleanup TODOs beside temporary dual-write/schema code and reference this tracker (#1267).

Done

Turn sessions — complete

Resume lease over SQL history. Nested routing/actors and ops metadata are out of Redis; TTLs are short; dashboard indexes are gone.

• #1268 — resume routing from SQL only
• #1286 — stop nested source/destination writes; explicit strip schemas
• #1298 — stop actor writes
• #1308 — 24h unfinished / 1h terminal; drop runtimeContext on terminal
• #1313 — thin recovery indexes; remove global dashboard list
• #1315 — remove ops metadata from Redis records
• #1266 closed without merge once Redis no longer owned nested source

Leftover: TODO(#1267) in packages/junior/src/chat/state/turn-session.ts for the temporary caller-to-SQL runtimeMetadata write. Resolve or re-home with a clear owner.

Thread/channel scratch — partial

• #1320 — Junior 7d TTL instead of chat-sdk 30d
• #1323 — drop rebuildable conversation stats/backfill from Redis

Still in Redis scratch today: processing/pendingAuth, vision cache, artifacts, sandbox ref, channel configuration.

Remaining

Decide authority and loss behavior before thinning each store. Do not move fields to SQL only to satisfy the slogan, and do not keep accidental durability in Redis.

1. Finish classifying thread/channel scratch

Treat each group separately:

• Processing / pending auth — valid ephemeral control. Keep in Redis; bind retention to the active pause/turn window where practical.
• Sandbox ref — ephemeral runtime linkage. Keep in Redis; TTL should match sandbox resumability.
• Vision summaries — expensive rebuildable cache. Keep as an explicitly versioned cache, or give summaries a durable home; do not relocate just for consistency.
• Artifacts — user-visible conversation context across turns. 7d expiry may be product-visible loss. Needs a durable conversation-owned projection, or an explicit cache-only decision.
• Channel configuration — durable user configuration, not scratch. Move off 7d Redis TTL.

2. Thin the conversation work mailbox

junior:conversation:* still mixes in-flight mailbox/control with duplicated durable conversation metadata (destination, actor, channel name, source, title) under a 7d TTL.

• Remove schema-v1 mailbox compatibility (TODO(v0.110.0)) once aged out
• Keep mailbox entries, execution status/lease, dedupe ids, recovery cursors
• Stop mirroring SQL-owned conversation metadata
• Define terminal/idle deletion or short grace instead of refreshing whole records for 7d
• Decision: must pending messages survive Redis loss? If yes, Redis cannot be their sole authority.

3. Redesign agent dispatch deliberately

junior:agent_dispatch:* still stores strict 7d records embedding actor, credential subject, destination, visibility, source, input, metadata, attribution, and status, plus legacy rollout fields through v0.116.0.

Before coding:

• Is a dispatch a durable product/API object whose idempotency/status must survive Redis loss or a >7d delay? If yes, SQL is canonical and Redis keeps locks/wake indexes only.
• If it is bounded operational state, document the 7d expiry contract and thin nested routing/authority via lookups where possible.
• Remove legacy execution fields and key/input compatibility after the stated release window.
• Split terminal retention from active retention.

Working order

1. Close turn-session rollout cleanup; mark turn sessions done.
2. Move channel configuration to durable storage.
3. Decide durable vs cache semantics for artifacts and vision; implement each separately.
4. Tighten processing/sandbox TTLs to real lease windows.
5. Thin mailbox metadata; remove schema-v1 compatibility.
6. Decide whether pending mailbox work must survive Redis loss; move only if yes.
7. Define dispatch durability/idempotency guarantees; SQL-canonicalize or explicitly bound/thin the Redis record.
8. Add terminal deletion/grace and remove expired compatibility code for mailbox and dispatch.

Out of scope

• OAuth user tokens and auth sessions
• SQL conversation history / events (already durable authority)

[sentry-junior]

Inventory: agent turn sessions (item 1)

Owner: packages/junior/src/chat/state/turn-session.ts (+ turn-session-keys.ts)

Keys / TTL

• Record: junior:agent_turn_session:{conversationId}:{sessionId}
• Global summary list: junior:agent_turn_session:index (cap 5_000)
• Per-conversation summary list: junior:agent_turn_session:conversation:{conversationId}:index
• TTL: 30d via chat-sdk THREAD_STATE_TTL_MS (AGENT_TURN_SESSION_TTL_MS = THREAD_STATE_TTL_MS)
• Not junior’s 7d JUNIOR_THREAD_STATE_TTL_MS

What redis stores

Full record (storedAgentTurnSessionRecordSchema, .strict()):

• Identity / lifecycle: conversationId, sessionId, sliceId, version, state, startedAtMs, lastProgressAtMs, updatedAtMs, surface, traceId, resumeReason, resumedFromSliceId, errorMessage
• Resume cursors into SQL: committedSeq, historyVersion?, turnStartSeq?
• Nested platform contracts: optional source (sourceSchema), destination (destinationSchema), actor / actors (actorSchema)
• Dispatch linkage: dispatchId?, dispatchOutcome?, resultMessageId?
• Ops/meta: channelName?, modelId?, reasoningLevel?, loadedSkillNames?, cumulativeDurationMs, cumulativeUsage?
• Volatile bootstrap only in redis: runtimeContext?: PiMessage[] (AGENTS.md / <runtime-turn-context> parts retained for unfinished resumes; not full transcript)

Summary indexes store the same summary fields (no committedSeq / runtimeContext / actors list), also strict-parsed.

On read, piMessages are not taken from redis — they are re-projected from SQL via committedSeq (+ optional runtimeContext restore while running / awaiting_resume).

Readers / writers (schema-coupled)

• Write/update: upsertAgentTurnSessionRecord, recordAgentTurnSessionSummary, updateAgentTurnSessionState
• Resume path: getAgentTurnSessionRecordForResume → agent-continue-runner, slack resume, oauth callbacks, reply-executor
• Dashboard/ops lists: listAgentTurnSessionSummaries*
• Also: agent-dispatch work path links dispatch ↔ turn session

Resume requires stored nested source / destination / actor today (agent-continue-runner fails closed if source missing).

Why shape changes hang around

1. 30d TTL — longest chat hangover in scope
2. Strict zod on record + summary — bad old shape → parse throw / skipped summary (resume can hard-fail; list warns and drops entry)
3. Shared nested contracts (source / destination / actor) live in redis, not just ids — same class as open temporary compat #1266
4. Duplicate shapes — full record key + two summary lists; list appends refresh list TTL independently of record rewrite patterns
5. SQL already owns durable pi history; redis is denser than “cursor + ids” for resume

Density vs need (inventory only)

Field group	Needed for short-lived resume?	Notes
committedSeq / historyVersion / turnStartSeq / status / slice	yes	true ephemeral resume metadata
runtimeContext	yes while unfinished	intentionally volatile; should die with short TTL
source / destination / actor full objects	currently yes (resume rebuilds routing from them)	highest schema-churn cost; couple redis to plugin-api contracts
usage / model / skills / channelName	ops/dashboard	could be sql or derived; not resume-critical
summary list copies of nested contracts	list/dashboard	multiplies hangover surface

Related

• Temporary source visibility coerce: #1266 (open; not a storage redesign)
• Tracker working order continues with thread/channel scratch next

[sentry-junior]

Analysis: turn sessions — ephemeral vs not, and what to improve

Lens: what must survive a pause, what can be rebuilt from SQL/other authority, what only exists because redis is currently the resume bag.

Already authoritative elsewhere (rebuildable)

Redis field	Rebuild from	Confidence
piMessages / provenance	SQL junior_conversation_events via committedSeq	high — already done on every read
destination	SQL junior_conversations.destination / destination row	high — written on every summary via recordActivity
source (session-stable parts)	SQL sessionSource (source_json)	high for locator; medium for per-message messageTs
actor (slack user)	SQL actor identity + conversation events / user message author	medium-high — continue path already re-resolves slack actor when possible
actors[]	instruction provenance on committed events	high — materialize path already falls back to provenance
channelName, usage, duration	SQL conversation aggregates (recordExecution)	high
modelId / skills / reasoning	turn events / live turn context if still needed	medium — ops/debug, not resume-critical
global + per-conversation summary lists	SQL conversation/execution rows + events	high for dashboard/ops; lists are a redis-shaped index, not a truth source

Truly ephemeral (cannot honestly rebuild today)

These are the only turn-session bits that are ops state for an in-flight run, not history:

Field	Why it’s ephemeral	If lost
state + resumeReason	lease/control for pause/continue	cannot know whether to continue vs ignore
version	optimistic concurrency for continue payloads	stale continues can double-fire or no-op wrong
sliceId / resumedFromSliceId	slice accounting / limits	weak rebuild; wrong slice breaks limits/metrics
committedSeq + historyVersion + turnStartSeq	exact resume cursor into SQL	without them, “continue” becomes “guess last boundary”
runtimeContext	volatile bootstrap (AGENTS.md / runtime-turn-context), never SQL	unfinished resume loses bootstrap; finished runs don’t need it
dispatchId / resultMessageId / dispatchOutcome	cross-store linkage for this run	partially recoverable from dispatch store / events, but not free today

Important: even the ephemeral set is only needed while the turn is non-terminal (running / awaiting_resume). After completed / failed / abandoned, redis is mostly a 30d ops cache of things SQL already got.

Pretends to be ephemeral but is treated as durable bag

Resume currently requires nested redis source (and uses redis destination / actor) even though SQL already holds conversation-level copies:

• agent-continue-runner fails closed if activeSessionRecord.source is missing
• getAwaitingAgentContinueRequest requires sessionRecord.destination
• queue wakeups only carry conversationId + sessionId + expectedVersion + destination on the continue request — so redis (or a rebuild step) must rehydrate routing

That is the core design smell: short-lived control plane fields are co-located with long-lived nested platform contracts, then given a 30d TTL and strict parse.

What “good” looks like for turn sessions

Keep in redis (short TTL, hours / low days, only while unfinished):

• identity: conversationId, sessionId
• control: state, resumeReason, version, sliceId
• cursors: committedSeq, historyVersion, turnStartSeq
• volatile: runtimeContext (drop on terminal)
• thin linkage ids only if still needed: dispatchId?, resultMessageId?

Do not keep as nested redis contracts:

• full source / destination / actor / actors — rebuild from SQL conversation + events at resume
• usage / model / skills / channelName — SQL only
• summary list copies of the above — delete or replace with SQL queries

TTL policy:

• active/unfinished: short (bounded by real resume window, not dashboard nostalgia)
• terminal: delete or tiny grace; do not keep 30d “history” in redis
• stop aliasing to chat-sdk THREAD_STATE_TTL_MS (30d)

Parse policy:

• if anything nested remains during transition: coerce/heal (like fix(plugin-api): Coerce legacy source visibility on read #1266), never .strict() fail-closed on resume for additive field drift
• long-term: schema so small that churn is boring

Improvement order (turn sessions only)

1. Resume rebuild path — load destination/sessionSource/actor from SQL conversation; stop requiring redis source
2. Shrink stored record to cursors + control + runtimeContext; drop nested contracts from writes
3. TTL split — short while unfinished; purge/skip rewrite on terminal (or explicit delete)
4. Retire summary indexes (or stop embedding nested contracts in them); dashboard reads SQL
5. Remove temporary source coerce once redis no longer carries source (fix(plugin-api): Coerce legacy source visibility on read #1266 becomes unnecessary for this store)

Non-goals / traps

• Moving full turn sessions into SQL as a new durable history table — fights the “redis ephemeral / SQL history” split and recreates hangover in postgres JSON
• Keeping 30d redis “for the dashboard” — that’s what junior_conversations + events are for
• Dual-writing fat objects “for a while” without shrinking readers — extends the hangover

Bottom line

Turn sessions should be a small resume lease over SQL history, not a 30-day mirror of routing + ops metadata. Most of the hangover surface is duplicated, rebuildable data; the irreducible ephemeral core is cursors + control + short-lived runtimeContext.

[sentry-junior]

Slice 1 in progress: resume rebuild from SQL

Implementing the first turn-session improvement from the analysis:

• Prefer turn-session redis source / destination when present
• If missing, rebuild from SQL junior_conversations.sessionSource / destination
• Unblock resume paths that currently fail closed on missing redis nested contracts

Out of this slice: shrinking writes, TTL split, dropping summary indexes, stop writing nested contracts.

Tracked call sites:

• getAwaitingAgentContinueRequest
• continueSlackAgentRun / stranded recovery
• OAuth + MCP OAuth resume paths

[sentry-junior]

Slice 1 PR

Draft PR: #1268

Implemented resume rebuild from SQL for missing turn-session source/destination:

• helper: resolveTurnSessionRouting
• wired into automatic continuation, stranded recovery, OAuth + MCP OAuth resume
• unit coverage for prefer-redis / fill-from-sql / partial rebuild
• component coverage for SQL rebuild success + still-fail when both missing

Still later slices: shrink writes, TTL split, drop summary-list fat copies, stop writing nested contracts.

[sentry-junior]

Slice 1 update: SQL is the only routing authority

PR #1268 now always loads destination + session source from junior_conversations on resume. No redis-prefer / SQL-fallback dual path.

Still later: stop writing nested contracts into turn-session redis, TTL split, drop fat summary indexes.