Bound the session drain, fix the handoff regressions, and add the ownership seam

[matej21]

What this is

The four commits already on this branch make the agent runtime survive a host
handoff. This PR fixes what they broke, and finishes the seams a host driver
actually has to plug into — a pool of pods where one dies and another takes over,
and where SIGTERM means drain within a budget.

Scope is deliberately the SDK side. A Postgres-backed event store is not here.

The blocking bug

beforeToolCall returning { action: 'replace' } with a new id made
executeToolCall emit tool_started/tool_completed under it, so the reducer
never cleared the original entry — and the rewritten tool_exec case re-enters
while anything stays pending. The agent re-ran the tool forever, appending two
events per iteration.

The id is the LLM's tool_use id and the result has to pair with it, so a hook
may now swap name and input but not identity. The loop also guards on head
progress, so a future path that leaves the head pending ends the turn instead of
spinning.

Draining

Nothing bounded a drain, and one settled failure refused it forever. Neither is
usable from a host with a shutdown budget.

Bounds. The worker effect drain lost the bound the stop path still had, so a
stalled append held the runtime lease and onSessionClose('parked') waited on it
without limit. Restored, and the park wait is bounded too — stragglers are
stopped so their state parks and the next host resumes them. parkSession takes
timeoutMs, which bounds the caller's wait without cancelling the drain.

The one that was not obvious: appends became serialised in SessionStore, which
is right — state has to follow the log — but it made one stalled write block
every later one, including the session_closed that close() has to write
first. close() could not return at all. writeQueueTimeoutMs now bounds each
append once it reaches the head of the queue, so a long queue of healthy writes
never trips it. An append that does not settle in time has an unknown outcome
and fences the store, because nothing may be ordered behind a write that may
still land. Writes queued behind it never reached the store, so they are refused
with EventAppendError, the fence as its cause. Hosts set it through
SessionManagerOptions, createSystem or SESSION_WRITE_QUEUE_TIMEOUT_MS.

A fenced runtime stops. A fenced store accepts no write, so a resident
runtime behind it was dead until something parked or disposed it — one transient
error from a custom store was enough. Every fence now revokes the runtime and the
manager drops it; the next access reloads from the log, which decides whether
the uncertain write landed. Until that write settles, the id refuses access as
unloading.

Retries. Five accumulators recorded failures and never cleared them: the
store failure, the session and agent scheduler errors, the service publication
failures. One recoverable append error or one failed wake downgraded every later
handoff of that runtime from park to hard revoke, for as long as the pod kept it
resident. They are reported once now, and park stays retryable after reporting —
the runtime never returns to ready, so it still admits no new work. A park that
found the runtime already gone stays terminal.

The seam that was missing

Fencing only worked against a host somebody had already revoked. Nobody can
revoke a host that is merely unreachable — that is what a partition is — so a
stale owner kept writing and kept retrying. The only signal it can ever get is
its own refused append.

SessionOwnershipLostError is that seam: a store bound to a host's lease throws
it once the lease moved on. It is a definite noncommit that also stops the
runtime — the store fences, the session revokes itself rather than parking
(a replacement owns the log, so there is nothing to drain), and the manager drops
the residency and marks the tenure lost. Reloading would replay the log and
restart services and workers only to fence again on the first write, so later
access on that host is refused with session_ownership_lost, and parkSession
rejects with SessionOwnershipLostError, until the host calls activateSession
again.

The epoch itself stays where it belongs: the host builds an EventStore bound to
its lease and passes it in. The SDK does not decide ownership.

Leak

tenure.entry exists to refuse re-admission while a disposed runtime still holds
resources, and nothing ever cleared it. Tenures are never pruned and
admission() creates one on first access, so every session id the process
touched kept its store, agents and plugin contexts reachable for the life of the
process. Cleared once the runtime is safe: Session.whenSafe() settles after
pending writes, the scheduler tail, tracked resources and local cleanup, so a
revoke during a stalled append releases the runtime once the append settles. getRuntimeCacheStats().retainedRuntimeCount makes
it watchable. The tombstone itself stays — forgetting it would let the next
access silently re-own a parked session.

Error channel

Scoping turned two things that could not fail into things that do. notify is
ephemeral and never persisted, and its callers are exactly the ones that cannot
handle a throw — an interval, a detached promise, a callback that outlived its
method; a late notification is dropped. RuntimeFileStore reads return Result
and threw a lease error before returning one; only a write can land behind a
replacement runtime, so reads pass through.

Event store

The branch first moved the file store to one immutable batch file per append.
That fixed two real defects of the JSONL log on main: a torn last line made
the whole session unloadable, and a partly written append was reported as a
definite noncommit, so a retry duplicated it. It also brought one file per
append with no compaction, and a listing that replayed every session's full log
at once. This PR returns to a single events.jsonl and fixes the two defects
directly:

• A line counts as committed once its newline is written. A trailing line
  without one is ignored by reads and dropped by the next append, which swaps in
  the log without it through a rename.
• A failed append is rolled back to the last committed line, so it definitely
  did not commit. When the rollback fails too, the outcome is unknown, the
  metadata is marked stale, and the next load decides it.
• Loading a session refreshes where the next append starts. A rollback that
  finds lines it did not write reports an unknown outcome and truncates nothing,
  so a host that takes a session back cannot delete what another host committed.

Listing reads only meta.json again, never an event log. meta.json is derived
and may lag the log: a failed metadata write no longer fails a committed append,
and the counters are rebuilt from the log on the next append, range read or load.
The on-disk format is main's, so older SDKs still read it.

Review follow-ups

• A lookup of a session id that has no session no longer leaves a tenure behind.
  Only a tenure that first access created is dropped; one a host activated stays,
  so a revoked id cannot be re-admitted. getRuntimeCacheStats().tenureCount
  makes the map watchable.
• The closed-session guard now throws ClosedSessionAppendError, a definite
  noncommit, so one refused hook event no longer fences every later write,
  session_reopened included.
• uploadAsync reports an upload that is already committed as processing when
  the runtime cannot start it, as parking already did. The next load starts it;
  returning an error made a retrying client upload it twice.
• Rebased onto main, which added context to user-chat.sendMessage. The
  delivery fingerprint covers it, so a retry with a different context conflicts.

Second review

• uploadAsync wrote pendingStart before its processing event. When that
  append definitely did not commit, the next load started the upload anyway and
  a retrying client got two attachments. The upload is now discarded.
• A worker reducer that threw after its sub-event committed left an event that
  failed every replay. The reducer now runs as a check before the append and
  again from current state after it.
• A failed service status publication made a revoked id unsafe forever, although
  the hook had already released the processes. A close hook now throws only when
  something is still held; the services hook re-throws a publication failure
  only on park, where it is retried.
• deliveryId is limited to 256 characters.
• The README states the event store error contract: EventAppendError is a
  definite noncommit, SessionOwnershipLostError stops the runtime, anything
  else is an unknown outcome and fences.

Smaller

performDisposal released its teardown lease as a plain statement, so a
rejecting agent drain skipped the release, the detach and markDisposed —
leaving activeCount above zero and every later waitForIdle hanging. A
worker's emit applied the reducer only while the runtime still ran, so an
append that committed during teardown left emit() resolving on a state behind
the log. callPluginMethod awaited the session-wide scheduler tail.

User-chat keeps delivery receipts for the session lifetime, including in the
replayed projection, so a retry dedupes exactly however late it arrives. Memory
grows with the number of distinct delivery IDs.

Deliberately not done

The fence classification stays as it was, but it is breaking for store
authors. An append slower than writeQueueTimeoutMs now counts as an unknown
outcome. It was suggested that only
EventAppendOutcomeUnknownError and EventLogCorruptionError should fence. They
should not: an unclassified failure from a custom store may still have landed, so
fencing unless the store said otherwise is the safe default, and
session-store.test.ts already documents it.

Terminal service status during park. A service that exits while the runtime
is parking cannot publish service_status_changed(stopped), so a replaying host
sees it as running. Publishing it without a lease does not fix this — the
unscoped path is gated by the same tryOperation and fails identically. It needs
either publishing before writes stop or reconciling on load. Documented in the
README as a known gap.

A single writer is still not enforced. The store caches where the next
append starts, so two stores over one directory interleave or cut each other's
lines, and a torn-tail repair can delete what the other committed. Loading
refreshes the cache, so a host that takes a session back must load before
appending. The README says so; a host that runs more than one instance fences
externally through SessionOwnershipLostError. The file store's per-session
caches are not pruned either, because the EventStore interface has no release
hook.

An upload whose owner was lost mid-commit. A revoked runtime cannot write,
so an upload whose processing event was refused by ownership loss cannot
discard itself, and the new owner may still start it.

A failed revoke cleanup. When a close hook reports resources still held after
a revoke, nothing retries it, and the id stays refused on that host until the
process restarts.

Snapshots. Taking over a session still replays its full log. That belongs
with the store that will serve a pool.

Stale service publication failures. A status publication that failed is
reported by the next stopService or park, not by the call that caused it.

An upload revoked mid-commit. If the runtime is revoked while
attachment_uploaded is being written, the event can land and the method still
reports an error. The runtime cannot tell whether the write landed before it was
detached.

Verification

bun run ts:build, bun run lint, and bun test packages/*/src packages/*/tests
— 1747 pass, 0 fail.

Each fix has a test that fails without it; for the review follow-ups and the
JSONL rollback and tail repair this was checked by removing the fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_015oMWoa8H4wY3Q1JLeqkkvs
https://claude.ai/code/session_01YKJFaSJo72tboWbPzjECzB