[jaredLunde]

watch_applied dropped its snapshot store on the tokio worker wherever the loop exited. fjall 3.1.x's Database::drop can deadlock while joining its worker pool: a blocking send(Close) into a full channel after the last worker has exited. On a single-worker runtime, which is what a CPUQuota=100% service gets, that froze the whole process. In production, instd stopped pulling NATS, logging and handling SIGTERM after a KV-watch heartbeat miss, and never reached the error path that restarts it. gdb on the wedged process showed fjall's thread counter at 0, the channel at 1000/1000, one parked sender and no worker threads. fjall 3.1.12 still has the same Drop.

Change

• The loop now only borrows the fold. On every exit (Ok or Err), the store is dropped on a dedicated thread, and watch_applied waits up to 30s for that close before returning.
• A normal close still finishes before the return, so callers can reopen the same path.
• A close that hangs leaves its thread behind; the runtime keeps going and the caller still gets the watch's error.
• This uses a std thread rather than spawn_blocking, because runtime shutdown waits on blocking tasks and would just move the hang to process exit.
• ARCHITECTURE.md and the watch_applied docs describe the new close behavior.
• Version bumped to 0.8.4.

Tests

• a_store_that_hangs_on_close_cannot_freeze_the_runtime runs on a current-thread runtime. Against the pre-fix code it hangs (killed at 30s); with the fix it passes.
• the_store_is_closed_before_watch_applied_returns checks that a normal close completes before the return.
• Full suite passes, including the NATS integration tests.
• All six stateright models pass. The change only runs after the loop has exited and its final flush has committed, so cursor authority is untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XoaQgogNq1z3dNoM9feGoH