Sprite storage wedges while VM stays running: all disk I/O hangs, exec unusable, /checkpoints and /services hang forever — twice in 8 days #230

Symptom

A sprite's disk I/O wedges permanently below the filesystem while the VM stays running. Everything that touches the overlay hangs forever; the supervisor keeps answering HTTP and refreshing its lease, so from the outside the sprite looks healthy while being completely unusable. The only remedy we have found is sprite destroy, i.e. total loss of the disk.

This has now happened twice in 8 days to the same workload, on two different sprites with different volumes:

• 2026-08-24 (supervisor 0.0.1-rc47), org junchao-yan, region sjc. Resolved by sprite destroy at ~14:20 UTC.
• 2026-09-01 (supervisor 0.0.1-rc48), org junchao-yan, sprite buzz-agent-276883e88d5a, id sprite-1a3bd078-3c45-4637-8b95-dc320b271ebd, created 2026-08-27. Wedge onset between 13:05 and 17:33 UTC. This sprite is in the wedged state right now — tell us if you want it preserved for inspection, otherwise we will destroy it to restore service.

Observed state (both occurrences)

• GET /v1/sprites/{name} → status steady running (no cold/warm flapping); lease refreshes succeed on schedule (refreshCount advancing).
• GET /v1/sprites/{name}/exec (session list) answers instantly.
• GET /v1/sprites/{name}/checkpoints and GET /v1/sprites/{name}/services hang forever (0 bytes, >25 s, repeatable).
• Supervisor /logs shows each incoming exec request and then nothing — no session start, no error, no container event.

Exec behavior differed by page-cache warmth:

• 08-24 (VM up ~8 h, cache warm): exec sessions were created, bash spawned, builtins worked — but any command touching a cold file entered uninterruptible D-state, immune to the supervisor's SIGTERM→SIGKILL. Even cat of a tiny file never returned. ~27 zombie exec sessions accumulated from client retries.
• 09-01 (VM re-entered running at 17:33 UTC, cache cold): even bash cannot spawn — no exec session is created at all; the client gets Error: connection closed after ~45 s (gateway timeout). SDK callers see their own budgets expire (exec did not finish within 30s).

Storage-layer evidence from sibling sprites, same org

• A sibling sprite logged a burst of JuiceFS staging errors during its suspend flush (2026-08-27 02:57 UTC): Open staging file /dev/fly_vol/juicefs/cache/<uuid>/rawstaging/chunks/0/57/…: invalid file size 0, data length <N> (cached_store.go:1038, uploadStagingFile) — dozens of chunks, each with a zero-length staging file that should have held data.
• A third incident on 2026-08-22 (different sprite, same org) was overlay corruption presenting as a container crash-loop: dumpe2fs on the preserved snapshot showed first error __ext4_find_entry err=EIO (a failed read from the loop image) followed by 75 EFSBADCRC directory-checksum errors — storage failing under the filesystem.

Workload correlate (possible repro direction)

The affected agent workload is fsync-heavy small writes: a coding-agent CLI journaling continuously to a SQLite database on the overlay (measured ~10k rows / 5.5 min), plus frequent suspend/resume cycles (~15 auto-checkpoints/day), each running the checkpoint-clone + staging-flush path. The wedge has followed this workload across two different sprites/volumes; sibling sprites without the SQLite churn have not wedged. This looks adjacent to #229 (sustained heavy I/O → overlay unmount + SIGKILL) but the failure mode here is a silent permanent hang, not a kill.

What we need

1. Root cause on the staging/overlay write path (rawstaging chunks with invalid file size 0 looks like the smoking gun).
2. Detection: a sprite in this state reports running and holds its lease, so nothing platform-side notices it is dead.
3. A disk-preserving recovery path — filed separately as a companion issue, since today the only exit is destroy.