Interrupting a run deletes finished uncommitted results, and can wedge the repo behind a stale index.lock

[EiffL]

An interrupted lc materialize can permanently delete finished results that were never committed, and can leave the repository wedged behind a stale .git/index.lock. Found while stress-testing concurrent execution and task dependencies on Perlmutter (2-node allocation, nid[004157-004158], direct mode).

The concurrency core itself came through clean — the problems are all on the interrupt/cleanup path. I've changed no code; this is a report.

What was tested

A fan-out/fan-in graph, 4 universes × (1 shared upstream → 6 dependents → 1 fan-in) = 32 tasks, run across both nodes. The shared upstream writes an 8 MB payload and its own sha256; every dependent then re-reads that payload 20 times and compares against the recorded digest, failing loudly on any mismatch. That targets the documented race in CLAUDE.md — "a dependent does run while its upstream is being annexed, and that is safe" — by keeping 6 readers hammering the file across the exact window in which the driver's git add swaps it for an annex link.

What held up (worth recording)

• No corrupt or short reads. 24 dependents × 20 re-reads each, all digests matched. The annex-swap-during-read claim holds under real multi-node load.
• Cascade integrity exact. Every dependent's input_versions[<upstream>] equalled that upstream's own recorded data_version.
• Commit scoping clean. 32 [DATALAD RUNCMD] commits, each touching exactly one output directory, despite other workers writing concurrently — git commit committing the whole index never picked up a neighbour.
• HEAD uniform across all 32 manifests; no cross-universe contamination; git annex fsck clean; thin hard links correct (link count 2); a second run did nothing and left HEAD unchanged.
• Failure cascade correct. Making one shared upstream exit 1 mid-run blocked exactly its 7 dependents with precise reasons (upstream did not finish: u3/base), left the other 3 universes untouched, ended with a clean tree, and exited 1.

So: normal operation, including a recipe failure mid-flight, behaves as designed.

Finding 1 — SIGINT destroys completed, uncommitted results (data loss)

The scenario that matters for research. Eight independent outputs, 12 MB each, in a project that had never been materialized. All eight recipes ran to completion and announced their bytes on disk; the driver had committed four of them when I sent a single SIGINT to the driver process.

before interrupt:  8 complete results on disk,  4 committed
after  interrupt:  5 complete results on disk,  4 committed

run_3, run_5, run_7 — directories removed entirely. Three finished, byte-valid results destroyed. lc status afterwards reports them as stale — no manifest — it has never been materialized, which is true and no consolation.

The path is deliberate, not accidental: materialize.py:594-600 restores everything still outstanding in its finally, and dataset.restore (dataset.py:329-345) begins with

_git(["clean", "-qfdx", "--", rel], cwd=directory)

git clean -qfdx deletes untracked files. On a first materialization there is no HEAD version to check back out, so this is pure deletion with no recovery anywhere — not in git, not in the annex.

On a re-materialization the damage is smaller but still real: the outputs revert to the previously committed version, so the old result survives and only the new (equally expensive) work is lost.

Why this is worse than it looks: the compute is unrecoverable. A recipe that costs a week of allocation is deleted by one Ctrl-C, and the run reports Aborted! with no mention that finished results were removed.

Finding 2 — the cleanup can leave a stale .git/index.lock, wedging the repository

In the 32-task run, the same interrupt produced:

Error: `git checkout -q HEAD -- results/u1/fan_1` failed:
fatal: Unable to create '.../.git/index.lock': File exists.

Afterwards the repository held a 0-byte .git/index.lock with no owning git process (checked: no git processes remained), 84 dirty files, and:

• every subsequent git add failed with the same message,
• lc materialize refused (dirty tree), and its remedy — git restore --staged --worktree results/ && git clean -fd results/ — also fails while the lock is present,
• nothing in lc's output mentions index.lock, so the user is left with a repository that appears broken.

Mechanism: SIGINT lands while the driver is inside a git child spawned by dataset.save (dataset.py:270); the finally then immediately runs more git commands against a repository whose index lock has not been released.

Recovery is rm .git/index.lock, which a researcher has no reason to know.

Finding 3 — restore aborts on the first failure, leaving a mixed state

dataset._git (dataset.py:372-377) raises ProjectError on any nonzero exit, and the finally loop at materialize.py:599-600 has no per-task guard. So the first failure (Finding 2's lock) skips the restore of every remaining outstanding output.

The result is a tree in three states at once, which I observed directly:

• some outputs restored,
• some deleted (Finding 1),
• one left staged but uncommitted (A results/base/run_1/... — the interrupt landed between git add and git commit).

Design note, separate from the bugs

Even with the above fixed, the intended behaviour is destructive for expensive artifacts:

• restore's contract is to delete uncommitted output ("wreckage"), and
• the dirty-tree refusal actively advises git clean -fd results/.

That is the right call for a three-second recipe and the wrong one for a week of allocation. A result that finished computing is not wreckage — the run merely failed to record it. Something closer to quarantine (leave the bytes, refuse, and say precisely what is unrecorded and why) would preserve the one thing that cannot be regenerated cheaply, and would make the dirty-tree refusal survivable rather than a prompt to delete real work.

Related invariants this touches, for whoever picks it up: "A run leaves the tree exactly as clean as it found it" and "the driver owns git, alone" in CLAUDE.md — both hold for normal completion and for recipe failure, and neither survives an interrupt.

Reproduction

1. Scaffold a project whose recipe writes a multi-MB file per output, with ~8 independent outputs and no prior materialization.
2. lc materialize inside a SLURM allocation (or locally — nothing here is venue-specific).
3. Once several recipes have finished but the driver is still committing, send SIGINT to the lc materialize process.
4. Compare on-disk outputs before and after; inspect git status and .git/index.lock.

Timing matters only in that the interrupt must land during the commit phase, which on a parallel filesystem is a wide window: 32 outputs took ~30 s of serialized commits, and 8 × 12 MB left four uncommitted at the moment of interrupt.

Environment: Perlmutter compute nodes, direct mode (no Landlock — attested fs: open), lightcone-cli at the nersc-seamless branch head, git-annex 10.20260717, uv 0.12.5.