docs: demonstrate cooperating-worker admission and cleanup

CHANGELOG.md

@@ -4,6 +4,10 @@ All notable changes to this project are recorded here. The format follows Keep a
 
 ## [Unreleased]
 
+### Added
+
+- A runnable cooperating-worker example (#40) acquires a path set in the mutation launcher, shows holder/note contention, allows unrelated work, and demonstrates renewal, superseded cleanup, failure cleanup and non-renewing TTL expiry. JSON receipts and behavior tests cover the golden path, an existing-output edge, and two concurrent isolated runs. The runbook distinguishes these controlled flows from unresolved #45 coherence work and defines external adoption validation as an unrun experiment.
+
 ## [0.7.0] - 2026-09-16
 
 ### Added

Makefile

@@ -1,6 +1,6 @@
 SHELL := /usr/bin/env bash
 # lib/*.sh are fragments of one script and only lint as the whole they build into (bin/git-locks).
-SCRIPTS := bin/git-locks test/test.sh scripts/hooks/pre-commit scripts/hooks/pre-push scripts/build.sh
+SCRIPTS := bin/git-locks test/test.sh examples/cooperating-workers/demo.sh examples/cooperating-workers/worker.sh scripts/hooks/pre-commit scripts/hooks/pre-push scripts/build.sh
 PREFIX ?= $(HOME)/.local
 
 .PHONY: build lint test test-docker install uninstall

README.md

@@ -216,6 +216,10 @@ When a claim finds an expired lock on a path it wants, it evicts the whole expir
 
 In summary, expiry is a field, not a process. A dead holder's lock is free the moment its time is up, and the next writer removes it as part of taking the path.
 
+## A runnable cooperating-worker example
+
+The [two-worker example](examples/cooperating-workers/README.md) reserves a path set before launching mutation, shows a competing worker who holds it and why, and lets unrelated work finish. It also demonstrates renewal, acquisition-aware cleanup, worker failure and the TTL boundary in an isolated store. The runbook defines an external adoption experiment as unrun. Its controlled flows do not resolve the observation-coherence failures tracked by [#45](https://github.com/git-stunts/locks/issues/45).
+
 ## Families and batches: all or nothing across locks
 
 One transaction per claim already makes a multi-path claim atomic; this section extends that to several locks at once, in two forms that share one mechanism. A child lock is tied to a parent so that the family lives and dies together, and a batch claims several independent locks in one stanza.

examples/cooperating-workers/README.md

@@ -0,0 +1,103 @@
+# Two cooperating workers
+
+This example puts reservation acquisition in the code that launches a mutation. Alice reserves two generated files before her worker starts. Bob receives a refusal naming Alice and her note, then completes unrelated work while Alice remains active. The same run demonstrates renewal, acquisition-aware cleanup, a failed worker, and the TTL boundary.
+
+The example exercises controlled local flows. It does not resolve the mixed-observation failures in [#45](https://github.com/git-stunts/locks/issues/45). Evaluate an external runner integration after the hardening work and that correctness gate; the adoption experiment below has not been run.
+
+## Run it
+
+From this checkout, with Bash 4+ and Git available:
+
+```bash
+./examples/cooperating-workers/demo.sh /tmp/locks-workers-review
+```
+
+Choose a fresh output path. An existing directory is refused before any file is overwritten. With no argument, the launcher creates a fresh temporary directory and prints its location. It uses this checkout's `bin/git-locks`; no installation, service, or package download is required.
+
+The launcher selects an explicit bare store at `<output>/store.git`, uses `<output>/work` as the workers' common artifact directory, and writes command receipts under `<output>/receipts`. It retains those files when it finishes. The demonstration does not modify this checkout's source files or project refs.
+
+The expected transcript is:
+
+```text
+Path set acquired before mutation; overlap refused; unrelated work completed; renewed acquisition released.
+Superseded cleanup preserved the replacement acquisition.
+Worker exit 17 propagated; its reservation was released.
+TTL expired while the command remained active; automatic renewal is not provided.
+Artifacts and JSONL receipts: <absolute output directory>
+```
+
+Inspect the evidence directly:
+
+```bash
+cat /tmp/locks-workers-review/receipts/worker-b-refusal.jsonl
+cat /tmp/locks-workers-review/receipts/before-renewal.jsonl
+cat /tmp/locks-workers-review/receipts/after-renewal.jsonl
+cat /tmp/locks-workers-review/receipts/replacement-survives.jsonl
+cat /tmp/locks-workers-review/receipts/expired-while-running.jsonl
+cat /tmp/locks-workers-review/receipts/final-doctor.jsonl
+```
+
+Each run records CLI `version`, selected store and checkout revision. The [recorded example](recorded-run.json) retains one observed run's CLI records and results against its named source revision. Object IDs and acquisition IDs change on later runs; compare their relationships and the actual outcomes.
+
+## What the launcher does
+
+The admission boundary is the `with` invocation in [demo.sh](demo.sh):
+
+```bash
+"${DEMO_BIN}" with --job build --holder alice \
+  --note 'regenerating API and types' --ttl 60 \
+  generated/api.txt generated/types.txt -- \
+  bash "${HERE}/worker.sh" worker-a build "${gate}" build
+```
+
+`with` obtains the whole path set before it invokes the mutation command. The worker records its admitted acquisition, writes both artifacts, and signals readiness through a gate file. Bob's launcher also uses `with`; a failed acquisition prevents Bob's mutation command from running. A prior `check` is not the admission mechanism.
+
+The gates control order without guessing how long a worker will take. Alice remains inside her command while Bob's conflicting launch is refused and Bob's independent launch writes `independent.txt`. The independent worker records Alice's live acquisition during its own execution. It then exits and releases its separate reservation.
+
+The launcher renews Alice's reservation with `extend`. The before/after records have different `record` object IDs and the same `acquisition` ID. When Alice's gate opens, her wrapper releases that original acquisition despite the renewal. Both generated paths become free.
+
+A separate case starts a wrapper with job name `reused`, then creates a replacement acquisition under that name. The old wrapper's cleanup reports `nothing` with `reason: superseded`; the replacement stays live and is released explicitly by its own acquisition ID.
+
+The failed-worker case returns status 17 after writing partial output. `with` propagates 17 and releases the reservation. Cleanup does not roll back the worker's file changes: `failed.txt` intentionally remains as partial output.
+
+Finally, a worker with TTL 1 waits at a gate. An observation at a simulated later clock reports the reservation expired while the command is still active. The launcher then opens the gate and lets cleanup complete. The example fixes `GIT_LOCKS_NOW=1000000` and explicitly advances one observation to `1000002`; this is a deterministic TTL demonstration, not a two-second benchmark. Real integrations should use the normal clock. `with` neither renews automatically nor terminates a command when its reservation expires. Choose a TTL appropriate to the workload and put any renewal policy in the runner.
+
+## Store and worktree meaning
+
+This example coordinates two workers accessing the same physical artifact directory. Its explicit store isolates the exercise and makes the sharing policy visible.
+
+In a project integration, the default separate store keeps coordination refs out of the project and is shared by linked worktrees. Reserving the same relative path across linked worktrees coordinates logical ownership; the files may be physically different. A runner must choose whether that shared logical ownership is the intended policy. Using distinct explicit stores intentionally creates independent coordination domains. Workers that should coordinate must select the same store and agree on relative path meaning.
+
+The reservations are cooperative. Other programs can write the files without using the launcher. These controlled runs do not prove the reader coherence or arbitrary interleaving properties tracked by #45.
+
+## Repeatable verification
+
+```bash
+python3 test/cooperating-workers.py
+make lint
+```
+
+The Python test requires the same `jsonschema` dependency as the existing suite; the example itself uses Bash and Git. `make test` runs the example test as part of the normal suite.
+
+The oracle parses the real command receipts and validates lifecycle JSON against the public schema. It checks acquisition before mutation, both reserved paths, refusal holder/note, absence of Bob's blocked mutation marker, unrelated progress during Alice's acquisition, renewal identity, release after renewal, superseded cleanup, status-17 cleanup, simulated expiry, and final store health.
+
+The golden run uses an output path containing spaces. An existing-directory case checks preservation of a sentinel file. Two complete demonstrations then run concurrently with separate stores and both must satisfy the same behavioral assertions. That is bounded stress of this example and its isolation, not arbitrary-schedule fuzzing or evidence of external adoption.
+
+## Adoption experiment, not yet run
+
+After the hardening gates, recruit one actual runner maintainer or integration user and agree on one existing generator or coding-worker task that mutates a known path set. Run it through the launch boundary above in a shared checkout or deliberately chosen artifact directory. Keep the experiment to that task and its existing workflow.
+
+Record the following before deciding whether to add features:
+
+| Question | Evidence to retain |
+| --- | --- |
+| Can the maintainer install and wire the launcher? | Setup minutes, commands changed, platform/runtime versions, and each obstacle. |
+| Is it useful beyond the first demonstration? | Number of runs on at least three workdays and whether the maintainer chose to keep using it. |
+| Is contention understandable? | The refusal shown to the user and their explanation of who held the paths, why, and what they did next. |
+| Does unrelated work keep moving? | A concrete blocked path set and an unrelated task that completed during it. |
+| Is lifecycle handling dependable for the task? | Renewal/expiry decisions, worker failures, interrupted runs, cleanup receipts and any unexpected artifacts. |
+| Does the sharing policy fit? | Shared checkout or linked-worktree layout, selected store, and any mismatch between logical paths and physical files. |
+
+Ask the maintainer: "What did this refusal tell you?", "Where would you place acquisition in your launcher?", "What happened when the worker exceeded its TTL?", and "Would you keep this in the workflow next week, and why?" Record their words rather than substituting an inferred adoption score.
+
+A useful result would be a maintainer who completes integration, uses it repeatedly, explains contention correctly, and wants to keep it. A passing local demo alone does not supply that evidence. No external maintainer has been recruited or contacted as part of this change, and no standalone-business conclusion follows from it.

examples/cooperating-workers/demo.sh

@@ -0,0 +1,142 @@
+#!/usr/bin/env bash
+# Local integration example. It retains all artifacts in one fresh directory.
+# The fixed clock makes expiry an explicit demonstration, not a timing guess.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${HERE}/../.." && pwd)"
+export DEMO_BIN="${ROOT}/bin/git-locks"
+if (($# > 1)); then
+  printf 'usage: bash examples/cooperating-workers/demo.sh [fresh-output-directory]\n' >&2
+  exit 2
+fi
+if (($#)); then
+  output="$1"
+  if [[ -e "${output}" ]]; then
+    printf 'output directory already exists: %s\n' "${output}" >&2
+    exit 2
+  fi
+  mkdir -p "${output}"
+else
+  output="$(mktemp -d "${TMPDIR:-/tmp}/git-locks-workers.XXXXXX")"
+fi
+output="$(cd "${output}" && pwd)"
+mkdir -p "${output}/work/generated" "${output}/receipts" "${output}/gates" "${output}/tmp"
+export DEMO_RECEIPTS="${output}/receipts"
+export GIT_LOCKS_STORE="${output}/store.git" GIT_LOCKS_NOW=1000000
+export TMPDIR="${output}/tmp"
+unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_COMMON_DIR GIT_PREFIX GIT_OBJECT_DIRECTORY GIT_NAMESPACE
+children=() gates=()
+cleanup() {
+  local gate pid
+  for gate in "${gates[@]}"; do : >"${gate}"; done
+  for pid in "${children[@]}"; do wait "${pid}" 2>/dev/null || true; done
+}
+trap cleanup EXIT
+trap 'exit 130' INT
+trap 'exit 143' TERM
+cd "${output}/work"
+"${DEMO_BIN}" version >"${DEMO_RECEIPTS}/version.jsonl"
+git -C "${ROOT}" rev-parse HEAD >"${DEMO_RECEIPTS}/source-revision.txt"
+"${DEMO_BIN}" store >"${DEMO_RECEIPTS}/store.jsonl"
+
+await_ready() { # gate pid: wait for the admitted worker's explicit ready signal
+  local gate="$1" pid="$2" attempt
+  for ((attempt = 0; attempt < 1500; attempt++)); do
+    [[ ! -f "${gate}.ready" ]] || return 0
+    if ! kill -0 "${pid}" 2>/dev/null; then
+      printf 'worker exited before signalling readiness\n' >&2
+      return 1
+    fi
+    sleep 0.02
+  done
+  printf 'launcher timed out waiting for worker readiness\n' >&2
+  return 1
+}
+
+expect_status() { # expected command...
+  local expected="$1" actual=0
+  shift
+  "$@" || actual=$?
+  if ((actual != expected)); then
+    printf 'expected exit %s, got %s: %s\n' "${expected}" "${actual}" "$*" >&2
+    return 1
+  fi
+}
+
+acquisition_from() { # VAR file: IDs emitted by this example are plain strings
+  local line
+  read -r line <"$2"
+  [[ "${line}" =~ \"acquisition\":\"([^\"]+)\" ]] || return 1
+  printf -v "$1" '%s' "${BASH_REMATCH[1]}"
+}
+
+# A owns both outputs before its mutation command starts. The command remains
+# gated while B attempts overlap and then completes unrelated work.
+gate="${output}/gates/build"
+gates+=("${gate}")
+"${DEMO_BIN}" with --job build --holder alice --note 'regenerating API and types' --ttl 60 \
+  generated/api.txt generated/types.txt -- bash "${HERE}/worker.sh" worker-a build "${gate}" build \
+  >"${DEMO_RECEIPTS}/worker-a.stdout.txt" 2>"${DEMO_RECEIPTS}/worker-a.jsonl" &
+worker=$!
+children+=("${worker}")
+await_ready "${gate}" "${worker}"
+expect_status 1 "${DEMO_BIN}" with --job competing --holder bob --note 'updating API' generated/api.txt \
+  -- bash -c 'printf "blocked mutation ran\n" >blocked-ran' \
+  >"${DEMO_RECEIPTS}/worker-b.stdout.txt" 2>"${DEMO_RECEIPTS}/worker-b-refusal.jsonl"
+"${DEMO_BIN}" with --job independent --holder bob --note 'writing unrelated notes' independent.txt \
+  -- bash "${HERE}/worker.sh" independent independent unused independent \
+  >"${DEMO_RECEIPTS}/independent.stdout.txt" 2>"${DEMO_RECEIPTS}/independent.jsonl"
+"${DEMO_BIN}" show --job build >"${DEMO_RECEIPTS}/before-renewal.jsonl"
+"${DEMO_BIN}" extend --job build --ttl 120 >"${DEMO_RECEIPTS}/renewal.jsonl"
+"${DEMO_BIN}" show --job build >"${DEMO_RECEIPTS}/after-renewal.jsonl"
+: >"${gate}"
+wait "${worker}"
+"${DEMO_BIN}" check generated/api.txt generated/types.txt >"${DEMO_RECEIPTS}/after-worker-a.jsonl"
+printf 'Path set acquired before mutation; overlap refused; unrelated work completed; renewed acquisition released.\n'
+
+# A later acquisition can reuse the name; the old wrapper releases only its
+# own acquisition, so the replacement survives that wrapper's cleanup.
+gate="${output}/gates/reused"
+gates+=("${gate}")
+"${DEMO_BIN}" with --job reused --holder alice reused.txt -- bash "${HERE}/worker.sh" old reused "${gate}" superseded \
+  >"${DEMO_RECEIPTS}/superseded-worker.stdout.txt" 2>"${DEMO_RECEIPTS}/superseded-worker.jsonl" &
+worker=$!
+children+=("${worker}")
+await_ready "${gate}" "${worker}"
+"${DEMO_BIN}" claim --job reused --holder bob reused.txt >"${DEMO_RECEIPTS}/replacement.jsonl"
+replacement_acquisition=''
+acquisition_from replacement_acquisition "${DEMO_RECEIPTS}/replacement.jsonl"
+: >"${gate}"
+wait "${worker}"
+"${DEMO_BIN}" show --job reused >"${DEMO_RECEIPTS}/replacement-survives.jsonl"
+"${DEMO_BIN}" release --job reused --acquisition "${replacement_acquisition}" >"${DEMO_RECEIPTS}/replacement-release.jsonl"
+printf 'Superseded cleanup preserved the replacement acquisition.\n'
+
+# A nonzero mutation status propagates through with while it releases the
+# reservation. Reservation cleanup does not roll back partial file writes.
+status=0
+"${DEMO_BIN}" with --job failing --holder alice failed.txt -- bash "${HERE}/worker.sh" failing failing unused failure \
+  >"${DEMO_RECEIPTS}/failing-worker.stdout.txt" 2>"${DEMO_RECEIPTS}/failing-worker.jsonl" || status=$?
+printf '%s\n' "${status}" >"${DEMO_RECEIPTS}/failing-worker.status"
+[[ "${status}" == 17 ]]
+"${DEMO_BIN}" check failed.txt >"${DEMO_RECEIPTS}/after-failure.jsonl"
+printf 'Worker exit 17 propagated; its reservation was released.\n'
+
+# Advancing only the observation's test clock shows that with neither renews
+# automatically nor terminates a still-running command when the TTL expires.
+gate="${output}/gates/ttl"
+gates+=("${gate}")
+"${DEMO_BIN}" with --job short --holder alice --ttl 1 short.txt -- bash "${HERE}/worker.sh" ttl short "${gate}" ttl \
+  >"${DEMO_RECEIPTS}/ttl-worker.stdout.txt" 2>"${DEMO_RECEIPTS}/ttl-worker.jsonl" &
+worker=$!
+children+=("${worker}")
+await_ready "${gate}" "${worker}"
+GIT_LOCKS_NOW=1000002 "${DEMO_BIN}" check short.txt >"${DEMO_RECEIPTS}/expired-while-running.jsonl"
+kill -0 "${worker}"
+printf 'yes\n' >"${DEMO_RECEIPTS}/worker-active-after-expiry.txt"
+: >"${gate}"
+wait "${worker}"
+printf 'TTL expired while the command remained active; automatic renewal is not provided.\n'
+"${DEMO_BIN}" list >"${DEMO_RECEIPTS}/final-list.jsonl"
+"${DEMO_BIN}" doctor >"${DEMO_RECEIPTS}/final-doctor.jsonl"
+printf 'Artifacts and JSONL receipts: %s\n' "${output}"