perf: measure historical directory token churn

CHANGELOG.md

@@ -4,6 +4,10 @@ All notable changes to this project are recorded here. The format follows Keep a
 
 ## [Unreleased]
 
+### Added
+
+- A calibrated historical directory-token benchmark (#39), comparing released prefix churn, repeated reuse, and live-lock controls. Small real-CLI calibration runs in the suite; large timing runs are informational. The retained macOS study contains 135 observations and reports multi-second reads of a released wide-10k store, with host-drift and synthetic-fixture limits.
+
 ## [0.7.0] - 2026-09-16
 
 ### Added

Makefile

@@ -1,6 +1,6 @@
 SHELL := /usr/bin/env bash
 # lib/*.sh are fragments of one script and only lint as the whole they build into (bin/git-locks).
-SCRIPTS := bin/git-locks test/test.sh scripts/hooks/pre-commit scripts/hooks/pre-push scripts/build.sh
+SCRIPTS := bin/git-locks test/test.sh scripts/hooks/pre-commit scripts/hooks/pre-push scripts/build.sh scripts/benchmark-directory-tokens.sh test/directory-token-churn.sh
 PREFIX ?= $(HOME)/.local
 
 .PHONY: build lint test test-docker install uninstall
@@ -14,6 +14,7 @@ lint:
 
 test:
 	bash test/test.sh
+	bash test/directory-token-churn.sh
 
 test-docker: # the same suite inside the official bash image, for a wall between the tests and your machine
 	docker run --rm -v "$(CURDIR)":/src -w /src bash:5.2 bash -c 'apk add --no-cache git python3 py3-jsonschema >/dev/null && git config --global user.email t@example.invalid && git config --global user.name t && bash test/test.sh'

README.md

@@ -454,3 +454,5 @@ The source is `lib/`, one module per section in numeric order (`000-prelude.sh`
 ## License
 
 Apache 2.0. See `LICENSE` and `NOTICE`.
+
+Historical directory-token measurements have a calibrated generator and an informational runner in [`scripts/benchmark-directory-tokens.sh`](scripts/benchmark-directory-tokens.sh). See the [benchmark protocol](docs/benchmarks/directory-tokens.md) for fixture semantics, resource bounds, and reproducible commands. Timings are not CI gates.

docs/benchmarks/directory-tokens-results.md

@@ -0,0 +1,46 @@
+# Directory-token churn results, 2026-09-22
+
+A released store with 10,000 distinct directory tokens and 10,000 reachable records had a median `check` latency of **3.641 seconds** in this run. Empty-store controls measured **0.140 seconds before** the matrix and **0.094 seconds after** it. The store had no job or path refs. Historical directory state belongs in performance measurements alongside live-lock count.
+
+These are 135 native macOS observations: nine scenarios, five operations, three repetitions each. Every measured command exited 0, expected output-line counts matched, and every post-operation ref fingerprint matched its starting state. The raw check range for wide 10k was **3.070–6.811 seconds**; its median should not be read as a stable latency guarantee.
+
+## Median operation latency
+
+Values below are rounded milliseconds. Full minimum/median/maximum distributions are in [summary.csv](results/2026-09-22/summary.csv), with every sample in [observations.csv](results/2026-09-22/observations.csv).
+
+| Scenario | Check | Exact claim | Prefix claim | List | Doctor |
+|---|---:|---:|---:|---:|---:|
+| Empty, before | 140 | 205 | 268 | 105 | 162 |
+| Wide 1k | 348 | 474 | 436 | 299 | 334 |
+| Wide 10k | 3641 | 2753 | 2433 | 2110 | 2462 |
+| Deep 1k prefixes | 201 | 260 | 296 | 193 | 261 |
+| Deep 10k prefixes | 1031 | 1144 | 1209 | 1158 | 1372 |
+| Reuse 1k acquisitions | 139 | 180 | 197 | 107 | 107 |
+| Reuse 10k acquisitions | 212 | 203 | 193 | 95 | 89 |
+| Live 1k control | 460 | 539 | 806 | 1160 | 3017 |
+| Empty, after | 94 | 165 | 163 | 71 | 71 |
+
+## What the comparisons show
+
+- **Released records still cost reads.** Wide 10k retained 10,000 refs and 10,000 distinct records, with zero jobs or paths. Median `list` was 2.110 seconds even though it emitted no lines; median `doctor` was 2.462 seconds.
+- **Keep ref count and reachable-record count separate.** Deep 10k retained the same 10,000 refs but only 1,000 distinct records. Its median `check` was 1.031 seconds. The fixed scenario order and host variation prevent attributing the entire difference to object count.
+- **Unreachable objects and retained authority have different costs.** Reuse 10k left 10 tokens pointing to one record, plus 10,000 loose historical objects on disk. Median `check` was 0.212 seconds, `list` 0.095 seconds, and `doctor` 0.089 seconds. This workload did not reproduce wide 10k's multi-second observations; it does not establish that unreachable-object growth is free under every storage layout.
+- **Live-lock work adds a separate cost.** The synthetic live 1k control had 3,000 refs and 1,000 records. Its median `list` was 1.160 seconds and `doctor` 3.017 seconds, compared with 0.299 and 0.334 seconds in the released wide 1k store.
+
+The [before/after inventories](results/2026-09-22/fixtures.csv) retain all ref, reachable-record, loose-object, and allocated-store counts. Wide 10k occupied 78.2 MiB after setup; deep 10k occupied 43.0 MiB; reuse 10k occupied 39.2 MiB. Each scenario accumulated six unreachable probe records during measured claim/release repetitions; the inventories expose those additions while ref fingerprints remained identical.
+
+## Provenance and limits
+
+The measured revision was `fe7cdb588ee16f0d90350865344e0b007bf4ffa0`, with binary blob `ff0444627cb9239b1a582ecfc9645fe22b68e611` and generator blob `2d934e9942ecd7aa2f080d93b7cb4e36b95688d8`. The git-locks executable is unchanged from the study's `01e39c3` main baseline. The final results commit adds data and this report without changing that measured code.
+
+The host was macOS 26.6.2, Darwin 25.6.0 arm64, MacBookPro18,3, 10 logical CPUs, and 16 GiB RAM. Bash was 5.3.9 and Git reported 2.54.0, Apple Git-157. Exact captured metadata is in [environment.txt](results/2026-09-22/environment.txt) and [hardware.txt](results/2026-09-22/hardware.txt). Hardware metadata came from `sw_vers` and `sysctl hw.model hw.ncpu hw.memsize` on the same host.
+
+Setup used synthetic loose Git objects and refs. Small wide/deep/reuse fixtures matched actual CLI claim/release records and refs after excluding acquisition IDs. The large history was not produced by thousands of CLI invocations. Setup, inventory, ref restoration, and release cleanup were excluded from operation timing. Native resource reports are concatenated, unchanged except for filename headers, in [native-time.txt](results/2026-09-22/native-time.txt). The largest native maximum-RSS report was 46.3 MiB during wide 10k `doctor`; it is not aggregate concurrent memory.
+
+The largest observed setup working footprint was **157.3 MiB**, measured with `du -sk` before setup inputs were removed. The 200 MiB check detects an excess after allocation; it is a planned workload budget and an observed check, not a preventive disk quota. The completed matrix removed every scenario store. Retained data is compact text.
+
+Caches were not cleared, and the host was shared with other development work. Empty controls changed from 140 to 94 milliseconds for `check`, and from 162 to 71 milliseconds for `doctor`. That drift, fixed operation order, wall-clock instrumentation, and three repetitions limit causal and statistical claims. There was no failed timing observation or partial large-matrix run. Earlier host disk pressure delayed the study before its full matrix started.
+
+These results support profiling snapshot reads at large retained-token counts and repeating the experiment after the hardening changes land. They do not justify deleting generation tokens or changing the concurrency protocol. Packed stores, garbage collection, cold caches, other machines, and long-running CLI history setup remain unmeasured. No timing threshold or retention-policy change is introduced.
+
+See the [protocol and reproduction commands](directory-tokens.md) for the calibrated generator, quick checks, and full matrix.

docs/benchmarks/directory-tokens.md

@@ -0,0 +1,41 @@
+# Historical directory-token benchmark
+
+This study asks how directory tokens retained after release affect command cost when no job or path refs remain. It compares ref count and distinct reachable records, since a deep path leaves several tokens pointing to the same record. Repeated reuse also leaves unreachable historical blobs on disk.
+
+The runner measures `check`, an exact claim, a prefix claim, `list`, and `doctor`. The live control measures the same successful commands against unrelated reservations. Large fixture setup uses Git directly and is excluded from operation timing. The fixture generator creates loose objects and loose refs, including distinct historical records, with the same post-release semantics as the CLI. Its deterministic acquisition IDs differ from real generated IDs.
+
+`test/directory-token-churn.sh` calibrates shallow, deep, and reused shapes against actual CLI claim/release sequences. It compares every retained ref and every record field except the acquisition ID, checks independent counts, and runs the real doctor. It deliberately contaminates a released fixture to verify rejection. Empty stores, live controls, invalid counts, missing stores, and existing output protection are also covered. Twelve seed-39 shape samples use hand-counted count/record oracles. A quick matrix exercises the timing/cleanup path without a latency assertion.
+
+## Workloads
+
+| Scenario | Requested scale | Retained directory tokens | Distinct reachable records | Live jobs |
+|---|---:|---:|---:|---:|
+| Empty, before and after the matrix | 0 | 0 | 0 | 0 |
+| Shallow/wide | 1,000 / 10,000 paths | 1,000 / 10,000 | 1,000 / 10,000 | 0 |
+| Deep, ten levels per path | 1,000 / 10,000 prefixes | 1,000 / 10,000 | 100 / 1,000 | 0 |
+| Repeated reuse of ten directories | 1,000 / 10,000 acquisitions | 10 | 1 | 0 |
+| Synthetic live control, shallow/wide | 1,000 paths | 1,000 | 1,000 | 1,000 |
+
+Each scenario has three repetitions of each operation. Claim cleanup runs outside the timer. Prefix claims create a retained probe token, so the runner removes that exclusively owned token with its expected object ID after release. A fingerprint verifies that all refs exactly match their starting state after every observation. The before/after inventories expose unreachable probe records left by those operations.
+
+## Reproduction
+
+Requirements are Bash 5, Git, standard Unix utilities, and native `/usr/bin/time` on macOS or GNU time on Linux. Small calibration is part of `make test`. No runtime dependency is added to git-locks.
+
+```bash
+bash test/directory-token-churn.sh
+bash scripts/benchmark-directory-tokens.sh run /tmp/locks-churn-quick quick
+bash scripts/benchmark-directory-tokens.sh run /tmp/locks-churn-results
+```
+
+Output directories must not exist. The fixture command refuses existing stores and bounds its input to 10,000 work units. The large matrix processes one store at a time in a private temporary directory, then removes that store. The planned peak footprint is below 200 MiB, including temporary input files, loose objects, and refs. The runner checks the working footprint after the setup transaction, before deleting setup inputs, and reports an excess above 200 MiB. This check happens after allocation; it is not a preventive disk quota. A failed run retains raw command output, metrics, partial CSVs, and its current temporary store for diagnosis.
+
+`environment.txt` identifies the measured revision, binary and generator blob IDs, Bash/Git/platform versions, and repetition count. `fixtures.csv` records setup time separately, ref/record/object counts, and allocated store KiB before and after operations, plus the observed setup working footprint. `observations.csv` retains every raw duration, native maximum RSS report, exit code, and stdout line count. Native resource reports are retained under `raw/`; `summary.csv` contains minimum, median, and maximum durations.
+
+## Interpretation limits
+
+Filesystem caches are not cleared. Setup and ref verification warm filesystem metadata, so these are repeated local observations on a shared host, not cold-storage measurements. `elapsed_us` uses Bash's wall clock around native time and the CLI; it includes the time wrapper. Native maximum RSS is a per-command resource report, not total concurrent process memory. Three repetitions support bounded descriptive comparisons, not a latency SLA or confidence interval.
+
+The fixture equivalence check covers reachable record semantics at small scale. Synthetic setup does not measure the cost of thousands of actual CLI claim/release invocations, historical transaction interleavings, or token reclamation. The benchmark adds no timing threshold and proposes no retention-policy change.
+
+The completed native macOS study and retained observations are in the [2026-09-22 results](directory-tokens-results.md).

docs/benchmarks/results/2026-09-22/environment.txt

@@ -0,0 +1,12 @@
+specimen_revision=fe7cdb588ee16f0d90350865344e0b007bf4ffa0
+binary_git_blob=ff0444627cb9239b1a582ecfc9645fe22b68e611
+generator_git_blob=2d934e9942ecd7aa2f080d93b7cb4e36b95688d8
+bash=5.3.9(1)-release
+mode=full
+repetitions=3
+git version 2.54.0 (Apple Git-157)
+Darwin 25.6.0 arm64
+started_utc=2026-09-22T16:28:15Z
+scratch=/tmp/locks-churn-measurement.trLdeM/locks-churn-matrix.rBBi2T
+Synthetic loose-object/ref fixtures; acquisition ids deterministic. Setup, inventory and cleanup excluded from timing. Filesystem caches not cleared. elapsed_us uses Bash EPOCHREALTIME around native time plus the CLI; max RSS is the native per-command resource report, not aggregate concurrent memory.
+completed_utc=2026-09-22T16:32:56Z

docs/benchmarks/results/2026-09-22/fixtures.csv

@@ -0,0 +1,19 @@
+scenario,phase,setup_us,dirs,jobs,paths,refs,reachable_blobs,loose_objects,disk_kib,setup_working_kib
+empty,before,95528,0,0,0,0,0,0,80,80
+empty,after,0,0,0,0,0,0,6,104,0
+wide-1000,before,4149531,1000,0,0,1000,1000,1000,8080,16184
+wide-1000,after,0,1000,0,0,1000,1000,1006,8104,0
+wide-10000,before,30998673,10000,0,0,10000,10000,10000,80080,161108
+wide-10000,after,0,10000,0,0,10000,10000,10006,80104,0
+deep-1000,before,937112,1000,0,0,1000,100,100,4480,8984
+deep-1000,after,0,1000,0,0,1000,100,106,4504,0
+deep-10000,before,13018614,10000,0,0,10000,1000,1000,44080,89108
+deep-10000,after,0,10000,0,0,10000,1000,1006,44104,0
+reuse-1000,before,1820118,10,0,0,10,1,1000,4120,8164
+reuse-1000,after,0,10,0,0,10,1,1006,4144,0
+reuse-10000,before,22319860,10,0,0,10,1,10000,40120,80164
+reuse-10000,after,0,10,0,0,10,1,10006,40144,0
+live-1000,before,3836279,1000,1000,1000,3000,1000,1000,16080,28356
+live-1000,after,0,1000,1000,1000,3000,1000,1006,16104,0
+empty-after,before,50016,0,0,0,0,0,0,80,80
+empty-after,after,0,0,0,0,0,0,6,104,0

docs/benchmarks/results/2026-09-22/hardware.txt

@@ -0,0 +1,6 @@
+ProductName:		macOS
+ProductVersion:		26.6.2
+BuildVersion:		25G83
+hw.model: MacBookPro18,3
+hw.ncpu: 10
+hw.memsize: 17179869184