perf: measure historical directory token churn

[flyingrobots]

Measurement status: resource-confounded initial run; fresh baseline pending. The workstation owner reported that the original timing run coincided with extreme RAM and disk exhaustion. Preserve the raw observations as a record of that run, but do not treat the latency values or ratios below as representative performance. The earlier report did not record concurrent system memory-pressure/swap telemetry, so the resource contribution cannot be quantified retrospectively. A fresh run will require recorded host preflight and monitoring during measurement.

A released store with 10,000 wide directory tokens and 10,000 reachable records measured a median check latency of 3.641 seconds, with zero job or path refs. Empty-store controls measured 0.140 seconds before and 0.094 seconds after the matrix. This PR retains a calibrated generator, an informational runner, raw observations, and a bounded report so historical directory state can be measured alongside live-lock count.

The study contains 135 native macOS observations: nine scenarios, five operations, and three repetitions. Every measured command exited 0, expected output counts matched, and every post-operation ref fingerprint matched its initial state. Wide/deep/reuse workloads distinguish retained refs, distinct reachable records, and unreachable historical objects; a synthetic live 1k control provides a separate comparison.

• Measured results and limitations
• Protocol and reproduction commands
• Raw observations and before/after inventories

Validation: observed RED before implementing fixture creation, missing-store refusal, and the quick timing matrix. Small wide/deep/reuse fixtures match actual CLI claim/release records and refs except for acquisition IDs. Independent counts, deliberate contamination, invalid inputs, existing-store preservation, 12 seed-39 shape samples, and all 25 quick-matrix observations pass. The normal pre-push main suite passes 452 tests, 0 failures, followed by the benchmark calibration; lint passes.

The measured code is frozen at fe7cdb5; e28f480 adds only results and documentation. Native macOS metadata and exact binary/generator blob IDs are retained. Setup and cleanup were excluded from timing. Caches were not cleared, scenario order was fixed, and the shared host drifted, so the report makes no SLA or causal multiplier claim. The largest observed setup working footprint was 157.3 MiB. The 200 MiB check detects excess after allocation; it is not a preventive disk quota. No timing CI threshold, runtime optimization, or retention-policy change is introduced.

Fixes #39.

[coderabbitai]

Warning

Review limit reached

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 68f1f391-d988-4a21-8971-17071d955943

📥 Commits

Reviewing files that changed from the base of the PR and between 01e39c3 and e28f480.

⛔ Files ignored due to path filters (3)

• docs/benchmarks/results/2026-09-22/fixtures.csv is excluded by !**/*.csv
• docs/benchmarks/results/2026-09-22/observations.csv is excluded by !**/*.csv
• docs/benchmarks/results/2026-09-22/summary.csv is excluded by !**/*.csv

📒 Files selected for processing (10)

• CHANGELOG.md
• Makefile
• README.md
• docs/benchmarks/directory-tokens-results.md
• docs/benchmarks/directory-tokens.md
• docs/benchmarks/results/2026-09-22/environment.txt
• docs/benchmarks/results/2026-09-22/hardware.txt
• docs/benchmarks/results/2026-09-22/native-time.txt
• scripts/benchmark-directory-tokens.sh
• test/directory-token-churn.sh

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}