A live daemon that is not holding the version-cohort marker poisons every index worker it forks, for its entire lifetime — no self-heal, and the conflict log names a build that does not exist

[halindrome]

Summary

If a daemon is alive but is not holding the VERSION_COHORT_DAEMON_FILE marker lock, every index
worker that same daemon forks rejects itself at startup and exits 1:

CBM index worker could not start: active daemon coordination could not be verified safely

The condition never clears on its own. The daemon keeps running, keeps forking workers, and every one
of them fails the same way until the daemon is killed by hand. On 2026-09-11 this produced 370
conflict records over 427 minutes (~52/hour) on my machine, and it stopped at the exact second I
killed the daemon.

This is why the workers fail. #2015 is about the cadence at which the watcher re-forks them once
they do. They are independent: I am running the #2075 backoff, it worked, and the wedge still produced
370 dead workers — see Impact #3.

Mechanism

version_cohort_active_daemon_presence() (src/daemon/version_cohort.c:805-839) classifies via the
daemon marker lock:

cbm_private_file_lock_try_acquire(VERSION_COHORT_DAEMON_FILE, EX)	lifetime probe	presence
BUSY (someone holds it)	—	COORDINATED ✅
OK (nobody holds it)	0	ABSENT ✅
OK (nobody holds it)	1 (a daemon IS alive)	UNCOORDINATED ❌

The worker then refuses in src/main.c:2787-2798:

cbm_version_cohort_daemon_presence_t worker_daemon_presence =
    cbm_version_cohort_daemon_presence_under_transition(worker_cohort_manager,
                                                        worker_endpoint, worker_transition);
if (worker_daemon_presence != CBM_VERSION_COHORT_DAEMON_ABSENT &&
    worker_daemon_presence != CBM_VERSION_COHORT_DAEMON_COORDINATED) {
    if (worker_daemon_presence == CBM_VERSION_COHORT_DAEMON_UNCOORDINATED) {
        (void)cbm_version_cohort_log_uncoordinated_daemon(&identity);
    }
    (void)fprintf(stderr, "CBM index worker could not start: active daemon coordination "
                          "could not be verified safely\n");
    goto worker_cleanup;
}

So the bottom row is the wedge: a daemon is alive, and it is not in the cohort. Since that daemon
is also the thing forking the workers, it is rejecting its own children — and because nothing makes
the daemon re-acquire the marker, the state is absorbing.

The conflict log actively misleads

cbm_version_cohort_log_uncoordinated_daemon() (src/daemon/version_cohort.c:963-980) writes a
hardcoded sentinel for the active side:

(void)snprintf(conflict.active_version, sizeof(conflict.active_version), "%s", "pre-cohort/unknown");
memset(conflict.active_build_fingerprint, '0', sizeof(conflict.active_build_fingerprint) - 1);

which lands in daemon-conflicts.ndjson as:

{"event":"daemon.version_conflict","timestamp_unix_s":1789149612,"reason":"build",
 "active_version":"pre-cohort/unknown",
 "active_build":"0000000000000000000000000000000000000000000000000000000000000000",
 "requested_version":"dev",
 "requested_build":"273d3ffe8585bb5f98bb5ed99abc1be6fece06fbe50f7fda849450d216ac4e99"}

The name asserts a diagnosis — an old pre-cohort binary is running — that is not true here. The
only runnable CBM binary on this machine is the 273d3ffe build in the requested_build field. (There
is one other copy on disk, a March nix build in a repo working tree, but it aborts in dyld on launch
and contains no watcher log keys at all, so it cannot have been the daemon.) The active daemon was
almost certainly the same build as the requester — it just wasn't holding the marker.

Reporting a zero fingerprint and a version string that no binary reports sends you looking for a stale
install that isn't there. A record saying "a live daemon holds no cohort marker; identity unknown"
would point at the actual fault.

Impact / diagnosability

1. No self-heal. Only kill <daemon-pid> clears it. Nothing in the daemon log names the
   condition; index_repository and index_status returned status=error throughout without
   surfacing the reason.
2. The reason is invisible from the daemon side. The daemon log records only:

   level=info msg=index.supervisor.reap outcome=exit_nonzero exit_code=1 signal=0
   level=warn msg=index.supervisor.worker_failed outcome=exit_nonzero exit_code=1 log=…/.worker-log-y4KhDv
   

   The actual sentence is written to a per-worker .worker-log-XXXXXX temp file, never promoted. You
   have to know to go read a randomly-named dotfile in the log directory.
3. Backoff bounds the rate but not the total. This machine is running the fix(watcher): back off re-forking a hard-failing index worker #2075 branch, so the
   rc < 0 backoff was active throughout and behaved exactly as designed — retries decayed to the
   INDEX_FAIL_CEILING_MS ceiling of one per 5 min per project, and watcher.index.sustained_failure
   fired at consecutive=10 for six projects. Over a 427-minute wedge that ceiling still permits ~85
   retries per project; liverpool-cleanup reached consecutive=81, i.e. it sat at the cap the whole
   time. Because the wedge never clears, dead workers accumulate linearly with daemon uptime no
   matter how good the backoff is — 370 conflicts across 8 watched projects in 7 hours. Capping the
   cadence was the right fix for Watcher re-forks a permanently-failing index worker forever — no backoff on rc < 0 (2233 workers in 4h43m); gap in #937 #2015 and is not a fix for this.
4. Those worker-log files are never pruned — 2618 of them here going back to 2026-09-01, every one
   a startup failure. Minor, but it is how I found the pattern.

Field data

~/.cache/codebase-memory-mcp/logs/daemon-conflicts.ndjson, 387 records, 386 of a single shape
(pre-cohort/unknown / all-zero → dev/273d3ffe):

date	conflicts	note
2026-09-04	1
2026-09-06	16
2026-09-11	370	10:52:48Z → 18:00:12Z, 427 min, ended at the daemon kill

The single remaining record is a genuine mixed-build conflict of the normal kind
(version | active=dev/273d3ffe → requested=0.10.8/2412e017), which is the mechanism working
as designed and is not what this issue is about.

The same cbm-daemon.log happens to contain a clean before/after for #2075, because the
watcher.index.err line only carries rc / consecutive on the patched build:

build	watcher.index.err lines	projects	max streak
unpatched (no rc= field)	2226	1 (add-translate-buttons)	n/a — unthrottled
patched (rc= present)	~560	8	81, at the 5-min ceiling

There is a sibling arm one branch earlier, src/main.c:2778-2786
(cbm_daemon_ipc_local_transition_seal_legacy() != 1), which prints "a pre-coordination or unverified
CBM generation is active". It produced a 2233-worker wave on 2026-09-01 — the same one quoted in
#2015 — and also calls cbm_version_cohort_log_uncoordinated_daemon(). If these share a cause, both
error strings should probably move with the fix.

Environment

• macOS 15 (Darwin 25.6.0), arm64
• codebase-memory-mcp dev, build 273d3ffe8585bb5f98bb5ed99abc1be6fece06fbe50f7fda849450d216ac4e99
  (local build of main at v0.10.8, plus the fix(watcher): back off re-forking a hard-failing index worker #2075 branch)
• Several concurrent MCP frontend clients (long-lived Claude Code sessions), watcher active on ~8 projects

Repro

I do not have a deterministic repro — it has appeared three times in eight days with no action on my
part that I can correlate. What I can say is that it is fully observable after the fact, and that
kill <daemon-pid> is a reliable recovery. If it would help, I am happy to run an instrumented build
that logs the marker-lock acquire/release path on the daemon side and report back the next time it
trips.

Questions for maintainers

1. Is the daemon expected to hold VERSION_COHORT_DAEMON_FILE for its whole lifetime? If so, is there
   a path that releases it while the daemon stays up — and should a daemon that finds itself outside
   the cohort re-acquire, or fail loudly and exit rather than keep serving?
2. Should a worker forked by a daemon consult the marker at all, versus inheriting a coordination
   token from the parent that forked it? The current check treats the parent as an untrusted stranger.
3. Would you take a PR that (a) replaces the pre-cohort/unknown sentinel with a record stating what
   was actually observed, and (b) promotes the worker's failure sentence into the daemon log so
   worker_failed carries the reason rather than a path to a temp file? Those are small and
   independent of whatever fixes the underlying coordination loss.