Skip to content

Govern a base with one run at a time, so concurrent runs stop deciding the same cluster repeatedly - #916

Merged
WaylandYang merged 1 commit into
devfrom
fix/governance-runs-one-at-a-time
Sep 25, 2026
Merged

WaylandYang merged 1 commit into
devfrom
fix/governance-runs-one-at-a-time

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Found while attributing the typed-graph bench's tokens (run on dev before #906). A third of one run's model tokens, about 5.3M of 16M, went to the governance agent, and not because there was that much to decide: 1346 review pairs received 8888 keep decisions, one pair ten decisions within 38 seconds from ten different runs. The duplicate-key errors on agent_decisions_open in the server log were the concurrent proposals colliding.

Cause. Every document's extraction enqueues a govern job for its base, and enqueue_unless_queued only skips a job that is still queued; with 64 workers each new job starts at once. Nine govern runs overlapped on one base, all read the same queue heads (the lock only marks stage = adjudicating, which the queue did not exclude), all asked the model about the same clusters, and close_review_auto's WHERE status = 'pending' let only the first close the row while every run recorded a decision.

Fix.

  • govern() takes a per-base session-level advisory lock, try only, the way the vector index build does. A run that does not get it exits and enqueues one govern for a minute later (enqueue_unless_queued_after, deduplicated against queued jobs only, so the running job does not suppress it), which covers pairs that arrive after the running job read its last queue head.
  • queue() and cluster_of() skip rows marked adjudicating.
  • close_review_auto returns how many rows it closed; a keep or already-merged outcome records no decision when the row was already closed by someone else.

#906 switched governance off in the bench base, which hides this from the bench but not from a real base with a running agent.

Verified on a fresh database: new store test a_base_is_governed_by_one_run (second lock attempt fails until the first releases; another base is another lock), the server's governance and adjudication tests, workspace clippy.

🤖 Generated with Claude Code

…iding the same cluster ten times

Every document's extraction enqueues a govern job for its base, and
enqueue_unless_queued only skips a job that is still queued; with 64
workers each new job starts at once. On the typed-graph bench nine
govern runs overlapped on one base, all read the same queue heads, and
1346 review pairs received 8888 keep decisions (one pair ten decisions
in 38 seconds from ten runs); a third of the run's model tokens went
there, and the duplicate-key errors on agent_decisions_open were the
concurrent proposals colliding.

govern() now takes a per-base session-level advisory lock (try only, as
the vector index build does). A run that does not get it exits and
enqueues one govern for a minute later, deduplicated against queued
jobs only, so pairs that arrive after the running job read its last
queue head are still picked up. The queue and cluster reads skip rows
the running job has marked adjudicating. close_review_auto reports how
many rows it closed, and a keep or already-merged outcome records no
decision when the row was closed by someone else.

Store test: the second lock attempt on a base fails until the first is
released; another base is another lock.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit b9e3aae into dev Sep 25, 2026
7 checks passed
@WaylandYang
WaylandYang deleted the fix/governance-runs-one-at-a-time branch September 25, 2026 09:53
WaylandYang added a commit that referenced this pull request Sep 25, 2026
…ith tokens by phase (#917)

Two groups on dev at 961c3c0 with gemini-3.5-flash, judged and with
errata: precision 91.0% / 88.5% before errata and 96.0% / 96.5% after;
same-sentence gold recall 11.3% / 13.2% before errata and 18.6% / 22.4%
after, with the errata retractions mostly wrong on this binary (72/98,
164/196), which is what #906's two-vote retraction addresses; 158k and
183k tokens per document. The token table by phase shows where they
went: the alignment window carried the rule proposals' per-item property
tables, the two-vote phrase alignment, and the governance agent deciding
the same pairs from up to ten concurrent runs (9481 decisions on 1421
pairs), fixed in #916.

The bench judge now fails fast when it would read a sealed key from
llm_settings instead of sending the ciphertext and collecting 401s.

Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
WaylandYang added a commit that referenced this pull request Sep 25, 2026
…tokens by phase (#918)

Same corpus, model and judge as the pre-#906 baseline, on dev with #906
and #916. Judged precision 81.0% / 79.0% before errata and 83.0% / 80.5%
after (the loss against the baseline is misworded facts from minimal
reasoning); same-sentence gold recall 14.2% / 14.3% before errata and
17.8% / 17.1% after, with errata now adding without retracting wrongly
(2 and 1 retractions, none judged stated); 31k and 34k tokens per
document, a fifth of the baseline and still five times 0044's ceiling,
seven tenths of it in alignment and rule proposals. 47 and 48 minutes a
group against three to five hours.

Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant