Skip to content

feat(boatstack): retromine — offline recurrence detection over transcripts - #242

Merged
bigboateng merged 1 commit into
mainfrom
boatstack/retromine
Jul 28, 2026
Merged

feat(boatstack): retromine — offline recurrence detection over transcripts#242
bigboateng merged 1 commit into
mainfrom
boatstack/retromine

Conversation

@bigboateng

Copy link
Copy Markdown
Contributor

Seventh slice of the published → merged program (follows #241) — the first half of Phase D. The program's general law: a recurring operator instruction is steady-state error — evidence the controller is missing a typed observation, verb, setpoint, or guard. This PR builds the sensor for that law.

What changed

New pure package internal/retromine:

  • Neutral event schema (JSONL: source, session_id, ts?, role: operator|agent|tool, text, index) any tool can emit — the miner is agent-agnostic by contract.
  • Adapters for Claude Code session JSONL and prefix-annotated plain text (User: / Agent: / >), plus the neutral format, with per-file content sniffing. Lossy projection is one-directional by design: unknown entry shapes are skipped (host formats grow constantly), but an unparseable line is a typed error naming its line — mis-parsing can never silently become "no recurrence found".
  • Deterministic recurrence detection: normalize (strip fenced payloads — pasted logs are payload, not instruction), token 3-shingles, Jaccard ≥ 0.6, greedy clustering in sorted order. Recurrence = ≥ 3 occurrences across ≥ 2 sessions (repetition inside one conversation is conversation, not steady-state error). Sub-4-token acknowledgements never enter the pool. Per-session event indexes are parser-assigned, so caller concatenation order cannot change the report.
  • No user-facing verb yet — retro derive (classifier + proposals + CLI) is the next slice.

Boundary conformance

  • Boundary: operator transcripts entering an analysis pipeline that ships publicly.
  • Control law: retro-derivation-is-offline-and-deterministic — no network, no subprocesses, no filesystem, no clocks, no randomness; identical inputs in any order produce identical output.
  • Required evidence: retromine_conformance_test.go — positive (planted 3×/2-session instruction clusters; both adapters project equivalent content to equivalent instruction streams; fenced-payload variance does not split clusters), negative (2 occurrences or 1 session never clusters), relation (input order cannot change the report), failure-state (malformed lines are typed errors), bypass (the package's import list is conformance-tested to grant no I/O capability at all — the boundary is structural, not behavioral), plus a golden report over synthetic fixtures.
  • Failure behavior: parse errors are typed and positional; nothing is ever partially, silently mined.
  • Release condition: test fixtures are synthetic only (everything under product-engineering-loop/ ships verbatim to the public repo); no real transcript content, paths, or session IDs anywhere.

Evidence

  • go build ./... && go vet ./... && go test ./... — pass.
  • Release note added: 2026-07-28-retromine-recurrence-detector.md.

Final slice next: retro derive — gap classification (observation/verb/setpoint/guard) + reviewable proposals, derive-only.

…ripts

New pure package internal/retromine: the sensor for the control law that a
recurring operator instruction is steady-state error. Neutral JSONL event
schema; adapters for Claude Code session JSONL and prefix-annotated plain
text (lossy projection: skip unknown shapes, typed error on unparseable
lines); deterministic recurrence detection (normalize, strip fenced
payloads, token 3-shingles, Jaccard >= 0.6, greedy clustering in sorted
order) with recurrence defined as >= 3 occurrences across >= 2 sessions.
Per-session event indexes are parser-assigned so caller concatenation order
cannot change the report.

Capability-free by construction: no network, subprocesses, filesystem,
clocks, or randomness — the import list is conformance-tested. Synthetic
fixtures only. No user-facing verb yet; retro derive follows.

control-law: retro-derivation-is-offline-and-deterministic

Disclosure-Reviewed: reviewed — public-safe only, private facet kept out of this commit
@bigboateng
bigboateng merged commit b37bfc3 into main Jul 28, 2026
12 checks passed
@bigboateng
bigboateng deleted the boatstack/retromine branch July 28, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant