settle: detection-quality report, honest claim reframe, restore decide subcommand - #78
Merged
Merged
Conversation
decide.go (decision-only stdin/stdout mode) didn't land in the #75 squash, but the Harbor integration depends on it. Restores the file + wires the 'decide' case in main.go. Build/vet/test green.
Adds docs/05-detection-quality.md: fixture correctness (0 FP/FN), ~93% precision at the redirect threshold via the recovery curve, ~10% firing rate on real traces, deny rung dormant. Reframes Status/docs-04 from Being-evaluated to Evaluated: net-neutral on Qwen/terminal-bench (workload not loop-bound); causal uplift on loop-prone workloads (small models, online evals) = next experiment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes out settle's evaluation honestly (options 5 + 6 from the "how do we prove settle" discussion), and fixes a merge gap.
1. Restore
decidesubcommand (fix)decide.go(decision-only stdin→verdict mode) didn't make it into the #75 squash-merge, but the Harbor integration (SETTLE_INTEGRATION.md) callssettle decide. Restored + wired intomain.go. Build/vet/test green.2.
docs/05-detection-quality.md(offline evaluation)What can be said without a causal experiment:
hidden-progresscase that an error-only detector would fail).settleis right ~93% of the time on real traces (via the recovery-probability curve incalibration.json); the deny rung (dwell ≥ 5) is effectively dormant on this corpus.3. Honest claim reframe (README + docs/04)
Being-evaluated→Evaluated: net-neutral on Qwen/terminal-bench— that workload's failures are step-budget/approach-bound, not loops (~13% loop-shaped, 0% deny-eligible). Stated as a result, not a shortcoming. Causal uplift remains open and belongs to a loop-prone population (smaller/cheaper models — settle's original thesis — and online evals), flagged as the next experiment.No invented numbers; every figure traces to
calibration.json, the fixtures, orqwen-budget200.🤖 Generated with Claude Code