Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ observable behavior and compatibility, not every internal refactor.

### Fixed

- Portable HTML finding-bound fixes now record against the HTML report contract
without requiring Qoder's `canvas.json`, and refresh `findings.json`,
`report.md`, and `report.html` to the same repair revision. Qoder split reports
retain their Canvas-sidecar validation boundary.

- Root CLI delegation failures now keep machine mode parseable: spawn errors,
signal termination, and output-buffer exhaustion each emit one stable JSON
error document, while normal child stdout, stderr, and numeric exit status
remain capability-owned.

- Checkup plan/apply is provider-aware: only `provider=qoder` can emit or execute
`qodercli` disable mutations. Other hosts keep candidates as `manual-review`
until a provider-native apply contract exists. `provider-home` source
Expand Down
114 changes: 114 additions & 0 deletions docs/specs/2026-08-01-html-finding-fix-recording.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Keep Portable HTML Fix Records Consistent

## Traceability

- Spec ID: `html-finding-fix-recording`
- Status: Implemented
- Review context: `QoderAI/better-harness#39`

## Intent

Complete the finding-bound repair flow introduced for portable HTML reports.
The fix recorder must recognize a valid HTML report without requiring Qoder's
Canvas sidecar, validate the correct report contract, and publish the updated
`findings.json`, `report.md`, and `report.html` as one consistent revision.
Qoder Canvas reports must retain their stricter split-artifact validation.

## Acceptance Scenarios

- **HFR-AC-1 (report-family resolution):** a compact findings document with
`report.md` and `report.html`, and without Canvas artifacts, is recognized as
an HTML report. A split Canvas report is recognized only through its Canvas
artifacts. Mixed, incomplete, or otherwise ambiguous artifact sets fail
closed with a stable report-context error.
- **HFR-AC-2 (HTML recording):** a valid result for an HTML finding records
`actualOutput`, its AI-authored Assignment Summary, and its independent repair
review without reading or creating `canvas.json`; the finding revision and
report repair progress advance exactly once.
- **HFR-AC-3 (three-artifact consistency):** after a successful HTML record,
`findings.json`, `report.md`, and `report.html` are regenerated from the same
updated report data. The HTML action callback carries the new expected
revision, and the report directory still contains only those three durable
report artifacts.
- **HFR-AC-4 (failure safety):** stale revisions, invalid result payloads,
invalid output paths, render-validation failures, and ambiguous report
families do not change any durable report artifact and do not consume the
result file.
- **HFR-AC-5 (Canvas isolation):** existing Qoder split reports still require
and validate their Canvas sidecar, and a fabricated or missing sidecar cannot
make an HTML report pass through the Canvas contract.
- **HFR-AC-6 (result lifecycle):** workflow guidance keeps the temporary result
outside the renderer-owned report directory and consumes it only after the
recorder returns `status: "pass"`.

## Non-goals

- Changing Loop Effectiveness scores or treating a repair review as delayed
outcome evidence.
- Changing the callback transport contract already shipped by PR #39.
- Redesigning Qoder Canvas rendering, durable Canvas ownership, or its split
report schema.
- Reconstructing or mutating historical reports without an explicit
finding-bound record operation.
- Guaranteeing cross-file atomicity across an operating-system or power loss;
the publish order instead keeps `findings.json` as the final commit marker.

## Plan and Tasks

1. Resolve report family from the durable sibling artifact signature before
validation: full findings, portable HTML, or Qoder Canvas. Reject mixed and
incomplete compact contexts.
2. Validate portable HTML findings with the compact task-loop contract and
preserve the existing full and Canvas validation paths.
3. Apply the result to an isolated clone, validate outputs, render Markdown and
HTML, and run the HTML evaluator before writing durable files.
4. Publish HTML and Markdown first and `findings.json` last so the revision file
remains the final commit marker for prepared artifacts.
5. Update the finding-bound-fix workflow to place temporary result payloads
outside the report directory and consume them only on recorder success.
6. Add focused tests using an actual three-artifact HTML report, plus failure
and Canvas-isolation coverage.

## Test and Review Evidence

- **HFR-AC-1/HFR-AC-2/HFR-AC-3:** add focused tests to
`test/task-loop-report.test.mjs` that render an HTML report, record one fix,
parse the resulting findings, and verify Markdown, HTML, callback revision,
repair progress, and durable artifact names.
- **HFR-AC-4:** snapshot all three artifacts before representative failures
and assert byte identity afterward; assert the temporary result remains.
- **HFR-AC-5:** retain existing split-report tests and add explicit missing and
mixed-sidecar rejection cases.
- **HFR-AC-6:** update and validate the shipped skill reference and package.
- **Regression:** run the focused report and CLI tests, `npm run pack:verify`,
and the full `npm test` suite.
- **Documentation integrity:** regenerate the Better Harness doc routing graph
and run `node --test test/doc-link-graph.test.mjs`.
- **Review readiness:** run `git diff --check`, inspect staged and unstaged
changes, and record changed modules, generated files, risks, and AI evidence.
- **Risk:** an interrupted multi-file publish can leave prepared Markdown or
HTML ahead of the old findings revision. Publishing `findings.json` last
keeps the old callback revision authoritative until all prepared views have
been installed; normal validation failures occur before any publish begins.

## Observed Validation Evidence

- The 16 focused `record-fix-output` tests passed, including the new portable
HTML success, pre-publish failure safety, and Qoder Canvas isolation cases.
- The HTML fixture proves that a successful record updates all three artifacts,
increments the embedded callback revision, advances repair progress, leaves
only `findings.json`, `report.md`, and `report.html`, and consumes an external
result only after success.
- The original three-artifact Codex report was exercised without mutation by
deliberately supplying a stale expected revision. The recorder returned
`STALE_FIX_OUTPUT_REVISION` instead of `ENOENT`, proving that the valid compact
HTML contract no longer attempts to read `canvas.json`.
- The combined report, renderer, Skill, CLI, and compatibility run passed 166
tests with one supported Windows symlink skip; its only failure was the same
pre-existing unguarded Windows symlink-permission case reproduced on the
unchanged baseline.
- The full repository run reached 1,044 passes and 6 supported skips. Its four
failures were all pre-existing Windows `EPERM` symlink-creation cases and did
not execute the changed recorder or HTML publication path.
- `npm run pack:verify` passed with 369 npm entries and 392 runtime-zip entries;
the regenerated doc graph and all six doc-link tests passed.
108 changes: 108 additions & 0 deletions docs/specs/2026-08-01-root-cli-machine-failure-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Keep Root CLI Machine Failures Parseable

## Traceability

- Spec ID: `root-cli-machine-failure-contract`
- Status: Implemented

## Intent

Keep automation able to classify failures that occur at the root CLI's
delegated-process boundary. After the root recognizes machine mode, a child
spawn failure or signal termination must produce one existing versioned error
envelope with a stable code and safe recovery hint. Human invocations should
retain short diagnostics, while successfully started child commands continue
to own their normal output bytes and exit status.

## Acceptance Scenarios

- **RCM-AC-1 (spawn failure):** when a pre-passthrough `--json` token selects
machine mode and the delegated process cannot start, stdout contains exactly
one parseable `format_version: "1.0"` error document with code
`DELEGATED_COMMAND_SPAWN_FAILED`, stderr contains no second envelope, and the
process exits with status 1.
- **RCM-AC-2 (signal termination):** when a delegated process terminates by
signal in machine mode, any incomplete child stdout or stderr is not relayed;
stdout instead contains exactly one parseable `format_version: "1.0"` error
document with code `DELEGATED_COMMAND_SIGNAL_TERMINATED`, a safe next step,
and exit status 1.
- **RCM-AC-3 (human diagnostics):** the same two failures without recognized
machine mode write no stdout and emit one concise diagnostic on stderr. A
spawn diagnostic may retain the operating-system error detail; a signal
diagnostic names the terminating signal.
- **RCM-AC-4 (normal child ownership):** delegated commands that return a
numeric status keep that status, stdout, and stderr byte-for-byte. The root
CLI adds no success or child-business-failure envelope.
- **RCM-AC-5 (output overflow):** when a delegated process starts, runs, and
exceeds the machine-mode output buffer, stdout contains exactly one parseable
`format_version: "1.0"` error document with code
`DELEGATED_COMMAND_OUTPUT_OVERFLOW` and a hint that narrows scope or drops
`--json`. Truncated child bytes and the raw `ENOBUFS` diagnostic are not
relayed, and the envelope is never confused with a start failure.

## Non-goals

- Changing capability-owned output schemas, messages, or business-failure
exit codes.
- Replacing synchronous argv-array dispatch, changing the command registry, or
adding shell-string execution.
- Activating the ADR's proposed `command-contract.v1` shape beyond the root
CLI's existing `format_version: "1.0"` envelope.
- Normalizing malformed argument bytes or redesigning all global-option
parsing.

## Plan and Tasks

1. Keep dispatch resolution in `scripts/better-harness-cli/cli.mjs`, and add a
single result-normalization path that receives the already-recognized
machine-mode state.
2. In machine mode, capture delegated stdout and stderr until `spawnSync`
returns. Relay both unchanged for numeric child statuses, but replace
incomplete output with the root error envelope for spawn and signal
failures.
3. Reuse the root `errorPayload`/`jsonDocument` writer for versioning and add a
narrow process-runner/output-writer seam so focused tests can deterministically
supply cross-platform spawn and signal results.
4. Add focused root CLI tests for spawn failure, signal termination, human
diagnostics, absence of a stderr envelope, and normal byte preservation.

## Test and Review Evidence

- **RCM-AC-1/RCM-AC-2:** run
`node --test test/better-harness-cli.test.mjs`; focused injected-result tests
must parse all of machine stdout with `JSON.parse`, assert the stable error
code and hint, and assert empty stderr even when the simulated child result
contains partial output.
- **RCM-AC-3:** the same focused tests assert empty stdout and bounded stderr in
human mode for spawn and signal failures.
- **RCM-AC-4:** compare delegated root CLI output with direct capability output
and assert identical stdout, stderr, and numeric exit status.
- **RCM-AC-5:** drive a real `spawnSync` overflow with a small `maxBuffer`, then
assert the resulting `ENOBUFS` shape produces the overflow envelope instead of
the spawn-failure envelope in machine mode and a bounded stderr diagnostic in
human mode.
- **Documentation integrity:** run
`node scripts/doc-link-graph/cli.mjs skills/better-harness` and
`node --test test/doc-link-graph.test.mjs` after adding this Markdown file.
- **Review readiness:** run `git diff --check` and inspect staged and unstaged
changes separately.
- **Risk:** machine-mode buffering delays delegated output until the child
exits and is bounded by Node's synchronous child-process buffer. Keep the
buffer explicit and large enough for current machine documents; buffer
exhaustion stays a stable machine envelope with its own overflow code rather
than leaking a raw diagnostic or truncated child bytes into machine stdout.

## Observed Validation Evidence

- `node --test test/better-harness-cli.test.mjs` passed 36 tests with one
pre-existing Windows symlink-permission skip. The focused injected-result
tests parse the complete spawn/signal stdout as one JSON document, assert
empty machine stderr, and prove partial child envelopes are discarded.
- The combined CLI, report, Skill, renderer, and compatibility run retained all
upstream leaf-command behavior and passed every root CLI and
`scripts-refactor-contract` assertion.
- The full repository run reached 1,044 passes and 6 supported skips. Its four
failures were pre-existing Windows `EPERM` symlink-creation cases; the same
failure was reproduced from the unchanged baseline and does not execute the
changed CLI normalization path.
- `npm run pack:verify`, the six doc-link tests, and `git diff --check` passed.
105 changes: 89 additions & 16 deletions scripts/better-harness-cli/cli.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import { createStyle, formatRows, shouldUseColor } from "./format.mjs";
const scriptsRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
const repoRoot = path.resolve(scriptsRoot, "..");
const programName = "better-harness";
const MACHINE_OUTPUT_MAX_BUFFER = 64 * 1024 * 1024;

const HELP_GROUPS = [
{
Expand Down Expand Up @@ -255,7 +256,11 @@ function hasHelpFlag(argv) {
}

function hasJsonFlag(argv) {
return argv.includes("--json");
for (const value of argv) {
if (value === "--") return false;
if (value === "--json") return true;
}
return false;
}

function optionValue(argv, name) {
Expand Down Expand Up @@ -492,26 +497,94 @@ export function resolveDispatch(argv = []) {
};
}

export function main(argv = process.argv.slice(2)) {
const dispatch = resolveDispatch(argv);
if (dispatch.kind === "help" || dispatch.kind === "json") {
process.stdout.write(dispatch.text);
return dispatch.exitCode;
function emitRootResult(result, { stdout, stderr }) {
if (result.kind === "help" || result.kind === "json") {
stdout.write(result.text);
return result.exitCode;
}
if (dispatch.kind === "error") {
process.stderr.write(dispatch.message);
return dispatch.exitCode;
stderr.write(result.message);
return result.exitCode;
}

function writeChildOutput(stream, value) {
if (value !== null && value !== undefined && value.length > 0) {
stream.write(value);
}
}

const result = spawnSync(process.execPath, [dispatch.script, ...dispatch.args], {
cwd: process.cwd(),
stdio: "inherit",
windowsHide: true,
});
function signalName(value) {
return typeof value === "string" && /^[A-Z0-9]+$/u.test(value)
? value
: "UNKNOWN";
}

function normalizeDispatchResult(result, {
machine,
stdout = process.stdout,
stderr = process.stderr,
} = {}) {
if (result.error?.code === "ENOBUFS") {
return emitRootResult(rootError({
code: "DELEGATED_COMMAND_OUTPUT_OVERFLOW",
message: "The delegated command produced more output than machine mode can buffer.",
hint: machine
? "Narrow the command scope, or rerun without `--json` to stream the full output."
: undefined,
machine,
}), { stdout, stderr });
}

if (result.error) {
process.stderr.write(`${result.error.message}\n`);
return 1;
return emitRootResult(rootError({
code: "DELEGATED_COMMAND_SPAWN_FAILED",
message: machine
? "Failed to start the delegated command."
: `Failed to start the delegated command: ${result.error.message}`,
hint: machine
? "Verify the Better Harness installation, then retry the command."
: undefined,
machine,
}), { stdout, stderr });
}

if (result.signal) {
const signal = signalName(result.signal);
return emitRootResult(rootError({
code: "DELEGATED_COMMAND_SIGNAL_TERMINATED",
message: `The delegated command terminated with signal ${signal}.`,
hint: machine
? "Retry the command; if it repeats, inspect the delegated capability diagnostics."
: undefined,
machine,
}), { stdout, stderr });
}

if (machine) {
writeChildOutput(stdout, result.stdout);
writeChildOutput(stderr, result.stderr);
}
return result.status ?? 1;
}

export function main(argv = process.argv.slice(2), runtime = {}) {
const {
spawn = spawnSync,
stdout = process.stdout,
stderr = process.stderr,
cwd = process.cwd(),
} = runtime;
const dispatch = resolveDispatch(argv);
if (dispatch.kind !== "dispatch") {
return emitRootResult(dispatch, { stdout, stderr });
}

const machine = hasJsonFlag(argv);
const result = spawn(process.execPath, [dispatch.script, ...dispatch.args], {
cwd,
stdio: machine ? ["inherit", "pipe", "pipe"] : "inherit",
windowsHide: true,
...(machine ? { maxBuffer: MACHINE_OUTPUT_MAX_BUFFER } : {}),
});

return normalizeDispatchResult(result, { machine, stdout, stderr });
}
Loading