Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Changelog

## Unreleased — after 0.2.0-rc.1

- Add explicit `init --command-only` scaffolding for CLI and CI maintenance. Start with one command criterion; retain input discovery, evidence states and overwrite refusal. The default scaffold continues to include a visual criterion.

## 0.2.0-rc.1 — prerelease candidate

- Normalize selected Intake snapshots consistently at every contract-file entry; validate the full source/requirement structure before admitting confirmed items. Keep explicit mappings and optional imports.
Expand Down
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,13 @@ Initialize a graph:
proofline init
```

For headless maintenance, the **unreleased source checkout** also supports
`node ./bin/proofline.js init /path/to/project --command-only`. It creates one
command criterion, so a CLI-only task can complete without removing an unrelated
screenshot criterion. Review the generated acceptance text and discovered input
paths before recording checks. The published `0.2.0-rc.1` archives keep their
original interface; see the [first-use maintenance case](docs/MAINTENANCE_CASE_2026-09-12.md).

Describe acceptance criteria in `proofline.json`:

```json
Expand Down
78 changes: 78 additions & 0 deletions docs/MAINTENANCE_CASE_2026-09-12.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# First-use observation: Workprint CI maintenance

On 2026-09-12, the Lab maintainer used the **published Proofline 0.2.0-rc.1 CLI**
while reviewing two existing Dependabot pull requests in Workprint:
[checkout #1](https://github.com/codex-improvement-lab/codex-workprint/pull/1)
and [setup-node #2](https://github.com/codex-improvement-lab/codex-workprint/pull/2).
This was actual internal maintenance, not an external-user trial or a replay
of the earlier [14/22-operation study](SIMPLIFICATION_RESULTS_2026-09-09.md).

## Task and evidence

Each PR changes one pinned GitHub Action in the existing workflow. The maintainer
checked the upstream tag/commit, relevant breaking changes, unchanged read-only
permissions, and all four Windows/macOS × Node 22/24 jobs. Each branch was refreshed
against main before verification and merged sequentially. The second refresh
included the first merge. A final check targets the combined main revision.

Proofline wraps a maintainer-authored, read-only GitHub API checker. The checker
requires an explicit target SHA, the expected diff and action pins, four successful
jobs, and successful test/release/build/package steps. Its script and target files
are tracked command inputs. Each PR and the combined result get separate receipts.

| Revision under review | Hosted CI | Result |
| --- | --- | --- |
| Refreshed checkout PR, `db68cfb0d320ffb00d72edec1a0120b7ce40892f` | [34678558827](https://github.com/codex-improvement-lab/codex-workprint/actions/runs/34678558827) | Four jobs passed; PR merged |
| Refreshed setup-node PR, `bd80dc70b8420dcf31d8b2e1f1fee47363641a4c` | [34678715695](https://github.com/codex-improvement-lab/codex-workprint/actions/runs/34678715695) | Four jobs passed; PR merged |
| Combined main, `eeec646c6663fad59805d5ac5e235107f6dd264d` | [34678840844](https://github.com/codex-improvement-lab/codex-workprint/actions/runs/34678840844) | Four jobs passed; only the workflow changed |

The final Proofline query returned three verified receipts and zero gaps. The
[redacted observation record](evidence/maintenance-2026-09-12.json) retains the
selected revisions, receipt hashes and measurement limits.

A command receipt here records **a Windows process reading GitHub-hosted CI
results**. It does not mean Proofline independently ran the remote tests or observed
a physical Mac. The API checker remains private because it uses the maintainer's
local credential helper; no credentials, local paths, raw receipt or command output
are published in this case note.

## First-use friction and resulting change

The released `init` scaffold creates a command criterion and a screenshot criterion.
This task has no visual acceptance condition, so the maintainer had to remove the
screenshot criterion while configuring the actual CI checks.

The **unreleased source checkout** adds one explicit option:

```sh
node ./bin/proofline.js init /path/to/project --command-only
```

It starts with one command criterion and still discovers existing conventional
input paths. Acceptance wording and input associations require review. The default
scaffold continues to include a visual criterion; existing manifests are never
overwritten. Missing evidence starts as missing, a failed command stays failed,
and changed tracked inputs make an old passing receipt stale.

The option is not present in the frozen 0.2.0-rc.1 release archives. CLI regression
and installed-package smoke cover a headless init/run/check loop, a spaced project
path, source invalidation, failed execution, and overwrite refusal.

## What this observation can support

It identifies one irrelevant criterion that needs removal in this class of task. It does not
establish net savings. Manifest customization, explicit target updates, and the
API checker all cost work. GitHub already retains the underlying CI history; the
additional Proofline record needs to earn its maintenance cost on subsequent use.

The observed product path used eight CLI invocations: one version probe, one init,
three recorded API checks, one readiness check, one gaps query and one Markdown
report. The custom checker, manifest customization and target-file writes are
additional work. This invocation count is not the earlier study's operation metric;
the scaffold option does not by itself remove a CLI invocation in this case.

The owner had already inspected notifications and initial PRs before starting the
record. There is no frozen comparison arm, complete task timer, model-usage record,
external-user observation, or repeat-use outcome. The earlier controlled study
remains unchanged. No artificial requirement change was introduced to demonstrate
Goal Delta.
10 changes: 5 additions & 5 deletions docs/evidence/macos-replacement-candidate.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
"product": "codex-proofline",
"gateVersion": 2,
"scope": "runtime-gate-tests-v1",
"id": "codex-proofline-macos-rc-24fc73aa5cbb10e1",
"contentSha256": "24fc73aa5cbb10e1beadd169d46f14ed5dd521f743002116959b88eaee4fede1",
"id": "codex-proofline-macos-rc-ec04ead371789c89",
"contentSha256": "ec04ead371789c89607f4e8ab063342fee7e2ea9fed3ac579479e44db66c573b",
"contentFiles": 53,
"installedVersion": "0.2.0-rc.1+codex.macos-24fc73aa5cbb10e1",
"installedVersion": "0.2.0-rc.1+codex.macos-ec04ead371789c89",
"status": "awaiting-full-real-mac-rerun",
"previousMacEvidenceImported": false,
"generatedOn": "2026-09-09",
"generatedOn": "2026-09-12",
"evidenceClass": "source-derived-candidate-identity-not-macos-execution",
"documentationUpdate": "Publication README and demo links refreshed; release v0.2.0-rc.1 remains frozen at 033903f6602b6f490e93df6d9364ec56b5ae35f3. No prior physical-Mac result transferred."
"documentationUpdate": "Unreleased command-only init and regression/package coverage. Release v0.2.0-rc.1 stays frozen; physical Mac and fresh-task activation remain pending."
}
61 changes: 61 additions & 0 deletions docs/evidence/maintenance-2026-09-12.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"schemaVersion": "proofline-maintenance-observation/1",
"date": "2026-09-12",
"evidenceClass": "actual-internal-maintenance-observation",
"project": "codex-improvement-lab/codex-workprint",
"prooflineVersion": "0.2.0-rc.1",
"prooflineRelease": "https://github.com/codex-improvement-lab/codex-proofline/releases/tag/v0.2.0-rc.1",
"observationEnvironment": "Windows / GitHub API readback",
"records": [
{
"evidenceId": "C-CHECKOUT/ci",
"head": "db68cfb0d320ffb00d72edec1a0120b7ce40892f",
"ci": "https://github.com/codex-improvement-lab/codex-workprint/actions/runs/34678558827",
"successfulJobs": 4,
"receipt": "sha256:702ba7cf6c6ff95af8735240c2bcf974ce2ac2425b2c0d54274398f535cafb5b"
},
{
"evidenceId": "C-NODE/ci",
"head": "bd80dc70b8420dcf31d8b2e1f1fee47363641a4c",
"ci": "https://github.com/codex-improvement-lab/codex-workprint/actions/runs/34678715695",
"successfulJobs": 4,
"receipt": "sha256:4ff887e5c64ffa25cf08ba40788adf49ab04f6ce0c644b2bcb486017f75ad4fd"
},
{
"evidenceId": "C-MAIN/ci",
"head": "eeec646c6663fad59805d5ac5e235107f6dd264d",
"ci": "https://github.com/codex-improvement-lab/codex-workprint/actions/runs/34678840844",
"successfulJobs": 4,
"receipt": "sha256:9264700d7bad3f688a15c5b0585f705107056e73ede25d7df923a714b4cddf9b"
}
],
"finalQuery": {
"verified": 3,
"missing": 0,
"stale": 0,
"declaredOnly": 0,
"failed": 0,
"returnedGaps": 0
},
"observedCliInvocations": {
"version": 1,
"init": 1,
"run": 3,
"check": 1,
"query": 1,
"report": 1,
"total": 8,
"definition": "Product-path calls only; additional checker authoring, manifest and target writes are not included. Not comparable to the earlier full-operation replay metric."
},
"limits": {
"externalUser": false,
"repeatUseMeasured": false,
"comparisonArm": null,
"completeTaskTime": null,
"modelUsage": null,
"goalDeltaExercise": false,
"physicalDeviceObservation": false,
"netBenefit": "unmeasured",
"receiptBoundary": "Hashes associate private local records with this summary; hashes alone do not prove record authenticity or task correctness."
}
}
11 changes: 10 additions & 1 deletion scripts/package-smoke.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,15 @@ try {
await access(path.join(packageRoot, ".codex-plugin", "plugin.json"));
await access(path.join(packageRoot, "skills", "proofline", "SKILL.md"));
await access(path.join(packageRoot, "hooks", "hooks.json"));
const headless = path.join(scratch, "headless project");
const headlessManifest = path.join(headless, "proofline.json");
invoke(["init", headless, "--command-only"]);
invoke(["check", "--manifest", headlessManifest, "--json"], 1);
invoke(["run", "AC-01/tests", "--manifest", headlessManifest, "--", process.execPath, "-e",
"require('node:assert/strict').equal(require('node:path').basename(process.cwd()), 'headless project')"]);
const headlessStatus = JSON.parse(invoke(["check", "--manifest", headlessManifest, "--json"]));
assert.equal(headlessStatus.ready, true);
assert.equal(headlessStatus.summary.totalProofs, 1);
invoke(["init"]);
const manifestPath = path.join(scratch, "proofline.json");
const manifest = JSON.parse(await readFile(manifestPath, "utf8"));
Expand Down Expand Up @@ -85,7 +94,7 @@ try {
assert.equal(run.status, 0, run.stderr);
}
assert.equal(JSON.parse(invoke(["query", "--manifest", selectedManifest, "--contract", "requirements-next.json", "--dependencies", "deps.json", "--gaps"])).evidence.length, 0);
process.stdout.write("Installed package smoke passed: base CLI/five states/reports/plugin, exact Goal Delta projections and JSON, Intake import, configuration checks, bound run/capture and target query.\n");
process.stdout.write("Installed package smoke passed: headless init loop, base CLI/five states/reports/plugin, exact Goal Delta projections and JSON, Intake import, configuration checks, bound run/capture and target query.\n");
} finally {
assert.ok(scratch.startsWith(path.join(os.tmpdir(), "proofline-package-smoke-")));
await rm(scratch, { recursive: true, force: true });
Expand Down
15 changes: 8 additions & 7 deletions src/cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ const HELP = `
Codex Proofline — proof, not promises

Usage:
proofline init [directory]
proofline init [directory] [--command-only]
proofline run <criterion/proof> [--environment <label>] -- <command> [...args]
proofline capture <criterion/proof> [--environment <label>] [--note <text>]
proofline claim <criterion/proof> --note <text>
Expand All @@ -43,6 +43,7 @@ Common option (all commands except init):
run/capture may bind --contract <target.json> --dependencies <dependencies.json> explicitly.
Contract files may be proofline-goal-contract/1 or a selected intake-requirements/1 snapshot.
Generated rechecks include --contract-digest <sha256> to refuse target drift.
init --command-only scaffolds one command criterion for headless workflows.

States:
verified · missing · stale · declared-only · failed
Expand All @@ -63,10 +64,10 @@ const VALUE_OPTIONS = new Set([
"profile-output",
"at", "status", "item", "evidence", "contract", "input", "contract-digest"
]);
const BOOLEAN_OPTIONS = new Set(["json", "help", "gaps", "affected"]);
const BOOLEAN_OPTIONS = new Set(["json", "help", "gaps", "affected", "command-only"]);

const COMMAND_OPTIONS = {
init: [],
init: ["command-only"],
run: ["manifest", "environment", "contract", "dependencies", "contract-digest"],
capture: ["manifest", "environment", "note", "observed-at", "contract", "dependencies", "contract-digest"],
claim: ["manifest", "note"],
Expand Down Expand Up @@ -165,7 +166,7 @@ function printStatus(result) {
process.stdout.write(`\n${result.ready ? "READY" : "EVIDENCE GAPS"}\n`);
}

async function initialize(directory) {
async function initialize(directory, { commandOnly = false } = {}) {
const root = path.resolve(directory || ".");
const manifestPath = path.join(root, "proofline.json");
await mkdir(root, { recursive: true });
Expand All @@ -191,7 +192,7 @@ async function initialize(directory) {
}
]
},
{
...(commandOnly ? [] : [{
id: "AC-02",
statement: "The user-visible result has current visual evidence.",
proof: [
Expand All @@ -204,7 +205,7 @@ async function initialize(directory) {
freshnessHours: 168
}
]
}
}])
]
};
try {
Expand Down Expand Up @@ -322,7 +323,7 @@ export async function runCli(argv = process.argv.slice(2)) {
if (options.help) return process.stdout.write(`${HELP}\n`);
validateOptions(command, options);
if (positionals.length > 1) throw new ProoflineError("init accepts at most one directory.");
await initialize(positionals[0]);
await initialize(positionals[0], { commandOnly: Boolean(options["command-only"]) });
return;
}

Expand Down
45 changes: 45 additions & 0 deletions test/cli.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,56 @@ test("init creates a manifest and refuses to overwrite it", async (t) => {
const first = await invoke(["init"], directory);
assert.equal(first.exitCode, 0);
assert.match(first.stdout, /Created/u);
const initial = JSON.parse(await readFile(path.join(directory, "proofline.json"), "utf8"));
assert.deepEqual(initial.criteria.flatMap(item => item.proof.map(proof => proof.kind)), ["command", "screenshot"]);
const second = await invoke(["init"], directory);
assert.equal(second.exitCode, 2);
assert.match(second.stderr, /Refusing to overwrite/u);
});

test("command-only init completes a headless evidence loop and keeps source invalidation", async (t) => {
const directory = await createTestDirectory("cli-command-only");
t.after(() => removeTestDirectory(directory));
const project = path.join(directory, "headless project");
await mkdir(path.join(project, "src"), { recursive: true });
const input = path.join(project, "src", "result.txt");
await writeFile(input, "ready", "utf8");
const initialized = await invoke(["init", project, "--command-only"], directory);
assert.equal(initialized.exitCode, 0, initialized.stderr);
const original = await readFile(path.join(project, "proofline.json"), "utf8");
const initial = JSON.parse(original);
assert.equal(initial.criteria.length, 1);
assert.deepEqual(initial.criteria[0].proof[0].inputs, ["src"]);
assert.equal((await invoke(["check", "--json"], project)).exitCode, 1);
const runArgs = ["run", "AC-01/tests", "--", process.execPath, "-e",
"require('node:assert/strict').equal(require('node:fs').readFileSync('src/result.txt', 'utf8'), 'ready')"];
const observed = await invoke(runArgs, project);
assert.equal(observed.exitCode, 0, observed.stderr);
const ready = await invoke(["check", "--json"], project);
assert.equal(ready.exitCode, 0, ready.stderr);
assert.equal(JSON.parse(ready.stdout).summary.verifiedProofs, 1);
await writeFile(input, "changed", "utf8");
const stale = await invoke(["check", "--json"], project);
assert.equal(stale.exitCode, 1);
assert.equal(JSON.parse(stale.stdout).criteria[0].proofs[0].status, "stale");
assert.equal((await invoke(runArgs, project)).exitCode, 1);
const failed = await invoke(["check", "--json"], project);
assert.equal(failed.exitCode, 1);
assert.equal(JSON.parse(failed.stdout).criteria[0].proofs[0].status, "failed");
assert.equal((await invoke(["init", "--command-only"], project)).exitCode, 2);
assert.equal(await readFile(path.join(project, "proofline.json"), "utf8"), original);
});

test("command-only is explicit, scoped to init and rejects duplicates without writing", async (t) => {
const directory = await createTestDirectory("cli-command-only-options");
t.after(() => removeTestDirectory(directory));
for (const args of [["init", "--command-only", "--command-only"], ["status", "--command-only"],
["init", "--command-only", "--output", "unexpected.json"]]) {
assert.equal((await invoke(args, directory)).exitCode, 2);
assert.deepEqual(await readdir(directory), []);
}
});

test("init prefills only present conventional inputs; doctor exposes configuration omissions", async (t) => {
const directory = await createTestDirectory("doctor");
t.after(() => removeTestDirectory(directory));
Expand Down
Loading