Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Contributing

Boatstack is a generated content distribution. Propose changes to workflow semantics, templates, evidence rules, or generated presentation in [Intelligence Flow](https://github.com/operatorstack/intelligence-flow/tree/9a5fd09f964901236c1e2708a8087f181617e9a9/examples/12-product-engineering-loop).
Boatstack is a generated content distribution. Propose changes to workflow semantics, templates, evidence rules, or generated presentation in [Intelligence Flow](https://github.com/operatorstack/intelligence-flow/tree/d37975f8c3960b03c701f029d4815f6d6814ed7c/examples/12-product-engineering-loop).

The Boatstack repository receives product/runtime changes through a generated pull request. Review the PR's `UPSTREAM.json`, tests, adapter diff, and context-size change; do not hand-edit generated output on `main`. `.github/workflows` is the exception: it is Boatstack's executable control plane, excluded from scheduled projection and changed only through a separate manually reviewed Boatstack PR.

Expand Down
33 changes: 33 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,43 @@ Install Boatstack in this repository from https://github.com/operatorstack/boats

Install Boatstack in its own infrastructure PR and merge that PR before starting a feature. This keeps one-time repository setup out of later product diffs.

## Start with two moves

1. Create and save a plan in your coding tool's Plan mode.
2. Run `/auto-plan`.

That is all you need to learn up front. Boatstack shows you one next action at a time through approval, building, tests, review, and PR preparation.

> The diagram below shows what Boatstack guides—not a checklist you need to memorize.

<p align="center">
<img src="assets/boatstack-journey.svg" width="960" alt="One feature moves from idea through planning, approval, building, tests, review, and pull request; its retained plans, decisions, gaps, evidence, and code state combine with the next idea to create the next plan">
</p>

## Use the model that fits your budget

Boatstack applies the same planning, approval, testing, review, and shipping requirements whichever coding model you choose. Lower-cost models remain an option without lowering the standard required to call the work complete.

<p align="center">
<img src="assets/boatstack-model-choice.svg" width="820" alt="Lower-cost, general, and frontier models enter the same Boatstack completion standard and produce a reviewed change with evidence">
</p>

> This does not mean every model performs equally. Boatstack makes the process less dependent on the model catching every mistake by itself.

> **Designed for model flexibility · Quality uplift evaluation in progress**

| Without Boatstack | With Boatstack |
|---|---|
| Quality depends heavily on the model catching every mistake itself | Planning, approval, tests, and review provide additional checks |
| Switching to a lower-cost model may also change the development process | The completion standard stays consistent across models |
| Important context disappears between features | Decisions, gaps, evidence, and code state inform the next feature |

- <!-- boatstack-claim:model-neutral-contract -->**Verified:** Boatstack uses the same completion requirements regardless of model, provider, or price.
- <!-- boatstack-claim:cross-model-failures -->**Observed:** benchmark runs exposed failures in protocol handling, context, verification, and recovery—not only model capability.
- <!-- boatstack-claim:lower-cost-outcomes -->**Being evaluated:** whether this measurably improves product quality or cost when using lower-cost models.

[See the evidence and paired evaluation design](docs/why-these-steps.md#model-choice-and-budget).

## From idea to PR

1. **Explore the idea in your host's Plan mode.** Save the plan, then run `/auto-plan`. Boatstack finds relevant repository facts and asks only for decisions the code cannot answer.
Expand Down
15 changes: 8 additions & 7 deletions UPSTREAM.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,11 @@
},
"files": {
".gitignore": "a7079e923a776f14f1bb3a6aa0a11a133a8e1dfb35af020f327623357b7e3957",
"CONTRIBUTING.md": "8469bb0fd7d8f8af51ea8ea414d1abad6a8163427e43338b9cf8449d701ff6d0",
"README.md": "8c8a299369720a400946fdc58d3cf404b74783f90794ecde52441030b8699677",
"CONTRIBUTING.md": "fe20dd1fbfa7e5b2cca253ce089808b388250ddecec603099688064a23e4e664",
"README.md": "0b817ceabe61d874f21f98362f550cb77f1819bfc3db6df40b69337f50b4641c",
"assets/boatstack-journey.svg": "c1f7fe2741f5e9ca66bb3fe9b103e6364ba5acbca8b7a8054768ffd85cf325ea",
"assets/boatstack-mark.svg": "c46e935f06fcfde3b37abfd579c1963b765b2337a0fa993f9538c9b652297e39",
"assets/boatstack-model-choice.svg": "979952c2fc6220d41426f9072186fca000fb3f388a4e775cc09cc1e830ebdde4",
"boatstack/SKILL.md": "4d80c39521763831655df0f98224769e0bf3caf535c0751ecf96c33bc33039e5",
"boatstack/agents/openai.yaml": "68a30a60859556c5a26e16d184594ca243a6043d99c8cf7d66b5dd6d50a93cd1",
"boatstack/assets/templates/adr.md": "c577a3c1c1319061f61deb053597e6e853657022185fe28b8f733327e2a78565",
Expand Down Expand Up @@ -59,22 +60,22 @@
"docs/account-recovery-walkthrough.md": "acd3558a95f48004f18a0590670de496e1cc9f0cd1d187f924615497f57e1d6f",
"docs/benchmark-corpus-audit.md": "f2d206fe8579a514f9da82b2c96c19b343ac004be67617e1bd34f0f8e0e5e6c6",
"docs/benchmark-submission-audit.md": "9518abdd17690729c6423f87cab20418ed47b0915b5faa44b9ef975e9e9c3b79",
"docs/evidence-engineered-coding.md": "6e02c6ff8e5f2d3f7da84b56a32038fcdb6d2dc4337bd359bdff121f138a5f11",
"docs/evidence-engineered-coding.md": "f14acdbaf6da6a11ad8ce73e5befde8af5f879571452b679d18a71a2c6f18191",
"docs/generated-files.md": "7f4d7cddff80d794361c26962b4c89b4e525eafb53776553be966258c6d867c6",
"docs/getting-started.md": "dccb5895bfbf6ac40309c6779f2a23dcdc27355e62a8a4858bb37c043e8de442",
"docs/public-claims.json": "93f058ee600bd3eb4a921292e6c6465b1a0fb66dae03e510f5fc9e1202dbbbd0",
"docs/public-claims.json": "73e41adf3c8ae7b6946145f65d80950842b16cef2350edca3d0936b9b6b21f93",
"docs/public-surface.md": "53d741f04b2928a6ee8c006d647a6d675a215e863412e5862cd67d48433bff76",
"docs/research-and-design.md": "d65c66e323037bda5d45aacef5d48afa6bf93da55901378891d235aca3a5684f",
"docs/safety.md": "7b9b5c515d36e683767ec8d3d9d6d119ac93650b2f629d351deadd4c600ed6a6",
"docs/troubleshooting.md": "67ec380fc24226d6afd1d4fe250a123edd153f88bc1e318ee2ef7635e0ae1520",
"docs/validation-and-evidence.md": "a9fe9274f3dc22b152094a307feda5d8c3ab099755100aef77bda13024cc3166",
"docs/why-these-steps.md": "7c6856c1cdff9946c771295184ce9fb552e0732f1f2ae8adf83aa72590bce9a7",
"docs/why-these-steps.md": "c45092ed7b4c8a913a83b4fcfa22494c76d864fb32ff5943febd6abe1f7de81f",
"examples/diagram-json/README.md": "061b583180e43bbd26618bbd9d3d79af4b75d7c8f37c66475640745a97328fbc",
"examples/diagram-json/approval.md": "bc421a825349923512d5cb0ce489310d3a4d7cbac35e661a693b4a32eec263d1",
"examples/diagram-json/compiled/evidence.md": "1ba1c989ade070a8ef9a508fbd788d100d7292f2dbacbb2bce895468019f619d",
"examples/diagram-json/compiled/tasks.json": "f040696f1f8bcedc4a8ed9816a61a49edbda970ec0cc3b28175ba37b73bbc896",
"examples/diagram-json/compiled/test-matrix.json": "6c6895c509271e4337f3c91d9f62ee3a2b34e768e78513784cb012506a328ecf",
"examples/diagram-json/plan.lock.json": "4c0694926165d2c5458e8ec1b7d0db7cdfbd6bbf8f7eb84fdb53e6d519149e5d",
"examples/diagram-json/plan.lock.json": "b10ea959364285f9ed551a31283f7690c19a921c25d291afb4fa354ae5c7f0aa",
"examples/diagram-json/plan.md": "3ad35cc3cbe48306e7ee401bd9e9047d25e46c8a6fe9679aa1b3f5e96ceea292",
"examples/diagram-json/questions.md": "1a0050041cac0a8d53e6ebfe04cbec4a298cdc8c50efeeb6fa15aeb663c5ec76",
"examples/diagram-json/request.md": "0808fc41c36779c404f4a3a121167da6e76cac56df526e70f9ed6d3e0d4c02ed",
Expand All @@ -87,7 +88,7 @@
"generator": "operatorstack/intelligence-flow:boatstack-distribution",
"schema_version": 1,
"source": {
"commit": "9a5fd09f964901236c1e2708a8087f181617e9a9",
"commit": "d37975f8c3960b03c701f029d4815f6d6814ed7c",
"path": "examples/12-product-engineering-loop",
"repository": "operatorstack/intelligence-flow"
}
Expand Down
36 changes: 36 additions & 0 deletions assets/boatstack-model-choice.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/evidence-engineered-coding.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,6 @@ Delivery and system improvement also remain separate. A failed task may suggest

## What is evidence-backed

The current moves were derived from the Intelligence Flow benchmark corpus and product-repository studies. The generated source commit is [`9a5fd09f964901236c1e2708a8087f181617e9a9`](https://github.com/operatorstack/intelligence-flow/tree/9a5fd09f964901236c1e2708a8087f181617e9a9/examples/12-product-engineering-loop).
The current moves were derived from the Intelligence Flow benchmark corpus and product-repository studies. The generated source commit is [`d37975f8c3960b03c701f029d4815f6d6814ed7c`](https://github.com/operatorstack/intelligence-flow/tree/d37975f8c3960b03c701f029d4815f6d6814ed7c/examples/12-product-engineering-loop).

The evidence supports specific failure mechanisms and guardrails. It does not establish that Boatstack is optimal, that control-theory notation proves software quality, or that one workflow dominates every team. Those are evaluation questions, so the distribution preserves measurements, provenance, gaps, and negative results.
43 changes: 38 additions & 5 deletions docs/public-claims.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"schema_version": 1,
"source_commit": "9a5fd09f964901236c1e2708a8087f181617e9a9",
"source_commit": "d37975f8c3960b03c701f029d4815f6d6814ed7c",
"statuses": ["verified", "observed", "still_being_evaluated"],
"claims": [
{
Expand All @@ -12,7 +12,7 @@
"readable_evidence": "why-these-steps.md#human-decisions",
"implementation": ["../boatstack/references/workflow.md", "../boatstack/plan.go"],
"verification": ["../boatstack/plan_test.go", "../boatstack/planning_test.go"],
"last_verified_version": "source:9a5fd09f964901236c1e2708a8087f181617e9a9"
"last_verified_version": "source:d37975f8c3960b03c701f029d4815f6d6814ed7c"
},
{
"id": "validation-provenance",
Expand All @@ -23,7 +23,7 @@
"readable_evidence": "why-these-steps.md#validation-provenance",
"implementation": ["validation-and-evidence.md", "../boatstack/plan.go"],
"verification": ["../boatstack/plan_test.go"],
"last_verified_version": "source:9a5fd09f964901236c1e2708a8087f181617e9a9"
"last_verified_version": "source:d37975f8c3960b03c701f029d4815f6d6814ed7c"
},
{
"id": "irreversible-operations",
Expand All @@ -35,7 +35,7 @@
"readable_evidence": "why-these-steps.md#irreversible-operations",
"implementation": ["safety.md", "../boatstack/safety.go", "../boatstack/hooks.go"],
"verification": ["../boatstack/safety_test.go", "../boatstack/hooks_test.go"],
"last_verified_version": "source:9a5fd09f964901236c1e2708a8087f181617e9a9"
"last_verified_version": "source:d37975f8c3960b03c701f029d4815f6d6814ed7c"
},
{
"id": "reviewer-ready-pr",
Expand All @@ -46,7 +46,40 @@
"readable_evidence": "why-these-steps.md#reviewer-ready-pr",
"implementation": ["../boatstack/pr.go", "getting-started.md"],
"verification": ["../boatstack/pr_test.go"],
"last_verified_version": "source:9a5fd09f964901236c1e2708a8087f181617e9a9"
"last_verified_version": "source:d37975f8c3960b03c701f029d4815f6d6814ed7c"
},
{
"id": "model-neutral-contract",
"public_claim": "Boatstack applies the same planning, approval, validation, review, and shipping requirements regardless of model, provider, or price.",
"status": "verified",
"originating_observation": "Changing models relocated the dominant benchmark bottleneck instead of producing one stable lower-cost-model or frontier-model failure profile.",
"safeguard": "One model-neutral workflow that branches on observable decisions, risk, tool outcomes, convergence, and evidence rather than model identity.",
"readable_evidence": "why-these-steps.md#model-choice-and-budget",
"implementation": ["research-and-design.md", "../boatstack/references/workflow.md"],
"verification": ["../boatstack/export_test.go", "../boatstack/planning_test.go"],
"last_verified_version": "source:d37975f8c3960b03c701f029d4815f6d6814ed7c"
},
{
"id": "cross-model-failures",
"public_claim": "The audited benchmark runs exposed failures in protocol handling, context, verification, and recovery in addition to differences in model capability.",
"status": "observed",
"originating_observation": "The audited Gemini, Qwen, and submission runs recorded malformed responses, context-window regression, unsupported verification, step exhaustion, near misses, and unsafe recovery behavior.",
"safeguard": "Trace safeguards to observed failure mechanics without treating model provider or price as the failure state.",
"readable_evidence": "why-these-steps.md#model-choice-and-budget",
"implementation": ["research-and-design.md"],
"verification": ["benchmark-corpus-audit.md", "benchmark-submission-audit.md"],
"last_verified_version": "source:d37975f8c3960b03c701f029d4815f6d6814ed7c"
},
{
"id": "lower-cost-outcomes",
"public_claim": "Whether Boatstack improves product correctness, completion cost, or delivery time when using lower-cost models is still being evaluated.",
"status": "still_being_evaluated",
"originating_observation": "The existing corpus supports specific failure mechanisms but contains no paired product-feature result proving a lower-cost-model outcome uplift.",
"safeguard": "Keep the outcome claim unpromoted until the same feature, model, budget, environment, and host are evaluated with and without Boatstack.",
"readable_evidence": "why-these-steps.md#model-choice-and-budget",
"implementation": ["research-and-design.md"],
"verification": ["benchmark-corpus-audit.md", "benchmark-submission-audit.md"],
"last_verified_version": "source:d37975f8c3960b03c701f029d4815f6d6814ed7c"
}
]
}
12 changes: 12 additions & 0 deletions docs/why-these-steps.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,18 @@ Those labels prevent an implementation test from being presented as proof that t

**Status:** product-workflow problem observed; projection behavior verified. Reviewer speed and acceptance quality still need blinded product-delivery evaluation.

## Model choice and budget

**What happened.** Across the audited benchmark runs, changing the model relocated the dominant bottleneck instead of removing failure. Gemini runs were dominated by near misses in one comparison, while Qwen runs exposed step exhaustion. Other recorded failures involved malformed protocol responses, context loss, unsupported verification claims, and unsafe recovery. A model name, provider, or price was not itself a reliable description of the active engineering problem.

**What Boatstack does.** Boatstack keeps one planning, approval, validation, review, and shipping contract across models. It reacts to observable conditions such as unanswered decisions, risk, reversibility, tool outcomes, convergence, and evidence. This lets a repository owner choose a lower-cost, general, or frontier model without silently changing what “ready” means.

**How we check it.** Export and projection tests verify the same workflow and gate vocabulary across supported coding hosts. The benchmark audits preserve the recorded trial coverage and the research record traces the model-dependent bottlenecks and failure mechanisms behind this design.

**Status:** the model-neutral contract is verified and the cross-model failure patterns are observed. Whether Boatstack improves correctness, cost, or delivery time for lower-cost models is **still being evaluated**; it is not a claim that models perform equally.

The paired product evaluation will use the same feature, lower-cost model, budget, environment, and coding host with and without Boatstack. An independent evaluator will compare correctness, regressions, review findings, cost, completion time, and the evidence available to support completion claims.

## What the experiments do and do not support

The current research covers thousands of locally available benchmark result records, preregistered comparisons, product-repository studies, and targeted trajectory inspection. It supports the mechanisms that Boatstack is designed to address. It does **not** yet support a claim that Boatstack improves feature success, cost, or delivery speed.
Expand Down
2 changes: 1 addition & 1 deletion examples/diagram-json/plan.lock.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"plan_path": "examples/diagram-json/plan.md",
"plan_sha256": "3ad35cc3cbe48306e7ee401bd9e9047d25e46c8a6fe9679aa1b3f5e96ceea292",
"schema_version": 1,
"source_commit": "9a5fd09f964901236c1e2708a8087f181617e9a9",
"source_commit": "d37975f8c3960b03c701f029d4815f6d6814ed7c",
"source_plan_path": "examples/diagram-json/source-plan.md",
"source_plan_sha256": "e10593ddaa7522ab80cc991d0a09399257139799e37f737794cd49d68a39985b",
"spec_path": "examples/diagram-json/spec.md",
Expand Down
Loading