diff --git a/README.md b/README.md index ebe6fb30..9d49b5b5 100644 --- a/README.md +++ b/README.md @@ -1,393 +1,194 @@ -

- Suites logo -

- -

Suites / Blackbox

- -**A system testing framework for executable specifications and runtime verification, built for developers and coding -agents.** - -Blackbox turns accepted specifications into repeatable verification against real applications and subsystems. Coding -agents can execute the specified behavior, inspect what happened, investigate failures, repair the implementation, and -rerun against the same accepted expectations. - -Expected behavior can be expressed directly in native Playwright tests or through optional executable Feature files -that generate a native suite. Each test attempt runs in a fresh isolated Sandbox, using response, state, and runtime -evidence to evaluate the specified claims. - -## Spec-Driven Verification - -Spec-Driven Verification checks a running system against an accepted specification. That specification can come from -product requirements, acceptance criteria, an API contract (like Swagger and OpenAPI), a Markdown document, or any -other source of agreed behavior. - -**Accepted behavior defines the checks; each execution supplies evidence.** +

BLACKBOX

-Developers and coding agents turn those requirements into executable expectations for review. They can write native -Playwright tests directly or generate suites from optional Feature files. Blackbox runs the selected application or -subsystem and gathers evidence for the specified claims. - -The example below follows the optional Feature-file workflow, from a Markdown specification through review and suite -generation to runtime verification: +

Code changes. Behavior is the contract.

- create-new-product.md becomes an accepted new-product.feature and generated new-product.spec.ts. Playwright runs the product subsystem, with product-service connected to PostgreSQL and Redis, before Blackbox verifies the execution report. + Independent system verification for I/O-driven software.

-Specifications in any format can become stale as implementation evolves. Blackbox keeps their accepted expectations -in the verification loop: review checks that executable expectations preserve the intended behavior, and repeated -execution makes departures from that behavior visible. - -This also fits spec-driven development (SDD) workflows such as Spec Kit, Kiro and Openspec. SDD guides -specification, clarification, planning, and implementation; Blackbox provides executable verification of the accepted -system behavior. - -[Spec-Driven workflows integration](docs/integrations/spec-driven-workflows.md) - - -## Installation - let your coding agent run Blackbox - -Ask your coding agent to set up Blackbox in your repository, starting with: - -```bash -$ npx @suites/blackbox-cli onboarding start -``` - -Blackbox provides a full skill system out of the box to let coding agents handle the operational work, from initial -setup to execution, investigation, and verification. The skills provide the procedures and context, while Blackbox’s CLI -returns structured feedback the agent can act on. - -### Agent Onboarding - -During onboarding, the agent inspects the repository, discovers services and dependencies, and derives the system -topology. It identifies useful application and subsystem boundaries, then creates or updates the Blackbox configuration -and supporting files. -

- Agent onboarding discovers services and dependencies, derives system boundaries, then creates and validates Blackbox configuration. -
- The agent derives runnable boundaries from the repository’s services and dependencies. + Apache 2.0 + Alpha

-The result describes what to run, how to start it, and how to act on and observe it. The agent reuses existing project -files where appropriate. For a Node application with an HTTP service, the setup could look like this: -

- Example project tree with blackbox.config.yaml and .blackbox catalog, driver, and instrumentation files. -
- Configuration references the files that start, drive, and observe the system. + See it work · + Run it · + Packages · + Alpha status · + Contribute

-`blackbox.config.yaml` is the configuration authority and references the supporting files. Capsules and Playwright use -that configuration to run the selected system and collect evidence. +Your agent writes the code. BLACKBOX verifies the running system. -[Initial setup](docs/getting-started/agent-onboarding.md) · -[Agent skills](docs/agents/skills.md) +BLACKBOX exercises real applications through their I/O boundaries, runs them with their dependencies in isolated +Sandboxes, and checks their responses, state changes and runtime effects against accepted expectations. It keeps +the execution evidence so developers and coding agents can see what was verified, what failed, and what remains +unknown. It does not matter which agent wrote the code or which methodology produced it. -## Behavior-driven verification +## Why BLACKBOX -[Behavior-driven development (BDD)](https://cucumber.io/docs/bdd/) uses concrete examples to establish a shared -understanding of expected behavior. Blackbox adopts that approach to verifying real applications and subsystems: -scenarios establish initial conditions, perform an action, and check the required outcomes. +Coding agents can rewrite thousands of lines in minutes. A 20,000-line change may alter nothing observable; a +one-line change can break a business rule. Reading the diff cannot settle which one you have, and reviewers cannot +keep up with the volume. -**These specifications describe what the running system must do:** its responses, state changes, and required runtime -effects, such as serving a product from cache. Their meaning should remain stable when classes, functions, or code -organization change while the required behavior stays the same. Native Playwright tests can express these expectations -directly. +Unit tests, integration tests and Playwright remain essential. BLACKBOX adds a different control point: the running +system, started fresh for every test, checked against expectations that stay fixed while implementations change, with +the evidence of what actually happened kept per execution. When the same agent writes the code and decides what +success means, a passing suite says less than it appears to. BLACKBOX keeps the accepted expectations separate and +reports what was not observed instead of treating it as a pass. -### Express scenarios with executable Feature files +## What BLACKBOX verifies -For this optional workflow, Blackbox adopts [Gherkin from Cucumber](https://cucumber.io/docs/gherkin/reference/). -A `Feature` describes a capability, a `Rule` groups scenarios illustrating a business rule, and each `Scenario` states -initial conditions (`Given`), an action (`When`), and expected outcomes (`Then`). +BLACKBOX verifies systems that can be exercised through defined input and output interfaces: APIs, network services, +message-driven systems, and applications with externally exercisable operations. In the alpha, tests drive the system +through an HTTP entrypoint and observe HTTP, PostgreSQL, Redis and SQS effects. -The agent drafts or refines the specification into Blackbox’s closed executable vocabulary for review. Supported steps -have defined setup, action, and verification semantics, without a separate -regex step-definition layer. Existing Gherkin may need refinement to use that vocabulary. - -The following example connects the source specification, reviewed Feature, and native Playwright suite around the same -response and required cache behavior: - -

- product-cache.md drives a reviewed product-cache.feature and its native product-cache.spec.ts suite. Review preserves intent, generation preserves alignment, and execution supplies evidence for the specified behavior. -

+Verification can look past the immediate output: persisted state, calls to dependencies, published messages and other +observable effects can all matter to an accepted requirement. The target is **system behavior at its I/O boundaries**, +not how the implementation was written. -
-View the specification and tests +**What BLACKBOX is not.** It is not a general-purpose verifier for arbitrary source code. It does not replace unit +tests, static analysis, code review or specialized tools for isolated algorithms, and it does not prove a system +correct. Its conclusions are limited to the expectations exercised, the evidence collected and the observation +coverage available. -**Specification · `product-cache.md`** +**Different implementation. Same accepted contract. Independently verified execution.** -```markdown -# Product details +## See it work -## Business rule +**The API returned success. Did the system do the right thing?** -Cached product details are served from cache. +A subscription service takes a payment, runs a fraud check and queues an order. The bundled test +[`subscription-system.spec.ts`](e2e/tests/playwright/subscription-system.spec.ts) starts eight services in a fresh +Sandbox (the application services, PostgreSQL, Redis, LocalStack for SQS, and a mock payment provider), sends real +requests, and checks three kinds of evidence: -## Acceptance example +| Check | Source | +| ------------------------------------------------------------------------ | ------------------------------------------- | +| The HTTP responses: for example `201` with the expected subscription | The responses | +| What the system stored: subscription, fraud audit, payment intent, Redis | The system's stored state | +| Calls to dependent services, database and cache writes, a queue message | Runtime effects observed with OpenTelemetry | -A product is already cached. When a client requests its details: +**Break it.** In `e2e/sut/src/public-api/redis-repository.ts`, make `setTier` write its cache entry under the wrong key, +a one-line mistake that changes nothing a caller can see: -- The response status is 200. -- The product is returned from cache. +```diff +- await this.#client.set(`user:${userId}:tier`, tier, { EX: 300 }); ++ await this.#client.set(`tier:${userId}`, tier, { EX: 300 }); ``` -**Feature · `features/product-cache.feature`** +In one recorded run against this change, the service's own unit tests (26 of 26) passed, because they replace Redis +with a fake. The BLACKBOX test for an eligible user started the real system, received the correct `201`, and then +failed on stored state: -The reviewed Feature expresses the same business rule and acceptance example: - -```gherkin -Feature: Product details - - Rule: Cached product details are served from cache - - Scenario: Return product details from cache - Given the product is already cached - When the client requests product details - Then the response status is 200 - And the product is returned from cache +```text +Expected: "pro" +Received: null + > expect(await clients.redis.get(`user:${expected.userId}:tier`)).toBe(expected.tier); ``` -**Native suite · `product-cache.spec.ts`** - -Generation preserves the `Feature → Rule → Scenario → steps` nesting and names. Within each scenario, `Given` arranges -the initial state, `When` applies the stimulus, and `Then` asserts the expected outcomes, following Arrange–Act–Assert -(AAA). `And` continues the preceding step type. The selected system and its isolated Sandbox provide the execution -context around that hierarchy. - -The planned native expression below retains the same capability, business rule, scenario, and steps. The existing -[Playwright E2E suite](https://github.com/suites-dev/blackbox/blob/8017b623d5c5da5f162f3ee828a703cb814ea1cf/e2e/tests/playwright/subscription-system.spec.ts) -illustrates the underlying system-test structure. This example requires a cache-only fixture: product 42 is present in -the cache and absent from the backing store, so its returned value can establish the response source. - -```ts -import { randomUUID } from 'node:crypto'; -import { expect, test } from '@suites/blackbox-playwright'; - -test -.system('products-system') -.feature('Product details') -.run(({ sandbox }) => { - sandbox.describe('Rule: Cached product details are served from cache', ({ suite }) => { - suite.test( - 'Scenario: Return product details from cache', - async ({ activities, request, sandbox, effects }) => { - const product = { id: '42', name: `Cached product ${randomUUID()}` }; - const url = (path: string) => new URL(path, sandbox.entrypoint.url).href; - - await test.step('Given the product is already cached', () => - activities.setup.request('seed cache-only product', request, async (scoped) => { - const seeded = await scoped.post(url('/__fixtures/products/cache-only'), { - data: product, - }); - expect(seeded.status()).toBe(204); - })); - - const response = await test.step('When the client requests product details', () => - activities.stimulus.request('request product details', request, (scoped) => - scoped.get(url('/products/42')), - )); - - await test.step('Then the response status is 200', async () => { - expect(response.status()).toBe(200); - }); - - await test.step('And the product is returned from cache', async () => { - expect(await response.json()).toEqual(product); - - await expect(effects).toSatisfy((e) => [ - e.exists(e.cache({ actor: 'catalog-api', operation: 'GET' })), - ]); - }); - }, - ); - }); -}); -``` +That is the case BLACKBOX exists for: the response is right, the unit tests are green, and the system did not do what +it was accepted to do. -**Draft, review, generate, and run** +This is one defect and one run, on one machine. The change above is not in the repository, so you reproduce it +yourself with the [steps below](#run-blackbox). Effect checks confirm that calls happened, not what they wrote; +the stored-state check is what caught this. -The agent drafts the Feature from the specification for review. Once its expectations are accepted, it validates the -Feature, generates the native suite, and runs it against the selected application or subsystem: +## How it works -```console -$ blackbox spec feature draft --file product-cache.md -$ blackbox feature validate --file features/product-cache.feature -$ blackbox feature suite emit --file features/product-cache.feature -$ npx playwright test +```text +CODE ─────────► VERIFY ─────────► SHIP +agents and BLACKBOX your existing delivery +people write runs the system, and production tooling +code checks accepted + behavior, returns + evidence + ◄── violations and missing evidence go back to the agent ── ``` -
- -The project-owned fixture endpoint seeds the cache and removes any backing-store copy of the product. Setup remains -separate from the stimulus. The final claim combines that controlled initial state, the returned product, and cache-read -evidence from the same execution; observing a cache read alone would not establish that the product was served from it. - -**Keep specifications connected as the system evolves.** Review checks that executable expectations remain faithful to -the accepted specification. Repeated execution checks those expectations against the running system. When intended -behavior changes, revise the specification and its executable expectations together. - -For the Feature workflow, `blackbox feature suite check` also detects deterministic drift between the accepted Feature -and its generated suite. After reviewing changes to the Feature, regenerate the suite and verify the new expectations. -Review maintains meaning, generation keeps the derived suite aligned, and execution supplies evidence for the exercised -claims. - -[Executable specifications](docs/specifications/index.md) · -[Feature files](docs/specifications/feature-files.md) +BLACKBOX does not write code and does not replace your CI, deployment or observability tools. Inside VERIFY: -## Developer-agent verification loop - -The developer approves the specification, behavioral claims, and verification policy. The coding agent uses that -guidance to implement the behavior, prepare system tests, and investigate failures. - -

- The developer approves intent and policy, the coding agent prepares and repairs system tests, and Blackbox returns runtime evidence and findings. Proposed changes to intent or policy return to the developer for review. -

- -Blackbox turns fresh runtime evidence into findings the agent can inspect and act on. The agent repairs the -implementation and reruns verification against the same accepted expectations. Proposed changes to the specification -or verification policy return to the developer for review. - -## Inspect the result and its evidence - -Playwright remains the test runner. Blackbox adds execution context and evidence to its terminal and HTML reports. -For the cache scenario, a report can look like this: - -

- Illustrative Product details report showing the cache scenario, steps L6 to L9, response and cache observations, coverage, and one passed result. -
- Claims and coverage appear alongside evidence from the same attempt. -

- -Claims are supported, refuted, or unresolved according to the available response, state, and runtime evidence. In this -example, the response establishes the status code. The cache claim combines the controlled initial state, returned -product, and runtime evidence from the same execution. - -Different claims require different evidence. Observing a database operation does not establish that a transaction -committed. Observing a message being published does not establish that a consumer completed its work. An operation that -was not observed is not automatically evidence that it never happened. - -Blackbox keeps those limits explicit so a passing verification does not claim more than the execution established. - -For suites generated from Feature files, **Executable Feature Coverage** shows which executable Feature lines and -Examples rows were actually exercised by the run. It measures coverage of the executable Feature; an exercised line -still needs evidence to support its claim. - -The HTML report connects scenarios, Sandbox execution, claims, observations, and retained evidence, including the -associated Feature when present: - -```console -$ npx playwright show-report +```text +Accepted expectations → Isolated execution → Runtime evidence → Result +tests that enter fresh Sandbox: responses, stored what held, what failed, +through the system's real app + deps state, effects, what was not observed +interfaces OpenTelemetry ``` -[Reports](docs/playwright/reports.md) · -[Verification](docs/verification/index.md) · -[Evidence](docs/verification/evidence.md) · -[Evidence qualification](docs/verification/evidence-qualification.md) +Accepted behavior, observed behavior and verification evidence stay separate. An observed effect is not automatically +desired, a passing run supports only the assertions that ran, and a result never claims to prove the whole system +correct. Agents may change implementations; they must not silently redefine what correct means. -## Investigate with Capsules +Expectations today are [Playwright](https://playwright.dev) tests that use BLACKBOX fixtures. Existing tests, +acceptance criteria and API contracts are useful sources for them. The alpha does not execute arbitrary requirement +documents. -When a claim is refuted or unresolved, the agent can use a Capsule to investigate. Capsules also support experiments -with candidate expectations before they become accepted behavior. +## Run BLACKBOX -| Mode | Purpose | -|------------------------|------------------------------------------------------------------------------| -| Playwright system test | Repeat accepted expectations against a fresh isolated Sandbox. | -| Capsule | Act on a running system, inspect evidence, test a hypothesis, and try again. | +BLACKBOX is `0.0.1-alpha` and **not on npm yet**. You need Node 22.15+, pnpm 9.15.4 and Docker. -A Capsule is a bounded interactive experiment. The agent can inspect what happened, change the implementation, and -repeat the experiment. When the investigation suggests a new expectation, it goes through review before becoming part -of the accepted specification. The example below carries that expectation into an accepted Feature and its Playwright -suite: +**Check your setup (about two minutes, no Docker).** This validates the bundled catalog. It is a configuration +check, not a verification run. -

- Capsule investigation moves from act, inspect, investigate, and retry to a candidate expectation, review, accepted Feature, and Playwright verification. -
- Candidate expectations require review before joining the accepted specification. -

- -**Observed behavior is not automatically accepted behavior.** A Capsule can validate or challenge a candidate -expectation; the accepted specification remains the source of what the implementation must satisfy. - -Stopping the environment does not discard the experiment. Its activities, observations, and evidence remain available -for inspection and reporting. - -[Capsules](docs/capsules/index.md) · -[Experiments](docs/capsules/experiments.md) · -[Capsule reports](docs/capsules/reports.md) · -[Playwright verification](docs/playwright/index.md) - ---- - -## Run the system your claim requires - -Blackbox can run a full application or a smaller subsystem. Use the smallest real system boundary that can answer the -claim being checked, including every participant needed to establish the behavior. - -A focused boundary lowers startup cost, memory use, and unrelated runtime noise. It also reduces the number of competing -explanations when an execution fails. A cross-service workflow still needs the participants required to establish that -workflow. - -The agent configures these boundaries during discovery. Configuration selects the system, drivers provide controlled -ways to act on it and inspect state, and instrumentation provides supported runtime observations: - -

- blackbox.config.yaml selects system boundary, drivers, and instrumentation for Capsule or Playwright execution. -
- Capsules and Playwright share the configured boundary, drivers, and instrumentation. -

- -Use unit tests for the behavior of a function or class. Use Blackbox when the claim concerns a running application or -subsystem, including when an internal implementation change could affect that behavior. +```sh +git clone https://github.com/suites-dev/blackbox.git +cd blackbox +pnpm install --frozen-lockfile +pnpm build +cd e2e && pnpm exec blackbox catalog validate && pnpm exec blackbox systems +``` -[System boundaries](docs/systems/system-boundaries.md) · -[Configuration](docs/systems/blackbox-config.md) · -[Systems](docs/systems/index.md) · -[Drivers](docs/systems/drivers.md) · -[Instrumentation](docs/systems/instrumentation.md) +**Run the real verification (about six minutes plus image pulls).** The subscription test installs BLACKBOX from a +disposable local package registry, as CI does, so you start a registry, publish the built packages to it, and run +the suite. The exact commands are in +[Alpha status](docs/alpha-status.md#run-the-bundled-example-from-source). The run writes a JUnit file and per-test evidence +attachments to `e2e/test-results/`. A one-command version does not exist yet. ---- +**Use it on your own application.** Install the skills, then ask your coding agent to connect BLACKBOX to the +repository: -## Keep verification consistent locally and in CI +```sh +pnpm exec blackbox skills install blackbox --codex # or --claude +``` -Local iteration and CI use the same accepted expectations. An agent can focus on selected scenarios and Capsule -experiments during development, while CI runs the suite selected by the repository's verification policy. +The skills guide the agent through discovering services and dependencies and writing the configuration, and the +CLI gives it commands and structured results. Setup against an external repository has not been validated end to +end in the alpha. See the [`@suites/blackbox` package](packages/blackbox/README.md) and the [skills installer](packages/skills/README.md). -Every physical Playwright attempt gets a fresh Sandbox. Isolation enables parallel execution; CPU, memory, and container -resources determine how much concurrency is practical. +## With your coding agent -For repositories using the Feature workflow, three checks protect different parts of verification: +Nothing about how you write code changes. Keep your agent, your editor and your process. -| Check | What it establishes | -|----------------------------------------------------|--------------------------------------------------------------------------------------------------| -| `blackbox feature suite check` | The generated suite matches the accepted Feature and current compiler inputs. | -| `blackbox feature verify` | Required scenarios ran, verdicts are supported, and runner policy matches the accepted baseline. | -| `blackbox feature change check --base origin/main` | Changes to accepted Features satisfy the repository's specification-change policy. | +1. **Connect.** The agent discovers services and dependencies and prepares configuration. +2. **Specify.** You review the expectations to verify. +3. **Verify.** BLACKBOX runs the system and checks them. +4. **Inspect.** You and the agent read results and evidence, including what was not checked. The agent can reproduce + a failure in a Capsule with `blackbox up`, `run`, `show` and `report`. +5. **Preserve.** The accepted expectations stay in the repository and rerun after every change. -For generated Feature suites, CI checks that the suite is current, runs it, and verifies the completed execution: +## Where we're going -```console -$ blackbox feature suite check -$ npx playwright test -$ blackbox feature verify -``` +**Every PR deserves a behavioral review.** A BLACKBOX check on a pull request should answer: what accepted behavior +was intentionally changed, what existing behavior was checked, which expectations were violated, what evidence +supports that, what could not be verified, and what needs investigation before merge. -The generated suite is derived from the accepted Feature. Source-specification-to-Feature alignment is a separate -semantic question from deterministic Feature-to-suite drift. Repositories that protect specifications separately from -implementation changes can also use `feature change check` against their chosen base branch. +It keeps two things apart: the **contract diff** (changes to accepted expectations) and the **execution diff** +(changes observed between runs or revisions). Neither is a summary of the source diff. -[CI verification](docs/playwright/ci.md) · -[Specification drift](docs/specifications/drift.md) +None of this exists in the alpha. Pull request reports, revision-aware comparison and impact planning are +roadmap. What exists is the execution and evidence layer they will rest on. ---- +## Open source -## Alpha and next steps +BLACKBOX is Apache-2.0. The code that evaluates behavioral claims is open and runs in your own infrastructure, so +you can read how a result was reached and keep your code and runs under your control. Issues and pull requests are +welcome; start with [CONTRIBUTING.md](CONTRIBUTING.md). -Blackbox is under active development. The executable Feature workflow described here is upcoming. APIs, command names, -and report formats may change before the stable release. +## Alpha status -Start with [coding-agent onboarding](#let-your-coding-agent-run-blackbox), or explore the guides below. +Interfaces will change. [Alpha status](docs/alpha-status.md) lists the audited revision, what works, what is in +source but unverified, and what is in development. Observations depend on instrumentation coverage and Node is the +primary instrumented runtime. BLACKBOX is not a hardened sandbox for hostile code; see [SECURITY.md](SECURITY.md). -[Getting started](docs/getting-started/index.md) · -[Initial setup](docs/getting-started/agent-onboarding.md) · -[CLI reference](docs/reference/cli/index.md) +[Packages](packages/blackbox/README.md) · [Contributing](CONTRIBUTING.md) · [Support](SUPPORT.md) · [License](LICENSE) diff --git a/docs/alpha-status.md b/docs/alpha-status.md new file mode 100644 index 00000000..a476d948 --- /dev/null +++ b/docs/alpha-status.md @@ -0,0 +1,100 @@ +# Alpha status + +BLACKBOX is `0.0.1-alpha`. This page is the trust contract for the alpha: what exists, how to run it, and what does +not exist yet. Other pages must not describe a capability more strongly than this one. + +**Audited revision:** `release/v0.0.1-alpha` at `9b15ea42`. Recheck when commands, exports or release availability change. + +## Distribution + +- **Packages are not published to npm.** Every `@suites/blackbox*` package is at `0.0.1-alpha.0`; the registry + returns 404 for `@suites/blackbox-cli`, and no tag or GitHub release exists. +- Until the alpha is published, the supported path is a **source checkout**. Install commands that use the + `@alpha` tag work only after publication. + +## Capability status + +| Status | Meaning | +| ------------- | ------------------------------------------------------------------------------------ | +| **Working** | Reproduced in the audited source revision (build, CLI or test run). | +| **In source** | Implemented in the repository; not reproduced in the audit or not publicly packaged. | +| **In review** | Described in docs or open pull requests; not in the audited source. | +| **Roadmap** | Direction only. | + +| Capability | Status | Evidence | +| ------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------------ | +| Workspace install and build | Working | `pnpm install --frozen-lockfile` and `pnpm build` succeeded. | +| CLI: `capsule up/run/show/down/ls/use/report`, `catalog list/validate`, `systems`, `skills list/install`, `driver install`, `inst install` | Working | Listed by `blackbox --help`; `catalog validate`, `systems` and `skills list` ran against `e2e/`. | +| CLI: `feature file draft/validate`, `feature step list`, `feature suite emit/validate`, `feature run verify` | In source | Listed by `blackbox --help`; not run in the audit. | +| Isolated Sandboxes and system execution | Working | Eight Playwright tests started Compose systems; see the run below. | +| Native Playwright (`test.system`, sandbox suites, clients, `step`) | Working | Same run. | +| Effects evaluation (`expect(effects).toSatisfy(...)`) | In source | Used by the bundled spec; its pass is part of the run below, but one test hit a startup failure first. | +| Reports and `blackbox-*` attachments | In source | Produced in the run; not inspected in the audit. | +| Capsule experiments and reports | In source | `capsule` commands registered; the maintainer journey exists, not run in the audit. | +| Agent skills | Working | `skills list` returns `blackbox`, `capsule`, `catalog`, `discovery`. | +| Node instrumentation and OTLP collection | In source | Packages build; exercised indirectly by the run. | +| `blackbox onboarding start` | **In review** | `blackbox onboarding start` fails with `command onboarding:start not found`. Not available yet. | +| Effects snapshots (capture, accept, compare) | Roadmap | | +| Editable scaffold generator; Spec Kit extension | Roadmap | | +| Behavioral PR reports, impact planning | Roadmap | | + +## Documentation limits + +- Install commands that use the `alpha` tag work only after the packages are published. +- The `blackbox feature` commands and the typed-client APIs are in source and were not run end to end in this audit. + +## Run the bundled example from source + +Prerequisites: Node.js 22.15 or later, pnpm 9.15.4, Docker with Compose (`docker info` must succeed), and network access +for images and dependencies on a fresh machine. + +```sh +pnpm install --frozen-lockfile +pnpm build +cd e2e +pnpm exec blackbox catalog validate # Catalog is valid ... Entries: 3 +pnpm exec blackbox systems # payment-mock, payment-mock-dist, subscription-system +``` + +The verification lane installs BLACKBOX into a throwaway consumer from a disposable local registry, as CI does. +Running the spec directly from the workspace does not work: `playwright test` in `e2e/` fails with `Stripping types +is currently unsupported for files under node_modules` on Node 25.8.1 and 22.23.3. Start the registry, publish the +built packages, then run the lane (from the repository root, after `pnpm build`): + +```sh +storage="$(mktemp -d)" +docker run --detach --name blackbox-test-registry \ + --user "$(id -u):$(id -g)" --publish 127.0.0.1:4874:4873 \ + --volume "$(pwd)/.github/verdaccio/config.yaml:/verdaccio/conf/config.yaml:ro" \ + --volume "$storage:/verdaccio/storage" verdaccio/verdaccio:6.10.3 + +tarballs="$(mktemp -d)" +pnpm --recursive --filter './packages/*' exec pnpm pack --pack-destination "$tarballs" +for t in "$tarballs"/*.tgz; do + env "npm_config_//127.0.0.1:4874/:_authToken=blackbox-e2e" \ + npm publish "$t" --registry http://127.0.0.1:4874/ --tag e2e --provenance=false +done + +pnpm test:e2e:playwright +docker rm --force blackbox-test-registry +``` + +The lane is a maintainer acceptance script: besides the Playwright suite it runs an effects proof and Feature +checks. Details: [maintainer acceptance guide](../maintainers/docs/capsule-bash-e2e.md). + +Without the registry the lane fails at once with `ECONNREFUSED 127.0.0.1:4874`. Artifacts are written to +`e2e/test-results/`. A one-command run that needs no registry does not exist yet. + +## Prerequisites + +- Node.js 22.15 or later; `@playwright/test >=1.61.0 <2` for the Playwright adapter. +- Docker with Compose for Sandbox execution. Catalog validation and skill installation do not start Docker. +- API-only tests need no browser download. + +## Limits + +- A passing run supports the assertions that ran under the observed conditions, not all behavior. +- Observations depend on instrumentation coverage. Missing or incomplete evidence stays inconclusive; an empty trace + does not prove an operation never happened. Keep authoritative state checks alongside effects assertions. +- Node is the primary instrumented runtime. +- BLACKBOX is not a hardened sandbox for hostile code. See [SECURITY.md](../SECURITY.md).