diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..9075a11 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule ".repos/effect"] + path = .repos/effect + url = https://github.com/Effect-TS/effect.git diff --git a/.repos/effect b/.repos/effect new file mode 160000 index 0000000..44d66f8 --- /dev/null +++ b/.repos/effect @@ -0,0 +1 @@ +Subproject commit 44d66f8d7c3f0fadc2194b2ccc55c7c8903b28db diff --git a/README.md b/README.md index 4a8951e..01d88fc 100644 --- a/README.md +++ b/README.md @@ -13,3 +13,7 @@ bun run catalog ``` See [`packages/drive/README.md`](packages/drive/README.md) for package usage. + +See [OpenCode Continuous Verification](docs/continuous-testing/README.md) for +the proposed 24/7 synthetic-journey, provider-contract, property, soak, logging, +review, and operations architecture targeting OpenCode V2. diff --git a/docs/continuous-testing/01-system-model.md b/docs/continuous-testing/01-system-model.md new file mode 100644 index 0000000..0b9ea33 --- /dev/null +++ b/docs/continuous-testing/01-system-model.md @@ -0,0 +1,471 @@ +# System Model + +This document defines the conceptual architecture of OpenCode continuous +verification. It focuses on ownership, lifetimes, failure domains, and the +contract between the always-on control plane and the OpenCode processes being +tested. + +Read [Environments and lanes](./02-environments-and-lanes.md) next for the +concrete lane types and isolation rules. + +## Objectives + +The system must: + +1. Keep at least one real OpenCode environment available continuously. +2. Exercise that environment with realistic TUI and SDK interactions. +3. Support deterministic mock inference before introducing real providers. +4. Explore fixed journeys and generated state-machine transitions. +5. Detect correctness failures, stuck work, crashes, recovery failures, and + performance regressions. +6. Preserve enough evidence to diagnose and replay failures. +7. Distinguish a product failure from a harness, infrastructure, or assertion + failure. +8. Continue reporting health when the OpenCode workload is unavailable. +9. Operate safely for long periods without unbounded storage, cost, or process + growth. + +## Non-Objectives + +The first system is not: + +- a production OpenCode hosting platform; +- a generic cloud scheduler inside `packages/drive`; +- a load test that tries to maximize requests per second; +- a replacement for unit, integration, or pull-request tests; +- a new model-provider abstraction in the Drive CLI; +- a promise that arbitrary generated agent output can be judged perfectly; +- a multi-tenant environment for untrusted external users; +- a reason to weaken the canonical OpenCode simulation protocol. + +## The Two-Plane Model + +The most important architectural separation is between the **control plane** +and the **workload plane**. + +```text +CONTROL PLANE WORKLOAD PLANE + +configuration OpenCode server +scenario registry TUI process(es) +scheduler and leases commands SDK client +attempt coordinator ------------------> simulated inference +run and checkpoint store controlled tools +artifact index <---------------- project and database +telemetry collector observations frames and recordings +alert evaluator +``` + +The workload plane is expected to fail. It is the object under test. The +control plane must therefore not share the same fate accidentally. + +For the first implementation, both planes may run on one machine, but they +remain separate processes and persistence domains. A lane crash must leave a +heartbeat gap or exit record that another process can observe. + +## Components + +### Scenario registry + +**Status: existing foundation, proposed monitoring metadata.** + +The authoritative OpenCode journey registry is +[`apps/catalog/scenarios/index.ts`](../../apps/catalog/scenarios/index.ts). +Each executable scenario already declares: + +- a stable flow ID; +- ordered states and checkpoint addresses; +- a response mode (`queue` or `serve`); +- a client-isolation policy; +- an Effect program that drives a real OpenCode instance. + +Continuous verification adds scheduling and operational metadata around that +registry. It does not create a competing registry. Examples of proposed +metadata are cadence, timeout, required lane capabilities, risk level, and +alert policy. + +### Scheduler + +**Status: proposed.** + +The scheduler decides which scenario or campaign is due. It emits work only +when an eligible lane has capacity. It is responsible for cadence, fairness, +jitter, disabled scenarios, and campaign budgets. + +It is not responsible for interpreting UI state or generating LLM chunks. Those +remain scenario and inference responsibilities. + +### Attempt coordinator + +**Status: proposed.** + +The coordinator owns the lifecycle of one attempt: + +1. Validate the work specification. +2. Acquire a compatible lane lease. +3. Create an immutable attempt record in `scheduled` state. +4. Prepare a fresh TUI/session or the explicitly requested reuse mode. +5. Execute the scenario with a deadline. +6. Record every reached checkpoint. +7. Capture failure evidence before cleanup. +8. Classify the outcome. +9. Release attempt-owned resources and the lane lease. +10. Emit metrics and evaluate alert policy. + +An attempt coordinator never silently reruns a failed UI action. If policy +requests a retry, it schedules a new linked attempt. + +### Lane supervisor + +**Status: proposed orchestration over existing Drive lifecycles.** + +A lane supervisor owns one persistent OpenCode environment. It starts the +server, attaches model and tool control, reports readiness, creates attempt +clients, observes process exits, and shuts everything down cooperatively. + +The existing +[`OpenCodeDriver`](../../packages/drive/src/driver/index.ts) and +[`defineScript`](../../packages/drive/src/script/types.ts) lifecycles provide +the process and resource foundation. The continuous system adds the outer +supervision and repeated-attempt policy. + +### Inference strategy + +**Status: deterministic and reactive primitives exist; strategy layer is +proposed.** + +The inference strategy turns an opened model exchange into controlled output. +The current controller supports queued output and a request-aware served +handler. The continuous system names, configures, and records the selected +strategy for every attempt. + +See [Inference simulation](./04-inference-simulation.md). + +### Run store + +**Status: proposed.** + +The run store persists low-volume, queryable operational truth: + +- environments and lanes; +- attempts and their status transitions; +- checkpoints and timings; +- failure classification; +- links between original, retry, and reproduction attempts; +- references to large artifacts; +- alert evaluation state. + +Large logs, frames, recordings, and event dumps do not belong directly in the +run store. They belong in the artifact store and are referenced by immutable +descriptors. + +### Artifact store + +**Status: local artifacts exist; indexed retention is proposed.** + +Drive already produces an artifact root, OpenCode logs, terminal frames, and +recordings. The artifact store adds stable naming, manifests, upload/retention, +redaction state, and integrity metadata. + +### Telemetry pipeline + +**Status: proposed.** + +The telemetry pipeline exports structured logs, traces, metrics, and +heartbeats. It should use Effect observability APIs in business code and an +OpenTelemetry layer at the application boundary. The run store remains the +authoritative per-attempt record; telemetry backends remain optimized views. + +### Alert evaluator + +**Status: proposed.** + +The evaluator turns attempt history, heartbeats, and metrics into actionable +notifications. It evaluates both event-based conditions and absence-based +conditions. For example, a server crash is an event; “no successful smoke +attempt in ten minutes” is an absence condition. + +## Resource Ownership + +Every resource has one owner and one release point. + +| Resource | Owner | Typical lifetime | Release condition | +| --- | --- | --- | --- | +| Environment configuration | Control plane | Deployment | Superseded by a versioned config | +| Lane lease | Attempt coordinator | One attempt | Attempt finalizer | +| Lane supervisor process | External process supervisor | Days or weeks | Rollout, maintenance, or crash | +| OpenCode server | Lane scope | Lane generation | Lane recycle or failure | +| Project fixture | Lane scope | Lane generation | Lane recycle | +| Persistent test database | Lane policy | Multiple generations if configured | Explicit retention/rebuild policy | +| Model controller | Lane generation | One server generation | Generation shutdown | +| TUI process | Attempt scope by default | One attempt | Attempt finalizer | +| Session | Attempt by default | Persisted in server database | Retention policy, not TUI cleanup | +| Controlled tool invocation | Tool exchange | One call | Success, failure, interruption, or controller close | +| Recording timeline | TUI scope | One attempt or sampled window | Finish/export during cleanup | +| Attempt record | Run store | Retention period | Archival policy | +| Failure artifact bundle | Artifact store | Longer retention period | Explicit expiration | + +Effect scopes should mirror these lifetimes. The lane layer owns the persistent +server and long-lived connections. `Effect.acquireUseRelease` or a scoped +attempt layer owns the TUI and per-attempt evidence writer. The application +runtime owns telemetry exporters and run-store connections. + +## Effect Architecture + +The proposed control plane is an Effect application. Services are behavioral +dependencies; layers construct and own them. A representative dependency graph +is: + +```text +Configuration + | + +--> ScenarioRegistry + +--> Scheduler --------------------+ + +--> RunStore | + +--> ArtifactStore v + +--> AlertSink AttemptCoordinator + +--> Telemetry | + +--> LanePool ---------------------+ + | + +--> scoped LaneSupervisor + | + +--> OpenCode Drive +``` + +Recommended service boundaries: + +- `ScenarioRegistry`: lookup and eligibility metadata; +- `Scheduler`: due work as a stream or queue; +- `LanePool`: compatible lane acquisition and release; +- `AttemptCoordinator`: one complete attempt workflow; +- `RunStore`: typed attempt and checkpoint persistence; +- `ArtifactStore`: artifact publication and retention metadata; +- `AlertSink`: outbound notification boundary; +- `Heartbeat`: control-plane and lane liveness publication; +- `Telemetry`: configured once as a top-level layer, not called as a domain + service for every metric. + +Service interfaces should remain focused. Do not create one `SoakService` that +contains scheduling, storage, OpenCode lifecycle, metrics, and alerting. + +Construction rules: + +- pure test doubles use `Layer.succeed`; +- connections and resource-owning implementations use `Layer.effect`; +- startup fibers that expose no service use `Layer.effectDiscard`; +- the final application layer is composed once and provided at the entrypoint; +- parameterized layer factories are called once at the boundary and their + values are reused so memoization remains effective. + +Reusable workflows should use named `Effect.fn` definitions. This produces +useful trace boundaries such as `AttemptCoordinator.execute`, +`LaneSupervisor.launch`, `Scenario.run`, and `Evidence.captureFailure`. + +## State Models + +### Environment state + +An environment is the promotion-level grouping of lanes. + +```text +planned -> provisioning -> active -> draining -> retired + | | + v v + failed degraded +``` + +An environment may remain `active` while one optional lane is degraded. The +promotion policy decides which lane classes are required. + +### Lane state + +```text +provisioning + | + v + ready <---------- recovering + | ^ + v | + leased ----failure----+ + | | + +------success----> ready + | + v + draining -> stopped +``` + +`ready` means both that the supervisor is alive and that the lane passed a +readiness probe. A process existing in a registry is not sufficient. + +`degraded` is represented as health metadata rather than a separate exclusive +state when the lane can still accept selected diagnostic work. A lane that +cannot safely accept normal work is `recovering` or `draining`. + +### Attempt state + +```text +scheduled -> leased -> preparing -> running -> collecting -> terminal + | | + +--> cancelling ---------+ + +terminal status: + passed | product_failed | harness_failed | infrastructure_failed | + timed_out | cancelled | inconclusive +``` + +State transitions are append-only events. The current state is a projection. +This makes interrupted coordinators diagnosable and lets a reconciliation +process identify abandoned attempts. + +## Failure Domains + +A reliable monitoring system identifies where failure originated without +pretending that classification is always perfect. + +### Product failure + +OpenCode violated an externally observable expectation: + +- a prompt disappeared; +- a session projection contradicted the UI; +- a tool never settled; +- the composer stayed unusable after terminal execution; +- the server or TUI crashed due to the exercised behavior; +- a recovery invariant failed after an injected outage. + +### Harness failure + +The scenario, model plan, assertion, or coordinator was invalid: + +- a queued response was unused because the scenario authored the wrong plan; +- a marker changed but the feature remained correct; +- the scenario referenced an impossible state; +- evidence capture itself failed after the primary behavior passed; +- a generated command violated its own precondition. + +### Infrastructure failure + +The hosting environment failed independently of the OpenCode behavior under +test: + +- disk full outside an intentional disk-pressure experiment; +- node eviction; +- telemetry backend outage; +- source checkout or dependency preparation failed; +- artifact upload failed after local evidence was retained. + +### Expected injected failure + +A chaos action deliberately causes a failure event. The injected event is not +the test failure. Failure occurs only if the declared recovery invariant does +not hold or the blast radius exceeds its envelope. + +### Inconclusive + +Evidence cannot safely attribute the result: + +- the control plane lost contact during a critical observation window; +- both product and host failed simultaneously without enough evidence; +- configuration skew invalidated the expected protocol contract. + +Inconclusive attempts count against monitoring freshness and must be visible. +They are not converted to passes. + +## Health Model + +Health has several independent dimensions: + +| Dimension | Example signal | +| --- | --- | +| Supervisor liveness | Heartbeat timestamp and process identity | +| Lane readiness | Simulation handshake, server health, disposable TUI probe | +| Functional health | Recent successful smoke journey | +| Performance health | Time to first output and journey latency windows | +| Resource health | RSS, CPU, file descriptors, disk, event-loop lag | +| Evidence health | Successful local capture and artifact publication | +| Control-plane health | Scheduler progress and attempt reconciliation | + +One green signal cannot substitute for the others. A server can answer a health +endpoint while every real prompt is stuck. Conversely, an artifact-upload +outage should not be labeled an OpenCode product regression. + +## Data Consistency + +The system favors understandable at-least-once coordination over a fragile +illusion of distributed exactly-once behavior. + +- Attempt IDs are allocated before work begins and are idempotency keys for + state updates. +- The lane lease has a deadline and owner identity. +- Checkpoint writes are idempotent by `(attemptId, ordinal)`. +- Artifact descriptors are immutable after publication; a failed upload may be + retried with the same content digest. +- Alert notifications use a deduplication key derived from policy and incident + window. +- A reconciliation loop terminalizes abandoned attempts or marks them + inconclusive after inspecting lane state. + +Inside one Drive controller, existing Deferred and semaphore semantics continue +to provide the stronger local guarantees described by the package +architecture. The control plane does not claim exactly-once execution across a +host crash. + +## Configuration Model + +Every configuration that influences behavior is versioned or captured: + +- OpenCode revision and build identity; +- Drive revision and package version; +- environment and lane configuration version; +- scenario ID and scenario-source revision; +- inference strategy and response-plan version; +- property seed, step budget, and generator version; +- chaos experiment specification; +- project fixture digest; +- permission and tool configuration; +- viewport and theme where UI assertions depend on them; +- protocol compatibility records. + +Configuration is decoded with Effect Schema at entry boundaries. A malformed +configuration prevents lane provisioning and produces a typed configuration +failure; it never falls back silently to an undocumented default. + +## Shutdown Semantics + +Graceful shutdown proceeds from new work toward owned resources: + +1. Mark the environment or lane as draining. +2. Stop scheduling new attempts to it. +3. Allow active attempts a bounded grace period. +4. Interrupt remaining attempts. +5. Capture cancellation evidence where safe. +6. Close attempt-owned TUIs and recordings. +7. Settle model and tool work. +8. Close the server generation and simulation connections. +9. Flush run-store and telemetry buffers. +10. Publish the final heartbeat/status and release the application scope. + +Interrupts remain interrupts. Cleanup code may suppress expected cancellation +noise, but it must not record a cancelled attempt as passed. + +## Design Review Checklist + +Any implementation proposal should answer: + +- Which plane owns this component? +- Which Effect scope owns each resource? +- What is the component's typed failure contract? +- Which failure classes are retryable? +- What identifier correlates its logs, spans, and persisted records? +- How is unknown input decoded? +- What happens if the process is interrupted between its two most important + writes? +- How does another component notice that it stopped making progress? +- Can a retry accidentally repeat a state-changing UI action? +- Does the proposal introduce OpenCode-specific vocabulary into + `packages/drive`? +- Does it add a backend-control CLI command or diverge from the canonical + frontend protocol? + +If those questions do not have crisp answers, the ownership boundary is not +ready. diff --git a/docs/continuous-testing/02-environments-and-lanes.md b/docs/continuous-testing/02-environments-and-lanes.md new file mode 100644 index 0000000..4bd11bb --- /dev/null +++ b/docs/continuous-testing/02-environments-and-lanes.md @@ -0,0 +1,566 @@ +# Environments and Lanes + +This document defines how OpenCode revisions are deployed for continuous +verification, which state persists, how work is isolated, and how concurrency +is scaled safely. + +The key distinction is: + +- an **environment** represents a promotion target or comparison set; +- a **lane** is one executable lifecycle and concurrency boundary inside that + environment. + +## Environment Model + +An environment groups everything needed to make a release decision about one +or more OpenCode revisions. + +Examples: + +- `v2-candidate`: current `origin/v2` revision under test; +- `baseline-candidate`: stable and candidate revisions tested side by side; +- `nightly-main`: one nightly commit with extended property and chaos budgets; +- `developer-pr-1234`: temporary environment for a branch or pull request. + +An environment records: + +- stable environment ID; +- exact OpenCode commit SHA for each variant; +- source repository identity; +- Drive revision or package version; +- lane specifications; +- configuration version; +- creation and retirement timestamps; +- promotion policy and required lane classes. + +Human-friendly branch names are labels, not identities. Every attempt records +the immutable commit SHA resolved when the environment was provisioned. + +## Lane Definition + +A lane owns: + +- one artifact root; +- one project fixture and OpenCode configuration; +- one OpenCode server generation at a time; +- one database policy; +- one simulated-model controller with one active response-routing mode for its + lifetime; +- one active Drive tool-controller attachment and registration generation; +- zero or more attempt-owned TUI processes; +- one concurrency limit; +- one health policy and one set of rules for replacing the lane generation. + +A lane is deliberately narrower than “all tests for this revision.” It is the +smallest unit that can be restarted, drained, compared, or declared unhealthy +without creating response-routing ambiguity. + +### Response-routing mode, in plain language + +Drive can answer model requests in two fundamentally different ways: + +- **queued mode**: a scenario loads response A, response B, and response C; + model requests consume them in that order; +- **served mode**: one request-aware function receives each model request and + decides what response to stream. + +The current `LlmController` does not allow a controller to start queueing +responses and later install a served handler. Therefore a queued-journey lane +and a reactive/property lane use different controller lifetimes. This bullet +does **not** mean one real AI model or one provider per lane. + +### Tool-controller attachment, in plain language + +Drive can intercept selected OpenCode tools—currently built-in shapes such as +`shell`, `webfetch`, `websearch`, and `write`—and let the test decide when each +call reports progress, succeeds, fails, or is interrupted. + +The lane owns one active controller connection. At a clean attempt boundary it +may install an attempt's complete tool profile, for example: + +```text +ordinary smoke + no controlled tools + +tool-success journey + control write; return a deterministic result + +interruption journey + control shell; hold it open until the test interrupts it +``` + +Only one registration generation is active at a time, so one attempt cannot +silently replace the handlers while another attempt is using them. The earlier +phrase “one tool-control configuration” did not mean that every journey in the +lane must use the same tools forever. + +### Replacing or recycling a lane, in plain language + +A persistent lane is long-lived, but not immortal. **Recycling** means: + +1. stop assigning new attempts; +2. finish or explicitly interrupt the current attempt; +3. collect required evidence; +4. shut down the TUI, controllers, and OpenCode server; +5. verify that processes, ports, and scoped resources are gone; +6. start a new lane generation and run its bootstrap smoke. + +The rules that trigger this replacement are the recycle policy. Examples are a +new OpenCode commit, changed configuration/protocol, a maximum lane age, +resource growth, failed health checks, or a lane frozen after failure evidence +is secured. Recycling is not retrying a scenario, and it never deletes or +rewrites the failed attempt. + +## Recommended Initial Lane Types + +### Persistent queued-journey lane + +Purpose: + +- run deterministic catalog journeys frequently; +- keep the same server alive across attempts; +- reveal session accumulation and long-lived resource problems. + +Properties: + +- inference mode: `queue`; +- attempt concurrency: `1`; +- fresh TUI and session per attempt; +- persistent server and selected database; +- fixture files reset between ordinary journeys; +- deterministic response plan authored by the scenario. + +This is the first lane to implement. + +### Persistent reactive lane + +Purpose: + +- run request-aware inference handlers; +- execute subagent flows; +- host stateful model-based campaigns. + +Properties: + +- inference mode: `serve`; +- attempt concurrency: `1` initially; +- handler routes by request body, session metadata, and active campaign state; +- fresh TUI/session unless a campaign declares reuse; +- separate from queued journeys because the controller response modes are + mutually exclusive. + +### Stateful-property lane + +Purpose: + +- generate many valid mid-flight action sequences; +- test invariants after every transition; +- accumulate a reproducible corpus of seeds. + +It may initially be implemented as a specialized reactive lane. It becomes a +separate lane class when its longer attempt deadlines and adaptive scheduling +would interfere with deterministic journey freshness. + +### Chaos lane + +Purpose: + +- inject server, provider, tool, network, and timing failures; +- verify declared recovery invariants; +- keep expected disruption away from the smoke signal. + +Properties: + +- explicit fault budget; +- no simultaneous unrelated experiment in the same lane; +- a supervisor capable of manual server kill/relaunch; +- stronger evidence capture and longer cooldown; +- never used as the only functional-health lane. + +### Real-inference canary lane + +Purpose: + +- verify the integration with one real provider; +- detect provider API, authentication, streaming, tool-call, and cost drift; +- compare deterministic product health with realistic model behavior. + +Properties: + +- low cadence and strict token/cost budget; +- dedicated credentials and egress policy; +- semantic or externally observable assertions, not exact output strings; +- no use as the sole release gate until its variance is understood. + +This lane is intentionally later than deterministic mock coverage. + +### Ephemeral reproduction lane + +Purpose: + +- rerun one failed attempt in a fresh environment; +- compare persistent-state and clean-state behavior; +- support developer-triggered replay by scenario address and seed. + +Properties: + +- created with the existing safe `OpenCodeDriver.use` lifecycle; +- exact revision, fixture, config, response plan, and seed from the source + attempt; +- artifacts retained regardless of pass or fail; +- result linked to, but never replacing, the source attempt. + +## Why Queue and Serve Need Separate Lanes + +The existing LLM controller exposes `queue`, `send`, and `serve`. Queued output +uses ordered matching between requests and response plans. A served handler +reacts to each request. The controller rejects mixing modes after one is +selected. + +This property is useful, not a limitation to erase. It makes a lane's response +semantics predictable. + +Trying to share one controller across unrelated concurrent queued journeys +would create ambiguity: the next request from TUI B could consume the response +authored for TUI A. Request-aware serving can support more concurrency later, +but only after session routing and handler state are explicitly designed. + +The initial rule is therefore: + +> One active attempt per lane. Add throughput by adding lanes. + +This is operationally simple and matches the current catalog runner, which +keeps scenario steps sequential while running independent variants in separate +processes. + +## Lifetime and Isolation Matrix + +| Resource | Ordinary persistent lane | Dedicated reuse experiment | Ephemeral reproduction | +| --- | --- | --- | --- | +| Host/pod | Persistent | Persistent | Attempt or short batch | +| Lane supervisor | Persistent | Persistent | Attempt | +| OpenCode server | Persistent across attempts | Persistent across attempts | Attempt | +| Database | Persistent by policy | Persistent | Attempt unless replay requires snapshot | +| Project artifact root | Lane generation | Lane generation | Attempt | +| Fixture files | Reset before attempt | Experiment-defined | Fresh | +| Model controller | Server generation | Server generation | Attempt | +| TUI | Fresh per attempt | Reused only if declared | Fresh | +| Session | Fresh per attempt | Reused only if declared | Fresh | +| Tool invocations | Call-scoped | Call-scoped | Call-scoped | +| Attempt evidence writer | Attempt | Attempt | Attempt | + +“Persistent” never means immortal. Every persistent resource has a generation +ID, start timestamp, owner process, and explicit recycle path. + +## Database Policies + +Drive uses an in-memory OpenCode database by default. A 24/7 environment needs +an explicit lane policy. + +### Memory database + +Use for: + +- ephemeral reproduction; +- deterministic smoke while the persistence contract is not under test; +- fast isolated diagnosis. + +The database disappears on server restart. Recovery scenarios must not expect +session continuity under this policy. + +### File-backed lane database + +Use for: + +- long-lived soak lanes; +- restart and rehydration scenarios; +- session-accumulation observation. + +The database path resolves inside the isolated artifact root through +`OPENCODE_DRIVE_DB`. Each lane uses its own file. Never point several lane +processes at the same database unless OpenCode explicitly supports that +topology and the test is designed for it. + +### Snapshot-based reproduction + +Later, a failed persistent attempt may publish a redacted database snapshot. +An ephemeral reproduction lane can restore it before replay. Snapshot restore +is optional evidence, not a prerequisite for the first release. It requires +strong retention and secret-redaction rules. + +## Fixture Isolation + +Catalog scenarios currently share a small project fixture and reset mutated +files between journeys. Continuous verification should extract that behavior +into one explicit preparation operation. + +An ordinary attempt preparation must: + +1. Verify that the lane generation is still current. +2. Close any stale attempt TUI left by reconciliation. +3. Restore declared fixture files atomically where practical. +4. Remove scenario-owned transient files. +5. Preserve lane-owned OpenCode state and database files. +6. Create a new session through observable UI or SDK behavior. +7. Verify the composer is actionable before starting the timed journey. + +Do not use `git reset --hard` against an unresolved or broad path from a +long-running process. Prefer an explicit fixture manifest or a prepared +directory swap whose target is validated inside the lane artifact root. + +## Client Isolation + +The existing executable-scenario metadata distinguishes shared and isolated +clients. Continuous verification interprets it as follows: + +- `shared` means scenarios may execute sequentially through one attempt client + when the runner intentionally batches them; +- `isolated` means the scenario receives a freshly launched TUI and the TUI is + closed after the scenario; +- the default monitoring behavior should still favor one scenario per attempt, + because that produces clearer latency, evidence, and failure attribution. + +Batching is an optimization and a special soak dimension, not the first +operational model. + +## Revision Preparation + +The catalog capture system already resolves revisions to immutable commits and +prepares detached worktrees. Continuous verification should reuse the same +principles: + +1. Resolve a configured ref to a commit SHA. +2. Record ref, SHA, commit timestamp, and source checkout identity. +3. Prepare dependencies with the lockfile enforced. +4. Validate the OpenCode simulation capabilities before marking the lane + ready. +5. Never mutate the prepared source checkout during attempts. +6. Use a separate artifact/project directory for runtime writes. + +Prepared source worktrees may be cached. Cache identity is the immutable +revision plus relevant build inputs, not a branch name. + +## Environment Promotion + +A safe candidate rollout is: + +```text +resolve revision + | + v +provision candidate lanes + | + v +protocol/readiness probe + | + v +short deterministic smoke + | + v +activate scheduled journeys + | + v +observe required window + | + +--> promote + | + +--> drain and retain evidence +``` + +Baseline lanes remain active during the observation window when comparison is +important. Candidate failures can then be compared against the same scenario +on the baseline revision without changing the source attempt. + +Promotion policy should name required signals, for example: + +- protocol negotiation succeeded; +- five consecutive smoke attempts passed; +- no required journey failed in the last hour; +- no property invariant failed in the configured campaign budget; +- resource slope stayed below the soak threshold; +- evidence pipeline remained healthy. + +## Lane Specification + +The persisted lane specification should be schema-backed. The following is an +illustrative proposed shape, not an existing public API: + +```ts +import { Schema } from "effect" + +const LaneKind = Schema.Literals([ + "queued-journey", + "reactive", + "property", + "chaos", + "real-inference", + "reproduction", +]) + +const DatabasePolicy = Schema.Literals([ + "memory", + "file", + "restored-snapshot", +]) + +export class LaneSpec extends Schema.Class("LaneSpec")({ + id: Schema.String, + environmentId: Schema.String, + kind: LaneKind, + revision: Schema.String, + database: DatabasePolicy, + concurrency: Schema.Int.check(Schema.isGreaterThanOrEqualTo(1)), + viewport: Schema.Struct({ + cols: Schema.Int.check(Schema.isGreaterThanOrEqualTo(1)), + rows: Schema.Int.check(Schema.isGreaterThanOrEqualTo(1)), + }), + maxAttemptMilliseconds: Schema.Int.check( + Schema.isGreaterThanOrEqualTo(1), + ), + recycleAfterAttempts: Schema.optionalKey( + Schema.Int.check(Schema.isGreaterThanOrEqualTo(1)), + ), + recycleAfterMilliseconds: Schema.optionalKey( + Schema.Int.check(Schema.isGreaterThanOrEqualTo(1)), + ), +}) {} +``` + +Important configuration choices such as permission policy, tool controls, +theme, and inference strategy either belong directly in the lane spec or in a +separately versioned referenced configuration. They must not exist only as +unrecorded environment variables. + +## Lane Eligibility + +Before leasing work, the lane pool checks: + +- lane state is `ready`; +- lane has an unused concurrency permit; +- OpenCode revision matches the work specification; +- response mode matches the scenario; +- required tools and protocol capabilities are available; +- database and reuse policies satisfy the scenario; +- no incompatible chaos experiment is active; +- remaining cost and resource budgets are sufficient; +- lane is not scheduled to drain before the attempt deadline. + +Eligibility failures do not count as scenario failures. They are scheduling or +capacity signals and can become an incident if they prevent freshness SLOs. + +## Health and Heartbeats + +Each lane publishes: + +- supervisor process identity; +- lane generation ID; +- last heartbeat timestamp; +- current state; +- OpenCode process identity and uptime; +- active attempt ID, if any; +- last successful readiness probe; +- last successful smoke attempt; +- current resource measurements; +- last recycle reason; +- protocol compatibility summary. + +Heartbeats must be written by the lane supervisor, not inferred solely from +OpenCode logs. The control plane detects missing heartbeats independently. + +## Recycling Policy + +There are two competing goals: + +- keep lanes alive long enough to expose lifetime bugs; +- recycle before unrelated resource exhaustion makes the environment useless. + +Use different policies for different lanes: + +- at least one **true soak lane** recycles only on rollout, explicit + maintenance, or unrecoverable failure; +- ordinary journey lanes may recycle after a high attempt count or maintenance + window; +- chaos lanes recycle after experiments that intentionally invalidate their + baseline; +- real-inference lanes may recycle with credential or provider configuration + rotation. + +Every recycle records a reason. A policy recycle is not a crash; a resource +threshold recycle is an operational warning and retains pre-recycle metrics. + +## Recovery Policy + +On unexpected server exit: + +1. Mark the active attempt with the observed exit evidence. +2. Classify it as product or infrastructure failure based on experiment and + host context; use `inconclusive` when attribution is unsafe. +3. Close attempt-owned resources. +4. Increment lane generation. +5. Apply bounded, jittered infrastructure retry policy. +6. Relaunch and perform protocol/readiness probes. +7. Return to `ready` only after the probe succeeds. +8. Escalate if the retry budget or freshness SLO is exhausted. + +Do not automatically replay the failed state-changing journey in the same +attempt. A diagnostic reproduction is separately scheduled. + +## Capacity Planning + +Initial capacity can be estimated from: + +- total journey duration per schedule interval; +- one active attempt per lane; +- property and chaos attempt duration percentiles; +- startup and recycle cost; +- desired baseline/candidate duplication; +- host CPU and memory per OpenCode server/TUI pair. + +If one queued journey lane takes 20 minutes to complete work scheduled every 10 +minutes, it is undersized even if the server is healthy. Add a lane, reduce +cadence, or shorten the matrix. Do not hide the backlog with overlapping queued +responses in one controller. + +## Recommended First Environment + +```text +environment: v2-continuous + +lane: v2-queue-1 + kind: queued-journey + server: persistent + database: file + concurrency: 1 + journeys: smoke + deterministic catalog flows + +lane: v2-reactive-1 + kind: reactive/property + server: persistent + database: file + concurrency: 1 + journeys: subagent + seeded lifecycle campaign + +lane: v2-repro + kind: reproduction + lifecycle: ephemeral + concurrency: 1 + work: on-demand failed-attempt replay +``` + +Add a chaos lane only after the smoke and evidence paths are trustworthy. Add a +real-inference lane only after deterministic product failures and provider +variance are distinguishable in dashboards and alerts. + +## Acceptance Criteria + +The lane subsystem is ready for the first 24/7 deployment when: + +- one server remains alive across at least 100 sequential attempts; +- every attempt gets a distinct TUI identity and session; +- fixture reset is verified and cannot escape the lane artifact root; +- queue and serve work cannot be scheduled onto the wrong lane; +- unexpected server exit produces a terminal attempt and a new lane generation; +- heartbeat absence is detected from outside the lane process; +- lane shutdown closes every TUI and server process; +- an ephemeral reproduction can run the same scenario at the exact revision; +- the run record identifies all relevant lifetimes and generation IDs. diff --git a/docs/continuous-testing/03-bot-orchestration.md b/docs/continuous-testing/03-bot-orchestration.md new file mode 100644 index 0000000..7280d55 --- /dev/null +++ b/docs/continuous-testing/03-bot-orchestration.md @@ -0,0 +1,581 @@ +# Bot Orchestration + +This document defines how continuous-verification bots select work, lease +lanes, execute attempts, handle deadlines, recover from coordinator failure, +and shut down safely. + +A bot is a deterministic orchestrator unless a scenario explicitly asks for +generated behavior. Calling it a bot describes its autonomous, recurring +operation; it does not imply that another LLM is deciding what to do. + +## Responsibilities + +The orchestration subsystem owns: + +- recurring schedules and campaign budgets; +- scenario eligibility and lane matching; +- leases and concurrency limits; +- attempt IDs and attempt state transitions; +- end-to-end deadlines; +- cancellation and draining; +- retry and reproduction requests; +- checkpoint timing wrappers; +- evidence collection coordination; +- stale-attempt reconciliation; +- liveness and progress heartbeats. + +It does not own: + +- OpenCode-specific scenario steps; +- model-output construction; +- UI protocol definitions; +- tool implementation semantics; +- telemetry exporter configuration; +- alert-delivery provider details. + +## Bot Identities and Worker Processes + +A **bot** is a durable logical test profile with its own purpose, cadence, +freshness objective, owner, and coverage report. It does not need one permanent +operating-system process. + +Examples: + +- `journey.smoke.queued` continuously runs essential catalog flows; +- `journey.recovery.reactive` runs interruption and recovery flows; +- `property.session-lifecycle` explores generated session transitions; +- `provider.openai.responses.native` owns the native OpenAI Responses contract; +- `provider.anthropic.messages.native` owns the Anthropic Messages contract; +- `provider.bedrock.converse.native` owns Bedrock request/framing behavior; +- `provider.azure.responses.native` owns Azure Responses construction and + transport cases; +- `provider..aisdk-fallback` owns an actual external AI SDK package and + the canonical adapter path; +- `provider.live.` runs a sparse budgeted drift probe. + +The scheduler materializes each bot's next work item. A shared worker pool may +execute many provider bots because their contract cases are finite and +ephemeral. Persistent journey/soak bots lease compatible long-lived lanes. + +This separation gives one visible health row per provider/package without +paying for one idle process or OpenCode server per provider. Run a dedicated +process only when isolation, credentials, native dependencies, or concurrency +require it. + +A bot definition contains: + +```text +BotDefinition + id + kind + owner + target selector + case/scenario selector + lane requirements + cadence and freshness objective + concurrency + timeout/resource/cost budgets + credential/network profile + evidence and alert policy + enabled/quarantine state + definition version/digest +``` + +Bot status is derived from durable attempts and schedule state, not from the +existence of a process named after it. A bot can therefore be `healthy`, +`failing`, `stale`, `blocked`, `disabled`, or `quarantined` independently of +the worker pool. + +## Units of Work + +### Schedule entry + +A schedule entry says when and under which policy a scenario should run. + +Proposed fields: + +- schedule entry ID; +- scenario or campaign ID; +- cadence policy; +- eligible environment or revision selector; +- eligible lane kinds; +- priority; +- maximum queue age; +- attempt timeout; +- retry classification policy; +- alert policy; +- enabled/disabled state; +- optional start and end windows. + +### Work specification + +A work specification is an immutable decision to run one scenario or campaign. +It records: + +- generated work ID; +- source schedule entry or manual trigger; +- target environment/revision; +- scenario ID; +- inference strategy; +- optional seed and step count; +- optional chaos plan; +- required protocol capabilities; +- timeout and evidence policy; +- creation timestamp and expiration. + +### Attempt + +An attempt is one execution of a work specification on one concrete lane. +Retries and reproductions create new attempts linked to the source attempt. + +This distinction prevents a retry from rewriting reliability history. One work +item may have multiple attempts, but each attempt has one immutable outcome. + +## Scheduler Design + +Timing policy should use Effect `Schedule`, not ad hoc `while` loops containing +mutable counters and sleeps. + +Recommended policies: + +- fixed or windowed cadence for smoke journeys; +- spaced cadence for ordinary journey cycles; +- cron cadence for nightly extended campaigns; +- jittered cadence across equivalent lanes to avoid synchronized load; +- bounded campaign iteration count for property tests; +- explicit cooldown after a chaos experiment. + +Example intent: + +```text +smoke every 60 seconds, aligned, small jitter +critical journeys every 5 minutes +full journey matrix every 30 minutes +property campaign 100 seeds per hour +reconnect chaos every 6 hours +extended soak report once per day +real provider canary every 30 minutes with a daily token budget +``` + +Schedules produce due work. They do not wait synchronously for a particular +lane. Due work enters a bounded queue or durable work table. Backlog age is +observable and can violate freshness SLOs even when individual attempts pass. + +## Fairness and Priority + +The initial priority order should be: + +1. heartbeat/readiness probes; +2. smoke journeys required for freshness; +3. diagnostic reproduction requested for an active incident; +4. critical deterministic journeys; +5. ordinary journey matrix; +6. property campaigns; +7. chaos and exploratory work. + +Lower-priority campaigns use quotas so they cannot starve indefinitely. A +simple approach is weighted round-robin across priority classes after reserving +capacity for smoke work. + +Do not let a large property campaign insert thousands of independent work rows +at once. Persist a campaign cursor and issue only enough seeds to fill the +configured concurrency window. + +## Lane Leasing + +A lane lease prevents overlapping attempts from consuming one simulation +controller or workspace unexpectedly. + +A lease contains: + +- lease ID; +- lane ID and lane generation ID; +- attempt ID; +- owner process identity; +- acquisition timestamp; +- expiration timestamp; +- renewal timestamp; +- concurrency slot number when the lane supports more than one. + +The first implementation uses one slot per lane. The design retains a slot +field so higher concurrency can be introduced without changing record +identity. + +Lease operations are idempotent: + +- acquiring for an already-owned attempt returns the same lease; +- renewal verifies owner and lane generation; +- release succeeds if the lease is already absent; +- a generation change invalidates every lease from the previous generation. + +A heartbeat fiber renews the lease while the attempt runs. Losing the lease +interrupts the attempt; it does not allow uncoordinated execution to continue. + +## Attempt Lifecycle + +The coordinator implements one named business operation, conceptually +`AttemptCoordinator.execute`. + +### 1. Decode and validate + +Decode the work specification through Effect Schema. Check that the scenario +exists and that requested inference, chaos, and database policies are +compatible. + +Validation failures are typed scheduling failures. They do not start a lane or +count as product failures. + +### 2. Allocate attempt identity + +Create an immutable attempt ID before acquiring mutable resources. Persist a +`scheduled` event with the exact configuration and source work ID. + +### 3. Acquire a compatible lane + +Ask `LanePool` for a lane satisfying the scenario requirements. Persist the +lease and transition to `leased`. + +If no capacity exists before the work expires, terminalize as +`infrastructure_failed` or a more specific scheduling failure. Do not call it a +scenario timeout because scenario execution never began. + +### 4. Prepare attempt resources + +Within an attempt scope: + +- verify lane generation; +- restore the project fixture; +- launch a named TUI; +- negotiate UI capabilities; +- create or navigate to a fresh session; +- start the local evidence manifest; +- install attempt correlation annotations. + +Preparation has its own deadline and failure classification. + +### 5. Execute the scenario + +Run the scenario with: + +- the adapted `Driver` whose `ui` points at the attempt TUI; +- the configured inference strategy; +- a checkpoint callback that records timing and optional frames; +- the attempt deadline; +- concurrent observation of lane/server failure. + +The current `OpenCodeDriver.use` already races user work with driver failure and +performs settlement at its safe lifecycle boundary. A persistent lane needs the +same principle at attempt granularity: scenario work races lane-generation +failure, while the lane itself remains alive after an ordinary scenario +failure. + +### 6. Collect evidence + +On success, collect the compact configured evidence. On failure, capture the +failure bundle before releasing the TUI when possible. Evidence failure is +recorded separately from the primary outcome. + +### 7. Classify outcome + +Inspect typed errors and, only at the coordinator boundary, the full Effect +`Cause` when defects or interrupts must be distinguished. + +Possible terminal classifications are defined in +[Observability and evidence](./08-observability-and-evidence.md). + +### 8. Release resources + +The attempt scope closes: + +- recording timeline; +- simulation client associated with the TUI; +- TUI process; +- local evidence writer; +- lease-renewal fiber; +- lane lease. + +Cleanup errors are appended to the attempt cause/evidence. They do not hide the +primary scenario failure. + +### 9. Emit follow-up work + +Policy may request: + +- no follow-up; +- one diagnostic retry in the same persistent lane; +- one clean ephemeral reproduction; +- an expanded property campaign near the failing seed; +- lane recovery or quarantine. + +Every follow-up receives a new attempt ID and a typed relationship to the +source attempt. + +## Checkpoint Wrapper + +Catalog scenarios receive a capture callback for each ordered state. The +monitoring adapter uses that callback as a checkpoint boundary. + +For every checkpoint: + +1. verify the attempt and lane generation are current; +2. record monotonic elapsed time and wall-clock timestamp; +3. append checkpoint status to the run store; +4. annotate the current span with checkpoint address and ordinal; +5. optionally capture a frame according to evidence policy; +6. update attempt progress heartbeat; +7. return control to the scenario. + +Checkpoint persistence should be idempotent by attempt ID and ordinal. A +duplicate callback with a different address is a harness invariant violation. + +## Timeouts + +Use layered deadlines rather than one opaque timeout: + +| Deadline | Purpose | Example reaction | +| --- | --- | --- | +| Queue age | Work waited too long for capacity | Freshness/capacity incident | +| Lane acquisition | Lease could not be obtained | Reschedule or fail scheduling | +| Preparation | TUI/session never became ready | Lane recovery and evidence | +| Scenario | End-to-end journey exceeded budget | Capture frame/events, classify | +| Checkpoint wait | One observable state did not arrive | Existing typed UI timeout | +| Evidence collection | Capture/upload is stuck | Preserve local bundle, mark evidence degraded | +| Graceful cleanup | Process refuses to close | Escalate termination and record cleanup failure | + +Timeouts produce typed errors. They are not generic strings, and they retain +which phase and checkpoint were active. + +## Retry Policy + +Retries have different semantics by failure class. + +### Never retry invisibly + +State-changing operations such as submit, click, approve, reject, or tool +completion are not retried inside an attempt. The system cannot generally know +whether the first operation committed before transport failure. + +### Retry safe infrastructure preparation + +Safe, idempotent infrastructure operations may use bounded retries: + +- artifact upload by content digest; +- heartbeat publication; +- read-only lane-health queries; +- opening a connection before any work is submitted; +- rebuilding an immutable source checkout. + +Use `Effect.retry` with a `Schedule`, retry only known retryable error tags, add +jitter for distributed workers, and expose attempt metadata in telemetry. + +### Retry a whole journey as a new attempt + +Policy may schedule a new attempt when: + +- an assertion appears intermittent; +- a clean-state comparison is useful; +- an infrastructure failure has recovered. + +The original failure remains visible. A pass on retry changes diagnosis, not +history. + +### Do not retry deterministic product failures automatically forever + +Repeated retries create load and alert noise without adding information. After +one persistent and one clean reproduction, deduplicate the incident and reduce +cadence until the revision or scenario changes. + +## Cancellation + +Cancellation sources include: + +- operator request; +- environment drain; +- lost lease; +- attempt deadline; +- lane generation failure; +- process supervisor shutdown. + +Effect interruption is the cancellation mechanism. The coordinator records +who requested cancellation and why, then interrupts the attempt fiber. Scoped +finalizers perform cleanup. + +Cancellation is not caught and converted to success. The terminal record is +`cancelled` unless another primary failure was already committed. + +## Draining + +When a lane or environment begins draining: + +1. scheduler stops assigning new work; +2. queued work is redirected or remains pending; +3. active attempts receive a grace deadline; +4. long property or soak attempts may checkpoint and stop at a safe boundary; +5. attempts exceeding grace are interrupted; +6. leases are released; +7. lane shutdown proceeds. + +Draining state and deadlines are visible in lane heartbeats. + +## Reconciliation + +The control plane periodically scans for inconsistent state: + +- `leased` attempt with expired lease; +- `running` attempt with no progress heartbeat; +- lane heartbeat missing while it owns an attempt; +- terminal attempt that still has an active lease; +- non-terminal attempt referencing an old lane generation; +- artifact upload pending beyond its budget; +- due schedule with no issued work; +- campaign cursor not advancing. + +Reconciliation is conservative. It may mark an attempt `inconclusive` and +release stale coordination records, but it never fabricates a passed outcome. + +## Backpressure + +Every queue is bounded or durable with an explicit age policy. + +- The scheduler does not generate unbounded future work. +- The attempt worker takes only work for which it can acquire capacity. +- Evidence upload uses a bounded local spool. +- Telemetry exporter backpressure cannot block lane cleanup indefinitely. +- Property campaigns issue a small rolling window of seeds. +- Alert delivery retries are bounded and deduplicated. + +Backlog depth and oldest-item age are metrics. Dropping low-priority work is an +explicit event with a reason, not silent loss. + +## Proposed Effect Services + +The following code is an illustrative architecture sketch, not a committed +public API: + +```ts +import { Context, Effect, Scope } from "effect" + +export class Scheduler extends Context.Service +}>()("ContinuousVerification/Scheduler") {} + +export class LanePool extends Context.Service Effect.Effect +}>()("ContinuousVerification/LanePool") {} + +export class RunStore extends Context.Service Effect.Effect + readonly appendEvent: ( + event: AttemptEvent, + ) => Effect.Effect +}>()("ContinuousVerification/RunStore") {} + +export class AttemptCoordinator extends Context.Service< + AttemptCoordinator, + { + readonly execute: ( + work: WorkSpecification, + ) => Effect.Effect + } +>()("ContinuousVerification/AttemptCoordinator") {} +``` + +In real code, all referenced models and errors should be Schema classes or +schema-backed tagged errors because they cross persistence and process +boundaries. + +The main worker workflow remains small: + +```ts +const worker = Effect.fn("ContinuousVerification.worker")(function* () { + const scheduler = yield* Scheduler + const coordinator = yield* AttemptCoordinator + const work = yield* scheduler.take + yield* coordinator.execute(work) +}) +``` + +Recurring execution applies a schedule at the worker boundary. Layer +composition, telemetry, platform services, and runtime execution occur once at +the application entrypoint. + +## Process Topology + +The first deployment can use: + +- one scheduler/control process; +- one lane-supervisor process per lane; +- one external supervisor such as systemd, Docker, or Kubernetes; +- one durable run store; +- one artifact root/spool per lane; +- one telemetry collector reachable by every process. + +The scheduler and lane supervisor may initially share code and configuration, +but should remain distinct process roles. If a lane process deadlocks while +driving OpenCode, the scheduler must still observe the missing heartbeat. + +## Testing the Orchestrator + +Use `@effect/vitest` and explicit layers. + +Unit coverage should include: + +- cadence and jitter using `TestClock`; +- priority and fairness; +- bounded campaign issuance; +- lease acquisition, renewal, expiry, and generation invalidation; +- every attempt state transition; +- idempotent checkpoint writes; +- timeout classification by phase; +- cancellation and finalizer execution; +- retry creation without source-attempt mutation; +- reconciliation of abandoned attempts; +- backpressure and queue expiration. + +Property tests should generate attempt-event sequences and assert that the +state projection never transitions from a terminal state, never owns two +exclusive leases, and never reports `passed` without a completed scenario. + +Integration coverage should use a fake lane layer first, then one real Drive +lane for lifecycle characterization. Tests that share a resource layer use the +`layer(...)` helper; tests requiring isolated instances use separate +`it.layer(...)` blocks. + +## Operational Metrics + +The orchestrator publishes at least: + +- work issued by schedule and scenario; +- pending work count and oldest age; +- lane acquisition wait duration; +- active attempts; +- attempt phase duration; +- lease renewal failures; +- reconciled abandoned attempts; +- retries and reproductions requested; +- work expired or dropped by reason; +- campaign seeds issued and completed; +- scheduler heartbeat age. + +Keep identifiers such as attempt ID out of metric labels. They belong in spans, +logs, and run records. + +## Acceptance Criteria + +The first orchestrator is ready when: + +- schedules run deterministically under `TestClock`; +- work never overlaps on a one-slot lane; +- a killed worker leaves an expired lease that reconciliation resolves; +- a scenario timeout captures evidence and releases its TUI; +- a retry appears as a linked new attempt; +- product failures are not retried as infrastructure operations; +- draining stops new work and closes active resources within a bound; +- queue age and missing progress can independently alert; +- the worker can run for 24 hours without growing an unbounded in-memory queue; +- all major workflows appear as named Effect spans. diff --git a/docs/continuous-testing/04-inference-simulation.md b/docs/continuous-testing/04-inference-simulation.md new file mode 100644 index 0000000..13bc6ce --- /dev/null +++ b/docs/continuous-testing/04-inference-simulation.md @@ -0,0 +1,527 @@ +# Inference Simulation + +This document defines how continuous verification controls model behavior. It +covers deterministic queued responses, request-aware served responses, +fault-injected output, and a later real-provider bridge. + +Model control belongs in Effect programs and scripts. It does not become a +Drive CLI command, an alias in the frontend protocol, or an OpenCode media +directory concern. + +## Goals + +Inference simulation must: + +- make important OpenCode behavior reproducible without a provider account; +- exercise text, reasoning, tool input, finish, disconnect, and timing paths; +- support simple authored journeys and stateful generated campaigns; +- detect unexpected or unused requests; +- preserve protocol compatibility and provider-neutral behavior; +- record exactly which response plan influenced an attempt; +- allow controlled nondeterminism only when the seed or distribution is known; +- provide a later path to real provider integration without changing scenario + and evidence contracts. + +## Existing Primitives + +[`packages/drive/src/llm/index.ts`](../../packages/drive/src/llm/index.ts) +defines schema-validated output values: + +- `Llm.text(text, options)`; +- `Llm.reasoning(text, options)`; +- `Llm.pause(milliseconds)`; +- `Llm.toolCall(call, options)`; +- `Llm.raw(chunk)`; +- `Llm.finish(reason)`; +- `Llm.disconnect()`. + +Text, reasoning, and tool-input output can be chunked and paced. The responder +plays those values onto the canonical backend simulation RPCs and guarantees a +terminal event when the authored stream omits one. + +The controller exposes: + +- `llm.queue(...)`: enqueue output for the next normal request and return; +- `llm.send(...)`: enqueue output and wait until the matched request completes; +- `llm.serve(handler)`: choose a response stream from each opened request; +- `llm.title(handler)`: control title requests separately. + +At settlement, queued mode detects unexpected model requests and unused +responses. Output after a terminal event is a controller failure. These +properties make the simulator useful as an oracle, not merely a stub. + +## Response Modes + +### Queued mode + +Queued mode is the default for deterministic, sequential journeys. + +Example: + +```ts +yield* driver.llm.queue( + Llm.reasoning("I will inspect the fixture."), + Llm.text("The fixture value is 42."), +) +yield* driver.ui.submit("Inspect the fixture") +yield* driver.ui.waitFor("The fixture value is 42.") +``` + +Strengths: + +- minimal authoring overhead; +- exact response sequence; +- settlement detects missing or extra exchanges; +- easy to understand in a failed journey; +- ideal for one active attempt per lane. + +Constraints: + +- responses match normal requests by order; +- unrelated concurrent clients can consume each other's plans; +- complex subagent or request-dependent behavior becomes awkward; +- the lane cannot switch to served mode after queueing begins. + +Use a dedicated queued lane and keep attempt concurrency at one. + +### `send` mode + +`send` is queued mode with a completion barrier. The call returns only after a +request consumed the output and the response finished. + +Use it when the script needs to synchronize with model completion directly. +Avoid it when UI observations are the intended assertion; waiting on UI state +usually gives stronger end-to-end evidence. + +An interrupted `send` withdraws an unmatched queued response. Once matched, +normal response lifecycle rules apply. + +### Served mode + +Served mode installs one request-aware handler: + +```ts +yield* driver.llm.serve((request, index) => + Stream.make( + Llm.text(`response ${index} for ${request.id}`), + ), +) +``` + +The handler receives the opened exchange and a normal-request index. It can +inspect the body, offered tools, session-related metadata exposed by the +protocol, and local campaign state. + +Strengths: + +- reacts to actual requests; +- supports subagent exchanges and variable request counts; +- supports a generated model of inference behavior; +- can delegate to a real provider bridge later. + +Constraints: + +- mutable handler state must be scoped and concurrency-safe; +- response routing must not rely on fragile full-body string matching; +- failures in the handler fail the controller; +- one served handler owns all normal exchanges in that controller generation; +- title exchanges need their own handler or documented default. + +Use a dedicated reactive lane. Start with one active attempt to keep handler +state and session ownership unambiguous. + +## Title Requests + +OpenCode may open a separate inference request to generate a conversation +title. Drive recognizes title requests and handles them outside normal request +sequencing, after in-flight normal jobs on which they depend. + +Every strategy must decide how titles behave: + +- use the default deterministic Drive title; +- configure `llm.title` once for the lane; +- route provider-backed title generation explicitly; +- disable title-sensitive assertions if the strategy intentionally varies + title text. + +Title requests must not consume a normal queued journey response. A response +plan and failure bundle should record whether a request was classified as a +title. + +## Strategy Model + +The continuous system introduces an app-owned **inference strategy** around the +existing controller. It is configuration and evidence vocabulary, not a new +wire protocol. + +Recommended strategy variants: + +```text +QueuedPlan + ordered response plans authored by a scenario + +ReactivePlan + request-aware deterministic handler + +GeneratedPlan + seeded handler selecting valid outputs from a model + +FaultPlan + wraps another plan with deliberate delays/disconnects/errors + +ProviderPlan + calls a real provider and translates its stream into Llm.Output +``` + +Each attempt records: + +- strategy variant and version; +- deterministic plan digest; +- seed when generation or jitter is involved; +- output and fault budgets; +- selected provider/model for real inference; +- request and response summary without secrets; +- compatibility mode used for tool-input streaming. + +## Deterministic Response Plans + +A response plan is a persisted, schema-validated description of intended +output. It should use the existing `Llm.Output` schema rather than defining a +parallel output vocabulary. + +Illustrative proposed model: + +```ts +import { Schema } from "effect" +import * as Llm from "opencode-drive/llm" + +export class PlannedExchange extends Schema.Class( + "PlannedExchange", +)({ + label: Schema.String, + expectedPromptMarker: Schema.optionalKey(Schema.String), + output: Schema.Array(Llm.Output), +}) {} + +export class QueuedResponsePlan extends Schema.Class( + "QueuedResponsePlan", +)({ + version: Schema.String, + exchanges: Schema.Array(PlannedExchange), +}) {} +``` + +The scenario may keep constructing output in code initially. The persisted +representation becomes useful when replaying a failure independently of a +changed source tree. It is not necessary to serialize functions or handler +closures. + +Plan validation should reject: + +- output after `finish` or `disconnect`; +- negative delays or invalid chunk sizes, already rejected by output schemas; +- duplicate tool-call indices or IDs where the scenario requires uniqueness; +- a tool call for a tool the request cannot offer; +- an empty plan when the scenario requires a normal exchange; +- unbounded pauses in a monitored journey. + +## Determinism + +Determinism has several layers: + +### Semantic determinism + +The same request receives the same logical text, tool calls, terminal reason, +and injected failures. + +### Timing determinism + +Chunk boundaries and delays follow the same sequence. + +### Scheduling determinism + +Concurrent fibers and external processes interleave identically. + +Full scheduling determinism is not realistic for a black-box multi-process +system. The goal is to capture enough inputs to reproduce the behavior at a +high rate and to separate deliberate timing variation from accidental hidden +randomness. + +The current text chunk helper varies chunk sizes with `Math.random`. That is +useful for naturally exercising boundaries, but it is not controlled by +Effect's `Random.withSeed`. Before calling a campaign fully replayable, either: + +- add an explicit deterministic chunk plan to the app-level response plan; +- provide a seeded randomness seam in the generic simulator if several callers + need it; or +- record the actual emitted chunk sequence in failure evidence and replay that + sequence through raw or fixed chunks. + +Do not claim seed-only reproduction while an unrecorded random source still +influences timing. + +## Request-Aware Routing + +A reactive handler should classify requests through structured data where +available. + +Recommended routing inputs: + +- title versus normal request classification; +- attempt ID known by the lane's active-attempt context; +- request ID; +- request index; +- offered tool names and schemas; +- normalized latest user prompt marker; +- parent/subagent relationship where observable; +- current campaign/model state. + +Avoid routing by `JSON.stringify(body).includes(...)` as a permanent design. +It is acceptable in a narrow existing fixture but fragile for an always-on +system. Decode the subset of request body required by the strategy and use +typed predicates. + +If the backend protocol does not expose a necessary stable identifier, first +decide whether that identifier belongs in the canonical OpenCode simulation +protocol. Do not invent a Drive-only wire field. + +## Tool Calls + +Tool-call simulation must preserve provider-neutral semantics. + +A plan declares: + +- call index; +- stable call ID within the exchange; +- offered tool name; +- JSON input; +- optional chunking/pacing; +- finish reason, normally `tool-calls` when applicable. + +The responder uses provider-neutral tool-input start/delta messages when the +endpoint advertises the capability. It falls back to the supported legacy raw +provider chunk when required by compatibility policy. + +Scenario assertions should inspect observable tool and server state, not only +the rendered label. Useful assertions include: + +- OpenCode offered the intended tool; +- streamed input became valid at the expected boundary; +- permission appeared before execution; +- exactly one invocation exists for the call ID; +- progress and terminal output have valid ordering; +- interruption settles the tool projection correctly; +- a recovery prompt can execute after failure. + +## Fault Injection + +The output vocabulary already provides several high-value faults: + +- `pause` introduces deterministic provider latency; +- `disconnect` terminates the simulated provider exchange; +- streamed tool input allows interruption before JSON is complete; +- output after a terminal event intentionally triggers controller validation; +- a handler can fail with a typed controller error; +- controlled tools can delay, fail, or wait for interruption. + +A `FaultPlan` wraps a normal strategy and records: + +- injection point; +- fault kind; +- delay or payload; +- expected OpenCode behavior; +- maximum recovery deadline; +- cleanup and cooldown requirements. + +Fault injection is never inferred from an ordinary error. If a disconnect was +not declared by the plan, it is an unexpected provider or transport failure. + +## Stateful Generated Inference + +A generated inference model chooses outputs based on current test state. For +example: + +```text +Idle request + -> stream text + -> stream reasoning then text + -> start tool input + -> disconnect + +Tool result request + -> finish with summary text + -> request another tool within budget + -> pause then finish +``` + +Generation rules must enforce: + +- only offered tools are called; +- tool indices and IDs are unique where required; +- terminal events end the output stream; +- step and output budgets prevent infinite tool loops; +- a command records its generation choice before executing it; +- every choice derives from a controlled random service or is captured as an + explicit trace item. + +The property model, not the inference handler alone, decides which transitions +are valid. See [Stateful property testing](./06-stateful-property-testing.md). + +## Real Provider Bridge + +Real inference is a later strategy implemented at the app/script boundary. +Its purpose is to introduce variable model behavior into a Drive-controlled +journey. It is not the provider-package compatibility harness. + +Conceptual flow: + +```text +OpenCode opened exchange + | + v +ProviderPlan decodes request subset + | + v +provider SDK / HTTP streaming call + | + v +translate provider events to Llm.Output + | + v +existing Drive responder and canonical simulation RPC +``` + +The bridge should close over a provider client constructed by a scoped Effect +layer. It must not instantiate SDK clients inside every handler call without a +lifecycle policy. + +Requirements: + +- credentials supplied through a secret provider, never attempt config; +- strict model allowlist; +- per-attempt token and time budget; +- daily lane cost budget and kill switch; +- retry only before the provider has emitted externally visible partial + output, unless the provider API offers a safe idempotency contract; +- provider request IDs captured in restricted logs, not metrics; +- content redaction before artifact publication; +- provider/model/version recorded in the attempt; +- deterministic mock lane remains the primary product oracle. + +Provider failover may later use an Effect `ExecutionPlan` when the purpose is +explicitly to test fallback behavior. Do not hide provider failure with +automatic fallback in a lane intended to monitor one provider. + +The bridge projects a real provider's output back through Drive's simulated +OpenAI Chat route. That still exercises the real OpenAI Chat decoding and the +full OpenCode session/UI stack, but it does not prove that OpenCode's selected +Anthropic, Gemini, Bedrock, Azure, Responses, WebSocket, or AI SDK package path +works. It also normalizes away some provider-native failures and metadata. + +Test those paths by executing the real installed package against a programmable +HTTP/WebSocket boundary as specified in [Provider and package contract +testing](./04-provider-package-contract-testing.md). Use a sparse production +provider canary only after deterministic contract coverage exists. + +## Assertions for Variable Output + +Exact strings remain appropriate for deterministic plans. Real or generated +output needs different oracles. + +Prefer externally observable properties: + +- a response reaches a terminal state; +- the session projection retains the user prompt and assistant parts; +- requested tool input validates against the offered schema; +- a declared file change occurred and has expected structural content; +- permission and form lifecycles settle; +- the composer returns to an actionable state; +- no internal transport defect is shown to the user; +- resource and latency budgets are respected. + +An LLM judge can provide supplemental diagnostics, but it must not be the only +oracle for critical correctness. A model judging another variable model creates +correlated failure and makes replay harder. + +## Response Evidence + +For each exchange, retain a redacted summary: + +- attempt and lane generation IDs; +- request ID and ordinal; +- title/normal classification; +- selected strategy and plan step; +- offered tool names; +- output item types and sizes; +- actual chunk count and timing summary; +- finish or disconnect terminal event; +- controller error, if any; +- start, first-output, and terminal timestamps. + +Full prompt and response content has stricter retention and redaction policy. +Metrics use low-cardinality types and durations, never raw prompt text or +request IDs. + +## Failure Classification + +Examples: + +| Observation | Likely classification | +| --- | --- | +| Scenario left an authored queued response unused | Harness failure | +| OpenCode made an unexpected extra normal request | Product or protocol drift; investigate before final classification | +| Handler called a tool not offered by OpenCode | Harness failure | +| OpenCode lost a prompt after valid streamed output | Product failure | +| Injected disconnect did not leave UI recoverable | Product failure | +| Provider credential expired | Infrastructure/configuration failure | +| Real provider returned rate limit within declared expectations | Provider/infrastructure signal, not automatically product failure | +| Drive emitted output after its own terminal plan | Harness or Drive defect | + +Classification rules live in configuration/code and are versioned. They should +not rely on free-form error-string matching when typed tags or protocol events +are available. + +## Testing the Strategy Layer + +Unit tests should cover: + +- schema decoding of every plan variant; +- terminal-event validation; +- title routing; +- offered-tool extraction; +- deterministic choice for the same seed; +- response trace recording; +- fault insertion at every supported point; +- budget exhaustion; +- cancellation during provider streaming; +- redaction and summary construction. + +Use `it.effect.prop` for generated response-plan laws. Useful properties: + +- generated streams contain at most one terminal event; +- no item occurs after a terminal event; +- every generated tool name belongs to the offered set; +- encoded and decoded response plans round-trip; +- step and token budgets bound the stream; +- replaying a recorded choice trace yields the same logical output. + +Integration tests should run plans through the actual `LlmController` and a +transport peer, reusing the package's current simulation tests. A small number +of live OpenCode tests validates end-to-end projection. + +## Acceptance Criteria + +Inference simulation is ready for continuous operation when: + +- every deterministic journey records a plan digest; +- queued and served strategies cannot share one lane accidentally; +- unexpected and unused exchanges terminalize the attempt visibly; +- title requests never consume normal queued responses; +- every deliberate fault is distinguishable from an unexpected failure; +- actual output type/chunk/timing summaries are captured; +- generated behavior is replayable from a seed plus recorded choice/chunk + trace; +- no strategy can exceed its time, step, or output budget; +- a real-provider strategy can be disabled globally without changing + deterministic lanes; +- no new backend-control CLI command or Drive-only protocol field is added. diff --git a/docs/continuous-testing/04-provider-package-contract-testing.md b/docs/continuous-testing/04-provider-package-contract-testing.md new file mode 100644 index 0000000..4ab4af3 --- /dev/null +++ b/docs/continuous-testing/04-provider-package-contract-testing.md @@ -0,0 +1,906 @@ +# Provider and Package Contract Testing + +This document defines the missing fidelity layer between Drive's deterministic +model simulation and sparse calls to real model providers. Its purpose is to +answer a concrete question: + +> Given the exact packages selected by OpenCode, what happens for this request, +> response, stream fragment, transport failure, or cancellation? + +The answer should come from executing those packages, not from reimplementing +what we think they do in a mock. + +## Target Snapshot + +This design was checked against the local `../opencode` `v2` ref at +`c53f4cfb094bb87852d0c3c8e83933e902e81283`. The remote-tracking `origin/v2` +was newer at the time of review, so implementation work must refresh the +inventory and record the exact tested commit in every compatibility report. + +The relevant V2 ownership is: + +- `packages/ai` owns the provider-neutral schema, native protocols, routes, + transports, provider entrypoints, and typed `AIError` model; +- `packages/core/src/model-resolver.ts` maps catalog package metadata to native + routes, package-like entrypoints, or the AI SDK fallback; +- `packages/core/src/aisdk.ts` adapts packages using the AI SDK provider + interface into canonical OpenCode events and errors; +- `packages/core/src/session/runner` owns retry, continuation, compaction, tool, + and durable session behavior; +- `packages/simulation/src/backend` replaces outbound HTTP in Drive mode and + exposes controlled inference through the canonical simulation protocol; +- `packages/http-recorder` records and replays real Effect HTTP and WebSocket + traffic; +- this repository controls the simulated backend and user-facing system through + Drive. + +These boundaries matter more than the raw number of dependencies. + +## The Central Rule + +**Mock the transport or an explicit service boundary; execute the package.** + +Do not create hand-written replacements for every provider SDK, parser, auth +helper, or package export. Such replacements only prove that the replacement +behaves as authored. They cannot reveal whether an installed package: + +- throws synchronously while constructing a model; +- rejects when preparing a request; +- returns a non-2xx provider error; +- emits an error event inside an otherwise successful stream; +- returns malformed usage or finish metadata; +- hangs after partial output; +- retries internally; +- reacts differently to cancellation before and after bytes are observed; +- changes its normalization behavior after a dependency upgrade. + +Instead, instantiate the actual package selected by the tested OpenCode lockfile +and route its network calls to a programmable local transport. The transport +can emit exact status codes, headers, bodies, SSE frames, WebSocket frames, +read failures, and timing. The package's observed output is the result under +test. + +There are narrow exceptions: + +- use `TestLLM` when a unit test is explicitly about a consumer of canonical + `LLMEvent`s and provider lowering is outside its scope; +- provide an Effect test layer for filesystem, clock, process, credential, or + catalog services when that service is the declared boundary under test; +- stub a package loader to test selection and loading failures without + installing arbitrary packages; +- use a deliberately fake implementation to test the port's consumer contract, + while keeping a separate contract suite for every real implementation. + +The exception must be visible in the test name and coverage metadata. + +## Four Different Things Commonly Called an Inference Mock + +These layers solve different problems and must not be treated as substitutes. + +| Layer | What is replaced | Real code still executed | Best use | Does not prove | +| --- | --- | --- | --- | --- | +| Canonical event fake | `LLMClient` through `TestLLM` | Session consumer above canonical events | Fast runner and consumer unit tests | Provider request lowering, framing, parsing, HTTP errors | +| Drive protocol simulation | Provider behavior controlled through backend RPC | OpenAI Chat request construction, in-memory HTTP route, SSE bytes, framing, schema decode, protocol state machine, runner, server, TUI | Deterministic end-to-end journeys and lifecycle faults | Other protocols, pre-response HTTP failures, many transport variants, AI SDK fallback parity | +| Programmable transport | Effect HTTP/WebSocket transport | Actual native route or AI SDK package plus its adapters | Provider/package contract probes and negative cases | The remote provider's current production behavior | +| Record/replay or live provider | Nothing below provider API, then replayed transport | Actual request path and real recorded response | Drift detection and realistic golden cases | Unrecorded failures, timing/backpressure when the recorder buffers | + +All four belong in the verification portfolio. Calling all of them “the mock” +would hide which behavior a passing test actually covered. + +## What Drive V2 Already Exercises + +Drive's default isolated project selects: + +```json +{ + "model": "simulation/gpt-sim-model", + "providers": { + "simulation": { + "package": "@opencode-ai/ai/providers/openai/chat" + } + } +} +``` + +In V2, `packages/simulation/src/backend/openai.ts` claims the real OpenAI Chat +endpoint. Controlled Drive items are encoded as OpenAI Chat chunks and streamed +as SSE ending in `[DONE]`. Downstream code then performs the normal: + +```text +canonical Drive output + | + v +OpenAI Chat-shaped SSE bytes + | + v +real SSE framing and event Schema + | + v +real OpenAI Chat protocol state machine + | + v +canonical LLMEvent stream + | + v +real Session runner, projections, server, and TUI +``` + +This is much stronger than returning an assistant string directly to the +session runner. Text chunking, reasoning chunks, tool-input assembly, tool +calls, finish reasons, incomplete streams, interruption, and durable session +effects pass through real production code. + +The V2 simulated network also denies any unregistered destination. A deterministic +Drive run cannot silently leak to a real provider. + +## What Drive Does Not Cover Today + +The current simulated backend registers one main inference route: OpenAI Chat. +Consequently, ordinary Drive journeys do not establish contract compatibility +for all of these paths: + +- OpenAI Responses over HTTP; +- Open Responses-compatible deployments; +- OpenAI Responses WebSocket channel execution; +- Anthropic Messages and Anthropic-compatible Messages; +- Gemini Developer API; +- Vertex Gemini, Chat, Responses, and Messages; +- Bedrock Converse and AWS event-stream framing; +- Bedrock Mantle Chat and Responses; +- Azure route selection and endpoint variants; +- OpenRouter and other provider-specific metadata/options; +- native package entrypoint construction and settings validation; +- dynamic AI SDK package loading and the `packages/core/src/aisdk.ts` adapter; +- authentication and endpoint resolution failures before a request is sent; +- non-2xx HTTP failures because the current Drive route normally returns 200; +- connection failure before headers, response-body read failure, truncated + framing, wrong content type, and backpressure; +- retry classification across all canonical error reasons; +- real-provider drift. + +This is the package-contract backlog. It is finite and can be generated from +routes, protocols, package mappings, and error categories; it is not an +unbounded requirement to mimic every dependency. + +## V2 Runtime Paths to Inventory + +The inventory must be generated from the tested ref on every compatibility +campaign. At the reviewed snapshot there are three principal resolution paths. + +### Direct native routes + +`ModelResolver` directly maps some catalog `aisdk:` metadata to native routes: + +- `@ai-sdk/openai` to OpenAI Responses; +- `@ai-sdk/anthropic` to Anthropic Messages; +- `@ai-sdk/openai-compatible` with an explicit URL to OpenAI-compatible Chat. + +These paths execute `@opencode-ai/ai` protocols and do not instantiate the +external provider package. + +### Native package-like entrypoints + +`AISDKNative.map` translates selected AI SDK package identities and settings to +export paths inside the single `@opencode-ai/ai` package. Examples include +Google, Azure, Bedrock, Bedrock Mantle, OpenRouter, xAI, and Vertex Messages. + +These are API slices, not separately published packages. Contract tests should +still test each exported `model(modelID, settings)` entrypoint because route, +auth, endpoint, defaults, and settings mapping differ. + +### Dynamic AI SDK fallback + +When no native mapping exists, production may load an external AI SDK provider +package and adapt its `LanguageModelV3` stream through +`packages/core/src/aisdk.ts`. + +This path requires two contracts: + +1. package loading and model construction from catalog settings; +2. conversion of AI SDK stream parts and `APICallError` values into canonical + `LLMEvent` and `AIError` values. + +The contract report must say which runtime path was exercised. Reporting only +the catalog provider and model would hide a native/fallback switch. + +## Provider Contract Harness + +The provider contract harness belongs primarily in `../opencode`, close to +`packages/ai` and `packages/core`. Drive consumes its summarized results in the +continuous-verification control plane; `packages/drive` must not absorb +OpenCode-specific provider inventories. + +Conceptual structure: + +```text +ProviderCase + route or package identity + model/settings/credential fixture + canonical LLMRequest + TransportScript + expected BehaviorFingerprint + | + v +real model construction / ModelResolver + | + v +real protocol or actual AI SDK package + | + v +programmable HttpClient / WebSocketConstructor + | + v +observed events, failure, timing, attempts, requests + | + v +canonical fingerprint + assertions +``` + +The first implementation should extend the existing test support rather than +introduce another framework. `packages/ai/test/lib/http.ts`, SSE helpers, +protocol-specific fixtures, `@opencode-ai/ai/testing`, and the Effect service +boundaries are already useful building blocks. + +## Programmable Transport + +A transport script is an ordered description of what the provider-facing code +will observe. It is lower-level than `Llm.Output` because its purpose is to +exercise framing, schemas, package behavior, and transport errors. + +Illustrative schema: + +```text +TransportScript + protocol + expected request matcher + attempts[] + +TransportAttempt + optional delay before headers + outcome: + HttpResponse + status + headers + body chunks[] + optional body failure after chunk N + RequestFailure + code + message + delivery phase + Hang + phase + WebSocketExchange + handshake outcome + expected client frames[] + server frames[] + close or failure +``` + +The implementation should provide an Effect `HttpClient.HttpClient` layer and, +where applicable, a scoped `Socket.WebSocketConstructor` layer. It should not +patch global `fetch` unless a specific external package offers no injectable +transport and that limitation is recorded. + +The harness records every attempted request before responding. It validates: + +- method, normalized URL, query, and selected safe headers; +- body against the provider-native schema or a stable semantic projection; +- attempt count and ordering; +- whether cancellation interrupted the response producer; +- whether every scripted response was consumed; +- whether an unexpected destination was attempted. + +An unexpected request fails loudly. Real network egress is denied in contract +tests. + +## Behavior Fingerprints + +We do not need to guess whether a package “throws.” We run a probe and capture +the complete observable shape. + +Each case produces a canonical behavior fingerprint: + +```text +BehaviorFingerprint + schemaVersion + targetRevision + lockfileDigest + packageIdentity + packageVersion + runtimePath: native-route | native-entrypoint | aisdk-fallback + protocol + requestDigest + transportScriptDigest + requests[] + events[] + terminal: + completed + failed + defected + interrupted + timed-out + failure?: + stage + class/tag + canonical reason + safe message pattern + status/code + retry metadata + cause summary + outputStarted + requestAttempts + retryDecision + sessionProjection?: safe summary +``` + +The event projection should retain event types, stable IDs normalized to +placeholders, finish reasons, usage presence, and provider-metadata keys. Large +text and opaque provider values are replaced with digests or safe summaries. + +Fingerprints serve three purposes: + +- assertions for behavior that is intentionally stable; +- reviewable diffs after package upgrades; +- input to differential tests between native and fallback implementations. + +A changed fingerprint is not automatically a regression. It is an explicit +compatibility review instead of an invisible behavior change. + +## Outcome Taxonomy + +Every probe must terminalize into exactly one harness outcome: + +| Outcome | Meaning | +| --- | --- | +| `completed` | A canonical terminal finish was consumed successfully | +| `failed` | The typed error channel produced an expected domain failure | +| `defected` | Code died, threw outside the declared error channel, or violated an invariant | +| `interrupted` | The test deliberately cancelled and all scoped resources closed | +| `timed-out` | The case did not terminalize inside its declared bound | +| `harness-failed` | Request matching, script consumption, or evidence capture was invalid | + +Never coerce defects, timeouts, or interruptions into a generic provider error. +Their difference is exactly what these tests need to reveal. + +## Canonical V2 Error Contract + +At the reviewed V2 snapshot, `@opencode-ai/ai` exposes these canonical reason +tags: + +- `InvalidRequest`, optionally classified as `context-overflow` or + `payload-too-large`; +- `NoRoute`; +- `Authentication`; +- `RateLimit`; +- `QuotaExceeded`; +- `ContentPolicy`; +- `ProviderInternal`; +- `Transport`; +- `InvalidProviderOutput`, optionally classified as `incomplete-stream`; +- `UnknownProvider`. + +`packages/core/src/session/to-session-error.ts` projects them to stable +session-facing types such as `provider.rate-limit`, `provider.auth`, +`provider.transport`, and `provider.invalid-output`. + +The runner's generic retry policy is also observable contract: + +- retry `RateLimit` and `ProviderInternal`; +- retry `Transport` only when delivery is absent or `not-sent`; +- retry an `InvalidProviderOutput` only for `incomplete-stream`; +- do not retry authentication, quota, policy, invalid request, no-route, or + unknown-provider failures; +- do not retry after visible output in the ordinary pre-output retry path; +- allow separate incomplete-stream continuation and context-overflow recovery + rules where the runner declares them. + +Provider/package probes assert the `AIError`. A smaller integration layer then +asserts the session projection, durable events, retry scheduling, and UI +recovery. Keeping these two assertions separate identifies whether a failure is +in parsing/classification or in session policy. + +## Failure Corpus + +The corpus should be systematic. Every applicable protocol/package receives +the shared core cases plus protocol-specific cases. + +### HTTP response cases + +- 200 with the smallest valid stream; +- 200 with an empty body; +- 200 with the wrong content type; +- 200 with whitespace or keepalive-only content; +- 204 and 304 responses; +- 400, 401, 403, 404, 408, 409, 413, 422, 429, 500, 502, 503, 504, and 529; +- a known structured provider error body; +- a plain-text error body; +- malformed JSON error body; +- empty structured message and code-only body; +- large body and truncation boundary; +- `Retry-After`, `retry-after-ms`, and provider rate-limit headers; +- request ID headers; +- a provider error disguised behind an unexpected status. + +The expectation is not that every status maps identically for every provider. +The expectation is that behavior is explicit and reviewed. + +### Transport cases + +- DNS/connect failure before request delivery; +- TLS or handshake failure; +- abort before headers; +- response-body read reset before any frame; +- response-body read reset after valid partial output; +- hang before headers; +- hang between frames; +- cancellation while waiting, reading, or decoding; +- middleware failure before and after request mutation; +- unexpected redirect or endpoint; +- connection close with delivery known, unknown, or accepted. + +### Framing and decoding cases + +- one event per chunk and many events per chunk; +- one event split across arbitrary byte chunks; +- UTF-8 code point split across chunks; +- comments, keepalives, blank lines, and provider-specific preambles; +- malformed JSON frame; +- valid JSON with missing required fields; +- unknown forward-compatible event; +- duplicate start, finish, or terminal event; +- data after terminal; +- clean EOF without a protocol terminal; +- provider error event within a 200 stream; +- inconsistent IDs or indices; +- negative, missing, fractional, or contradictory usage fields. + +### Tool cases + +- complete tool call in one event; +- arguments split at every byte boundary; +- empty arguments where the protocol treats them as `{}`; +- invalid JSON arguments; +- unknown tool name; +- duplicate tool-call ID; +- parallel calls with interleaved fragments; +- hosted/provider-executed call and result; +- hosted call with missing result; +- local tool call followed by an ordinary finish; +- finish reason inconsistent with emitted calls; +- interruption before arguments parse and during tool execution. + +### Request-lowering cases + +- empty and minimal messages; +- chronological system updates; +- text, image, PDF, and provider-supported media; +- cache hints and cache usage; +- portable generation options at limits; +- provider-specific options; +- raw HTTP overlays; +- tool-choice variants; +- malformed history that compatibility patches may accept; +- large context and payload boundaries; +- unknown provider-defined strings that should remain forward-compatible. + +### Model construction and package cases + +- missing, invalid, and empty credentials; +- endpoint derived from required variables; +- unresolved `${VARIABLE}` placeholders; +- base URL normalization and provider path selection; +- unknown package export; +- package module without the `model` contract; +- `model(...)` throwing synchronously; +- settings mapping with unsupported types; +- catalog variant overlay ordering; +- native mapping selected when expected; +- fallback selected when no mapping exists; +- a native/fallback selection change captured as a fingerprint diff. + +## Discovering Unknown Errors + +The corpus above captures known categories. Property and mutation probes find +cases we did not anticipate. + +The discovery loop is: + +1. Construct a valid provider-native request and valid response transcript. +2. Confirm it completes through the actual package. +3. Apply one controlled mutation. +4. Run with a strict deadline and real cancellation. +5. Capture the resulting fingerprint, including defects and hangs. +6. Shrink the mutation to the smallest reproducing transcript. +7. Classify it as acceptable, a product defect, an upstream package defect, or + an unsupported input. +8. Promote the minimized case into the permanent regression corpus. + +Useful mutations include: + +- delete one required field; +- replace a value with every JSON primitive type; +- duplicate, reorder, or omit an event; +- split bytes at a generated boundary; +- truncate at every frame boundary; +- replace a known enum with an unknown string; +- change one ID between start/delta/end; +- inject one provider error before or after output; +- cancel at every lifecycle checkpoint; +- vary status, code, message, and retry headers independently. + +Property generation should produce valid cases more often than invalid noise. +A protocol-specific generator knows the state machine and can deliberately +violate one rule at a time. Pure arbitrary JSON fuzzing is supplementary. + +## Shrinking + +A useful failure is a small failure. The shrinker should minimize in this +order: + +1. remove whole transport attempts; +2. remove frames not needed to reproduce; +3. remove unrelated fields; +4. shorten strings and arrays; +5. reduce chunk count and delay; +6. reduce the request history; +7. normalize generated IDs. + +The stored replay artifact contains the minimized case and the original seed. +Shrinking runs in an ephemeral process so a stuck or defective candidate does +not poison a persistent lane. + +## Native-versus-Fallback Differential Testing + +During the V2 migration, some provider identities can execute through native +`@opencode-ai/ai` code while others still use an AI SDK package. Differential +testing is the fastest way to find semantic gaps. + +For one canonical request and semantically equivalent wire transcripts: + +```text + canonical request + / \ + v v + native protocol AI SDK package + adapter + \ / + v v + normalized fingerprints + | + v + semantic comparison +``` + +Compare stable semantics, not implementation accidents: + +- request roles/content/tool schemas; +- event ordering and tool-call assembly; +- normalized finish reason; +- usage and cache usage; +- provider-executed tool markers; +- canonical error reason and retry metadata; +- cancellation and partial-output behavior. + +Allow explicit differences for provider-native metadata and features. Every +allowance has an owner, rationale, and expiry/review condition. A broad +“snapshots differ” waiver is not sufficient. + +Differential tests are particularly valuable before changing `ModelResolver` +or `AISDKNative.map`, because the same catalog model may silently move from one +runtime implementation to another. + +## Record/Replay and Live Provider Probes + +The target already includes `@opencode-ai/http-recorder` and many committed +provider recordings. Reuse them for high-realism success and tool-loop cases. + +Recordings provide: + +- real request shapes; +- real headers after redaction; +- real provider event variants; +- realistic optional fields and metadata; +- deterministic replay without cost or provider availability. + +They do not replace programmable negative cases. The recorder currently +buffers HTTP responses, so it cannot faithfully test streaming timing, +cancellation, or backpressure. WebSocket replay preserves frame chronology but +not transport timing and does not capture every handshake/failure dimension. + +A small live lane refreshes confidence that recordings still resemble current +providers: + +- allowlisted provider/model only; +- minimal prompts and token limits; +- strict daily budget; +- no arbitrary tool execution; +- credentials isolated from Drive artifacts; +- response/request IDs in restricted evidence; +- new recordings reviewed and redacted before commit; +- live failure does not automatically imply an OpenCode product regression. + +Live calls should refresh selected cassettes and emit drift reports. They should +not run every generated malformed-input case against a real provider. + +## Package Upgrade Gate + +Every change to the lockfile or provider mapping computes the impacted set: + +```text +changed package/version + | + +--> provider entrypoints importing it + +--> protocols/transports importing it + +--> ModelResolver mappings selecting it + +--> existing contract cases tagged for those paths +``` + +The gate runs: + +1. compile/type contract tests; +2. model construction and request-lowering tests; +3. shared error/fault corpus; +4. protocol-specific valid and malformed streams; +5. recorded provider cases; +6. native/fallback differential cases where applicable; +7. a small Drive end-to-end smoke if the canonical OpenAI Chat simulation path + or session projection changed. + +The report includes added, removed, and changed fingerprints. Reviewers approve +intentional changes in the same pull request. + +## Continuous Cadence + +Package contracts do not need a persistent TUI lane. They are mostly ephemeral, +parallel, and cheap. Their role in the 24/7 system is: + +| Cadence | Campaign | +| --- | --- | +| Every pull request | Impacted deterministic package contracts and core Drive smoke | +| On merge to `v2` | Full deterministic native protocol and resolver matrix | +| Hourly | Small rotating malformed/fault corpus against the deployed revision | +| Nightly | Broader property/mutation campaign with shrinking | +| Daily or budgeted | Selected live-provider drift probes | +| On dependency update | Full affected-package fingerprints and differential parity | +| Weekly | Refresh coverage inventory and find untested routes/mappings | + +The control plane treats these as a separate `provider-contract` lane kind. +They publish into the same run/evidence store but do not pretend to be user +journeys. + +## One Logical Bot Per Provider Contract + +Use one logical bot profile per independently meaningful provider contract. +This is close to “one bot per AI package/provider,” but the split follows the +runtime behavior under test rather than marketing brand alone. + +Split a bot when any of these differ: + +- package or package-like export path; +- native versus AI SDK fallback implementation; +- semantic protocol, such as Chat, Responses, Messages, Gemini, or Converse; +- transport, such as HTTP, WebSocket channel, or AWS event stream; +- auth/endpoint construction substantial enough to have an independent + contract; +- credential/network profile for live probes; +- owner or release decision. + +Illustrative V2 bot inventory: + +```text +provider.openai.chat.native +provider.openai.responses-http.native +provider.openai.responses-websocket.native +provider.openai-compatible.chat.native +provider.open-responses.native +provider.anthropic.messages.native +provider.anthropic-compatible.messages.native +provider.google.gemini.native +provider.google-vertex.gemini.native +provider.google-vertex.chat.native +provider.google-vertex.responses.native +provider.google-vertex.messages.native +provider.amazon-bedrock.converse.native +provider.amazon-bedrock.mantle-chat.native +provider.amazon-bedrock.mantle-responses.native +provider.azure.chat.native +provider.azure.responses.native +provider.openrouter.chat.native +provider.xai..native +provider..aisdk-fallback +provider.resolver.matrix +provider.error-projection.session +``` + +Generate the concrete list from the pinned target ref; this illustration is not +a hard-coded registry. + +Each provider bot runs: + +1. entrypoint/model construction; +2. request-lowering assertions; +3. smallest valid text stream; +4. tool-call and continuation cases when supported; +5. the applicable shared HTTP/transport error corpus; +6. protocol-specific malformed streams; +7. cancellation at declared phases; +8. recorded cases; +9. optional live drift sub-profile; +10. fingerprint comparison with its last reviewed baseline. + +Shared protocol code should not cause uncontrolled duplication. Use a layered +matrix: + +- the protocol bot runs the full framing/malformed-event corpus; +- every provider bot runs construction, endpoint/auth/options, a valid stream, + representative error, and its recordings; +- a resolver bot verifies catalog identity selects the intended runtime path; +- a session projection bot runs each canonical `AIError`/retry class through + the runner; +- provider-specific cases extend, rather than copy, the common corpus. + +These are logical scheduler identities. A bounded ephemeral worker pool can run +them in parallel and isolate each case in its own Effect scope/process. Give a +bot a dedicated long-running worker only for unique native dependencies, +credentials, network policy, or provider rate limits. + +Provider bot health includes: + +- last scheduled, started, completed, and successful attempt; +- pinned OpenCode commit and package version; +- deterministic contract outcome; +- current fingerprint review state; +- recording age; +- optional live-probe age and budget state; +- known gap/quarantine reason; +- worker/infrastructure blocking reason. + +One broken shared worker must make affected bots `stale` or `blocked`; it must +not leave their previous green state looking current. + +## Coverage Manifest + +The inventory and coverage report should be machine-readable. Suggested rows: + +```text +ProviderContractTarget + id + packageIdentity + resolvedPackageVersion + runtimePath + providerEntrypoint + protocol + transport: http | websocket | eventstream + modelResolverCases[] + validCases[] + errorCases[] + propertyCampaigns[] + recordings[] + liveProbe?: policy + owners[] + knownGaps[] +``` + +Generate the initial manifest from source exports, resolver mappings, and test +metadata; then require human ownership and gap rationale. The report should +flag: + +- a runtime path with no valid completion case; +- a protocol with no malformed-stream case; +- an entrypoint with no construction case; +- a retryable error with no session-runner integration case; +- a package version that changed without a fresh report; +- a mapping that changed from fallback to native without differential review; +- a committed recording no longer referenced by a test; +- a live-supported provider whose latest successful probe is too old. + +## Evidence and Redaction + +Contract evidence often contains provider-native requests, which are more +sensitive than ordinary pass/fail metrics. + +Persist by default: + +- digests and safe structural projections; +- package identity and version; +- status, safe header names, error tags, and finish reason; +- event type/order with normalized IDs; +- request count, timing, and cancellation points; +- minimized malformed payload when it contains generated fixture data only. + +Restrict or redact: + +- authorization and API key headers; +- provider query credentials; +- user prompt and model output content; +- tool arguments/results; +- complete provider error bodies that may echo request content; +- cloud account, project, region, deployment, and request identifiers where + policy requires it. + +The target executor intentionally retains detailed HTTP context in typed errors +for diagnosis. Artifact publication must therefore redact again at the +evidence boundary; typed errors are not automatically safe to publish. + +## Ownership + +Keep ownership aligned with the repositories: + +- native route, protocol, error-classification, resolver, AI SDK adapter, and + recorder tests live in `../opencode`; +- generic simulation-controller laws live in `packages/drive` here; +- OpenCode-specific coverage manifests, campaign policy, dashboards, and + review UI live in `apps/catalog` here; +- scripts may orchestrate both repositories and pin their exact revisions; +- no provider/package taxonomy is imported from `apps/catalog` into + `packages/drive`; +- no backend package-control convenience commands are added to the frontend + simulation CLI. + +## Initial Implementation Sequence + +### Phase 1: inventory and fingerprint + +1. Pin `../opencode:v2` by commit in a campaign manifest. +2. Generate native protocols, provider package entrypoints, resolver mappings, + fallback identities, and installed versions. +3. Define the fingerprint schema and safe normalization rules. +4. Wrap the existing `packages/ai` valid tests to emit fingerprints. +5. Publish a coverage report without adding new faults yet. + +### Phase 2: shared HTTP error corpus + +1. Extract a reusable programmable Effect HTTP transport from existing test + helpers where that reduces duplication. +2. Run the shared status/body/header matrix through native request execution. +3. Assert canonical `AIError` reasons and retry metadata. +4. Add runner integration cases for each retry class and partial-output state. +5. Add strict per-case timeouts and script-consumption checks. + +### Phase 3: protocol mutation + +1. Add state-aware transcript generators for OpenAI Chat, Responses, + Anthropic, Gemini, and Bedrock. +2. Mutate one protocol rule per case. +3. Capture and shrink defects, hangs, and unexpected error categories. +4. Promote minimized discoveries into permanent fixtures. +5. Feed results into the continuous control plane. + +### Phase 4: resolver and package parity + +1. Test every package-like entrypoint's construction/settings contract. +2. Test dynamic loader and AI SDK adapter outcomes. +3. Add native-versus-fallback comparisons for migration candidates. +4. Gate resolver mapping changes on reviewed differential reports. + +### Phase 5: recorded and live drift + +1. Index existing cassettes in the coverage manifest. +2. Fill high-risk recording gaps identified by V2 `STATUS.md`. +3. Add low-cost allowlisted live probes. +4. Automate age, drift, redaction, and budget alerts. + +## Acceptance Criteria + +This layer is ready when: + +- every runtime provider path says whether it is native, package-like, or AI + SDK fallback; +- tests execute actual installed package code instead of hand-written package + replicas; +- every supported protocol has a valid stream, malformed stream, transport + failure, HTTP failure, cancellation, and tool-call case; +- every canonical `AIError` reason has an intentional session projection and + retry expectation; +- synchronous throws, typed failures, defects, hangs, and interruptions remain + distinguishable in reports; +- generated failures shrink to replayable fixtures; +- package upgrades produce reviewable behavior-fingerprint diffs; +- resolver changes cannot silently switch runtime implementation; +- deterministic tests deny accidental real network access; +- recorded/live tests have budget, redaction, and ownership controls; +- Drive end-to-end coverage and provider/package contract coverage are reported + separately, then correlated in one dashboard. +- every independently meaningful provider/package/protocol path has a logical + bot with its own cadence, freshness, owner, and visible health; +- provider bots share ephemeral workers by default without sharing mutable case + state or credentials. diff --git a/docs/continuous-testing/05-scenarios-and-journeys.md b/docs/continuous-testing/05-scenarios-and-journeys.md new file mode 100644 index 0000000..8cec284 --- /dev/null +++ b/docs/continuous-testing/05-scenarios-and-journeys.md @@ -0,0 +1,520 @@ +# Scenarios and Journeys + +This document defines how existing executable catalog flows become monitored +synthetic journeys, how new journeys should be authored, and how capture, +reproduction, and 24/7 verification share one source of truth. + +## Principle: One Executable Journey Model + +The catalog already has an executable-flow model in +[`apps/catalog/catalog/flow.ts`](../../apps/catalog/catalog/flow.ts). A flow +defines: + +- a stable flow ID; +- title, group, and description; +- a non-empty ordered state list; +- metadata for each state; +- an Effect program that drives a real `Driver`; +- an ordered checkpoint callback. + +The adapter `executableScenario(...)` adds response mode and client-isolation +metadata. The registry in +[`apps/catalog/scenarios/index.ts`](../../apps/catalog/scenarios/index.ts) is +already authoritative for capture and state reproduction. + +Continuous verification must reuse that exact registry. Creating a second +“monitoring scenarios” directory with copied steps would cause IDs, waits, +fixtures, and expected behavior to drift. + +The same journey can serve three consumers: + +```text +ExecutableFlow + | + +--> catalog capture: checkpoint -> frame artifact + | + +--> reproduction: stop at selected checkpoint -> frame artifact + | + +--> monitoring: checkpoint -> timing + evidence + run event +``` + +The consumer supplies checkpoint behavior. The scenario owns user behavior and +assertions. + +## What Counts as a Journey + +A journey is a bounded, meaningful workflow with observable success criteria. + +Good examples: + +- submit a prompt and observe a completed assistant response; +- stream a patch call, approve permission, verify file and transcript state; +- reject a tool and verify recovery; +- answer a question form and verify the session projection; +- create a subagent, observe parent and child completion, open the child; +- restart the server and verify transcript rehydration; +- run concurrent tools and verify their settlement ordering. + +A journey is not: + +- a single low-level protocol call with no user outcome; +- an arbitrary sequence of keystrokes without state assertions; +- a screenshot script that only sleeps and captures pixels; +- an unbounded soak loop; +- a property campaign containing many generated attempts; +- a broad “test everything” script whose failure cannot identify a feature. + +## Checkpoints Are Assertions + +`executeFlow` verifies that checkpoints occur in the declared order and that +every declared checkpoint is reached. The scenario normally reaches a +checkpoint only after one or more `ui.waitFor`, SDK, filesystem, or lifecycle +assertions. + +For monitoring, a checkpoint means: + +- the journey reached a named observable state; +- all preceding scenario assertions passed; +- elapsed time can be attributed to a meaningful product phase; +- optional evidence can be captured without duplicating journey logic. + +Do not add checkpoints for every keystroke. Add them for states a developer or +operator would recognize, such as “permission visible,” “tool running,” +“assistant settled,” or “composer actionable again.” + +## Assertion Layers + +A strong journey uses more than one observation surface where useful. + +### UI assertion + +Examples: + +- visible transcript marker; +- semantic UI element state; +- composer focus/actionability; +- permission or form visibility; +- absence of an internal error string. + +Prefer semantic UI state and stable node identity when the canonical protocol +provides it. Text markers remain useful but can drift with copy changes. + +### SDK/server assertion + +Examples: + +- session exists and has expected parent relationship; +- prompt and assistant parts are present in the server projection; +- no pending form or permission remains after terminal state; +- shell or tool invocation has expected status; +- a queued prompt has one owner. + +### Filesystem assertion + +Examples: + +- declared file changed as expected; +- rejected operation did not mutate the fixture; +- Git worktree is clean or has the expected diff; +- output file stays inside the isolated project root. + +### Lifecycle assertion + +Examples: + +- response reached terminal state within a bound; +- TUI remains alive after interruption; +- server generation changed after restart; +- reconnect restored an actionable client; +- no unsettled simulated work remains at attempt cleanup. + +Use the narrowest sufficient set. Duplicating the same assertion across every +surface adds fragility without diagnostic value. + +## Journey Categories + +### Smoke journeys + +Characteristics: + +- under a minute in healthy conditions; +- deterministic mock output; +- very small fixture; +- no intentional disruptive failure; +- validates the essential prompt-to-response path; +- runs frequently and powers freshness alerts. + +A smoke journey must be reliable enough that one failure is informative, while +alert policy may still require consecutive failures before paging. + +### Critical feature journeys + +Characteristics: + +- permission, tool, form, session, and subagent workflows; +- deterministic response plans; +- explicit end-state cleanup assertions; +- moderate cadence. + +Most existing catalog lifecycle flows fit here. + +### Recovery journeys + +Characteristics: + +- include interruption, provider disconnect, server restart, or rejection; +- assert recovery and post-failure reuse; +- run in a compatible reactive or chaos lane; +- retain more evidence than ordinary success runs. + +### Diagnostic probes + +Characteristics: + +- target a known race or issue; +- may need many attempts; +- preserve seed and issue-specific evidence; +- may be excluded from release gates until stabilized; +- should migrate from `test/manual` when they become continuously valuable. + +### Visual catalog flows + +Some flows exist primarily to capture a design state. They are not +automatically good 24/7 health journeys. Monitoring eligibility should be +explicit so visual-only states do not consume operational capacity or page on +copy-only differences. + +## Operational Metadata + +The existing `ExecutableScenario` keeps source-of-truth identity and execution +metadata. Continuous verification needs additional app-owned policy without +polluting the generic flow model. + +Proposed monitoring metadata: + +```text +scenarioId +enabled +tier: smoke | critical | extended | diagnostic +cadence policy reference +timeout +eligible lane kinds +required protocol capabilities +required tool controls +database policy +reuse policy +evidence policy +alert policy +estimated healthy duration +owner/team +known issue or quarantine reference +``` + +Store this as a map keyed by registered scenario ID or as a typed wrapper built +from the registry. Type-level checks should prevent metadata for unknown IDs and +should identify required registered scenarios with missing policy. + +Flow identity, taxonomy, and state metadata remain in the catalog definitions. +Operational policy does not move into `packages/drive`. + +## Scenario Lifecycle Contract + +Every monitored scenario has four phases. + +### Preconditions + +Declare and verify: + +- eligible inference mode; +- required tools and permissions; +- Git fixture requirement; +- database persistence requirement; +- viewport/theme sensitivity; +- server and client reuse policy; +- required simulation capabilities. + +The scheduler uses static preconditions for lane matching. The runner verifies +dynamic preconditions immediately before execution. + +### Preparation + +Preparation belongs to the monitoring adapter when it is common across +journeys: + +- reset fixture files; +- launch a fresh TUI; +- create a fresh session; +- verify actionable composer; +- set attempt correlation context. + +Scenario-specific preparation remains inside the scenario: + +- create special SDK state; +- configure a specific controlled tool response; +- open a feature-specific starting screen. + +### Execution + +The scenario queues or serves model output, performs actions, waits for +observable conditions, and reaches checkpoints. + +Every wait has a meaningful deadline. Use `ui.waitFor` rather than large +unconditional sleeps. A short sleep can be appropriate to capture an intended +mid-stream visual state, but it must not replace a completion condition. + +### Postconditions + +Before returning success, verify the state that matters beyond the final +rendered string: + +- model response terminalized; +- tool/form/permission state settled; +- expected files or session records exist; +- no unintended pending work remains; +- composer is actionable when that is part of the contract. + +Drive settlement provides a final guard for queued model and tool work at the +end of a finite driver run. A persistent lane also needs per-attempt +postconditions because lane settlement occurs much later. + +## Fresh Session Preparation + +The current catalog runner opens a new session through the TUI command palette. +That is valuable end-to-end coverage. The monitoring adapter should preserve it +for user-facing journeys. + +Preparation sequence: + +1. wait for at least one semantic UI element or known home state; +2. open the command palette; +3. select `New session` through stable UI control; +4. wait for an actionable composer; +5. record the selected session ID from the SDK when available; +6. verify it differs from the previous attempt's session unless reuse is + declared. + +An SDK-created session may be faster for specialized non-UI setup, but it does +not replace the smoke coverage of creating and navigating sessions through the +real client. + +## Fixture Reset + +Fixture reset must be deterministic and explicit. + +Recommended design: + +- define the canonical fixture contents in one module; +- compute a fixture digest stored in every attempt; +- restore only declared scenario-owned paths; +- remove a declared set of transient paths; +- verify reset result before executing; +- never touch `.opencode`, the lane database, logs, or artifact directories + unless the lane policy explicitly requests it; +- fail preparation if a path escapes the project root after normalization. + +A scenario that needs a materially different fixture should declare a fixture +profile. Avoid incrementally mutating a shared fixture until it happens to be +usable. + +## Stable Markers + +Use markers that represent behavior rather than incidental prose. + +Preferred order: + +1. canonical semantic element or node identity; +2. stable role/label/state from the simulation protocol; +3. deliberately authored deterministic mock text; +4. concise product copy that is itself under test; +5. timing-only capture as a last resort for a mid-stream visual state. + +When product copy changes legitimately, update exact markers in the scenario. +Do not replace them with long unconditional sleeps or overly broad substring +matches that could pass on the wrong screen. + +## Scenario IDs and Versioning + +Flow and state addresses are user-facing reproduction identities. Preserve +them when behavior remains semantically the same. + +Change an ID when: + +- the scenario now represents a different user outcome; +- the checkpoint meaning changed incompatibly; +- replaying an old address against new code would be misleading. + +Do not change an ID merely because implementation or copy changed. + +Every attempt additionally records the source revision of the catalog app and +a scenario-definition digest. This lets an old failure point to the exact +journey implementation even when the stable ID remains. + +## Captures in Monitoring + +The authoritative catalog artifact is a normalized terminal frame. Monitoring +uses the same `ui.capture` output. + +Evidence policy controls frequency: + +- smoke success: final frame sampled or omitted; +- ordinary success: metadata and checkpoint timings only; +- property success: no frames unless sampled; +- any failure: current frame, recent event evidence, and logs; +- visual regression campaign: frame at every declared checkpoint; +- recording: failure-focused or sampled because encoding every success is + expensive. + +PNG remains a derived artifact. Do not change the canonical OpenCode protocol +or endpoint contract to support monitoring storage. + +## Reproduction + +The existing catalog supports replay through a canonical +`/` address. Continuous verification extends the +reproduction specification with: + +- source attempt ID; +- exact OpenCode and scenario revisions; +- lane configuration version; +- response plan digest or recorded reactive trace; +- seed and action trace for generated runs; +- fixture digest and optional redacted snapshot; +- requested terminal checkpoint or full journey; +- output artifact destination. + +Reproduction produces a new linked attempt. It may stop at a selected +checkpoint for visual diagnosis or run the full journey for outcome comparison. + +## Scenario Failure Classification + +Examples: + +| Failure | Classification guidance | +| --- | --- | +| `ui.waitFor` timed out and server projection also lacks state | Product failure likely | +| Text marker changed but semantic node and behavior are correct | Harness assertion drift | +| Checkpoints arrived out of declared order | Harness or product lifecycle regression; inspect evidence | +| Flow completed without a declared checkpoint | Harness failure unless product skipped an expected state | +| Unused queued response remains | Harness plan mismatch or product stopped requesting; inspect request history | +| TUI process exited during a valid action | Product failure unless host evidence indicates infrastructure | +| Frame capture failed after all postconditions passed | Evidence/harness failure; product outcome may remain passed with degraded evidence policy | + +Automatic classification should remain conservative. Ambiguous cases are +`inconclusive` and enter triage; they do not become passes. + +## Authoring Checklist + +Before registering a monitored journey, verify: + +- The journey tests a named user outcome. +- Its flow ID and checkpoint addresses are stable and descriptive. +- It declares the correct queue/serve mode. +- It declares client isolation and fixture requirements. +- Every state is reached only after a meaningful observation. +- Every wait has a bounded timeout. +- Deterministic text is authored by the response plan, not hoped for from a + real provider. +- Tool calls use offered names and schema-valid inputs. +- Cleanup/postconditions prevent pending work from leaking into the next + attempt. +- Failure evidence will identify the active phase and session. +- The scenario succeeds repeatedly against the baseline revision. +- A known failure can be reproduced by address, revision, and plan/seed. +- The operational cadence and alert tier match its reliability. + +## Review Policy + +Scenario changes should be reviewed like product code because they define the +monitoring oracle. + +A pull request changing a monitored scenario should explain: + +- which product behavior changed; +- whether IDs or checkpoint meanings changed; +- whether lane requirements or timeout changed; +- whether baseline history remains comparable; +- how the response plan changed; +- how the scenario was repeated to check flakiness; +- whether generated artifacts were regenerated where required. + +Do not weaken an assertion solely to make a candidate revision green. Either +fix the product, update a legitimately changed contract, or quarantine the +journey with an owner, reason, and expiration. + +## Quarantine + +Quarantine is a visible operational state, not deletion. + +A quarantine record includes: + +- scenario ID; +- affected revisions or environments; +- reason and issue link; +- owner; +- start time and expiration; +- reduced cadence or alert behavior; +- evidence from the last unquarantined failure. + +Quarantined journeys still run at a reduced cadence when safe so recovery is +detected. Their failures do not page according to the normal policy, but they +remain on dashboards and release reports. + +## Migrating Manual Probes + +The manual TUI regression directory contains high-value candidates. Migration +steps: + +1. Identify the user-visible invariant and required lane type. +2. Extract reusable fixture and assertion helpers into app-owned scenario code + when they are OpenCode-specific. +3. Define a stable flow or campaign ID. +4. Replace unrecorded environment variables with decoded campaign config. +5. Add deterministic response and fault plans. +6. Add failure evidence and replay metadata. +7. Characterize the probe against a known baseline over many attempts. +8. Register it with diagnostic cadence first. +9. Promote it to critical policy after its harness false-positive rate is + acceptable. + +The seeded lifecycle probe is better represented as a property campaign than a +single deterministic flow; see +[Stateful property testing](./06-stateful-property-testing.md). + +## Proposed Directory Shape + +```text +apps/catalog/ + catalog/ + flow.ts existing executable flow contract + scenarios/ + index.ts existing authoritative registry + ... existing OpenCode journeys + continuous/ + policy.ts monitoring policy keyed by scenario ID + fixture.ts reset profiles and digests + run-scenario.ts monitoring adapter and checkpoint wrapper + registry.ts validated registry + policy projection + errors.ts schema-backed app errors + scripts/ + continuous-runner.ts application entrypoint +``` + +Names may change during implementation, but ownership should not: OpenCode +journeys and their policy remain under `apps/catalog`; the published package +must not import them. + +## Acceptance Criteria + +Scenario integration is ready when: + +- capture, reproduction, and monitoring use the same registered flow program; +- monitoring records every ordered checkpoint without changing scenario code; +- one scenario can run in a fresh attempt TUI against a persistent server; +- fixture reset and postconditions prevent ordinary cross-attempt leakage; +- queue/serve and client-isolation requirements drive lane selection; +- every monitored journey has owner, timeout, cadence, and alert metadata; +- text-marker drift is distinguishable from server-state failure; +- failures can be reproduced with the same flow address and exact revision; +- quarantines are visible and expire; +- no OpenCode-specific flow ID or monitoring taxonomy is added to + `packages/drive`. diff --git a/docs/continuous-testing/06-stateful-property-testing.md b/docs/continuous-testing/06-stateful-property-testing.md new file mode 100644 index 0000000..12bcf26 --- /dev/null +++ b/docs/continuous-testing/06-stateful-property-testing.md @@ -0,0 +1,653 @@ +# Stateful Property Testing + +This document defines the generated testing layer for OpenCode's long-lived and +concurrent behavior. It explains what “property testing” means in this project, +which parts are useful, and how failures remain understandable and replayable. + +## Terminology + +Several related techniques are often grouped under property testing: + +**Property-based testing** +: Generate many inputs and assert a rule that should hold for all of them. + +**Stateful property testing** +: Generate commands whose validity and expected result depend on the current + abstract state. + +**Model-based testing** +: Maintain a small reference model and compare the real system after each + command with the model's predicted state. + +**Metamorphic testing** +: Apply a transformation that should preserve or predictably change the result, + even when the exact output is not known in advance. + +**Fuzzing** +: Explore large or malformed input spaces, usually with weaker semantic + knowledge. Stateful property testing uses more domain knowledge than generic + fuzzing. + +For OpenCode's TUI and session lifecycle, the primary method is **stateful +model-based property testing**. Fixed catalog journeys remain the first line of +defense; generated campaigns explore the ordering space between them. + +## Why This System Needs a State Model + +An arbitrary keystroke fuzzer will spend most of its time producing irrelevant +or invalid interaction. The interesting bugs occur at valid boundaries: + +- a second prompt arrives while the first response is streaming; +- interruption occurs between tool-input fragments; +- the server restarts after durable admission but before visible delivery; +- a controlled tool completes while the provider stream fails; +- a TUI reconnects while a permission or form is pending; +- a queued input is promoted after completion, interruption, or recovery; +- a session is opened from another client during an active execution. + +A model knows when those actions are legal and what must remain true afterward. +This concentrates generated work on meaningful lifecycle interleavings. + +## Existing Seed + +[`packages/drive/test/manual/tui-regressions/lifecycle-properties.ts`](../../packages/drive/test/manual/tui-regressions/lifecycle-properties.ts) +already demonstrates the approach. It has: + +- a controlled random seed; +- an abstract lifecycle state; +- preconditioned actions; +- separate transitions for submit, queued submit, reasoning, text, tool input, + tool execution, completion, interruption, and provider disconnect; +- UI and server invariants; +- a failure file containing seed, trace, state, events, and terminal frame. + +The continuous system should first make this campaign operable and shrinkable, +then generalize its reusable pieces. It should not replace it with an unrelated +property framework merely to gain terminology. + +## Test Architecture + +```text + generated command + | + +------------+-------------+ + | | + v v + reference model predicts real command executes + legal transition through Drive/OpenCode + | | + +------------+-------------+ + v + observation snapshot + | + invariants + model comparison + | + trace item persisted +``` + +The reference model is intentionally smaller than OpenCode. If it reproduces +the complete product implementation, it will reproduce the same bugs and be +too expensive to maintain. + +## Campaign, Case, Step, and Trace + +**Campaign** +: One configured exploration run: target revision, model version, seed range, + step budget, action weights, and lane policy. + +**Case** +: One generated initial state and command sequence. A case has one root seed. + +**Step** +: One chosen command, its pre-state, execution, observation, invariant results, + and post-state. + +**Trace** +: The ordered replay artifact for a case. It records actual choices rather than + assuming the random generator can be reconstructed forever. + +A 24/7 bot schedules bounded campaigns. It never runs one unbounded property +loop whose partial work disappears when the process stops. + +## Reference State + +Start with a lifecycle model that tracks only properties needed to select +commands and assert ownership. + +Illustrative model: + +```text +ModelState + generation + server + tui + controller + session + id? + execution: idle | pending | streaming | terminal | unknown + activePrompt? + queuedPrompts[] + visiblePrompts[] + visibleAssistantParts[] + activeToolCalls[] + settledToolCalls[] + pendingPermission? + pendingForm? + inference + requestId? + phase: none | opened | reasoning | text | tool-input | finished | disconnected + outputStarted + fixture + expectedFiles + budgets + commandsRemaining + modelStepsRemaining + restartsRemaining + faultsRemaining +``` + +Use explicit `unknown` or observation-unavailable states when the public +surface cannot determine a fact. Do not invent certainty to make the model +easier. + +The persisted model is schema-versioned. A trace records the model version that +interpreted it. + +## Observed State + +After each command, collect the smallest useful snapshot from independent +surfaces: + +- TUI semantic tree or normalized frame; +- current session ID and selected client state; +- server session/message projection; +- pending inbox, permission, form, and tool state when exposed; +- Drive LLM pending request summary; +- controlled tool invocation summary; +- server/TUI/controller generation IDs; +- fixture digest and selected file facts; +- recent correlated OpenCode events; +- process liveness. + +Observation is bounded by a deadline. An observation timeout is a real case +outcome; the model must not silently use stale prior state. + +Not every invariant needs every surface. The snapshot collector can lazily +obtain expensive evidence only when the current command or a failed invariant +requires it. + +## Command Contract + +Every generated command defines: + +```text +Command + id and schema version + parameters + precondition(ModelState) -> boolean + expected transition(ModelState) -> ModelState or allowed states + execute(Driver, parameters) -> Effect + observe requirements + postcondition(before, observation, after) + quiescence policy + timeout + destructive/fault budget cost + shrink(parameters) +``` + +Command selection evaluates preconditions first. A generator that repeatedly +chooses illegal commands is a generator-quality defect, not product coverage. + +The expected transition may be a set when external scheduling makes more than +one state legitimate. That set must remain narrow and explainable; “anything +can happen” is not a model. + +## Initial Command Set + +### Session and prompt commands + +- create a new session through the TUI; +- submit a prompt while idle; +- submit or steer a second prompt while work is active; +- change or cancel a queued input where public behavior supports it; +- open another existing session; +- return to the active session; +- start a second TUI client in a dedicated multi-client campaign. + +### Inference progress commands + +- release one reasoning fragment; +- release one text fragment; +- begin tool input; +- release one tool-input fragment; +- finish a tool call; +- finish the provider exchange; +- disconnect before output; +- disconnect after output; +- leave the stream paused while another action occurs. + +The response plan must expose deterministic gates. Sleeping for a random period +does not establish which lifecycle phase was reached. + +### Tool and form commands + +- report controlled tool progress; +- complete a controlled tool successfully; +- fail a controlled tool; +- answer a question form; +- reject or approve permission when a scenario intentionally configures it; +- interrupt while a tool or form is pending. + +### Lifecycle commands + +- interrupt the active session; +- restart the OpenCode server; +- restart the TUI; +- detach and reattach the Drive controller; +- wait for a declared stable boundary; +- reconcile/reobserve after an ambiguous transport outcome. + +### Fixture commands + +- read a declared file through the UI/tool path; +- apply a controlled edit; +- verify Git/file digest; +- restore the fixture at a legal campaign boundary. + +Never generate destructive shell commands or arbitrary paths. Generated file +operations are chosen from a fixture-owned allowlist. + +## Preconditions + +Examples: + +| Command | Preconditions | +| --- | --- | +| Submit idle prompt | TUI actionable, session selected, no active inference | +| Queue second prompt | Active execution, queue budget available | +| Emit text delta | Matching Drive request open, no terminal event | +| Start tool input | Request offered at least one controlled tool, no terminal | +| Complete tool | Matching invocation active and controller attached | +| Interrupt | Session is known and execution may be active | +| Restart server | Restart budget available, no lane-wide maintenance lock | +| Answer form | Exactly one matching pending form is observable | + +Preconditions are based on the model plus recent observation. When observation +contradicts the model, fail the invariant before choosing another command. + +## Core Invariants + +### Prompt ownership + +Every synthetic prompt has exactly one logical owner: + +- admitted and pending; +- delivered into visible/projected history; +- cancelled according to product semantics; +- or rejected with an explicit terminal error. + +It must not be lost, duplicated, or simultaneously counted as pending and +delivered when those representations are intended to be exclusive. + +### Monotonic durable history + +Once a durable user or assistant fact is observed in the authoritative session +projection, later observations do not erase it unless a declared product +operation such as revert changes the visible history according to its contract. + +Client rendering may temporarily lag. The invariant uses bounded eventual +visibility rather than requiring every surface to update in one instant. + +### Exactly one terminal outcome per logical step + +A started logical step eventually has one terminal outcome: ended or failed. +Retries may create physical attempts without consuming a new logical step as +defined by V2 session semantics. + +### Tool settlement + +Every locally called tool eventually reaches exactly one terminal state. +Interruption, rejection, provider failure, and server recovery must not leave a +tool permanently `streaming` or `running`. + +### No output after terminal + +No provider or tool output is accepted after its terminal event. If the test +deliberately sends late output, rejection itself is the expected behavior and +the session remains usable. + +### Actionable recovery + +After a bounded terminal or recovery sequence, either: + +- the composer becomes actionable and another prompt can be accepted; or +- the UI presents a stable, user-actionable error/recovery state declared by + the scenario. + +An endless spinner or inert composer violates the invariant. + +### One active execution owner + +Within the current single-process V2 execution model, concurrent resumes for +the same session coalesce or join according to the product contract. They do +not create conflicting simultaneous model executions. + +### Cross-session independence + +Work in one session must not consume another session's queued Drive response, +tool completion, permission answer, or UI selection. Test this only in a served +handler or otherwise explicitly routed lane; ordinary queued mode intentionally +cannot safely support unrelated concurrent requests. + +### Resource settlement + +After each case: + +- no case-owned request remains pending; +- no case-owned tool remains active; +- no case-owned child process or TUI remains alive; +- no response queue entry remains unused; +- all scoped recorders and files are closed. + +A cleanup failure fails the case even if earlier behavior passed. + +## Temporal Properties + +Many important properties include time but should not be encoded as arbitrary +sleeps. + +Examples: + +- after submit, a request opens within the request-start deadline; +- after releasing a text chunk, it becomes visible within the projection/UI + deadline; +- after interruption, the request disappears from `llm.pending` within the + settlement deadline; +- after server restart, the TUI reconnects or shows an actionable failure + within the recovery deadline; +- after terminal execution, no active tool remains beyond the cleanup deadline. + +Record the actual duration. A timeout fails with the expected transition, +current observation, and recent event trace. + +## Quiescence + +Some assertions require a stable boundary; others intentionally inspect +mid-flight state. + +Each command declares one policy: + +- `immediate`: observe directly after the command; +- `condition`: wait for a named semantic condition; +- `eventual`: repeatedly observe until the invariant holds or the deadline + expires; +- `terminal`: wait for session execution to settle; +- `none`: the next command intentionally races the current work. + +Global “wait until everything is idle” logic would erase the interleavings the +campaign exists to test. + +## Generation Strategy + +Use weighted state-dependent selection. + +Example initial weights: + +```text +idle: + submit prompt 60 + open/new session 20 + restart server 5 + restart TUI 5 + inspect stable state 10 + +streaming: + emit next fragment 35 + queue/steer prompt 20 + interrupt 15 + disconnect 10 + restart server 5 + inspect mid-flight 15 +``` + +Weights are campaign configuration and recorded with the case. Coverage data +should influence later tuning, but production failures must not silently mutate +the generator during a replay. + +Favor useful traces: + +- ensure most cases reach at least one complete prompt/response; +- reserve a bounded percentage for early failure paths; +- cap repeated no-progress actions; +- bias toward lifecycle boundaries not recently covered; +- use separate campaigns for multi-client, restarts, and destructive faults so + ordinary lifecycle exploration remains productive. + +## Randomness and Replay + +A seed is necessary but insufficient when the system contains uncontrolled +randomness or version-dependent generators. + +Persist: + +- root seed; +- generator and model version; +- action-weight configuration digest; +- every selected command and parameter; +- actual inference chunk plan; +- target and Drive revisions; +- fixture digest; +- timing/fault choices; +- any observed nondeterministic branch selected by the product. + +Replay consumes the recorded trace directly. It does not regenerate commands +from the seed unless validating the generator itself. + +The current Drive text chunking uses `Math.random`, which is not controlled by +an Effect seeded random service. Generated campaigns must use explicit chunk +plans or record emitted chunks before claiming deterministic replay. + +## Shrinking Stateful Failures + +Naively removing commands can make later commands illegal. Stateful shrinking +must replay candidates through the reference model. + +Shrink order: + +1. remove suffix after the first failed invariant; +2. remove whole command ranges while preserving preconditions; +3. remove independent session detours; +4. reduce restart/fault count; +5. shrink prompt text to stable markers; +6. shrink tool input structures; +7. merge or remove inference fragments; +8. reduce timing delays toward boundary values; +9. normalize IDs and fixture data. + +Every shrink candidate runs in a fresh ephemeral environment. Persistent lane +state is useful for finding the failure but is not a safe substrate for repeated +candidate replays. + +Stop shrinking when: + +- the time budget expires; +- a stable minimum is reached; +- the failure becomes non-reproducible; +- infrastructure prevents safe replay. + +Retain the smallest reproducing trace, the original trace, and shrink history. + +## Metamorphic Properties + +Metamorphic cases broaden coverage without requiring exact model prose. + +Examples: + +- splitting one valid text delta into more chunks preserves final text and + terminal state; +- combining adjacent text deltas preserves final text; +- changing harmless JSON object key order in tool arguments preserves parsed + input; +- reopening the same completed session from a fresh TUI preserves durable + transcript facts; +- repeating a read-only navigation sequence preserves server state; +- restarting a client after durable completion preserves the same session + projection; +- using a queue versus an equivalent reactive response plan preserves the + user-visible terminal outcome; +- replaying a recorded provider cassette produces the same canonical event + fingerprint after normalization. + +Only declare a metamorphic relation after confirming it is a product contract. +Provider-specific chunk or metadata differences may be intentionally visible. + +## Concurrency Campaigns + +Concurrency needs explicit ownership and stronger routing. + +Use separate campaigns for: + +- two sessions executing concurrently; +- two TUI clients observing one session; +- parallel tool calls within one model step; +- controller detach/reattach with pending work; +- server restart while client and controller reconnect; +- queued steering arriving at a safe step boundary. + +These campaigns use served inference with request-aware correlation. They must +not share an ordinal response queue across unrelated sessions. + +Concurrency assertions focus on causal and per-source order, not a single total +order where the product allows legitimate interleavings. + +## Coverage + +Track semantic transition coverage rather than raw command count: + +- state entered; +- `(state, command)` pair; +- `(state, command, outcome)` triple; +- adjacent command pair; +- fault injection phase; +- recovery transition; +- invariant evaluated; +- tool/provider terminal combination; +- server/TUI/controller generation change; +- partial-output status at failure. + +Coverage dimensions use bounded enumerations. Seeds, session IDs, prompts, and +request IDs do not become metric labels. + +A campaign report identifies unreachable or never-selected transitions. That +may reveal bad weights, impossible preconditions, missing instrumentation, or +dead product behavior. + +## Failure Classification + +| Observation | Classification guidance | +| --- | --- | +| Model transition was wrong but product behavior matches documented contract | Harness model defect | +| Generated command violated its own precondition | Generator/harness defect | +| Product lost or duplicated a prompt | Product failure | +| Observation endpoint timed out while process health is good | Product or observability failure; preserve as inconclusive until triage | +| Replay cannot reproduce because chunk randomness was unrecorded | Harness reproducibility failure | +| Persistent lane fails but clean replay passes | Product state-leak candidate, not a pass | +| Case exceeds host resource budget | Infrastructure or product leak depending attribution evidence | +| Shrinker cannot reproduce original failure | Original remains failed; shrink result is supplemental | + +## Running Continuously + +Use two lane types: + +### Persistent discovery lane + +- retains server/database state across cases; +- runs bounded cases repeatedly; +- uses a fresh session and TUI by default; +- occasionally runs declared reuse cases; +- records lane age and cumulative counts; +- recycles on policy, never silently after a suspicious failure. + +### Ephemeral replay/shrink lane + +- starts from the exact target revision and fixture; +- replays a recorded trace; +- retries only as linked diagnostic attempts; +- performs shrinking in isolated cases; +- never overwrites the discovery evidence. + +If a failure depends on accumulated persistent state, preserve or snapshot the +lane data according to security policy before recycling it. + +## Effect Structure + +The implementation should expose focused services such as: + +- `CampaignGenerator` for seeded command choice; +- `ReferenceModel` for valid transitions; +- `Observation` for bounded product snapshots; +- `InvariantEvaluator` for typed results; +- `TraceStore` for append-before-execute records; +- `CaseRunner` for scoped command execution; +- `Shrinker` for ephemeral candidate search. + +Campaign configuration and trace values use Effect Schema. Each case and each +shrink candidate runs in its own scope. Commands use named `Effect.fn` +boundaries so spans identify the command and phase without logging prompt +content. + +Expected command, observation, and invariant failures remain typed. Defects and +interruptions retain their native meaning. + +## Test the Tester + +The property harness needs its own deterministic tests: + +- generated commands always satisfy their preconditions; +- budgets make every generated case finite; +- the same explicit trace produces the same model transitions; +- trace encoding/decoding round-trips; +- append-before-execute survives interruption; +- invalid traces fail with a typed replay error; +- shrinking never emits a trace invalid under the reference model; +- each known seeded fixture triggers its expected invariant; +- fake observations verify every invariant's positive and negative cases; +- cancellation closes scopes and marks the case interrupted. + +Use small property tests for pure model and trace laws, plus selected live Drive +integration cases for the execution boundary. + +## Initial Campaigns + +Start with three narrow campaigns: + +1. **Prompt lifecycle**: idle submit, queued submit, reasoning/text progress, + completion, interruption, and disconnect. +2. **Tool lifecycle**: streamed input, valid and invalid arguments, controlled + tool progress, completion, failure, and interruption. +3. **Restart recovery**: durable prompt admission, server restart at selected + boundaries, transcript rehydration, and post-recovery prompt. + +Do not start by combining every command. Each campaign should first produce +stable, replayable failures and useful transition coverage. + +## Acceptance Criteria + +Stateful property testing is operational when: + +- every case is bounded by command, time, inference, tool, and fault budgets; +- commands are selected only when their preconditions hold; +- core prompt, history, terminal, tool, recovery, and resource invariants run + after relevant transitions; +- every choice and actual inference chunk is replayable from the stored trace; +- failures preserve model state, observation, recent events, logs, and frame; +- shrink candidates run in isolated environments and retain the original + failure evidence; +- persistent discovery and ephemeral replay results are linked but never + conflated; +- generated action coverage is visible without high-cardinality metrics; +- a failing property identifies one command and one invariant, not merely a + random seed; +- fixed catalog journeys remain simple and are not replaced by generated + campaigns. + diff --git a/docs/continuous-testing/07-soak-and-chaos-testing.md b/docs/continuous-testing/07-soak-and-chaos-testing.md new file mode 100644 index 0000000..4982309 --- /dev/null +++ b/docs/continuous-testing/07-soak-and-chaos-testing.md @@ -0,0 +1,569 @@ +# Soak and Chaos Testing + +This document defines long-running workload tests and controlled failure +experiments for the always-on OpenCode environment. + +Soak testing and chaos testing share infrastructure, but they answer different +questions: + +- **Soak testing** asks whether normal use remains healthy after hours, days, + and accumulated state. +- **Chaos testing** asks whether the system preserves declared safety and + recovery properties when one known fault occurs at a known lifecycle phase. + +Neither technique means “run random destructive commands against a machine.” + +## Objectives + +The combined program should reveal: + +- memory, handle, file descriptor, process, and storage growth; +- performance degradation with session/history/database age; +- stale client, simulation controller, and server state; +- failure to release requests, tools, forms, permissions, or execution claims; +- restart and reconnection defects; +- incorrect retry or continuation after partial provider output; +- data loss or duplication across interruption boundaries; +- correlated failures that ordinary isolated tests never encounter; +- an environment that is alive but no longer completing useful work. + +## Principles + +### State the hypothesis first + +Every experiment declares the failure being injected and the property expected +to survive. “Kill things and see what happens” produces ambiguous evidence and +unsafe automation. + +### One primary fault per experiment + +Start with one controlled fault. Combining faults is valuable later, after each +component failure is understood independently. + +### Preserve the first unexpected state + +Do not immediately restart and erase evidence. Freeze the affected lane, collect +bounded artifacts, then reproduce in an ephemeral environment. + +### Bound the blast radius + +Every fault has an explicit target, duration, budget, and cleanup check. The +injection mechanism must be incapable of selecting the host, workspace root, +unrelated container, or real user process by an unresolved glob or broad +environment variable. + +### Keep control plane independent + +The scheduler, heartbeat evaluator, artifact writer, and alert path must not run +inside the only OpenCode process they monitor. A killed server must remain +observable. + +## Soak Workload Model + +A soak is not one huge scenario. It is a sequence of bounded attempts against a +persistent lane, with periodic health samples and explicit lane-age evidence. + +```text +lane starts + | + +--> attempt 1 --> health sample + +--> attempt 2 --> health sample + +--> ... + +--> maintenance checkpoint + +--> ... + +--> declared recycle or failure freeze +``` + +Each attempt remains independently attributable and replayable. The lane adds +accumulated context: + +- server generation and start time; +- database age and size; +- session/message counts; +- total prompts and model steps; +- cumulative tool invocations; +- TUI/controller reconnect counts; +- prior successful and failed attempts; +- host/process resource samples. + +## Workload Profiles + +### Baseline conversation soak + +- fresh TUI and session per attempt; +- short deterministic text response; +- occasional reasoning and multiple chunks; +- normal completion only; +- high frequency and low artifact volume. + +This profile establishes the lowest-noise trend for resource leaks and latency +drift. + +### Tool lifecycle soak + +- controlled read, edit, search, question, and shell-like fixture tools; +- bounded success, declared failure, and progress updates; +- permission policy varied by scenario; +- fixture reset between attempts; +- exact tool settlement assertions. + +### Session-history soak + +- selected sessions deliberately reused; +- history grows across a declared number of steps; +- compaction and context limits exercised; +- reopen from fresh TUI clients; +- response and projection latency measured against history size. + +### Multi-session soak + +- several sessions progress concurrently through a request-aware served model; +- sessions use distinct markers and tool calls; +- assertions check routing and isolation; +- concurrency remains below a declared lane capacity. + +### Client reconnect soak + +- server stays persistent; +- fresh TUIs connect and disconnect repeatedly; +- selected attempts keep one TUI across a server generation change; +- stale subscriptions, duplicated events, and retained terminal instances are + monitored. + +### Mixed realistic soak + +Use only after individual profiles are stable. A weighted schedule combines +ordinary conversations, tools, navigation, interruptions, and bounded restarts. +It is useful for discovery but weaker for diagnosis, so failures trigger replay +through the narrowest matching profile. + +## Soak Durations + +Use progressive qualification: + +| Stage | Typical duration | Purpose | +| --- | --- | --- | +| Local qualification | 15–30 minutes | Find immediate lifecycle leaks and harness errors | +| Pull-request extended | 1–2 hours, selected changes | Catch short accumulation regressions | +| Nightly | 6–12 hours | Cross multiple maintenance and workload cycles | +| Continuous | Repeated bounded attempts until a declared lane-replacement trigger | Detect long-age drift and rare interleavings | +| Release qualification | 24–72 hours | Compare candidate against a stable baseline | + +Duration is configuration, not the success condition. A soak succeeds only if +all attempt and trend invariants pass and the final lane cleanup/recycle is +healthy. + +## Resource Sampling + +Sample at a fixed low cadence and around attempt boundaries: + +- resident and virtual memory; +- CPU time and recent utilization; +- open file descriptors or handles; +- child process count; +- thread count where meaningful; +- event-loop delay if available; +- database bytes, WAL bytes, and row counts; +- artifact/log disk bytes; +- active sessions, executions, requests, tools, and clients; +- connection/reconnect counts; +- attempt and checkpoint latency distributions. + +Record raw samples in bounded artifacts or a time-series backend. Metrics use +lane and revision dimensions, not process IDs or session IDs as labels. + +## Leak Detection + +A single high value is not necessarily a leak. Evaluate: + +- absolute safety ceiling; +- baseline-adjusted slope over a minimum window; +- post-attempt return toward a steady band; +- step changes correlated with one scenario; +- monotonic growth in a resource that should be bounded; +- comparison with an idle control lane; +- candidate versus baseline revision under matched workload. + +Example policy: + +```text +suspect memory leak when all hold: + lane age >= 2 hours + completed attempts >= 200 + robust RSS slope > configured bytes/attempt + RSS does not return within steady-state band after cooldown + candidate slope materially exceeds baseline slope +``` + +Thresholds are initially observational. Promote them to gates only after enough +healthy history establishes variance. + +## Lane Maintenance and Recycling + +Continuous does not mean immortal. Lanes need declared lifecycle policy. + +Here, recycling means replacing the entire lane generation—server, +controllers, attempt-owned clients, and optionally its retained database—not +merely clearing a session. A continuous soak may keep a lane for days when lane +age is the thing being tested, while ordinary smoke lanes may use a shorter +scheduled maximum age. Both also recycle on revision/configuration changes or +unhealthy state. + +Recycle reasons: + +- scheduled maximum age; +- tested revision changed; +- configuration or protocol version changed; +- resource ceiling approached; +- maintenance window; +- lane frozen after a failure and evidence completed; +- unrecoverable health state. + +Every recycle records a reason and performs: + +1. stop admitting work; +2. allow or interrupt the active attempt according to deadline; +3. capture final health/resource state; +4. settle controllers and clients; +5. terminate lane-owned processes; +6. verify no process or port remains; +7. archive or delete state according to policy; +8. start a new generation and run a bootstrap smoke. + +An unexpected crash is not mislabeled as a scheduled recycle. + +## Chaos Experiment Model + +Every chaos experiment has a versioned specification: + +```text +ChaosExperiment + id + hypothesis + eligible lane kind + steady-state probe + trigger phase + fault + target resolver + duration/budget + expected transient observations + recovery action, if any + recovery invariants + abort conditions + cooldown + evidence policy + owner +``` + +The runner validates the target identity immediately before injection. It logs +the resolved explicit target and generation. It never kills by fuzzy process +name alone. + +## Steady State + +Before injecting a fault, prove a small useful behavior works: + +- control plane heartbeat fresh; +- lane reports the expected revision/configuration; +- OpenCode server health succeeds; +- simulation controller is attached when required; +- a short smoke attempt completed recently; +- no unrelated attempt is active; +- resource use is below abort thresholds. + +If steady state is absent, the experiment is `not-started` or +`inconclusive`; it is not a failed recovery test. + +## Fault Catalog + +### Simulated provider disconnect + +Mechanism: `Llm.disconnect()` at a declared stream phase. + +Trigger points: + +- before output; +- after reasoning only; +- after partial text; +- during tool-input JSON; +- after a tool call is emitted; +- after a local tool settles but before provider terminal. + +Assertions depend on delivery and output state: retry, continuation, explicit +failure, and tool settlement must match V2 policy. The session must remain +reusable. + +### Provider pause or hang + +Mechanism: a gated or never-ending simulated response within an outer timeout. + +Assertions: + +- pending UI state remains internally consistent; +- interruption succeeds; +- timeout/cancellation closes the provider invocation; +- no late output is accepted; +- another session remains usable in a concurrent campaign. + +### Malformed provider stream + +Drive's raw output can inject some OpenAI Chat event errors; the provider +contract harness covers broader wire faults. + +Assertions: + +- error becomes the expected canonical type; +- no hidden retry occurs after output; +- partial output and terminal failure are durably coherent; +- internal decoder defects are not leaked as confusing UI state. + +### Controlled tool delay/failure + +Mechanism: Drive tool controller waits, reports progress, fails, or is +interrupted. + +Assertions: + +- exactly one terminal tool state; +- failure is model-visible only through the declared tool error contract; +- defects remain defects; +- interrupted work cannot mutate the fixture later; +- next model/session action follows policy. + +### TUI process termination + +Mechanism: terminate the explicit lane-owned TUI process. + +Assertions: + +- server and session remain healthy; +- controller state is not consumed by the dead client unexpectedly; +- a replacement TUI can connect and rehydrate; +- no orphan terminal/renderer process remains. + +### OpenCode server termination + +Mechanism: terminate the explicit lane-owned server generation, then let the +supervisor restore it or invoke the declared restart operation. + +Trigger points: + +- idle; +- immediately after prompt admission; +- while provider request is open; +- during local tool execution; +- after terminal provider output but before all client observations; +- with queued input awaiting promotion. + +Assertions: + +- process death is detected promptly; +- write-ahead execution/recovery behavior matches V2 contract; +- durable facts survive; +- stale active tools settle on recovery; +- TUI reconnects or presents an actionable state; +- post-recovery smoke passes; +- no duplicate model execution is silently claimed as exactly-once behavior. + +### Drive controller detach + +Mechanism: close the backend controller connection while preserving the server. + +Assertions: + +- attachment generation changes; +- pending invocation handling matches the canonical protocol; +- reconnection does not attach two active controllers; +- response plans are not silently reassigned across attempts; +- settlement reports unresolved work. + +### Network denial or route miss + +Mechanism: request an unregistered simulated destination or make a contract +transport fail at a declared phase. + +Assertions: + +- no real egress occurs; +- typed transport error retains safe diagnostic context; +- retry follows delivery policy; +- lane remains controllable. + +### Database pressure + +Begin with non-destructive conditions: + +- slow database operations through an injectable test boundary; +- bounded WAL/database growth; +- many sessions/messages; +- lock contention generated by supported concurrent actions. + +Disk-full, file corruption, and forced I/O errors are later experiments in an +ephemeral disposable volume. Never run them against a shared or user-owned +database. + +### Host resource pressure + +CPU, memory, descriptor, and disk pressure are later container-level tests. +They require: + +- a dedicated host or container; +- explicit cgroup/resource limit; +- supervisor and artifact store outside the constrained unit where possible; +- hard abort threshold; +- no credentials or persistent shared state; +- automatic cleanup verification. + +## Failure Timing Matrix + +For every lifecycle fault, classify the injection phase: + +```text +before admission +after durable admission, before delivery +after delivery, before model request +request opened, no output +partial reasoning/text +partial tool input +tool executing +provider terminal, tools pending +step settlement +client projection only +``` + +Coverage reports phase/fault pairs. This is more useful than counting total +chaos runs. + +## Recovery Invariants + +The common recovery contract is: + +1. The injected fault is observed and correlated with the experiment. +2. No unrelated lane or control-plane component is affected. +3. Durable admitted input is retained or explicitly rejected. +4. No tool or provider request remains permanently active. +5. No duplicate terminal outcome is created. +6. Process/controller generations converge to one active owner. +7. The TUI reaches a stable actionable or explicit error state. +8. A post-recovery synthetic prompt completes within its deadline. +9. Resource usage returns below the cooldown ceiling. +10. Cleanup verifies the fault mechanism itself left no residue. + +Some faults intentionally fail the active attempt. Recovery success does not +rewrite that attempt to passed. + +## Abort Conditions + +Stop an experiment immediately when: + +- target identity no longer matches the leased lane generation; +- an unrelated process or path could be selected; +- control-plane heartbeat is lost; +- host resource hard limit is approached; +- artifact storage or redaction is unavailable; +- a real credential or network destination appears in a deterministic lane; +- more than the allowed number of attempts/sessions would be affected; +- cleanup cannot be guaranteed. + +Abort terminalizes the experiment distinctly from product failure. + +## Scheduling and Exclusivity + +Chaos experiments require an exclusive lane lease. The scheduler: + +- drains ordinary work; +- verifies steady state; +- marks the lane `experimenting`; +- runs one experiment; +- performs cooldown and post-recovery smoke; +- returns the lane to `ready` or freezes/recycles it. + +Use Effect `Schedule` for campaign cadence and health sampling. Do not put the +entire experiment inside an automatic retry. A repeated fault injection is a +new linked attempt. + +## Evidence + +Retain: + +- experiment spec and digest; +- exact target and resolved process/container identity; +- pre-fault steady-state result; +- injection timestamp and phase proof; +- process generation changes; +- correlated OpenCode/session/simulation events; +- resource samples around the fault; +- frame immediately before and after recovery; +- active request/tool/session summaries; +- post-recovery smoke result; +- cleanup verification; +- original and reproduction attempt links. + +Evidence collection is bounded. A log storm must not exhaust the host while a +fault is active. + +## Baseline Comparison + +For release soak tests, run matched workload profiles against baseline and +candidate revisions when capacity allows. + +Compare: + +- success and timeout rates; +- checkpoint latency quantiles; +- resource slope per completed attempt; +- database growth per session/message; +- reconnect/restart recovery time; +- number of residual active entities after attempts; +- lane recycle/crash frequency. + +Pin workload, fixture, Drive revision, configuration, and response-plan digests. +Without matched inputs, a difference is a signal for investigation rather than +a release verdict. + +## Runbooks for a Soak Failure + +1. Stop admitting new work to the lane. +2. Preserve the first failed attempt and current lane health. +3. Determine whether a declared chaos fault was active. +4. Capture bounded process, storage, event, and frame evidence. +5. Run a clean ephemeral replay of the attempt. +6. If clean replay passes, snapshot or retain persistent state according to + policy and attempt a stateful reproduction. +7. Compare resource trends with the control lane. +8. Classify product, harness, infrastructure, expected fault, or inconclusive. +9. Recycle only after evidence requirements are met. + +## Initial Experiments + +Begin with high-value faults already supported by Drive: + +1. provider disconnect before output; +2. provider disconnect after partial text; +3. interruption during streamed tool input; +4. interruption during controlled tool execution; +5. OpenCode server restart while idle; +6. OpenCode server restart after a completed persisted session; +7. repeated fresh TUI reconnect to a persistent server; +8. 12-hour baseline conversation soak. + +Only then add restart-during-execution and multi-fault experiments. + +## Acceptance Criteria + +Soak and chaos testing is operational when: + +- every soak consists of bounded attributable attempts; +- lane age and cumulative workload accompany all resource samples; +- leak alerts use both absolute ceilings and trend evidence; +- every chaos experiment has a hypothesis, explicit target, trigger proof, + recovery invariant, abort condition, and cleanup check; +- one experiment holds an exclusive lane lease; +- fault injection cannot resolve to unrelated processes, paths, or hosts; +- active attempt failure and recovery success remain separate outcomes; +- persistent failures freeze evidence before recycle; +- a post-recovery smoke verifies useful behavior, not only process liveness; +- no destructive storage or resource fault runs outside a disposable isolated + environment; +- baseline and candidate soak comparisons record matched workload inputs; +- the control plane detects a lane that is alive but no longer completing work. diff --git a/docs/continuous-testing/08-log-files-and-review-ui.md b/docs/continuous-testing/08-log-files-and-review-ui.md new file mode 100644 index 0000000..2bff6ba --- /dev/null +++ b/docs/continuous-testing/08-log-files-and-review-ui.md @@ -0,0 +1,624 @@ +# Log Files and Review UI + +This document defines the local log-file format, collection pipeline, and human +review experience for continuous verification. + +The core decision is: + +> Every process writes bounded local files first; structured attempt events are +> indexed into a unified timeline; `apps/catalog` presents a searchable review +> UI over those records and artifacts. + +This preserves evidence during exporter outages and makes local reproduction +easy, while still giving operators something much better than opening several +raw files in separate terminals. + +## Current Foundation + +[`packages/drive/src/log.ts`](../../packages/drive/src/log.ts) currently: + +- writes Drive messages to `logs/opencode-drive.log` when + `OPENCODE_DRIVE_LOG` is configured; +- prints friendly success/error messages to stderr; +- locates OpenCode's most recent `opencode*.log` file beneath the run + artifacts; +- can forward error text to an owner log through + `OPENCODE_DRIVE_OWNER_LOG`; +- treats log-file writes as best effort so logging does not change CLI + behavior. + +[`RunReport`](../../packages/drive/src/driver/report.ts) currently returns the +artifact root, retention state, recordings, and protocol compatibility. This is +a useful finite-run handoff, but it is not yet a structured operational log or +attempt timeline. + +Keep that lightweight behavior for generic Drive CLI use. Build the richer +24/7 logging contract in the app/control-plane layer and add generic Drive +structured hooks only when they benefit multiple consumers. + +## Goals + +- every attempt has discoverable local logs even if remote telemetry is down; +- logs from scheduler, worker, Drive, OpenCode, TUI, inference, and tools can be + viewed on one timeline; +- operators can filter and search without downloading a bundle; +- a log row links to its checkpoint, frame, session summary, exchange, tool, + error, and artifact where applicable; +- live tail is available for active attempts; +- raw files remain downloadable and usable with ordinary command-line tools; +- content, cardinality, size, and retention are bounded; +- logging failure is visible but cannot change a product pass into a false + failure unless required evidence policy says the attempt is inconclusive; +- viewer features do not modify OpenCode's simulation protocol. + +## Two File Classes + +### Structured JSONL event logs + +One JSON object per line, emitted by components controlled by this system. +These are the canonical local diagnostic stream and can be indexed reliably. + +Examples: + +- control-plane events; +- lane-worker lifecycle; +- scenario checkpoints; +- Drive operations and controller summaries; +- inference/tool lifecycle summaries; +- evidence/redaction/cleanup events; +- resource samples at bounded cadence. + +### Raw process logs + +Unmodified or minimally framed stdout/stderr from OpenCode, TUI, build tools, +and other child processes. These preserve diagnostics the structured layer does +not understand. + +Raw logs are linked into the timeline by process, time window, and generation. +They are not parsed as authoritative attempt state. + +## Proposed File Layout + +```text +/logs/ + lane.jsonl + processes/ + opencode.stdout.log + opencode.stderr.log + tui-.stdout.log + tui-.stderr.log + drive.stdout.log + drive.stderr.log + attempts/ + / + attempt.jsonl + timeline.jsonl + inference.jsonl + tools.jsonl + resource.jsonl + process-excerpts/ + log-manifest.json +``` + +The exact directory names are implementation details of `apps/catalog` or the +control plane. `packages/drive` continues to expose an artifact root rather +than application-specific attempt taxonomy. + +The attempt directory may contain hard links or references to lane process log +ranges rather than copying the same bytes. Portable failure bundles materialize +bounded excerpts. + +## Structured Log Entry + +Illustrative schema: + +```text +LogEntry + schemaVersion + timestamp + monotonicElapsedMs? + sequence + level: trace | debug | info | warn | error | fatal + event + message? + component + deploymentId + laneId? + laneGenerationId? + attemptId? + workItemId? + campaignId? + scenarioId? + checkpointId? + phase? + process: + role? + generation? + correlation: + exchange? + tool? + sessionPlaceholder? + outcome? + error?: safe typed summary + fields?: bounded JSON object + sensitivity + redactionVersion +``` + +Required fields are stable and schema-decoded. `fields` is not an unlimited +escape hatch: values must satisfy size, nesting, and key allowlists or be +replaced with a digest/summary. + +## Event Names + +Use namespaced stable event names: + +```text +control.scheduler.tick +control.work.enqueued +control.attempt.reconciled +lane.generation.started +lane.health.changed +lane.resource.sampled +attempt.started +attempt.phase.changed +scenario.checkpoint.entered +scenario.checkpoint.completed +driver.ui.operation.started +driver.ui.operation.failed +driver.llm.request.opened +driver.llm.response.item +driver.llm.response.terminal +driver.tool.invocation.opened +driver.tool.invocation.terminal +opencode.process.exited +evidence.artifact.published +evidence.redaction.failed +attempt.terminal +``` + +The stable event drives filtering and grouping. `message` is concise human +context and may evolve. + +Do not emit one row for every rendered pixel or high-frequency polling cycle. +Inference chunk rows should be summaries or enabled only for failure-focused +debug policy. + +## Sequences and Ordering + +Every structured writer assigns a monotonic sequence within its file. The +attempt coordinator also assigns an attempt timeline sequence to the events it +owns. + +Across processes: + +- retain source sequence; +- retain wall-clock timestamp; +- record process generation; +- use protocol/request/checkpoint causality when available; +- do not claim a strict total order from timestamp alone; +- mark rows whose order is approximate. + +The indexed timeline may interleave sources for viewing, but raw source order is +always recoverable. + +## Writer Design + +Use an Effect `LogWriter` service with one scoped instance per process or lane. + +Responsibilities: + +- validate/encode entries with Effect Schema; +- add identity, timestamp, sequence, and redaction version; +- append newline-delimited UTF-8 JSON; +- batch low-priority entries through a bounded queue; +- synchronously or transactionally preserve critical lifecycle markers as + configured; +- flush at attempt checkpoint/terminal and process drain; +- rotate by size/time; +- expose queue/drop/write health metrics; +- close through a scope finalizer. + +The service returns typed failures for required audit/run-event writes. Ordinary +debug logging remains best effort and cannot fail product execution. + +Do not call synchronous append for every output chunk in high-volume lanes. +The current Drive logger is fine for its small CLI messages; the continuous +writer needs bounded batching and explicit flush boundaries. + +## Durability Tiers + +Not every row needs the same durability. + +### Tier 1: authoritative run events + +Attempt creation, phase changes, selected command intent, checkpoint result, +fault injection, and terminal outcome belong in the run store or a durable +journal before/with side effects. JSONL is a local mirror. + +### Tier 2: required failure evidence + +Failure summary, process exit, current frame reference, and artifact manifest +must be flushed before cleanup where possible. Missing data marks evidence +degraded. + +### Tier 3: diagnostic logs + +Debug messages, resource samples, and raw child output use bounded asynchronous +files and may drop under extreme backpressure. Drop counts remain visible. + +This prevents an overloaded debug stream from blocking model/session execution +while preserving the events needed for reconciliation. + +## Rotation + +Rotate lane/process logs by configured size and time, for example: + +```text +lane.jsonl +lane.jsonl.1 +lane.jsonl.2 +``` + +Requirements: + +- an active attempt can still resolve the file/range containing its events; +- rotation is atomic from the writer's perspective; +- retained segments are immutable; +- compression happens after close and outside critical attempt paths; +- total bytes per lane and process have a hard quota; +- oldest success-only segments expire before unresolved failure evidence; +- deletion uses validated lane-owned paths; +- the manifest records gaps, rotations, and drops. + +## Child Process Capture + +For every spawned process: + +- capture stdout and stderr separately; +- prefixing/color is optional for an interactive console but raw file bytes + should not gain ambiguous human prefixes; +- record process role, generation, command digest, start, and exit in structured + logs; +- bound line length and total bytes; +- handle output without a final newline; +- preserve undecodable bytes through a safe representation or binary artifact; +- never allow child output to write arbitrary terminal control into the + operator console by default; +- flush/close descriptors during drain. + +OpenCode's own rotating logs remain separate artifacts. The collector indexes +their discovered paths and bounded relevant excerpts. + +## Attempt Log Manifest + +Every attempt produces a manifest: + +```text +LogManifest + schemaVersion + attemptId + target/config revisions + sources[]: + source id + component/process role + path or artifact reference + byte range? + first/last timestamp + first/last sequence + bytes + digest + complete + dropped entries/bytes + sensitivity + redaction status + timeline reference +``` + +The viewer uses the manifest rather than scanning directories heuristically. + +## Indexing Pipeline + +```text +local JSONL/raw files + | + +--> live tail stream + | + +--> attempt terminal collector + | + v + validate + redact + index + | + +--------+---------+ + | | + v v + searchable rows immutable artifacts +``` + +Index at bounded checkpoints and terminalization. A background reconciler can +finish indexing after worker restart using the manifest and source sequences. + +Indexing is idempotent by `(source, sequence, digest)`. Duplicate uploads do not +create duplicate viewer rows. + +## Review UI Ownership + +`apps/catalog` owns the OpenCode-specific review experience because it already +owns flow IDs, state taxonomy, captures, and review UI. The generic Drive +package must not import it or learn provider/scenario taxonomies. + +The review UI consumes an API or static exported run bundle. It does not read +arbitrary server filesystem paths supplied by the browser. + +## Review UI Information Architecture + +### Fleet view + +Shows: + +- bot/profile status, including one row per provider contract bot; +- lane state and latest generation; +- active attempts; +- last success/failure and freshness; +- queue age; +- current target revision; +- active alerts and frozen lanes. + +### Attempt list + +Filter by: + +- time range; +- target revision; +- bot, scenario, campaign, or provider; +- outcome and classification; +- lane kind; +- error/invariant tag; +- runtime path and protocol; +- expected chaos fault; +- evidence completeness; +- replay/retry relationship. + +### Attempt detail + +Header: + +- outcome/classification; +- target and harness revisions; +- scenario/campaign/bot; +- lane generation; +- duration and last checkpoint; +- seed/trace/plan/package fingerprints; +- retry/replay links; +- triage owner/status. + +Main panes: + +1. normalized timeline; +2. terminal frame/screenshot and checkpoint captures; +3. session/inference/tool state summaries; +4. structured logs; +5. raw process logs; +6. artifacts and reproduction command/spec; +7. provider fingerprint diff when applicable. + +### Comparison view + +Compare: + +- failed attempt against its last success; +- candidate against baseline revision; +- original property failure against minimized replay; +- native provider path against AI SDK fallback; +- before/after chaos recovery; +- two package fingerprint versions. + +Align by checkpoint/event type rather than raw line number. + +## Timeline Interaction + +The timeline should support: + +- compact phase bands for prepare, execute, evidence, cleanup; +- nested checkpoint spans; +- icons/colors for process, UI, inference, tool, provider, and fault events; +- jump from error to nearest frame and raw log excerpt; +- expand one row to show safe structured fields and causal links; +- collapse repeated resource/heartbeat events; +- “show only warnings/errors”; +- “show around failure” time window; +- copy stable event/attempt IDs; +- display approximate-order warning for cross-process rows; +- keyboard navigation and shareable filtered URL. + +Avoid a decorative waterfall that hides actual messages. The textual event list +remains primary and accessible. + +## Log Viewer + +The log pane needs: + +- component/source toggles; +- level and event filters; +- exact and token search over permitted fields; +- time-window brush linked to the timeline; +- virtualized rendering for large files; +- preserved whitespace for raw logs; +- JSON tree and raw-line toggle for structured entries; +- line wrapping toggle; +- correlation highlighting for exchange/tool/checkpoint placeholders; +- ANSI/control-sequence-safe rendering; +- dropped/truncated/gap indicators; +- download current safe view and original authorized artifact; +- live follow mode with pause and unseen-row count; +- permanent link to a source sequence, not a fragile visual row index. + +Search should not send restricted content to a third-party service. + +## Live Tail + +Active attempts may stream new indexed entries over an app-owned SSE or +WebSocket endpoint. + +Rules: + +- authorize by attempt sensitivity; +- send already redacted structured rows; +- use source sequence for resume after reconnect; +- bound per-client buffer; +- drop/coalesce low-priority rows with an explicit gap marker; +- disconnect slow clients without affecting the attempt; +- final UI reloads the immutable terminal index after completion. + +This endpoint is part of the review/control application, not the canonical +OpenCode simulation protocol and not `--command.ui.*`. + +## Local Developer Experience + +Every retained run should print or return: + +- artifact root; +- Drive log path; +- OpenCode log path or pattern; +- attempt manifest path when running through the control layer; +- optional local review URL. + +Provide a small app/control-plane command to pretty-print or tail JSONL: + +```text +verification logs --follow +verification logs --component inference --level warn +verification open +``` + +Names are illustrative. They are not Drive frontend protocol commands and must +not be added as `--command.ui.*` aliases. + +Raw files remain compatible with `tail`, `jq`, `rg`, and editors. + +## Redaction and Access + +Default structured logs contain no raw prompt, response, tool argument/result, +HTTP body, credential, environment value, or arbitrary file content. + +Represent content as: + +- known synthetic marker; +- type and byte/character count; +- digest; +- schema-validity result; +- restricted artifact reference. + +Raw process logs pass through redaction and secret scanning before publication. +The unredacted local source, if retained at all, has restricted access and short +retention. + +The viewer visibly labels sensitivity and redaction status. It never fetches a +quarantined artifact into an ordinary page. + +## Performance and Backpressure + +- bounded writer queue per process; +- separate critical and diagnostic channels; +- maximum event and line size; +- sampled/coalesced chunk and resource logs; +- rotation and total byte quotas; +- index batches; +- viewer pagination/virtualization; +- bounded live-tail fan-out; +- stop large campaigns when artifact/log quota is threatened; +- preserve terminal/failure rows before success debug data. + +Expose writer queue depth, write latency, dropped entries, raw bytes, rotation, +index lag, and viewer query latency. + +## Failure Semantics + +| Failure | Attempt effect | +| --- | --- | +| One debug append fails | Continue; mark log degraded and count drop | +| Required attempt journal write fails before action | Do not execute untracked action; fail/inconclusive as harness | +| Raw child log exceeds quota | Truncate/rotate, continue, show gap | +| Redaction fails | Quarantine artifact; continue collecting safe manifest | +| Index unavailable | Retain local files and retry; viewer shows delayed evidence | +| Local disk hard threshold reached | Stop new work and preserve essential terminal records | +| Viewer unavailable | Test execution continues; alert review-plane outage | + +Logging must never silently convert a failure to success. Conversely, a best- +effort debug log failure does not imply OpenCode behavior failed. + +## Testing + +Test the complete path: + +- Schema encode/decode and forward version handling; +- one JSON object per line under concurrent writers; +- source and attempt sequence monotonicity; +- flush on checkpoint, terminal, interruption, and drain; +- crash recovery from a partial final line; +- rotation while an attempt is active; +- manifest byte ranges and digest verification; +- dropped-entry and truncation markers; +- ANSI, control character, invalid UTF-8, and huge-line handling; +- secret sentinel redaction in structured and raw logs; +- indexing idempotency; +- live-tail reconnect/resume and slow-client backpressure; +- viewer filtering and permanent links; +- comparison alignment by checkpoint/event; +- authorization and quarantine behavior; +- exporter/index outage with local evidence preserved; +- retention deleting only validated targets. + +## Implementation Sequence + +### Step 1: stable JSONL schema + +- define app-owned `LogEntry` and `LogManifest` Schemas; +- emit lane/attempt/checkpoint lifecycle events; +- mirror authoritative run events into JSONL; +- retain existing Drive/OpenCode raw logs; +- add schema/digest validation. + +### Step 2: attempt timeline + +- normalize worker, scenario, inference, tool, and process events; +- collect bounded raw excerpts; +- build `timeline.jsonl` at terminalization; +- add failure bundle links. + +### Step 3: read-only review UI + +- add bot/attempt list and detail routes in `apps/catalog`; +- implement filters, timeline, log pane, frame pane, and artifact list; +- serve static/local bundles first if that is simpler; +- add access/redaction labeling. + +### Step 4: live operation + +- incremental indexing; +- live tail with resume; +- fleet freshness/status view; +- comparison and triage annotations; +- remote artifact integration. + +## Acceptance Criteria + +Logging and review are ready when: + +- every process has bounded local structured or raw files; +- every attempt has a schema-validated log manifest; +- critical attempt events are durable independently of best-effort debug logs; +- a failed attempt can be understood in one review page without manually + correlating timestamps across terminals; +- the viewer filters by bot/provider/scenario, component, event, level, phase, + outcome, and time; +- timeline rows link to frames, inference/tool summaries, raw excerpts, and + artifacts; +- active attempts can be followed without backpressuring execution; +- source sequence and raw files remain available for exact diagnosis; +- rotation, truncation, drops, and indexing gaps are explicit; +- default logs and views contain no sensitive content; +- `apps/catalog` owns the OpenCode-specific viewer and `packages/drive` remains + generic; +- no log/viewer command is added to the canonical `ui.*` simulation protocol. + diff --git a/docs/continuous-testing/08-observability-and-evidence.md b/docs/continuous-testing/08-observability-and-evidence.md new file mode 100644 index 0000000..41d99b2 --- /dev/null +++ b/docs/continuous-testing/08-observability-and-evidence.md @@ -0,0 +1,790 @@ +# Observability and Evidence + +This document defines how the continuous-verification system proves what ran, +detects when useful work stops, and preserves enough evidence to diagnose a +failure without leaking sensitive content. + +The design separates three concerns: + +- **telemetry** supports aggregate health, trends, dashboards, and alerts; +- **run records** are the durable source of truth for attempts and outcomes; +- **artifacts** preserve bounded high-detail evidence for selected runs. + +Logs alone are not a run database, and a dashboard is not a reproduction +artifact. + +## Objectives + +The evidence system must answer: + +- Which OpenCode and Drive revisions were tested? +- Which lane, scenario, seed, response plan, package path, and configuration ran? +- What was the last meaningful checkpoint reached? +- Did the product fail, the harness fail, infrastructure fail, or did an + expected injected fault occur? +- Was any output visible before the failure? +- Were retries, continuations, restarts, and controller generations involved? +- Is the environment completing useful work now? +- Can the exact action and response trace be replayed? +- Which evidence is safe to show broadly, and which is restricted? + +## Signals + +### Run records + +Durable, queryable entities representing scheduled work, attempts, checkpoints, +failures, and artifacts. They are authoritative for pass/fail accounting. + +### Traces + +OpenTelemetry spans describe execution timing and causality across scheduler, +lane, scenario, Drive, and selected OpenCode boundaries. + +### Metrics + +Low-cardinality counters, gauges, and histograms support aggregate reliability, +latency, capacity, freshness, and resource analysis. + +### Logs + +Structured diagnostic events explain local decisions and failures. Logs carry +correlation fields but are not parsed as the primary state machine. + +The concrete JSONL/raw file layout, writer behavior, indexing, live tail, and +`apps/catalog` review experience are specified in [Log files and review +UI](./08-log-files-and-review-ui.md). + +### Artifacts + +Frames, PNGs, recordings, event excerpts, response traces, minimized property +cases, resource samples, and redacted logs retained according to policy. + +## Identity Hierarchy + +Use explicit stable identifiers: + +```text +deployment_id + lane_id + lane_generation_id + work_item_id + attempt_id + checkpoint_id / exchange_id / artifact_id +``` + +Additional links: + +- `parent_attempt_id` for retry, replay, or shrink relationship; +- `discovery_attempt_id` for minimized property reproductions; +- `baseline_attempt_id` for candidate comparison; +- `experiment_id` for chaos attempts; +- `campaign_id` for property, soak, or provider-contract campaigns. + +IDs are fields in records and spans. They are normally not metric labels. + +## Run Record Model + +All records are schema-versioned and immutable after terminalization except for +explicit review annotations. + +### Work item + +Represents the scheduler's intent: + +```text +WorkItem + id + kind: journey | property | provider-contract | soak | chaos | replay + scenario/campaign/experiment identity + priority + requested target revision + configuration version + eligible lane kinds + createdAt + notBefore + deadline + trigger: cadence | commit | manual | alert | replay + deduplication key +``` + +### Attempt + +Represents one execution: + +```text +Attempt + id + workItemId + parent links + laneId and laneGenerationId + kind + state + outcome and classification + target: + opencode revision + drive revision + catalog revision + lockfile digest + configuration: + config version/digest + fixture digest + scenario definition digest + response plan digest + inference strategy/version + seed/trace digest + provider/package/runtime path when applicable + timing: + scheduled, leased, started, first checkpoint, terminal + phase and last checkpoint + process generations + retry/replay ordinal + error summary + evidence completeness +``` + +### Checkpoint attempt + +```text +CheckpointRecord + attemptId + checkpoint address + ordinal + enteredAt + completedAt + outcome + assertion summaries[] + optional artifact references[] +``` + +### Failure + +```text +FailureRecord + attemptId + stage + typed category/tag + classification + safe message + cause summary + outputStarted + expected fault correlation? + retryability assessment + first observedAt + related process/request/tool safe summaries + triage status and owner +``` + +### Artifact metadata + +```text +ArtifactRecord + id + attemptId + kind + media type + content digest + bytes + storage key + sensitivity + redaction status + createdAt + expiresAt + producer version +``` + +The database stores metadata and safe summaries. Large content lives in an +artifact store. + +## Attempt State Machine + +```text +scheduled + | + v +leased -> preparing -> running -> collecting -> terminal + | | | | + +----------+----------+-----------+ + failure/interruption +``` + +Terminal outcomes: + +- `passed`; +- `failed`; +- `inconclusive`; +- `cancelled`; +- `timed-out`; +- `not-started` because preconditions or steady state were absent. + +Classification is separate: + +- `product`; +- `harness`; +- `infrastructure`; +- `expected-fault`; +- `provider-drift`; +- `security-policy`; +- `unknown`. + +For example, an attempt may be `failed` and classified `harness`, or +`inconclusive` and classified `infrastructure`. Do not overload one enum with +both meanings. + +## Atomicity and Append Order + +Record intent before executing side effects: + +1. create attempt; +2. record selected lane generation and configuration; +3. append command/checkpoint intent; +4. execute; +5. append observation and result; +6. terminalize once; + +If the worker dies between steps 4 and 5, reconciliation sees a non-terminal +attempt and classifies it using lane/process evidence. It does not disappear. + +Attempt terminalization uses a compare-and-set or transactional guard. Late +worker updates cannot overwrite a reconciler's terminal record. + +## Tracing Model + +Recommended span hierarchy: + +```text +verification.work_item + verification.attempt + lane.prepare + scenario.execute + scenario.checkpoint + driver.ui.operation + driver.llm.exchange + driver.tool.invocation + evidence.collect + lane.cleanup +``` + +Provider-contract attempts use: + +```text +verification.attempt + provider.resolve + provider.request + provider.transport.attempt + provider.protocol.decode + provider.fingerprint +``` + +Stateful cases add one span per generated command, not one span per random +number or frame. + +Use named `Effect.fn` boundaries for meaningful operations. Provide one +OpenTelemetry layer at the application edge. Domain services emit spans and +metrics but do not choose exporters. + +## Span Attributes + +Safe, bounded examples: + +- `verification.kind`; +- `verification.scenario` from a registered finite set; +- `verification.phase`; +- `verification.outcome`; +- `verification.classification`; +- `lane.id`, if lane count is bounded; +- `lane.kind` and `lane.generation` on traces/logs; +- `target.revision` on traces and records, not necessarily metrics; +- `inference.strategy`; +- `provider.protocol` and `provider.runtime_path`; +- `fault.kind` and `fault.phase`; +- `checkpoint.id` from the scenario registry; +- `error.tag` from a finite union; +- `output_started`. + +Never attach raw prompts, model output, tool inputs/results, request bodies, +session IDs, request IDs, artifact paths, or arbitrary error messages as metric +labels. Restricted traces may opt into selected content only through explicit +policy. + +## Metrics + +### Scheduling and freshness + +- `verification_work_items_total{kind,state}`; +- `verification_attempts_total{kind,outcome,classification}`; +- `verification_queue_age_seconds{priority,kind}`; +- `verification_attempt_start_delay_seconds{kind}`; +- `verification_last_success_age_seconds{journey_tier,lane_kind}`; +- `verification_last_completion_age_seconds{lane_kind}`; +- `verification_scheduler_heartbeat_age_seconds`; +- `verification_lane_heartbeat_age_seconds{lane}`. + +Freshness metrics are essential. If the entire worker loop stops, no ordinary +failure counter increases. + +### Journey latency + +- `verification_attempt_duration_seconds{scenario_group,lane_kind,outcome}`; +- `verification_checkpoint_duration_seconds{checkpoint_group,lane_kind}`; +- `verification_time_to_first_output_seconds{strategy,lane_kind}`; +- `verification_recovery_duration_seconds{fault_kind}`. + +Avoid one histogram label value for every individual high-churn test. Use +bounded groups and query exact attempt records for detail. + +### Inference and tools + +- `verification_llm_exchanges_total{strategy,outcome,terminal}`; +- `verification_llm_output_items_total{type}`; +- `verification_llm_exchange_duration_seconds{strategy}`; +- `verification_unexpected_llm_requests_total{lane_kind}`; +- `verification_unused_llm_responses_total{lane_kind}`; +- `verification_tool_invocations_total{tool_group,outcome}`; +- `verification_tool_duration_seconds{tool_group,outcome}`; +- `verification_active_llm_requests{lane}`; +- `verification_active_tools{lane}`. + +### Lanes and processes + +- `verification_lane_state{lane,state}` as a single current-value series; +- `verification_lane_age_seconds{lane}`; +- `verification_lane_generation_total{lane,reason}`; +- `verification_process_restarts_total{role,reason}`; +- `verification_controller_reconnects_total{lane}`; +- `verification_lane_rss_bytes{lane,role}`; +- `verification_lane_open_handles{lane,role}`; +- `verification_lane_database_bytes{lane}`; +- `verification_lane_artifact_bytes{lane}`. + +### Properties and contracts + +- `verification_property_cases_total{campaign,outcome}`; +- `verification_property_steps_total{campaign,command}`; +- `verification_property_transition_coverage_ratio{campaign}`; +- `verification_shrink_duration_seconds{campaign}`; +- `verification_provider_contracts_total{protocol,runtime_path,outcome}`; +- `verification_provider_fingerprint_changes_total{protocol,review_state}`; +- `verification_live_provider_probe_age_seconds{provider_group}`. + +Metric cardinality is reviewed before deployment. + +## Logs + +Emit structured JSON or an equivalent structured format with: + +- timestamp and level; +- event name; +- deployment/lane/generation/attempt correlation; +- component and phase; +- safe typed error fields; +- process role and revision; +- bounded contextual fields. + +Examples of useful event names: + +```text +attempt.leased +attempt.phase.changed +scenario.checkpoint.completed +llm.request.opened +llm.response.terminal +tool.invocation.settled +lane.health.failed +lane.recycle.started +attempt.evidence.degraded +attempt.terminalized +alert.fired +``` + +Do not rely on human prose as the only classifier. A message may accompany a +stable event name and typed fields. + +### Log streams + +Keep separate logical streams or fields for: + +- control plane; +- lane worker; +- OpenCode server stdout/stderr; +- TUI stdout/stderr and renderer diagnostics; +- Drive controller; +- provider-contract harness; +- artifact/redaction service. + +The attempt evidence collector creates bounded excerpts using correlation and +time windows. It does not copy an unbounded lane log into every failure bundle. + +## Artifact Bundle + +A failed attempt bundle should contain a manifest plus selected files: + +```text +attempt.json +failure.json +timeline.jsonl +checkpoints.json +lane-health.json +processes.json +inference-trace.json +tool-trace.json +session-events.jsonl +frame.json +screenshot.png optional derived view +recording.* policy-dependent +property-trace.json generated cases +provider-fingerprint.json contract cases +logs/ + runner.jsonl + opencode.stderr.log + tui.stderr.log +``` + +The manifest lists every expected artifact and whether it is present, omitted +by policy, unavailable, failed redaction, or failed collection. + +One artifact failure must not erase the primary attempt result. Evidence +completeness is a separate field and may trigger a harness alert. + +## Timeline + +Build one normalized timeline from run events, not by guessing order from +multiple wall-clock log files. + +Timeline entries include: + +- monotonic sequence within the attempt; +- wall time and monotonic elapsed time; +- source component; +- phase and event type; +- stable entity placeholder; +- safe summary; +- links to detailed artifacts. + +Cross-process clock skew is possible. Causal sequence from the worker and +protocol events takes precedence over tiny timestamp differences. Record host +clock diagnostics when deployment spans machines. + +## Frames, Screenshots, and Recordings + +The canonical UI evidence is `ui.capture`, which returns a renderer-neutral +RGBA terminal frame through the OpenCode simulation protocol. + +Drive's deliberate local `ui.screenshot` command calls `ui.capture`, renders a +PNG, and prints its absolute path for a standalone command. Continuous +verification stores the canonical frame and may derive a PNG for review. It +does not add `ui.screenshot` to OpenCode or expose media-directory bookkeeping +through normal usage. + +Capture policy: + +- always capture current frame on failure when the UI connection is healthy; +- capture selected before/after frames for recovery experiments; +- sample success frames; +- retain full recordings only for failures, explicit visual campaigns, or a + low sample rate; +- bound frame/recording count and byte size per attempt. + +## Session and Event Evidence + +Store only the slice needed to reason about the attempt: + +- selected session safe metadata; +- relevant message/part types and statuses; +- prompt markers or content digests; +- pending inbox/tool/form/permission summaries; +- correlated durable event names and sequence; +- server generation/recovery claim summaries; +- before/after projection digests. + +Full user content is unnecessary for deterministic synthetic prompts. Use +known fixture markers and hashes. Real-provider or imported-session content has +stricter restricted handling. + +## SLOs and SLIs + +These are test-system objectives, not necessarily end-user product SLOs. + +### Synthetic availability + +```text +successful smoke attempts +------------------------- +eligible completed smoke attempts +``` + +Exclude only explicitly classified harness/infrastructure windows according to +documented policy; publish both raw and adjusted views to prevent convenient +reclassification. + +### Completion freshness + +Age since the last successful smoke for each required lane/revision. + +This catches dead schedulers, blocked lanes, and silent telemetry failures. + +### Attempt latency + +Time from lease to declared final checkpoint, plus checkpoint and first-output +breakdown. + +### Recovery success + +Fraction of eligible chaos experiments that reach all recovery invariants and +post-recovery smoke within deadline. + +### Evidence completeness + +Fraction of failed attempts with all required safe artifacts collected and +verified. + +### Reproducibility + +Fraction of deterministic failures reproduced by exact replay within a bounded +number of linked attempts. This is diagnostic quality, not a reason to dismiss +non-reproducing failures. + +## Initial Objectives + +Start with conservative objectives and tune after observing variance: + +- scheduler heartbeat age below 2 minutes; +- each required smoke lane has a completed attempt within 5 minutes; +- each target revision has a successful essential smoke within 10 minutes; +- 99% of healthy deterministic smoke attempts finish within their scenario + timeout; +- 100% of attempts reach a terminal record through execution or reconciliation; +- 100% of failed attempts retain the run/failure manifests; +- 95% of failed UI attempts capture a terminal frame when the frontend + connection remains available; +- no lane exceeds hard resource or artifact quotas. + +Do not page on tight latency objectives until a stable baseline exists. + +## Alerts + +### Page-worthy + +- no control-plane heartbeat; +- no successful essential smoke across redundant lanes beyond the freshness + threshold; +- all lanes for the active target unhealthy; +- repeated data-loss, duplication, or security-policy invariant; +- artifact store or run database unavailable such that failures cannot be + retained; +- unplanned real network egress or credential exposure signal; +- runaway resource/cost threshold. + +### Ticket or non-urgent notification + +- one deterministic scenario repeatedly failing while other smoke passes; +- candidate-only regression; +- provider contract fingerprint drift; +- one stale recording/live provider probe; +- gradual resource slope; +- evidence degradation; +- quarantined diagnostic campaign finding a known failure. + +### Alert shape + +An alert includes: + +- concise symptom and affected target; +- first and latest failed attempt links; +- last known success; +- lane/revision/configuration; +- current phase and classification confidence; +- whether an intentional fault was active; +- dashboard and runbook link; +- mute/deduplication key. + +Do not attach sensitive artifacts to alert payloads. + +## Alert Evaluation + +Evaluate absence-of-signal alerts from an independent process reading durable +heartbeats/run records. A worker cannot reliably alert that it has stopped. + +Use consecutive windows or multi-lane corroboration for noisy failures, while +immediately surfacing high-severity invariants. Every alert rule has a unit test +against synthetic time series or records. + +## Dashboards + +### Fleet overview + +- active target revisions; +- lane state, age, generation, and last heartbeat; +- queue depth/age; +- last successful smoke age; +- attempts by outcome/classification; +- active incidents and frozen lanes. + +### Scenario reliability + +- success/timeout/failure rate by scenario group; +- checkpoint latency distributions; +- first failure and last success; +- recent failure clusters by error tag; +- baseline versus candidate. + +### Inference and package fidelity + +- Drive exchange outcomes and unexpected/unused work; +- canonical protocol coverage; +- native versus fallback runtime counts; +- contract fingerprint changes; +- recorded/live provider freshness; +- canonical error/retry coverage. + +### Soak and capacity + +- lane resource series against attempt count and age; +- database/artifact growth; +- process restarts and controller reconnects; +- recovery duration; +- trend comparison with control lane. + +### Evidence health + +- artifact failures and redaction failures; +- bundle completeness; +- storage bytes and retention backlog; +- reproduction rate and shrink duration. + +## Triage Workflow + +1. Open the immutable failed attempt, not only the alert summary. +2. Confirm target, lane generation, scenario/campaign, and deliberate faults. +3. Inspect the normalized timeline and last checkpoint. +4. Compare UI, server projection, inference/tool trace, and process health. +5. Check whether a retry/replay exists; do not let it overwrite first failure. +6. Classify with evidence and confidence. +7. Link issue/owner and optional known-failure signature. +8. Reproduce in an ephemeral lane using recorded inputs. +9. Add a regression case or update harness contract after resolution. + +Review annotations are append-only audit records. Original machine evidence is +never edited. + +## Failure Signatures + +Group failures using stable fields: + +- scenario/campaign and phase; +- typed error tag; +- failed invariant ID; +- last checkpoint; +- process exit role/code; +- normalized top stack frames where safe; +- provider protocol/runtime path; +- expected fault kind and phase; +- output-started state. + +Do not group solely by free-form message. Similar-looking timeout messages can +have unrelated causes, and dynamic IDs fragment groups. + +Signatures support deduplication, not automatic dismissal. Every occurrence +remains an attempt record. + +## Retention + +Example policy: + +| Data | Success | Failure | +| --- | --- | --- | +| Attempt/checkpoint metadata | 90 days or longer | 1 year or issue lifetime | +| Aggregate metrics | 13 months | Same | +| Structured logs | 7–14 days | Failure excerpts 90 days | +| Frames | Sampled, 7–30 days | 90 days or issue lifetime | +| PNGs | Derived/sampled | 90 days | +| Recordings | Sampled or campaign-specific | 30–90 days | +| Property traces | Summary only | Original and minimized, 1 year | +| Provider cassettes | Version-controlled when approved | Version-controlled | +| Restricted live content | Minimal, shortest practical | Policy-controlled | + +Deletion jobs produce auditable counts and never follow artifact-supplied paths +outside the configured store. + +## Redaction + +Redact at collection and verify before publication. + +Sensitive classes: + +- credentials, tokens, cookies, and authorization headers; +- provider/account/project/deployment identifiers; +- raw prompts and outputs from non-synthetic sources; +- tool arguments/results and file contents; +- user paths and environment variables; +- HTTP bodies that may echo secrets; +- repository remotes and Git credentials. + +Use deterministic placeholders where comparison needs identity consistency. +Store redaction status and redactor version in artifact metadata. + +A redaction failure quarantines the artifact and marks evidence degraded. It +does not upload the unverified file to the normal review surface. + +## Backpressure and Failure of Observability + +Telemetry must not take down the product-under-test or fill the disk. + +- metrics/log exporters use bounded queues; +- artifact capture has per-attempt byte/time limits; +- low-priority success artifacts are dropped before failure manifests; +- run-record terminalization has a durable local fallback when the central + store is temporarily unavailable; +- a full artifact store stops new expensive campaigns while preserving smoke + metadata; +- exporter failure emits a local health signal and evidence-degraded outcome; +- raw log rotation has hard limits. + +The system alerts separately when observability is degraded. + +## Clock and Duration Rules + +- wall time identifies records and supports cross-system correlation; +- monotonic time measures durations within one process; +- store timestamps in UTC; +- record configured timezone only for human scheduling; +- never derive duration by subtracting wall clocks across hosts without clock + quality evidence; +- retain schedule evaluation time so delayed work can be explained. + +## Testing Observability + +Test: + +- Schema round trips and version decoding; +- exactly-once terminalization under worker/reconciler races; +- append-before-execute recovery; +- cardinality allowlists; +- redaction positive and adversarial cases; +- artifact manifest completeness and digest verification; +- retention target resolution safety; +- alert rules including absence of heartbeat; +- trace parenting across scoped fibers; +- exporter/backpressure failure behavior; +- timeline normalization with skewed timestamps; +- UI capture failure without loss of primary outcome. + +## Acceptance Criteria + +Observability is ready when: + +- every scheduled work item and started attempt reaches a durable terminal or + reconciled state; +- every attempt records exact revisions, configuration, lane generation, and + scenario/campaign identity; +- freshness alerts detect silent absence of useful work; +- metrics have reviewed bounded dimensions; +- failure bundles include a manifest and report missing evidence explicitly; +- failed attempts are immutable and retries/replays are linked separately; +- traces use meaningful Effect operation boundaries and one edge exporter + layer; +- raw prompt, model, tool, credential, and path content is absent from default + telemetry; +- artifacts are redacted and digest-verified before normal publication; +- an observability failure cannot silently turn a failed attempt into a pass; +- dashboards distinguish product reliability, harness health, infrastructure, + and provider/package fidelity. diff --git a/docs/continuous-testing/09-deployment-and-operations.md b/docs/continuous-testing/09-deployment-and-operations.md new file mode 100644 index 0000000..af45bc6 --- /dev/null +++ b/docs/continuous-testing/09-deployment-and-operations.md @@ -0,0 +1,670 @@ +# Deployment and Operations + +This document defines how continuous verification runs as a 24/7 service. The +first deployment favors a simple, inspectable topology over a large distributed +platform. It can scale later without changing attempt, lane, and evidence +contracts. + +The topology is implementable entirely in TypeScript/Effect. A hybrid Elixir +implementation is evaluated separately in [Elixir control-plane +option](./09-elixir-control-plane-option.md); it changes control-plane runtime, +not Drive, provider, scenario, or evidence contracts. + +## Operational Goals + +- useful smoke work completes continuously; +- a failed or hung OpenCode process cannot hide from supervision; +- every process and resource has one owner; +- target revisions and configurations are immutable within a lane generation; +- deployments and lane recycling do not erase unresolved failures; +- the system can be drained, upgraded, and restored predictably; +- real-provider credentials and network policy remain isolated from normal + deterministic lanes; +- operators have concrete runbooks rather than needing to understand all source + code during an incident. + +## Recommended First Topology + +Run one control-plane process and a small number of colocated lane workers on a +dedicated host or container runtime. + +```text +host / node + control plane + scheduler + reconciler + freshness evaluator + run-record writer + artifact uploader + + lane queued-1 + lane worker + OpenCode server + fresh TUI per attempt + Drive backend/frontend controllers + isolated project + data directories + + lane reactive-1 + same process family, separate ports/state + + ephemeral worker pool + finite OpenCodeDriver.use runs + provider contract processes + replay/shrink jobs +``` + +The control plane and lane workers may initially be one deployable application +with separate supervised processes. They remain logical services with separate +health and failure domains. + +## Why Colocate a Lane + +OpenCode Drive currently manages local child processes, loopback endpoints, +terminal clients, artifact directories, and simulation connections. Keeping one +lane's worker, server, controllers, and TUI on the same node avoids premature +remote-control protocol design. + +Scale by adding lane units. Do not let unrelated workers attach to one backend +controller or share one ordinal LLM response queue. + +## Process Tree + +Every lane generation records and owns an explicit process tree: + +```text +lane supervisor + lane worker + opencode server + opencode TUI A + optional TUI B for declared scenario + renderer/terminal children owned by TUI +``` + +The supervisor records PID plus a stronger generation identity such as process +start time and lane token. A recycled PID must not be mistaken for the old +process. + +All subprocesses receive: + +- lane and generation correlation; +- explicit isolated data/config/cache paths; +- loopback port allocation; +- target revision/binary path; +- bounded environment allowlist; +- output redirection with rotation; +- shutdown deadline. + +## Supervisor Responsibilities + +The supervisor: + +- starts control-plane and lane processes; +- restarts only according to component policy; +- records crash loops and exit reasons; +- enforces resource limits; +- delivers termination then bounded force-kill to explicit children; +- starts no new work during drain; +- verifies that stopped generations release ports and processes; +- remains independent of OpenCode health. + +Use the platform supervisor available in the chosen environment: + +- systemd or launchd for a dedicated host; +- Kubernetes Deployments/StatefulSets or Jobs for a cluster; +- a container runtime restart policy plus an external freshness evaluator; +- a process supervisor for local development only. + +An application-level Effect supervisor coordinates fibers and attempts, but it +does not replace an operating-system-level process supervisor. + +## Service Boundaries + +Suggested Effect services: + +- `WorkQueue` stores and leases work; +- `Scheduler` creates cadence and revision-triggered work; +- `LaneRegistry` tracks generations, capabilities, health, and leases; +- `LaneRuntime` owns one process/state topology; +- `AttemptCoordinator` runs one bounded attempt; +- `RunStore` writes durable work and attempt records; +- `ArtifactStore` publishes verified evidence; +- `TargetResolver` resolves immutable OpenCode/Drive artifacts; +- `HealthEvaluator` computes lane readiness and freshness; +- `AlertSink` delivers notifications; +- `Clock` and configuration services support deterministic tests. + +Construct live layers once at the process edge. Each lane generation and attempt +runs in a child scope with finalizers. + +## Storage Layout + +Use separate declared roots. Example conceptual layout: + +```text +state/ + control/ + queue.db + run-buffer/ + lanes/ + / + generations// + opencode-data/ + project/ + logs/ + runtime/ + retained-failures/ + artifacts-staging/ + target-cache/ +``` + +The actual paths are deployment configuration. No cleanup command derives a +recursive target from an unset variable, home directory, workspace root, or +glob. + +Separate: + +- control-plane durable state; +- lane-persistent OpenCode state; +- per-attempt temporary state; +- artifact staging; +- immutable target cache. + +## Databases + +The run store may begin with SQLite on one host if: + +- one process owns writes or locking is well understood; +- backups and integrity checks are configured; +- the control plane can recover non-terminal attempts after restart; +- artifacts remain outside database blobs; +- migration version is recorded and tested. + +Move to a network database when multiple control-plane replicas or nodes need +concurrent leases. Do not introduce it only to appear production-like. + +Each persistent lane has its own OpenCode database or explicitly isolated +database namespace. A shared database across lanes would make state attribution +and chaos safety much harder. + +## Artifact Storage + +Use local staging plus an object store or durable filesystem. + +Upload sequence: + +1. write artifact to attempt-owned staging path; +2. close and compute digest; +3. redact/validate according to artifact kind; +4. write immutable destination object; +5. verify digest/size; +6. create artifact record; +7. delete staging copy after policy permits. + +If the central store is unavailable, preserve the manifest and essential +artifacts locally within a quota. Stop expensive campaigns before overwriting +unuploaded failure evidence. + +## Port Allocation + +Every lane owns an explicit port lease covering: + +- OpenCode server/API; +- frontend simulation endpoint; +- backend inference/tool simulation endpoint; +- any preview or fixture service declared by the scenario. + +Allocate from a configured loopback range or let the runtime assign available +ports and persist the resolved values. Validate ownership before launch. Release +only after the process generation stops. + +A port in use by an unknown process marks the lane unhealthy; never terminate +the unknown process automatically. + +## Target Artifacts + +Do not mutate a checked-out target under a running lane. Resolve one immutable +artifact per revision: + +- clean Git worktree pinned to commit; +- built package/binary digest; +- container image digest; +- or a development command plus exact checkout commit for local-only use. + +Attempt records include both requested revision and resolved commit/artifact +digest. + +For `../opencode:v2`, the resolver fetches or observes the ref, resolves it to a +commit, and builds a new target artifact. Existing lane generations continue on +their pinned artifact until drained. + +## Configuration + +Configuration is schema-validated at startup and versioned by digest. + +Groups: + +- run-store and artifact endpoints; +- target repository/ref/build command; +- lane definitions and capacities; +- scheduler cadences and priorities; +- scenario/campaign policy; +- timeouts and retry schedules; +- resource and retention quotas; +- telemetry exporters; +- alert routing; +- network/credential profiles; +- maintenance and rollout policy. + +Reject unknown fields for safety-critical configuration. Secrets are references +to a secret provider, not values serialized into the configuration snapshot. + +Config reload policy: + +- scheduling/alert thresholds may update atomically when supported; +- lane topology, target, network, or credential profile changes create a new + lane generation; +- an active attempt keeps the configuration digest with which it started; +- invalid reload leaves the previous valid configuration active and alerts. + +## Deployment Profiles + +### Local development + +- one ephemeral worker; +- optional one persistent lane; +- local SQLite and artifact directory; +- no real-provider credentials by default; +- visible TUI optional; +- manual dashboard or CLI status. + +### Dedicated single host + +- one control plane; +- two or more deterministic lanes; +- OS supervisor; +- local run database with backups; +- remote artifact/telemetry destination; +- container or OS-level resource limits; +- independent external freshness check. + +This is the recommended first 24/7 production-like deployment. + +### Cluster + +- replicated/stateless schedulers with leader or idempotent scheduling; +- network work/run store; +- one pod/unit per persistent lane; +- Jobs for ephemeral attempts; +- object artifact store; +- node pools separated by credential/network profile; +- pod disruption and rollout controls. + +Adopt only when capacity, isolation, or availability needs justify it. + +## Lane Bootstrap + +One generation boot sequence: + +1. acquire lane-generation record; +2. resolve and verify target artifact; +3. prepare isolated paths and ports; +4. write configuration and fixture; +5. launch OpenCode server in simulation mode; +6. attach Drive controllers; +7. launch a fresh TUI; +8. verify handshakes and declared capabilities; +9. run bootstrap smoke; +10. publish `ready` only after smoke passes; +11. close bootstrap TUI and admit normal work. + +Process liveness alone never makes a lane ready. + +## Lane Health + +Health has separate dimensions: + +```text +processHealth +protocolHealth +controlHealth +workFreshness +resourceHealth +evidenceHealth +``` + +Lane states: + +- `starting`; +- `ready`; +- `leased`; +- `draining`; +- `experimenting`; +- `degraded`; +- `frozen`; +- `recycling`; +- `stopped`. + +A lane may have live processes but be `degraded` because its controller is +detached or no work has completed recently. + +## Heartbeats + +The control plane writes a heartbeat independent of scheduled work. Each lane +worker writes: + +- generation; +- state; +- active attempt if any; +- process health summary; +- last successful control probe; +- last attempt completion; +- resource summary; +- timestamp and monotonic sequence. + +An external evaluator checks both. Heartbeats are leases with expiry, not +unbounded rows. + +## Scheduling + +Use a durable work queue for at-least-once delivery and attempt idempotency. + +Scheduler inputs: + +- fixed cadence; +- target revision change; +- package/lockfile impact; +- replay/shrink request; +- alert-triggered diagnostics; +- maintenance windows; +- live-provider budget windows. + +Avoid synchronized bursts by applying controlled schedule jitter. Priority and +backpressure rules are defined in [Bot orchestration](./03-bot-orchestration.md). + +## Rollout of a New OpenCode Revision + +Use generation replacement: + +1. resolve candidate ref to immutable commit; +2. build and verify artifact; +3. create candidate lanes without touching baseline lanes; +4. run protocol handshake and bootstrap suite; +5. run essential deterministic journeys; +6. run impacted provider/package contracts; +7. start candidate soak while baseline remains available; +8. promote candidate as active target after gate policy; +9. drain old lanes; +10. retain old artifact until failure/reproduction policy permits deletion. + +A broken candidate cannot prevent baseline health signals. + +## Drive and Catalog Rollout + +Drive, catalog scenarios, and control-plane code also have revisions. Treat a +harness rollout as a canary: + +- run old and new harness against the same known target where possible; +- compare attempt outcomes and protocol compatibility; +- validate scenario-definition and response-plan digests; +- verify evidence and redaction; +- move a subset of lanes first; +- retain the previous deployable artifact for rollback. + +A harness rollout that changes many failures should initially classify them as +potential harness drift until compared against the old runner. + +## Protocol Compatibility + +On lane bootstrap, record: + +- frontend and backend handshake profiles; +- OpenCode server name/version; +- protocol capability set; +- Drive compatibility policy and result; +- unsupported optional capabilities; +- exact copied protocol schema version/digest when available. + +CLI `--command.ui.*` names and payloads stay identical to the canonical OpenCode +frontend protocol. Backend model control remains in scripts and Effect programs. + +If a canonical protocol change is required, update OpenCode first, copy it into +Drive, update the CLI directly, and run both repositories' protocol tests. Do +not add aliases to smooth over incompatible versions. + +## Draining + +Drain sequence: + +1. mark component/lane `draining` and stop new leases; +2. wait for active attempt to reach a safe boundary up to a deadline; +3. interrupt according to attempt type and record cancellation; +4. collect required evidence and terminalize/reconcile; +5. settle Drive responses/tools; +6. close TUI clients; +7. stop server/controllers; +8. verify process/port release; +9. close scopes and storage handles; +10. mark stopped. + +System shutdown is not a successful attempt outcome. + +## Restart Policy + +### Control plane + +Restart automatically on unexpected exit, then reconcile leases and attempts. +Crash loops alert and stop repeated rapid restart according to supervisor +policy. + +### Lane worker + +Restart may create a new lane generation. Do not adopt the old process tree by +guessing. First inspect explicit runtime metadata and kill only verified +lane-owned remnants. + +### OpenCode server + +Ordinary unexpected exit fails the active attempt and freezes or recycles the +lane according to evidence policy. A chaos experiment may expect supervision to +restart it while preserving experiment correlation. + +### TUI + +An unexpected exit fails that attempt. Start a new TUI for the next attempt only +after cleanup; do not retry the same state-changing UI action invisibly. + +## Reconciliation After Control-Plane Restart + +On startup: + +1. load non-terminal work, attempts, and lane generations; +2. expire stale leases using persisted expiry and current time; +3. query lane heartbeats/process identity; +4. ask live workers for active attempt where protocol supports it; +5. terminalize orphan attempts as interrupted/infrastructure or leave them + pending only under a bounded recovery rule; +6. requeue work as a new linked attempt when policy permits; +7. detect duplicate generation ownership and quarantine affected lanes; +8. resume schedules from durable last-enqueue markers. + +At-least-once work delivery does not imply exactly-once UI side effects. +Attempt IDs and append-before-execute records prevent silent duplication. + +## Backups and Restore + +Back up: + +- run/queue database; +- configuration history; +- artifact manifests and object-store durability metadata; +- approved provider cassettes through source control; +- optional retained persistent-lane snapshots for unresolved failures. + +Do not treat ordinary lane state as the only copy of critical evidence. + +Restore drills verify: + +- database integrity and migrations; +- work/attempt reconciliation; +- artifact link/digest validity; +- scheduler does not enqueue an uncontrolled backlog; +- secrets are re-resolved, not restored from plaintext backup; +- a bootstrap smoke completes after restore. + +## Capacity Planning + +Measure per lane: + +- CPU/RSS at idle and under each workload profile; +- process and handle count; +- database and artifact growth; +- attempts/hour; +- median and tail scenario duration; +- property shrink concurrency; +- provider-contract parallelism; +- build/target cache size. + +Reserve capacity for: + +- one baseline lane during candidate rollout; +- one frozen failure without immediate deletion; +- an ephemeral replay/shrink worker; +- telemetry/artifact buffering during transient outages. + +Admission control rejects or delays work before host pressure destabilizes all +lanes. + +## Cost Controls + +Deterministic simulation cost is host/storage. Real-provider lanes add monetary +cost. + +- per-call token/output limits; +- per-attempt and daily budgets; +- provider/model allowlist; +- concurrency cap; +- circuit breaker after repeated provider failures; +- global kill switch independent of deployment; +- cost estimates in attempt records; +- alerts before hard budget exhaustion; +- no fallback to a more expensive provider unless the experiment explicitly + tests it. + +## Maintenance + +Scheduled tasks: + +- lane recycle by age; +- database integrity and backup; +- artifact retention and staging cleanup; +- target/build cache pruning; +- provider cassette age report; +- coverage manifest refresh; +- dependency and base-image updates; +- restore drill; +- security credential rotation; +- alert-route test. + +Maintenance creates run records or audit events. It does not silently suppress +freshness alerts; declared maintenance windows are visible to alert evaluation. + +## Operator Commands + +The control plane may expose operational commands such as: + +- list lane health; +- drain/freeze/recycle one lane; +- enqueue a registered scenario/campaign; +- replay an attempt; +- acknowledge/classify a failure; +- enable/disable a scheduler policy; +- activate real-provider global kill switch; +- inspect artifact manifest; +- show exact target/configuration. + +These are control-plane operations, not additions to OpenCode's simulation +frontend CLI. They operate on stable IDs, require authorization, and write an +audit record. + +## Runbooks + +### No successful smoke + +1. Check independent control-plane and lane heartbeat age. +2. Inspect queue age and active leases. +3. Determine whether all lanes share target/config/harness revision. +4. Run one ephemeral bootstrap smoke on the last known-good target. +5. Freeze the first relevant failure evidence. +6. Roll back harness/config only with evidence that it caused the outage. +7. Restore at least one known-good deterministic lane. + +### Lane heartbeat stale + +1. Verify supervisor/process state from outside the worker. +2. Check host resource and disk state. +3. Avoid killing by process name; resolve recorded generation identity. +4. Preserve logs/runtime metadata. +5. mark active attempt interrupted through reconciliation; +6. start a new generation and bootstrap smoke. + +### Lane alive but work hung + +1. Inspect active attempt phase and deadline. +2. Check LLM/tool pending summaries and controller attachment. +3. Collect current frame/session/process evidence. +4. Let configured attempt timeout/interrupt execute. +5. Freeze if cleanup does not settle. +6. reproduce ephemerally before recycle where possible. + +### Artifact store unavailable + +1. Stop visual/property/soak campaigns that produce large evidence. +2. Continue essential smoke metadata only if the durable local buffer is safe. +3. Monitor local quota. +4. Restore store and upload digest-verified backlog. +5. alert on any evidence degradation or discarded sampled success artifact. + +### Candidate regression + +1. Keep baseline lanes active. +2. compare exact scenario/config/response-plan inputs; +3. run ephemeral replay on baseline and candidate; +4. run impacted package contracts when inference-related; +5. block promotion, not the baseline service; +6. link minimized evidence to the source change. + +## Disaster and Safety Conditions + +Immediately stop admission when: + +- filesystem target validation fails; +- unknown processes occupy lane ports or paths; +- deterministic lane attempts real network egress; +- secret/redaction policy fails broadly; +- run database cannot durably terminalize attempts and local fallback is full; +- host resource limits threaten the control plane; +- duplicated lane-generation ownership is detected. + +Prefer a visible outage over unsafe untracked work. + +## Acceptance Criteria + +The deployment is ready for unattended 24/7 operation when: + +- control-plane and lane processes are supervised outside OpenCode; +- every lane generation owns explicit processes, paths, ports, target, and + configuration; +- bootstrap smoke, not process liveness, gates readiness; +- stale heartbeat and stale successful-work alerts are externally evaluated; +- non-terminal attempts reconcile after control-plane restart; +- rollout creates new immutable target generations and preserves baseline; +- drain and recycle verify all child processes, ports, and scoped resources are + released; +- failed lanes preserve evidence before recycle; +- storage, artifact, target-cache, resource, and provider-cost quotas are + enforced; +- backup restore and alert routing are tested; +- operational commands are audited and do not modify the canonical simulation + CLI protocol; +- at least one last-known-good deterministic lane can be restored without + depending on the failing candidate. diff --git a/docs/continuous-testing/09-elixir-control-plane-option.md b/docs/continuous-testing/09-elixir-control-plane-option.md new file mode 100644 index 0000000..e50b94b --- /dev/null +++ b/docs/continuous-testing/09-elixir-control-plane-option.md @@ -0,0 +1,427 @@ +# Elixir Control-Plane Option + +This document evaluates using Elixir/OTP for the 24/7 control plane while +keeping OpenCode Drive and provider-contract execution in TypeScript/Effect. + +## Recommendation + +Elixir is a good fit for the **outer operational service**. It is not a good +replacement for the inference simulator, provider-package harness, canonical +OpenCode protocol client, or catalog scenarios. + +The only Elixir architecture recommended here is therefore a hybrid: + +```text +Elixir/OTP control plane + scheduling, durable jobs, supervision, heartbeats, API, alerts + | + versioned worker contract + | +Bun/TypeScript workload workers + Drive, OpenCode protocol, scenarios, provider packages, Effect scopes + | + v + OpenCode server/TUI/provider transports +``` + +Do not rewrite `packages/drive` or `../opencode/packages/ai` in Elixir. That +would duplicate the exact TypeScript contracts we need to test. + +Between Erlang and Elixir, choose Elixir unless the team already operates an +Erlang codebase. Both use BEAM/OTP; Elixir offers a more approachable language, +Mix, Ecto, Phoenix, and the surrounding application tooling needed here. + +## What Elixir Would Buy Us + +### Explicit supervision trees + +OTP supervisors define child start, shutdown, restart, and restart-intensity +policy. A `DynamicSupervisor` can own the dynamically changing set of lane +coordinators, while ordinary supervisors own durable services such as the +scheduler, log ingestor, and alert evaluator. See the official +[`Supervisor`](https://hexdocs.pm/elixir/Supervisor.html) and +[`DynamicSupervisor`](https://hexdocs.pm/elixir/DynamicSupervisor.html) +documentation. + +### Cheap isolated coordinators + +One BEAM process can represent each: + +- lane generation; +- active attempt; +- live log subscription; +- heartbeat evaluator; +- provider-bot status aggregator. + +Those processes isolate control-plane failures and communicate through +messages. Durable truth still belongs in the database. + +### Mature durable scheduling + +Oban can provide database-backed scheduled work, queues, priorities, retryable +jobs, cancellation, and periodic insertion. Its queue concurrency is useful for +separating smoke, provider-contract, replay, property, and live-provider work. +See the official [Oban](https://hexdocs.pm/oban/Oban.html) and [queue +documentation](https://hexdocs.pm/oban/defining_queues.html). + +Oban uniqueness is insertion-time deduplication, not a guarantee that matching +jobs never execute concurrently. The verification design still needs attempt +IDs, lane leases, idempotent reconciliation, and explicit concurrency policy; +see Oban's [unique jobs +documentation](https://hexdocs.pm/oban/unique_jobs.html). + +### Operational web/API layer + +Phoenix can expose the fleet, attempt, artifact, and live-log APIs. Phoenix +LiveView could implement a real-time viewer, but the recommended initial design +keeps the existing React review UI in `apps/catalog` and feeds it from the +Elixir API/SSE or WebSocket endpoint. This preserves current UI ownership and +avoids rewriting the catalog. LiveView remains a reasonable later option; its +process-and-diff model is described in the official [LiveView +documentation](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html). + +## What Elixir Would Not Solve + +Elixir does not automatically solve: + +- provider behavior and error compatibility; +- OpenAI/Anthropic/Gemini/Bedrock framing; +- TypeScript package loading; +- Drive's queue-versus-serve inference semantics; +- OpenCode session assertions; +- OS process-tree containment; +- exactly-once UI actions; +- artifact redaction; +- useful-work freshness. + +Those remain explicit application contracts. + +Most workload processes are external Bun/OpenCode programs. OTP can supervise a +port owner, but an external process is not a BEAM process. Official Elixir +documentation warns that closing a port or crashing the VM does not necessarily +terminate a long-running external OS process. Use a container, process group, +or platform supervisor and verify descendants during cleanup; see +[`Port`](https://hexdocs.pm/elixir/Port.html) and +[`System.cmd/3`](https://hexdocs.pm/elixir/System.html#cmd/3). + +## Recommended Ownership Split + +### Elixir control plane owns + +- bot registry and enable/quarantine state; +- periodic scheduling and work insertion; +- durable job/attempt state; +- lane leases and generation registry; +- `DynamicSupervisor` for lane coordinator processes; +- worker process launch/monitoring; +- heartbeat and useful-work freshness; +- log-file indexing and live-tail fan-out; +- artifact metadata and retention jobs; +- alert evaluation and delivery; +- operator API and audit log; +- provider credential/network profile selection; +- global live-provider and chaos kill switches. + +### TypeScript/Effect worker owns + +- `OpenCodeDriver` and simulation connections; +- exact canonical `ui.*` protocol usage; +- queued/served inference handlers; +- controlled tool behavior; +- executable catalog scenarios and checkpoint assertions; +- V2 provider/package contract execution; +- programmable HTTP/WebSocket transport scripts; +- property model commands that directly use Drive/OpenCode types; +- frame, recording, and attempt-local artifact production; +- scoped cleanup of every resource it opens; +- a typed terminal result for the control plane. + +### `../opencode` owns + +- native provider routes and errors; +- actual AI SDK fallback behavior; +- provider-contract tests and fingerprints; +- canonical simulation behavior and protocol. + +## Suggested OTP Tree + +```text +Verification.Application + Verification.Repo + Oban + Verification.RunStore + Verification.BotRegistry + Verification.Scheduler + Verification.FreshnessEvaluator + Verification.ArtifactStore + Verification.LogIndex + Verification.Alerting + Verification.LaneSupervisor DynamicSupervisor + Verification.Lane one per active lane generation + Bun lane worker / OpenCode tree external, explicitly contained + Verification.WorkerSupervisor Task/Dynamic supervisor as needed + VerificationWeb.Endpoint +``` + +Do not put one permanent BEAM process under the tree for every logical provider +bot. Provider bots are durable definitions and jobs. A bounded worker queue +executes them. A process may temporarily represent an active bot attempt. + +## Lane Process + +One `Verification.Lane` process tracks only control state: + +```text +lane ID and generation +target/config digests +worker process identity +ports/paths/container identity +state: starting | ready | leased | draining | frozen | recycling +active attempt +last heartbeat and useful completion +resource summary +``` + +It does not hold the only copy of durable attempt or lane state. On restart it +reconstructs from the database and revalidates the external worker identity. + +The lane process serializes commands such as: + +- bootstrap; +- lease attempt; +- drain; +- collect/freeze evidence; +- recycle; +- stop. + +## Meaning of Recycle in OTP + +Recycling a lane maps naturally to supervised replacement, but it is not simply +“let the process crash and restart.” + +The coordinator first: + +1. marks the generation draining; +2. stops new work; +3. terminalizes or interrupts the active attempt; +4. asks the Bun worker to settle and collect evidence; +5. terminates the explicit external process/container tree; +6. verifies ports and paths are released; +7. persists the old generation terminal state; +8. starts a new generation child; +9. waits for bootstrap smoke before marking it ready. + +An unexpected crash follows a related recovery path, but remains recorded as a +crash rather than a planned recycle. + +## Worker Contract + +The hybrid succeeds only if the cross-language boundary is small and versioned. + +### Job input + +```text +WorkerJob + protocolVersion + attemptId + kind + exact target/harness revisions + config/fixture/plan digests and references + scenario/campaign/provider contract identity + deadline and budgets + artifact root + restricted correlation token +``` + +### Worker events + +```text +WorkerEvent + protocolVersion + source sequence + attempt/lane generation identity + timestamp and elapsed time + event type + phase/checkpoint + safe fields +``` + +### Terminal result + +```text +WorkerResult + protocolVersion + attemptId + outcome + typed failure summary + output-started state + checkpoint summary + artifact/log manifests + compatibility/fingerprint result + cleanup result +``` + +Use JSON-compatible values with a published schema and compatibility tests in +both languages. Do not expose Effect types, Elixir structs, closures, or stack +objects across the boundary. + +## Transport Between Elixir and Bun + +### Finite jobs + +The simplest reliable first contract is file plus process exit: + +1. Elixir writes immutable job JSON; +2. Elixir starts a Bun worker with explicit arguments, environment, workdir, + process containment, and result/log paths; +3. the worker writes structured JSONL and raw logs to attempt-owned files; +4. the worker atomically writes a terminal result; +5. Elixir monitors exit, validates the result, and reconciles missing results. + +This avoids treating arbitrary stdout from dependencies as a control protocol. + +### Persistent lanes + +After the finite contract is stable, use a loopback Unix socket or authenticated +loopback endpoint for commands, heartbeats, and live events. Keep files as the +durable fallback. + +Do not invent another version of OpenCode's simulation protocol. This is a +control-plane-to-worker contract above Drive. + +## Logs and Viewer + +The TypeScript worker remains the authoritative producer of attempt-local JSONL +and raw process logs because it owns the processes and semantic events. + +Elixir: + +- tails/indexes structured files by sequence; +- stores searchable safe rows; +- publishes live updates through Phoenix; +- detects gaps, truncation, and stale writers; +- retains manifests and artifact metadata; +- applies access policy. + +The existing React `apps/catalog` viewer consumes: + +- fleet/bot status API; +- attempt list/detail API; +- paginated log/timeline API; +- live SSE/WebSocket updates; +- authorized artifact URLs. + +This gives the operational benefit of Phoenix without discarding the existing +catalog UI. + +## Provider Bots in Elixir + +One logical provider bot becomes a durable bot-definition row plus scheduled +Oban jobs. + +Example: + +```text +provider.anthropic.messages.native + cadence: hourly deterministic, daily recording-age check + queue: provider_contract + concurrency key: target + contract ID + worker job: run target provider-contract report case set + freshness: last deterministic success < 2 hours +``` + +The Oban worker does not reimplement Anthropic. It launches the TypeScript +contract worker in `../opencode`, validates its terminal report, and updates bot +health. + +Provider credentials are supplied only to separate live-probe jobs and never to +deterministic contract jobs. + +## Error and Retry Boundaries + +Keep two retry systems from fighting each other: + +- OpenCode owns provider/session retry semantics under test; +- the TypeScript attempt worker records those behaviors but does not hide them; +- Oban may retry safe infrastructure preparation or an unstarted job; +- once a state-changing attempt starts, an Oban retry creates a new linked + attempt rather than reusing the identity; +- Elixir never interprets a worker process crash as success; +- a missing terminal result enters reconciliation. + +Configure Oban retries conservatively. The durable attempt record, not Oban's +job state alone, is the test result. + +## Effect Boundary + +Inside the Bun worker, keep the existing Effect design: + +- services for Drive, target, attempt logging, artifact writing, and contract + execution; +- live/test implementations supplied through layers at the worker entrypoint; +- one scoped attempt lifecycle; +- typed errors distinct from defects and interruption; +- `Schedule` only for declared polling/retry behavior inside that boundary; +- OpenTelemetry/log context derived from the Elixir-provided attempt identity. + +Elixir does not replace those guarantees; it supervises the worker runtime from +outside it. + +## Costs + +The hybrid adds: + +- a second language and build/deployment toolchain; +- a versioned cross-language protocol; +- likely Postgres/Ecto if using Oban conventionally; +- duplicate schema validation implementations or generated schemas; +- more integration and local setup; +- harder debugging when ownership is unclear; +- a need for maintainers comfortable with OTP. + +It is a poor choice if nobody intends to maintain Elixir or if the project will +remain a small local test script. + +## Comparison + +| Design | Advantages | Costs | Recommendation | +| --- | --- | --- | --- | +| TypeScript/Effect only | One language, direct Drive imports, fastest first attempt | More application-owned supervision/job durability | Best default MVP | +| Elixir control plane + TS workers | OTP supervision, durable scheduling ecosystem, strong operational API | Two runtimes and a real protocol boundary | Best long-term option if team knows/wants Elixir | +| Rewrite Drive/provider logic in Elixir | One control-plane language in theory | Duplicates TS contracts and no longer tests actual packages | Do not do | +| Raw Erlang control plane | Same OTP strengths | Less ergonomic application/UI ecosystem for this team/repo | Use only with existing Erlang expertise | + +## Decision Spike + +Before committing the full architecture, build one narrow vertical slice: + +1. an Elixir application with a supervisor, database, and one durable job; +2. one Bun `verify-once` worker running the deterministic smoke scenario; +3. versioned job/event/result JSON schemas; +4. JSONL tail/index into one attempt page in the existing catalog UI; +5. kill the Bun worker, Elixir lane process, and Elixir app at different phases; +6. prove reconciliation and external-process cleanup; +7. measure local setup and debugging cost. + +Decision gate: + +- choose the hybrid if recovery is materially simpler, the boundary stays + small, and at least one maintainer is comfortable owning it; +- remain TypeScript/Effect-only if the cross-language overhead dominates or the + team would depend on one Elixir specialist. + +## Acceptance Criteria for the Hybrid + +- Elixir never imports or reimplements Drive/OpenCode provider semantics; +- TypeScript workers can run independently from a job file for local replay; +- every cross-language value is schema-versioned and validated on both sides; +- durable attempt state survives either runtime restarting; +- external OpenCode/Bun process trees cannot become orphans silently; +- Oban retry cannot duplicate a state-changing attempt under the same attempt + identity; +- provider bots remain logical definitions rather than permanent processes; +- `apps/catalog` continues to own the OpenCode-specific review UI; +- JSONL/raw logs remain usable when Phoenix or the database is unavailable; +- an Effect-only deployment remains possible until the Elixir spike proves its + value. + diff --git a/docs/continuous-testing/10-security-and-safety.md b/docs/continuous-testing/10-security-and-safety.md new file mode 100644 index 0000000..4b15bf0 --- /dev/null +++ b/docs/continuous-testing/10-security-and-safety.md @@ -0,0 +1,530 @@ +# Security and Safety + +This document defines the security boundary for an always-on system that runs +an AI coding agent, synthetic prompts, tools, terminals, provider clients, and +fault experiments. + +The verification environment is test infrastructure, but it still executes +code and handles credentials. Treat it as a potentially hostile workload, not +as a trusted shell script that happens to run continuously. + +## Security Objectives + +- deterministic lanes cannot reach undeclared external networks; +- OpenCode and test tools cannot read or modify user or host data outside their + assigned workspace; +- real-provider credentials are available only to the narrow lanes that need + them; +- generated prompts, model output, tool input, fixtures, and cassettes cannot + inject control-plane commands; +- secrets and sensitive content do not enter ordinary telemetry or artifacts; +- a compromised lane cannot control the supervisor or other lanes; +- resource and cost abuse is bounded; +- destructive chaos actions target only explicit disposable resources; +- dependencies, target revisions, and test artifacts are attributable and + reviewable; +- operational mutations require authorization and produce audit records. + +## Threat Model + +Potentially untrusted inputs include: + +- the OpenCode target revision under test; +- provider package code and transitive dependencies; +- real model output; +- generated or recorded provider payloads; +- repository fixture content; +- plugin and MCP output; +- tool arguments selected by a model; +- cassettes and artifacts loaded from storage; +- scenario/config changes; +- malformed protocol frames; +- a compromised lane process. + +Protected assets include: + +- host filesystem and user data; +- source repositories outside the fixture; +- Git and package-registry credentials; +- provider API keys and cloud credentials; +- control-plane database and artifact credentials; +- other lanes and their state; +- production networks and services; +- review users who open logs, frames, or HTML-like artifacts; +- provider and infrastructure budgets. + +## Trust Boundaries + +```text +operator / CI identity + | + v +control plane ------ run/artifact stores + | + authenticated lease + | + v +lane supervisor boundary + | + v +OpenCode + TUI + Drive controllers + fixture tools + | + deny-by-default egress + | + +--> no network in deterministic lanes + +--> allowlisted provider endpoints in live lanes +``` + +The control plane never treats a lane-supplied path, PID, URL, artifact, or +classification as trusted without validation. + +## Lane Isolation + +The first single-host deployment should use containers or a comparably strong +OS sandbox per lane when practical. + +Each lane receives: + +- a dedicated unprivileged user/identity; +- a private writable workspace and OpenCode data directory; +- read-only target artifact where possible; +- no mount of the user's home, SSH directory, cloud config, Docker socket, or + control-plane state; +- explicit CPU, memory, process, descriptor, and disk quotas; +- loopback-only simulation endpoints; +- declared egress policy; +- a minimal environment allowlist; +- no host PID namespace or privileged capabilities; +- a separate temporary directory. + +Avoid sharing writable package caches between untrusted target executions. A +read-only verified cache or per-generation cache is safer. + +## Filesystem Safety + +All scenario and tool paths are resolved relative to an explicit fixture root. + +Before any write, restore, move, or delete: + +1. parse and normalize the requested relative path; +2. reject absolute paths and parent traversal; +3. resolve symlinks according to policy; +4. verify the final target remains inside the lane-owned root; +5. reject mount points and protected control directories; +6. operate on an explicit path, not a broad glob; +7. record the action and result. + +Never use the user's home directory, workspace root, `/`, or an unresolved +environment variable as a recursive cleanup target. + +Fixture reset owns a declared path set. It does not recursively replace the +entire lane directory, which also contains OpenCode state, logs, and runtime +metadata. + +## Process Safety + +Process termination uses explicit generation metadata: + +- lane ID; +- process role; +- PID; +- observed start time or process handle; +- executable/command digest where available; +- parent-child relationship. + +Before a destructive signal, revalidate that identity. If it no longer matches, +stop and mark the lane unsafe. Do not kill by fuzzy command substring or by +assuming a port owner belongs to the lane. + +Graceful termination precedes force kill. Force kill remains scoped to the +verified child process tree. + +## Network Policy + +### Deterministic lanes + +- deny external egress at the container/host firewall layer; +- allow loopback simulation endpoints and explicitly required local fixture + services; +- V2's simulated Effect HTTP client also denies unregistered destinations; +- DNS need not be available; +- fail and alert on any attempted undeclared destination. + +Application-level route denial is defense in depth, not a substitute for +network policy. + +### Provider contract lanes + +Deterministic programmable-transport and cassette replay tests deny all real +egress. Recording mode is an explicit, separately authorized operation with a +provider endpoint allowlist. + +### Live-provider lanes + +- separate identity and node/lane profile; +- allow only required provider/auth endpoints; +- block metadata-service access unless a tested cloud auth flow explicitly + needs it in an isolated environment; +- enforce model/provider allowlist; +- cap request count, tokens, time, and spend; +- no fallback to arbitrary endpoint from model or prompt content; +- record safe destination identity. + +## Credentials + +Use a secret manager or deployment-native secret provider. Credentials are: + +- injected only into the process/layer that needs them; +- never written to scenario config, attempt records, traces, or fixture files; +- scoped to test accounts/projects with minimal permissions; +- rotated independently; +- omitted from deterministic lanes; +- unavailable to generated tool commands; +- revoked when a lane image or dependency is suspected compromised. + +Prefer short-lived credentials. For cloud providers, isolate project/account and +apply hard service quotas. + +The control plane stores secret references and credential-profile names, not +secret values. + +## Environment Variables + +Build subprocess environments from an allowlist. Do not inherit the full +operator shell environment. + +Commonly sensitive variables to exclude include: + +- Git/SSH credentials; +- package registry tokens; +- cloud/provider credentials not selected for the lane; +- database and artifact-store credentials belonging to the control plane; +- desktop/session tokens; +- unrelated application secrets; +- proxy variables that bypass egress policy. + +Record variable names supplied to the lane, never sensitive values. + +## Tool Safety + +Drive-controlled tools should be the default in deterministic scenarios. + +Every tool registration declares: + +- stable name and schema; +- capability class; +- read/write/network/process effects; +- path or destination allowlist; +- timeout and output limit; +- whether user permission is expected; +- whether it is valid in generated campaigns. + +Generated models choose only from currently offered controlled tools. Tool +arguments are schema-decoded and then independently policy-validated. + +Shell-like testing uses a dedicated fixture command surface or sandbox. Do not +turn arbitrary model text into a host shell command. + +Tool output is bounded. Binary or huge output becomes a safe digest/summary and +restricted artifact when needed. + +## Prompt and Model-Output Safety + +Synthetic prompt strings are data. They do not interpolate into shell commands, +file paths, SQL, metric names, or artifact keys without escaping and validation. + +Real model output is untrusted even when the prompt is synthetic: + +- it cannot alter scheduler policy; +- it cannot select provider credentials or endpoints; +- it cannot widen tool permissions; +- it cannot choose host paths; +- it cannot emit HTML/terminal control that review tools execute; +- it cannot mark its own attempt passed. + +Pass/fail derives from registered assertions and run policy. + +## Terminal and Rendering Safety + +Frames and logs may contain control sequences or crafted Unicode. + +- store canonical frame data as structured pixels/cells, not executable + terminal replay when possible; +- escape terminal output in web review UI; +- serve downloaded artifacts with safe content type and disposition; +- sanitize filenames and never use prompt text as a path; +- render PNGs in an isolated process with resource limits; +- bound terminal dimensions and frame count; +- do not allow links in model output to become privileged control-plane + navigation without safe URL handling. + +## Artifact Security + +Artifacts have sensitivity classes: + +- `public-synthetic`: reviewed deterministic fixture content; +- `internal`: normal logs/frames with safe synthetic data; +- `restricted`: real provider content, HTTP context, environment or repository + detail; +- `quarantined`: redaction failed or content type is unsafe/unknown. + +Artifact storage keys are generated from IDs, not caller paths. Upload verifies +size, digest, type, redaction status, and quota. + +Review/download access follows sensitivity. Object-store URLs are short-lived +and audited for restricted content. + +## Redaction + +Redact: + +- authorization, cookies, API keys, signed URLs, and credential-like values; +- query parameters and JSON fields configured by provider; +- environment values; +- provider/account/project/deployment/request IDs according to policy; +- user paths and repository remotes; +- prompt, response, and tool content outside safe synthetic fixtures; +- error bodies that echo a request. + +Use multiple layers: + +1. avoid collecting content; +2. redact at source adapter; +3. redact artifact before publication; +4. scan complete artifact for credential patterns and known secret values; +5. quarantine on scan failure. + +Never log a secret merely to prove the redactor catches it in a live lane. Use +synthetic sentinel secrets in tests. + +## HTTP Recorder Safety + +The target recorder has secure defaults and scans cassettes. Operational policy +adds: + +- record only from dedicated test credentials/accounts; +- review cassette diffs before commit; +- allowlist necessary non-sensitive matching headers; +- stabilize account-specific paths with redaction functions; +- reject recordings containing unrecognized credential formats; +- never automatically overwrite an existing cassette; +- version cassette schema and recorder version; +- run replay with egress denied to prove it is self-contained; +- remove stale cassettes through explicit reviewed targets. + +Record/replay is not a license to preserve full production conversations. + +## Package and Supply-Chain Safety + +Target and harness builds should record: + +- Git commit and clean/dirty status; +- lockfile digest; +- resolved package versions; +- build artifact/image digest; +- base image/runtime version; +- dependency provenance available from the build system; +- source of dynamically loaded provider packages. + +Controls: + +- use lockfile-frozen installation; +- avoid runtime installation in persistent lanes where possible; +- isolate package caches; +- scan dependencies/images according to organizational policy; +- require review for new postinstall/native code; +- do not load a package specifier derived from model output; +- test loader failures and unexpected package exports; +- rotate credentials after confirmed dependency compromise. + +Provider packages run with the same suspicion as the OpenCode target. + +## Control-Plane Authorization + +Read-only status and broad synthetic artifacts may have wider access than +mutations. + +Require authenticated authorization for: + +- enabling/disabling schedules; +- launching live-provider or chaos work; +- draining/freezing/recycling lanes; +- changing network/credential profiles; +- replaying restricted attempts; +- accessing restricted artifacts; +- changing retention/quota policy; +- acknowledging or reclassifying failures. + +Every mutation records actor, request, resolved targets, before/after policy, +and outcome. + +## Configuration Safety + +Schema validation rejects: + +- unknown lane/network/credential profile names; +- paths outside approved roots; +- overlapping lane ports or writable roots; +- real-provider configuration in deterministic lanes; +- unbounded time, output, step, retry, or cost settings; +- destructive chaos without disposable resource declaration; +- artifact retention beyond policy; +- scenario IDs absent from the registry; +- unsupported protocol commands or aliases. + +Policy validation runs both at config load and immediately before the sensitive +operation, because external state may have changed. + +## Resource Limits + +Per lane and attempt, bound: + +- CPU and memory; +- process/thread/descriptor count; +- writable disk and database size; +- log and artifact bytes; +- terminal dimensions and frame count; +- request/output/tool payload bytes; +- model steps and tool invocations; +- wall-clock duration; +- concurrent sessions/clients; +- provider tokens, requests, and cost. + +Cross hard thresholds by stopping admission and safely terminating the narrowest +owner. Preserve an essential failure manifest before large artifacts. + +## Cost Abuse and Provider Safety + +Real-provider probes have: + +- daily and monthly hard budgets enforced outside the model call; +- per-provider/model maximum output; +- concurrency one initially; +- simple non-sensitive prompts; +- no user-supplied prompt endpoint; +- no arbitrary hosted tools; +- circuit breaker after authentication/quota/repeated transient errors; +- independent kill switch; +- usage reconciliation against provider billing where feasible. + +A compromised scenario cannot increase budgets or choose a premium model. + +## Chaos Safety + +Every destructive experiment validates: + +- dedicated disposable target; +- exact lane generation and resource identity; +- exclusive lease; +- steady state; +- blast-radius limit; +- abort thresholds; +- supervisor/control plane outside the target where required; +- cleanup mechanism tested without the fault; +- no real credentials or shared state unless explicitly necessary; +- evidence capacity. + +Disk corruption, disk-full, cgroup pressure, and broad network failure remain +disabled until run inside disposable isolated resources. + +## Data Retention and Deletion + +Minimize collection and expire data by class. Deletion jobs: + +- select objects from database IDs/prefixes generated by the system; +- validate store/root boundaries; +- delete in bounded batches; +- record counts and failures; +- preserve legal/security holds explicitly; +- never follow symlinks or artifact-supplied filesystem paths; +- retry idempotently; +- verify orphan staging data separately. + +Persistent lane state is not retained indefinitely merely because a test once +failed. Snapshot the minimum required evidence and apply the failure retention +policy. + +## Incident Response + +### Suspected secret exposure + +1. Stop affected lane and artifact publication. +2. Quarantine relevant artifacts/logs/cassettes. +3. Revoke and rotate the credential. +4. identify target/harness revision and access history; +5. scan storage for the exposed value and related formats; +6. remove through approved incident process; +7. add synthetic regression sentinel; +8. restore only after redaction and isolation are verified. + +### Unexpected network egress + +1. Block the lane profile at network policy. +2. Freeze attempt and capture safe destination/process evidence. +3. Revoke potentially exposed credentials. +4. determine whether application-level simulated network was bypassed; +5. audit other lanes with the same artifact; +6. require an explicit regression test before re-enable. + +### Filesystem escape attempt + +1. Stop the lane without following the requested target. +2. preserve normalized path, symlink, process, and policy evidence; +3. verify host/shared paths were not changed; +4. rotate sensitive credentials if readable data may have been exposed; +5. fix both application validation and container mount policy. + +### Runaway resource or cost + +1. Activate the narrow kill switch; +2. stop admission; +3. terminate explicit lane/provider work; +4. retain bounded manifests and counters; +5. reconcile provider usage and host resource impact; +6. lower limits or fix loop before restoring. + +## Security Tests + +Automate: + +- path traversal, absolute path, symlink escape, and mount-boundary tests; +- cleanup target validation with unset/hostile configuration; +- process PID reuse/identity mismatch; +- unknown port ownership behavior; +- egress denial and route miss; +- environment allowlist verification; +- fake credential sentinel scanning in every artifact format; +- malicious error body echoing request credentials; +- terminal escape and review UI encoding; +- oversized frame/log/body/tool output; +- untrusted artifact filename/content type; +- unauthorized control-plane mutation; +- expired artifact access; +- cost/step/time budget enforcement; +- chaos target mismatch abort; +- package loader with unexpected export and malicious specifier input; +- replay proving no real network use. + +Run security-policy smoke before declaring a new deployment profile ready. + +## Acceptance Criteria + +The system is safe for unattended operation when: + +- each lane runs with isolated identity, storage, processes, resources, and + network policy; +- deterministic and replay lanes have external egress denied; +- live credentials exist only in allowlisted live profiles with hard budgets; +- target and tool paths are normalized and proven inside explicit roots; +- process signals target revalidated generation identities; +- arbitrary model output cannot become a shell command, path, endpoint, + permission, scheduler policy, or pass verdict; +- default telemetry contains no prompt/model/tool content or secrets; +- every artifact is typed, size-bounded, redaction-verified, and access-classed; +- redaction failure quarantines rather than publishes; +- destructive chaos requires a disposable isolated target and exclusive lease; +- control-plane mutations are authorized and audited; +- lockfile, package versions, target commit, and build digest are recorded; +- incident kill switches work independently of the workload; +- restore and security regression tests are exercised regularly. + diff --git a/docs/continuous-testing/11-implementation-roadmap.md b/docs/continuous-testing/11-implementation-roadmap.md new file mode 100644 index 0000000..dbfc81e --- /dev/null +++ b/docs/continuous-testing/11-implementation-roadmap.md @@ -0,0 +1,746 @@ +# Implementation Roadmap + +This document turns the architecture into reviewable increments. Each milestone +must produce a usable result, tests, evidence, and an operational rollback. The +roadmap deliberately establishes run identity and logs before starting an +unattended loop: a 24/7 system that cannot explain its failures is only a log +generator. + +## Guiding Priorities + +1. Pin and describe exactly what is tested. +2. Make one finite attempt durable and reviewable. +3. Build provider/package fidelity around real code and fake transport. +4. Run logical provider bots continuously. +5. Add persistent OpenCode journey lanes. +6. Add generated state exploration, soak, and controlled chaos. +7. Add sparse real-provider drift checks last. + +## Repository Ownership + +### This repository + +`packages/drive` owns only generic capabilities: + +- simulation protocol client copied from canonical OpenCode; +- driver lifecycle; +- provider-neutral scripted output; +- generic tool control; +- frame/screenshot and recording primitives; +- compact generic run report; +- reusable generic logging hook only if multiple callers need it. + +`apps/catalog` owns OpenCode-specific continuous-verification behavior: + +- bot IDs and definitions; +- flow/scenario selection; +- provider/package coverage taxonomy; +- schedules and alert policy; +- attempt/run schemas specific to the application; +- evidence bundles and retention policy; +- review/log UI; +- reproduction entrypoints; +- dashboards and operator views. + +### `../opencode` + +The target repository owns: + +- provider protocol and route tests; +- native package entrypoint tests; +- ModelResolver and AI SDK fallback tests; +- canonical `AIError`, session projection, and retry tests; +- programmable transport helpers close to `packages/ai`; +- HTTP/WebSocket cassettes; +- canonical simulation protocol changes. + +Cross-repository scripts pin both revisions and collect results. Do not move +OpenCode provider behavior into Drive just to avoid coordinating two pull +requests. + +## Proposed Application Layout + +Names are provisional, but ownership should resemble: + +```text +apps/catalog/ + continuous/ + schema/ + bot.ts + work.ts + attempt.ts + log.ts + artifact.ts + bots/ + journeys.ts + providers.ts + properties.ts + soak.ts + scheduler/ + lane/ + runner/ + evidence/ + store/ + target/ + provider-contract/ + telemetry/ + scripts/ + verify-once.ts + verify-service.ts + verify-replay.ts + verify-export.ts + src/ + verification/ + FleetView.tsx + AttemptList.tsx + AttemptDetail.tsx + Timeline.tsx + LogViewer.tsx + ArtifactViewer.tsx +``` + +The first implementation may use fewer files. Extract only stable concepts; +avoid a directory per one-line wrapper. + +## Definition of Done for Every Milestone + +- Effect Schemas decode all durable/config inputs; +- resources and background fibers are scoped; +- expected failures are typed; +- interruption remains interruption; +- tests cover success, expected failure, defect/cleanup, and cancellation where + relevant; +- exact target/harness/config versions appear in output; +- no raw sensitive content enters default logs; +- documentation and runbook are updated; +- a rollback or disable mechanism exists; +- package/app ownership rules remain intact; +- no frontend simulation CLI alias or backend control command is introduced. + +## Milestone 0: Freeze the Contracts + +### Outcome + +One checked-in architecture set and one machine-readable target audit establish +what the first system will test. + +### Work + +- keep this documentation set as the design baseline; +- resolve local `../opencode:v2` to an immutable commit for every run; +- record that the reviewed local ref was + `c53f4cfb094bb87852d0c3c8e83933e902e81283`, while treating it only as the + planning snapshot; +- inventory V2 native protocols, package-like entrypoints, `AISDKNative` + mappings, dynamic fallback identities, canonical errors, retry classes, and + existing recordings; +- snapshot Drive frontend/backend protocol capabilities; +- define initial bot IDs and owners; +- choose local paths and quotas for development without hardcoding user home + paths in production configuration; +- decide the first deployment profile: dedicated single host is recommended. + +### Deliverables + +- `target-audit.json` generated for an exact OpenCode commit; +- provider/bot coverage manifest draft; +- configuration Schema and sample config; +- architecture decision record for local JSONL plus run store; +- risk register for real-provider credentials and fault experiments. + +### Acceptance + +- rerunning the audit on the same commit is deterministic; +- audit changes visibly when provider exports/mappings/packages change; +- every proposed bot points to real discovered targets or is marked planned; +- no source checkout is modified during audit. + +## Milestone 1: One Durable Finite Attempt + +### Outcome + +A developer can run one existing catalog journey and receive a durable attempt +record, structured logs, artifacts, and a terminal outcome. + +### Work + +- define `WorkItem`, `Attempt`, `CheckpointRecord`, `FailureRecord`, `LogEntry`, + `LogManifest`, and `ArtifactRecord` Schemas; +- implement a small local run store, likely SQLite or an append journal plus + indexed metadata; +- wrap one existing executable scenario rather than copying it; +- record exact OpenCode/Drive/catalog commits and configuration digests; +- emit JSONL for attempt, phases, checkpoints, LLM summaries, tools, processes, + evidence, and cleanup; +- retain current Drive and OpenCode raw logs; +- build a failure manifest with frame and bounded log excerpts; +- reconcile an attempt interrupted by killing the runner process in a test; +- expose a script such as `verify-once` that returns non-zero on failed or + inconclusive outcome. + +### Initial scenario + +Use a deterministic prompt-to-text smoke flow with: + +- fresh isolated OpenCode instance; +- one queued response; +- one final UI checkpoint; +- server projection assertion; +- frame on failure; +- Drive settlement. + +### Tests + +- successful attempt; +- UI wait timeout; +- unused and unexpected LLM response; +- OpenCode process exit; +- artifact capture failure with primary outcome preserved; +- cancellation during scenario; +- runner death after intent but before terminalization; +- log rotation/truncation at small test limits; +- secret sentinel absent from published bundle. + +### Acceptance + +- every started attempt reaches terminal state directly or via reconciliation; +- rerunning the script produces a new linked-independent attempt; +- the attempt can be diagnosed from its manifest and files; +- no existing catalog capture behavior changes. + +### Control-plane technology decision gate + +After this finite worker contract exists, run the vertical slice in [Elixir +control-plane option](./09-elixir-control-plane-option.md). This is the correct +decision point: before implementing the durable scheduler, but after the Bun +worker's job/event/result boundary is concrete. + +Do not choose Elixir by rewriting the finite worker. Compare an Elixir host and +an Effect-only host around the same worker and attempt fixtures. + +## Milestone 2: Log Viewer and Attempt Review + +### Outcome + +The finite attempt is understandable in one read-only `apps/catalog` view. + +### Work + +- add an attempt list with outcome, target, scenario/bot, duration, last + checkpoint, and evidence status; +- add an attempt detail header and normalized timeline; +- add virtualized structured/raw log panes with source, level, event, time, and + text filters; +- link timeline rows to frames, inference/tool summaries, and artifacts; +- add stable URL addressing for attempt and source sequence; +- show rotation, dropped rows, truncation, redaction, and approximate ordering; +- support local static bundle or local API first; +- add before/after comparison skeleton; +- keep capture catalog browsing intact. + +### Tests + +- render a successful and failed fixture bundle; +- filter by component/event/level; +- jump from failed checkpoint to log row and frame; +- malicious ANSI/HTML-like content is rendered inert; +- large log uses bounded DOM/rendering; +- missing/corrupt artifact is visible, not a page crash; +- restricted/quarantined artifact is not fetched; +- URL deep link restores filter and row. + +### Acceptance + +- a reviewer does not need multiple terminal windows to understand the sample + failure; +- raw JSONL remains downloadable and CLI-friendly; +- viewer is read-only and does not add OpenCode `ui.*` commands. + +## Milestone 3: Provider Contract Harness in V2 + +### Outcome + +The actual V2 inference code is tested against programmable transport, and its +observed behavior is stored as a fingerprint. + +### Work in `../opencode` + +- define a behavior-fingerprint Schema and normalization; +- generalize existing Effect HTTP test helpers only where reuse is real; +- execute native routes through actual `RequestExecutor` and protocol code; +- cover the shared HTTP status/body/header corpus; +- cover response read failure, partial output, cancellation, malformed frame, + incomplete stream, and tool-input assembly; +- capture typed `AIError`, request count, output-started, and events; +- assert session-facing projection and retry behavior in focused Core tests; +- index existing cassettes by provider/protocol; +- add a target command that emits a machine-readable report for orchestration. + +### First contracts + +1. OpenAI Chat, because Drive uses it end to end; +2. OpenAI Responses HTTP, because V2 directly selects it for + `@ai-sdk/openai`; +3. Anthropic Messages, because V2 directly selects it for + `@ai-sdk/anthropic`; +4. OpenAI-compatible Chat with explicit URL; +5. resolver and canonical error/retry matrix. + +### Work in this repository + +- run the target report as a finite app-owned work item; +- import only the report Schema/data, not target source modules; +- persist fingerprints/artifacts under attempts; +- show fingerprint diff and runtime path in the viewer; +- define logical provider bot profiles for the first contracts. + +### Tests + +- actual package path receives the request; +- real egress denied; +- synchronous construction throw remains distinct from typed stream failure; +- hang times out and interrupts transport producer; +- malformed stream shrinks or saves a minimal fixture; +- fingerprint IDs normalize while semantic change still diffs; +- package/lockfile change marks prior report stale. + +### Acceptance + +- no hand-authored provider package replica is used as the primary oracle; +- every first contract has valid, error, malformed, tool, and cancellation + coverage as applicable; +- canonical error and retry mappings are explicit; +- target report identifies exact commit, lockfile, package version, protocol, + and native/fallback path. + +## Milestone 4: Scheduler and Logical Provider Bots + +### Outcome + +Provider/package contracts run continuously with one visible bot status per +meaningful provider/protocol path. + +### Work + +- implement durable schedule entries and work queue; +- implement leases, deadlines, attempt creation, and reconciliation; +- register bots from the generated coverage manifest plus app-owned policy; +- run bots through a bounded ephemeral worker pool; +- expose bot states: healthy, failing, stale, blocked, disabled, quarantined; +- compute last success/completion age independently of worker liveness; +- add fleet/provider matrix view; +- alert when a bot becomes stale or its reviewed fingerprint changes; +- run impacted bots on target/lockfile change and rotate full matrix hourly or + nightly according to cost; +- implement backpressure and priority for replay versus routine rotation. + +### Initial bot profiles + +- `provider.openai.chat.native`; +- `provider.openai.responses-http.native`; +- `provider.anthropic.messages.native`; +- `provider.openai-compatible.chat.native`; +- `provider.resolver.matrix`; +- `provider.error-projection.session`. + +### Tests + +- schedule idempotency across restart; +- lease expiry and late worker result; +- one worker executes different bots without shared state; +- one bot failure does not stop others; +- worker outage turns bot stale instead of leaving green status; +- definition/config change creates new attempt identity; +- high-priority replay is fair and cannot starve freshness work; +- clean shutdown drains or terminalizes active work. + +### Acceptance + +- service can run unattended for 24 hours; +- every bot completes according to cadence or alerts as stale; +- no bot requires a permanently idle process unless isolation policy says so; +- fingerprint changes are reviewable and cannot silently become green. + +## Milestone 5: Always-On Deterministic Journey Bots + +### Outcome + +At least two persistent OpenCode lanes continuously run high-signal synthetic +user journeys. + +### Work + +- implement lane registry/generation lifecycle; +- add one queued lane and one reactive lane; +- pin target artifact, ports, paths, database, and configuration per generation; +- bootstrap through handshake, capability validation, and smoke; +- lease one attempt at a time initially; +- launch fresh TUI/session per ordinary attempt; +- adapt existing executable scenarios to attempt/checkpoint records; +- add freshness metrics and independent evaluator; +- implement drain, freeze, recycle, and post-recycle bootstrap; +- add process supervision outside OpenCode; +- preserve failure evidence before recycle. + +### First journeys + +- submit and complete text; +- reasoning then text; +- one controlled tool success; +- tool rejection/failure recovery; +- provider disconnect before output; +- provider disconnect after partial output; +- create/reopen session; +- server restart after completed persisted session. + +### Tests + +- controller attachment loss; +- queued and served modes cannot mix; +- TUI process crash; +- OpenCode process crash; +- lane worker restart and reconciliation; +- stale heartbeat/freshness alert; +- scheduled recycle; +- unexpected port owner abort; +- local artifact store outage/quota. + +### Acceptance + +- an essential smoke completes at least every configured few minutes; +- process-alive but no-useful-work condition alerts; +- persistent lane age/state is visible; +- clean ephemeral replay links to the persistent failure; +- one failed lane does not remove all target coverage. + +## Milestone 6: Scenario Registry Expansion + +### Outcome + +The existing catalog registry drives smoke, critical, recovery, and diagnostic +monitoring without duplicated scenario code. + +### Work + +- add app-owned monitoring metadata keyed by registered scenario ID; +- validate eligibility, lane type, timeout, fixture, evidence, and alert policy; +- add fixture profiles and deterministic reset digests; +- wrap ordered checkpoints with timing/evidence; +- migrate stable manual regression probes into registered monitored policy; +- add quarantine/known-issue status without rewriting failures as passes; +- add per-scenario reliability and checkpoint views; +- add exact reproduction spec and script. + +### Acceptance + +- no second copied journey DSL exists; +- every monitored scenario has bounded waits and postconditions; +- capture, reproduce, and monitoring use the same executable flow source; +- visual-only flows are not accidentally page-worthy monitors. + +## Milestone 7: Stateful Property Campaigns + +### Outcome + +The existing lifecycle property probe becomes a continuously scheduled, +replayable, shrinkable campaign. + +### Work + +- extract/reference a versioned model and command trace; +- replace or record uncontrolled chunk randomness; +- emit command intent before execution; +- collect bounded observation after each relevant transition; +- enforce prompt ownership, history, tool settlement, terminal, recovery, and + resource invariants; +- run discovery on persistent reactive lane; +- replay and shrink in ephemeral lanes; +- record semantic transition coverage; +- show original/minimized traces side by side. + +### Initial campaigns + +- prompt lifecycle; +- tool lifecycle; +- restart recovery. + +### Tests + +- generator precondition laws; +- bounded case budgets; +- trace round trip and direct replay; +- shrink validity; +- append-before-execute interruption; +- known seeded failures; +- cleanup after cancellation. + +### Acceptance + +- one failed invariant names command, pre-state, observation, and expected rule; +- seed plus explicit trace reproduces all harness-controlled choices; +- shrinking never destroys original evidence; +- transition coverage guides weights without mutating replay behavior. + +## Milestone 8: Soak and Safe Chaos + +### Outcome + +Persistent lanes detect resource/state drift and execute initial controlled +recovery experiments. + +### Work + +- sample process/database/resource health; +- run baseline conversation and client-reconnect soak profiles; +- compute absolute ceilings and observational trend/slope reports; +- implement exclusive chaos experiment lease/spec; +- add steady-state probe, trigger proof, recovery invariants, abort conditions, + cooldown, and cleanup; +- add supported Drive faults first; +- add explicit server/TUI/controller generation actions; +- implement freeze-before-recycle and post-recovery smoke; +- compare candidate and baseline soak profiles. + +### Initial experiments + +- provider disconnect before/after output; +- interruption during tool input/execution; +- TUI termination and replacement; +- server restart idle and after completed state; +- controller detach/reconnect; +- 12-hour baseline conversation soak. + +### Acceptance + +- each experiment has one declared fault and exact target; +- active attempt failure and recovery outcome are both visible; +- resource trend relates to lane age and completed workload; +- no destructive host/storage fault is enabled yet; +- failure evidence survives lane recycle. + +## Milestone 9: Full Provider Matrix and Differential Parity + +### Outcome + +Every discovered V2 provider/package/protocol path has a logical bot and an +explicit coverage status. + +### Work + +- expand bots to Responses WebSocket, Gemini, Vertex variants, Bedrock Converse + and Mantle, Azure, OpenRouter, xAI, and remaining fallback packages; +- split by semantic API/transport where required; +- run full protocol corpus once per shared protocol and representative subsets + per provider; +- add model construction/auth/endpoint/options cases per entrypoint; +- add native-versus-AI-SDK differential fingerprints for migration candidates; +- gate `ModelResolver` mapping changes on reviewed parity; +- fill recording gaps identified by target `packages/ai/STATUS.md`; +- expose known gap, owner, and review expiry. + +### Acceptance + +- generated inventory has no silently uncovered runtime path; +- each provider bot has cadence, freshness, owner, version, and current result; +- shared corpus reuse avoids uncontrolled duplicate cost; +- fallback-to-native switches are visible before rollout. + +## Milestone 10: Sparse Live Provider Drift + +### Outcome + +Budgeted live probes determine whether selected providers have drifted from +recordings and deterministic assumptions. + +### Work + +- create isolated credential/network profiles; +- implement provider/model allowlists and hard budgets; +- add global kill switch and circuit breakers; +- run minimal text and representative tool calls; +- refresh selected cassettes through explicit reviewed workflow; +- compare canonical fingerprint and optional fields; +- classify provider drift separately from OpenCode product failure; +- reconcile usage/cost; +- alert on probe age and credential/quota failure. + +### Acceptance + +- deterministic lanes still have no provider credentials/egress; +- live probes stop at budget and can be disabled independently; +- recordings are redacted/scanned/reviewed; +- a live provider outage cannot erase deterministic OpenCode health. + +## Milestone 11: Production Hardening + +### Outcome + +The system can be operated by someone other than its author. + +### Work + +- OS/container supervision and resource limits; +- backup/restore and reconciliation drill; +- external heartbeat/freshness evaluator; +- retention and safe deletion jobs; +- artifact/upload outage handling; +- credential rotation and secret-exposure drill; +- control-plane authorization and audit log; +- rollout/rollback of target and harness generations; +- baseline/candidate capacity reservation; +- incident dashboards and runbooks; +- quarterly chaos/security exercise; +- SLO review from observed healthy variance. + +### Acceptance + +- 7-day unattended qualification with controlled maintenance; +- restart/restore does not lose or duplicate attempt outcomes; +- alert routes are tested; +- all hard resource/cost/security limits are enforced; +- an operator follows documented runbooks to restore known-good smoke; +- unresolved failures retain their first evidence through rollout/recycle. + +## Cross-Milestone Test Strategy + +### Pure and Schema tests + +- configuration decoding; +- state-machine transitions; +- schedule decisions; +- fingerprint normalization/diff; +- redaction; +- failure classification; +- fixture/path validation; +- coverage-manifest derivation; +- log/timeline ordering. + +Use property tests for algebraic laws such as round trips, bounded generators, +idempotent reconciliation, and valid shrink traces. + +### Service tests + +Use Effect test layers for clock, queue/store, target resolver, artifact store, +transport, and alert sink. Test scoped lifecycle and cancellation, not only +successful return values. + +### Process integration + +- worker/control restart; +- child output and exit capture; +- port/path ownership; +- drain/force-stop deadlines; +- local file rotation/quota; +- static review bundle generation. + +### Real OpenCode integration + +Run from package directories according to repository guidance. Keep live target +cases narrow and evidence rich. Protocol changes require both OpenCode +simulation tests and the full Drive suite. + +## Rollout Strategy + +For every service milestone: + +1. run finite local qualification; +2. shadow without alerts/gates; +3. compare with existing manual/capture workflows; +4. enable recording and dashboards; +5. enable non-page notifications; +6. establish healthy variance; +7. enable release gates or pages only for stable high-signal bots; +8. keep global disable and last-known-good runner. + +Do not page on a new generated campaign until its harness and failure rate are +understood. + +## Initial Capacity + +Start small: + +- one control-plane process; +- one queued persistent lane; +- one reactive persistent lane; +- two to four ephemeral provider/replay workers; +- one local run database; +- local staged plus durable artifact store; +- no live provider bot until deterministic matrix is useful. + +Scale based on queue age, freshness, scenario duration, shrink load, and host +resource data. + +## First Four Pull Requests + +An actionable opening sequence: + +### PR 1: attempt and JSONL schemas + +- app-owned Schemas; +- one finite smoke wrapper; +- local attempt directory and manifests; +- tests with fixture-only fake store; +- no daemon yet. + +### PR 2: timeline and read-only viewer + +- terminal bundle normalization; +- attempt fixture pages in catalog; +- log filter/timeline/frame integration; +- security rendering tests. + +### PR 3: target provider report + +- changes in `../opencode` for first contract fingerprints/common error corpus; +- orchestration script here pins target and imports report artifact; +- provider result fixture in viewer. + +### PR 4: durable scheduler and provider bots + +- work queue, leases, reconciliation; +- first logical provider bot registry; +- finite worker pool; +- freshness status and 24-hour shadow run. + +This order produces value for the hardest inference/package problem before +introducing persistent OpenCode lane operations. + +## Decisions to Revisit With Evidence + +Do not decide these prematurely: + +- SQLite versus network run database; +- single host versus cluster; +- exact lane count and concurrency; +- remote log search backend versus run-store index; +- memory/resource slope thresholds; +- how many provider bots need dedicated processes; +- live-provider cadence and models; +- whether generic structured logging belongs in `packages/drive`; +- whether a canonical simulation protocol needs additional stable request + identity for concurrent routing. + +The initial implementation records the data needed to make these decisions. + +## Overall Acceptance + +The project has reached its intended first mature state when: + +- exact `v2` revisions run continuously under independent supervision; +- essential synthetic journeys and every meaningful provider/package contract + have fresh visible bot health; +- provider mocks execute actual package code behind controlled transports; +- persistent lanes catch accumulated-state failures and ephemeral lanes replay + them; +- stateful campaigns produce minimized, replayable invariant failures; +- soak and initial chaos experiments verify recovery safely; +- every attempt has append-only records, JSONL/raw log files, bounded artifacts, + and one useful review page; +- absence of useful work alerts independently of ordinary failures; +- deterministic work has no real network or credentials; +- target, harness, scenario, plan, fixture, package, and configuration versions + are attributable; +- failures remain failures even when retries or replays pass; +- operators can restore a last-known-good lane using documented runbooks; +- `packages/drive` remains generic and the canonical OpenCode simulation + protocol remains authoritative. diff --git a/docs/continuous-testing/README.md b/docs/continuous-testing/README.md new file mode 100644 index 0000000..fdd530d --- /dev/null +++ b/docs/continuous-testing/README.md @@ -0,0 +1,330 @@ +# OpenCode Continuous Verification + +This documentation defines a proposed always-on test environment for +OpenCode. The environment keeps real OpenCode processes running, exercises +them continuously with synthetic users, controls model output through OpenCode +Drive, checks observable invariants, and retains enough evidence to explain a +failure after the fact. + +The system combines several testing disciplines: + +- **Synthetic monitoring** continuously executes known user journeys against a + running environment. +- **Canary testing** verifies one exact OpenCode revision before or while it is + promoted. +- **Soak testing** leaves the system under realistic activity long enough to + expose leaks, stale state, and lifecycle drift. +- **Stateful model-based property testing** generates valid action sequences + and checks invariants after every transition. +- **Chaos testing** introduces controlled failures and verifies recovery. + +The short name used throughout these documents is **continuous verification**. +It describes the product as a whole without confusing it with any one testing +technique. + +## Status + +This is an architecture and implementation plan. It distinguishes existing +repository behavior from proposed behavior: + +- **Existing** means the capability is present in this repository now. +- **Proposed** means the capability belongs to the continuous-verification + system but has not been implemented yet. +- **Later** means the capability is intentionally outside the first usable + release. + +No document in this directory changes the canonical OpenCode simulation +protocol. In particular, model control remains in Effect programs and scripts; +it does not become a new set of CLI commands. + +## Why Build This + +Unit and integration tests answer whether a known case passes in a controlled +run. They are necessary, but they do not answer several operational questions: + +- Does a real OpenCode server remain useful after thousands of sessions? +- Can a TUI recover after the server disappears and returns? +- Do queued prompts, streaming output, permissions, forms, tools, and + interruptions remain consistent under unusual interleavings? +- Did a change increase time to first output or leave more work stuck in a + pending state? +- Can a failure be replayed with the same revision, seed, response plan, and + action trace? +- Will somebody be notified when no test has completed successfully for ten + minutes, even if no individual run emitted a clean failure? + +Continuous verification answers those questions by treating the test system as +a continuously operated service rather than a command run occasionally by a +developer. + +## Existing Foundation + +The repository already contains most of the execution substrate. + +| Capability | Current owner | How it is reused | +| --- | --- | --- | +| Isolated OpenCode server and project | [`packages/drive`](../../packages/drive) | One real server per test lane | +| Headless or visible TUI control | [`OpenCodeDriver`](../../packages/drive/src/driver/index.ts) | Synthetic user input and UI assertions | +| Simulated model output | [`driver/llm-controller.ts`](../../packages/drive/src/driver/llm-controller.ts) | Deterministic, reactive, and fault-injected inference | +| Runtime tool control | [`packages/drive/src/tool`](../../packages/drive/src/tool) | Delayed, failed, interrupted, and concurrent tools | +| Executable OpenCode journeys | [`apps/catalog/scenarios`](../../apps/catalog/scenarios) | High-signal synthetic monitoring flows | +| Ordered flow checkpoints | [`apps/catalog/catalog/flow.ts`](../../apps/catalog/catalog/flow.ts) | Assertion and timing boundaries | +| Revision and theme variants | [`capture-opencode-drive.ts`](../../apps/catalog/scripts/capture-opencode-drive.ts) | Baseline and candidate lane planning | +| Seeded lifecycle state machine | [`lifecycle-properties.ts`](../../packages/drive/test/manual/tui-regressions/lifecycle-properties.ts) | Stateful property campaign seed | +| Frames and recordings | [`packages/drive/src/frame`](../../packages/drive/src/frame), [`recording`](../../packages/drive/src/recording) | Failure evidence | +| Run artifact and compatibility report | [`driver/report.ts`](../../packages/drive/src/driver/report.ts) | Input to the richer attempt record | + +The missing product is the control and evidence layer around these primitives: +a scheduler, persistent lanes, typed run records, telemetry export, artifact +retention, dashboards, alerts, and operational supervision. + +## Target Architecture + +```text + CONTINUOUS-VERIFICATION CONTROL PLANE + + scenario registry scheduler run store artifact store alerts + | | ^ ^ ^ + | v | | | + +-----------> attempt coordinator ----------+------------+ + | + lease + scenario + revision + seed + | + +-----------------------+-----------------------+ + | | | + v v v + PERSISTENT LANE A PERSISTENT LANE B EPHEMERAL LANE + queued mock journeys reactive/property clean reproduction + OpenCode server OpenCode server OpenCodeDriver.use + fresh TUI per attempt fresh TUI per attempt + persistent DB/state persistent DB/state + | | + +----------- telemetry and evidence -----------+ +``` + +The control plane decides what should run and records what happened. A lane +owns the OpenCode processes that execute it. Keeping those responsibilities +separate is important: a broken OpenCode process must not prevent the control +plane from noticing that the lane is unhealthy. + +## Core Decisions + +### Use a hybrid of persistent and ephemeral execution + +A persistent lane catches long-lived-state failures. An ephemeral lane gives a +clean comparison and a reliable reproduction path. A passing ephemeral run +does not erase a persistent-lane failure; it is diagnostic evidence. + +### Keep one OpenCode server per lane + +OpenCode Drive currently launches local child processes and loopback simulation +endpoints. The first deployment should therefore colocate one runner and one +OpenCode server on the same host or in the same container/pod. Parallelism is +achieved with more lanes, not by allowing unrelated workers to compete for one +simulation controller. + +### Use a fresh TUI and session per ordinary attempt + +The server and selected database persist. The client used by one synthetic +journey does not. This preserves long-lived server state while bounding UI +contamination and giving every attempt a clear lifecycle owner. + +Some dedicated soak experiments deliberately reuse a session or TUI. Those +experiments must say so explicitly; reuse is not the default. + +### Separate queued and served inference lanes + +The LLM controller has two mutually exclusive response modes: + +- queued responses are simple and deterministic for sequential journeys; +- a served handler reacts to request content and supports model-based tests or + subagent routing. + +A controller does not switch between these modes after work begins. Separate +lanes keep the behavior honest and prevent response cross-talk. + +### Treat checkpoints as operational assertions + +Existing catalog flows already wait for meaningful UI conditions and visit +ordered checkpoints. Continuous verification wraps those checkpoints with +timing, evidence, and run-record updates. It does not duplicate the journeys in +a second monitoring DSL. + +### Retain the first failure, not the best retry + +Retries may determine whether a failure is intermittent, but they never rewrite +the original attempt to success. Every retry is a new attempt linked to the +first one. This is essential for measuring reliability rather than measuring +the effectiveness of retries. + +### Preserve package and application ownership + +[`packages/drive`](../../packages/drive) remains a generic published package. +OpenCode-specific flow IDs, taxonomies, scenario selection, evidence review, +and continuous-verification policy remain in +[`apps/catalog`](../../apps/catalog). The package must never import the app. + +Generic capabilities should move into Drive only after more than one caller +needs them and only when they do not introduce OpenCode catalog vocabulary. + +## Documentation Map + +1. [System model](./01-system-model.md) defines the components, lifetimes, + failure domains, and system-wide invariants. +2. [Environments and lanes](./02-environments-and-lanes.md) defines persistent + and ephemeral execution, isolation, revisions, concurrency, and lane health. +3. [Bot orchestration](./03-bot-orchestration.md) defines scheduling, leases, + attempt lifecycles, timeouts, retries, and Effect service boundaries. +4. [Inference simulation](./04-inference-simulation.md) defines queued, + reactive, fault-injected, and real-provider response strategies. + Its companion, [Provider and package contract + testing](./04-provider-package-contract-testing.md), defines how the actual + V2 native routes and AI SDK packages are exercised against programmable + transports instead of being reimplemented as mocks. +5. [Scenarios and journeys](./05-scenarios-and-journeys.md) defines how catalog + flows become monitored journeys and how new journeys are authored. +6. [Stateful property testing](./06-stateful-property-testing.md) defines the + model, commands, invariants, generation, replay, and shrinking strategy. +7. [Soak and chaos testing](./07-soak-and-chaos-testing.md) defines long-lived + workloads, fault experiments, recovery invariants, and safety envelopes. +8. [Observability and evidence](./08-observability-and-evidence.md) defines run + records, spans, logs, metrics, artifacts, SLOs, dashboards, and alerts. + [Log files and review UI](./08-log-files-and-review-ui.md) specifies the + append-only local JSONL/raw files and the `apps/catalog` timeline/log viewer. +9. [Deployment and operations](./09-deployment-and-operations.md) defines + process topology, supervision, rollout, configuration, runbooks, and + recovery. + [Elixir control-plane option](./09-elixir-control-plane-option.md) evaluates + a hybrid OTP/Oban/Phoenix control plane with Bun/Effect workload workers. +10. [Security and safety](./10-security-and-safety.md) defines isolation, + credentials, permissions, egress, resource limits, retention, and cost + controls. +11. [Implementation roadmap](./11-implementation-roadmap.md) breaks the design + into reviewable milestones with acceptance criteria. + +## Suggested Reading Paths + +For a first implementation: + +1. Read [System model](./01-system-model.md). +2. Read [Environments and lanes](./02-environments-and-lanes.md). +3. Implement the first milestone in + [Implementation roadmap](./11-implementation-roadmap.md). +4. Use [Observability and evidence](./08-observability-and-evidence.md) as the + run-record contract rather than inventing fields during implementation. + +For scenario authors: + +1. Read [Scenarios and journeys](./05-scenarios-and-journeys.md). +2. Read [Inference simulation](./04-inference-simulation.md). +3. Use [Stateful property testing](./06-stateful-property-testing.md) only when + fixed journeys no longer cover the important ordering space. + +For inference and provider work: + +1. Read [Inference simulation](./04-inference-simulation.md) for deterministic + end-to-end model behavior. +2. Read [Provider and package contract + testing](./04-provider-package-contract-testing.md) for native protocol, + transport, package, and error compatibility. +3. Treat their reports as complementary coverage, not interchangeable mocks. + +For operators: + +1. Read [Deployment and operations](./09-deployment-and-operations.md). +2. Read [Observability and evidence](./08-observability-and-evidence.md). +3. Read [Soak and chaos testing](./07-soak-and-chaos-testing.md). +4. Read [Security and safety](./10-security-and-safety.md). + +## System-Wide Invariants + +Every implementation phase preserves these rules: + +1. Every attempt identifies its exact OpenCode revision, scenario, lane, + inference strategy, and configuration version. +2. Every generated action campaign records a replayable seed and action trace. +3. Every process, TUI, simulation connection, recording, and temporary file has + exactly one lifecycle owner. +4. Expected failures use typed error values. Defects and interrupts retain + their distinct meanings. +5. Cancellation closes resources; it is not converted into a successful run. +6. No retry of a state-changing UI operation happens invisibly inside an + attempt. +7. The absence of successful attempts is observable independently of ordinary + failure reporting. +8. A failed attempt is immutable. Reproduction and retries create linked + attempts. +9. Secrets and unredacted prompts never appear in metrics, span attributes, or + artifact paths. +10. OpenCode-specific concepts remain in `apps/catalog`; generic lifecycle and + simulation behavior remain in `packages/drive`. +11. CLI `--command.ui.*` names and payloads remain identical to the canonical + frontend simulation protocol, except for Drive's documented local + `ui.screenshot` behavior. +12. Backend model control remains in programs and scripts, never convenience + CLI commands. + +## Glossary + +**Attempt** +: One execution of one scenario with one lane, revision, configuration, and + optional seed. A retry is another attempt. + +**Bot** +: A long-running scheduler participant that selects and executes journeys. A + bot is not necessarily powered by an LLM. + +**Campaign** +: A related sequence of property or chaos attempts, usually sharing a revision + and configuration but using different seeds or faults. + +**Checkpoint** +: An ordered, meaningful state reached by an executable catalog flow. In the + continuous system it is also a timing and evidence boundary. + +**Control plane** +: The scheduler, lease coordination, run store, artifact index, configuration, + and alert evaluation that continue to observe the workload plane. + +**Environment** +: A named collection of lanes testing one promotion target, such as a baseline + and candidate OpenCode revision. + +**Inference strategy** +: The mechanism that produces simulated model responses: queued, reactive, + fault-injected, or provider-backed. + +**Journey** +: A deterministic scenario that represents a user-visible workflow and its + assertions. + +**Lane** +: One lifecycle and concurrency boundary containing an OpenCode server, + project, database policy, model controller, and runner. + +**Lane generation** +: One concrete lifetime of a lane's target, server, controllers, paths, ports, + configuration, and optional retained database. + +**Model** +: In property-testing documents, the small expected-state machine maintained by + the test. In inference documents, model means the language-model provider. + The surrounding context disambiguates the term. + +**Run** +: A human grouping of attempts, such as one scheduled journey plus its + diagnostic reproduction. Persisted APIs use the more precise word + `attempt`. + +**Recycle** +: Drain and replace a whole lane generation after a declared trigger such as a + target/configuration change, maximum age, failed health check, or resource + ceiling. It is not an attempt retry and does not erase failure evidence. + +**Synthetic user** +: A controlled TUI or SDK client that performs real user-visible actions + against the test environment. + +**Workload plane** +: The OpenCode server, TUI processes, tools, and simulated model exchanges being + tested.