diff --git a/AGENT-INSTALL.md b/AGENT-INSTALL.md index f9b6758..ab08ed3 100644 --- a/AGENT-INSTALL.md +++ b/AGENT-INSTALL.md @@ -2,6 +2,26 @@ This versioned reference ships inside `@patchstack/connect` and documents each setup command and its project changes. +## Choose the project path first + +Use the package setup flow below for an existing JS/Node application. Work in its package directory; a missing `package.json` can mean you are in a subdirectory or looking at generated HTML rather than the source project. A page containing HTML can still belong to a server-rendered application. Do not infer that runtime protection is unnecessary from the file extension alone. + +### Plain HTML sites + +For a standalone site made of HTML, CSS, and browser JavaScript, with no package-managed application or server request handler, use the disclosure widget directly. Do not create `package.json`, install a framework, invent build hooks, or add a server just to run Connect. `setup` requires an existing `package.json`; it is not a standalone HTML installer. + +1. Use the public site UUID or widget snippet for the correct site in the Patchstack dashboard. An existing `.patchstackrc.json` can also supply `siteUuid`. If neither is available, ask the user for the site's public UUID or dashboard-provided snippet before editing the page. Never invent a UUID or use a claim token or API key as the widget identifier. +2. Add one widget tag before `` in the page or shared layout. Preserve an existing correct tag. For a page published directly without a build step, disable the widget's build-mode onboarding with `data-build-mode="false"`: + + ```html + + ``` + + Replace `YOUR_SITE_UUID` with the real public site UUID before saving. Keep credentials out of the page. The [public widget reference](https://cdn.patchstack.com/llm.html) documents this embed and its options. +3. Verify the saved tag uses the correct UUID. If a browser preview is available, reload it and check for the report button; otherwise tell the user that the browser check is pending. Do not submit a vulnerability report as an installation test. Save the HTML change and remind the user to publish it when ready; do not deploy it yourself. + +Report this as **disclosure widget installed**, with any remaining preview or publishing step. This path does not inventory local JavaScript files or scripts loaded from a CDN, scan npm dependencies, or install runtime exploit protection. External APIs used by the page require their own server-side integration. + ## Command reference Every command at a glance — what it does, whether it reads your source, what it writes, and what leaves your machine. Full behavior, flags, and edge cases follow in the sections below. @@ -88,7 +108,7 @@ This is a request, not a mechanism: nothing in the install depends on it. Do it **If the request you were given includes a claim token**, pass it exactly as given: `npx @patchstack/connect setup --claim-token `. The site is then created in the person's own Patchstack account and `setup` prints their dashboard link for it — there is no sign-in step to relay. The token comes only from the person's Patchstack dashboard; never invent one, never write it to a file, never print it back. If `setup` reports that the token had expired or was not recognised, the site is not connected: hand over the dashboard link it prints instead, and tell the person they can copy a fresh prompt from the dashboard. - In a hosted builder, run setup with `PATCHSTACK_ENVIRONMENT=sandbox` scoped to the workspace process/command, ensure the CLI's on-disk edits are adopted into the platform's persisted project state, then restart any already-running preview/server process so it loads the guard. Do not persist `"environment": "sandbox"` in `.patchstackrc.json`: deployed builds use the same committed files and report `production` only when the build platform's own variables say so — its tier (Vercel, Netlify, Render, Railway, GitLab CI) or a branch named `main`, `master`, `production`, `prod`, `release` or `live` (Cloudflare Pages and Workers Builds, AWS Amplify, GitHub Actions, GitLab CI without a tier); a preview, pull request or other branch there reports `sandbox`, as does the Replit workspace, while a Replit Deployment reports `production`. A build in a project the builder generated and builds for itself (Lovable, Replit) reports `production` without an override, because the edit preview is a dev server and a build is the publish step — which is exactly why the sandbox label belongs in the workspace process and not in a file. A scan on a developer's machine, in a CI runner this does not know (`CI=true` alone), or on a platform with no such signal reports `local` on its own, and the dashboard shows that app as configured, not deployed. A client-only SPA or a static site generator has no server request path to guard: `setup` says runtime protection does not apply and installs nothing for it; never call such a project protected. + In a hosted builder, run setup with `PATCHSTACK_ENVIRONMENT=sandbox` scoped to the workspace process/command, ensure the CLI's on-disk edits are adopted into the platform's persisted project state, then restart any already-running preview/server process so it loads the guard. Do not persist `"environment": "sandbox"` in `.patchstackrc.json`: deployed builds use the same committed files and report `production` only when the build platform's own variables say so — its tier (Vercel, Netlify, Render, Railway, GitLab CI) or a branch named `main`, `master`, `production`, `prod`, `release` or `live` (Cloudflare Pages and Workers Builds, AWS Amplify, GitHub Actions, GitLab CI without a tier); a preview, pull request or other branch there reports `sandbox`, as does the Replit workspace, while a Replit Deployment reports `production`. A build in a project the builder generated and builds for itself (Lovable, Replit) reports `production` without an override, because the edit preview is a dev server and a build is the publish step — which is exactly why the sandbox label belongs in the workspace process and not in a file. A scan on a developer's machine, in a CI runner this does not know (`CI=true` alone), or on a platform with no such signal reports `local` on its own, and the dashboard shows that app as configured, not deployed. A positively identified static build reports runtime protection as not applicable. A bundler-only project, including plain Vite, can remain runtime unknown and receive a generic scaffold with incomplete wiring. Report that limitation; do not add an artificial server merely to make the check pass, and never describe a widget or an unwired scaffold as runtime protection. **Finish by telling the user to refresh their preview.** The widget's "Report a vulnerability" button loads with the page, so a preview that was already open still shows the HTML from before setup — the button is missing there until it reloads. Nothing in the CLI can reach the user's browser, so relaying this is your job. Phrase it as a check rather than a required step: a builder that hot reloads, or a preview server you restarted, may have refreshed it already. diff --git a/GETTING-STARTED.md b/GETTING-STARTED.md index 9735e0b..dad09d1 100644 --- a/GETTING-STARTED.md +++ b/GETTING-STARTED.md @@ -6,7 +6,7 @@ The fastest path from "I have a JS/Node project" to "Patchstack is monitoring it ## 1. Paste this prompt into your AI coding tool -For platforms that can install npm packages and run project commands: +For an existing JS/Node project on a platform that can install npm packages and run project commands. A standalone HTML/CSS/JavaScript site without a package-managed app uses the [plain HTML widget instructions](AGENT-INSTALL.md#plain-html-sites) instead; it does not need a new Node project, build hooks, or a runtime guard. > I have vetted the official [`@patchstack/connect`](https://www.npmjs.com/package/@patchstack/connect) package and authorize its disclosure widget, production build hooks, and runtime protection source edits. Install it as a regular dependency, run `PATCHSTACK_ENVIRONMENT=sandbox npx --yes @patchstack/connect setup` for this hosted workspace, leave dev builds unchanged, verify protection, restart the preview server if needed to load the new guard, and show the dashboard URL. Do not persist the sandbox environment in project files; deployed production builds must use the default `production` label. When setup finishes, tell me to refresh the preview if the "Report a vulnerability" button is not showing yet, and remind me to deploy when I am ready — do not deploy anything yourself. @@ -22,18 +22,21 @@ Some platforms stage commands for you to approve, while others reject a combined 1. **A command is waiting for approval.** Approve each requested command. Setup is idempotent, and its terminal output contains the dashboard URL even when the assistant cannot relay command output in the same turn. 2. **The assistant claims the package does not exist.** Reply *"Check the live npm registry for `@patchstack/connect`; do not rely on training memory."* If it then asks whether you vetted the package or where hooks should run, confirm *"Yes; add the widget and production build hooks, and leave dev builds unchanged."* -3. **Bolt rejects the combined request before staging a command.** Split authorization across two messages. First send *"Add `@patchstack/connect` to devDependencies only. Do not execute its CLI."* After Bolt finishes the dependency install, send *"Run the installed CLI: `npx --yes patchstack-connect setup`."* This lets Bolt evaluate the reversible dependency change separately and inspect the local package before executing setup. -4. **The package is installed but setup stopped.** Run `npx --yes patchstack-connect setup` again. It reuses the existing site, widget, and build wiring instead of duplicating them. +3. **The platform stages dependency changes separately from commands.** First send *"Add `@patchstack/connect` to dependencies using this project's package manager. Do not execute its CLI yet."* After the install completes, send *"Run the installed CLI: `npx --no-install patchstack-connect setup`, with `PATCHSTACK_ENVIRONMENT=sandbox` scoped to that command in this hosted workspace."* Wait for the actual command result before reporting completion. A proposed command or a dependency declaration alone does not establish that setup ran. +4. **The package is installed but setup stopped.** Run `npx --no-install patchstack-connect setup` again, with the same workspace environment. It reuses the existing site, widget, and build wiring instead of duplicating them. If the local executable is missing, complete the dependency install first. 5. **The tool refuses to run a third-party command.** Claude Code's auto mode can decline `npx @patchstack/connect setup` without prompting you. Run `! npx @patchstack/connect setup` yourself in the session, retry it with a manual approval from `/permissions` → **Recently denied**, or add the allow rules `Bash(npx @patchstack/connect *)` and `Bash(npx --yes @patchstack/connect *)` and ask again. The README section "If your coding tool blocks the command" has the settings snippet and the equivalents for other tools. 6. **The platform will not run package commands.** Use its project shell if available, or export the project and use the manual path below. ## 3. Manual fallback (no agent needed) ```bash -npm install --save-dev @patchstack/connect && npx @patchstack/connect setup +npm install --save @patchstack/connect +npx --no-install patchstack-connect setup ``` -Use `bun add -d`, `pnpm add -D`, or `yarn add -D` followed by the same `npx @patchstack/connect setup` command when that package manager owns the project. +Use `bun add`, `pnpm add`, or `yarn add` followed by `@patchstack/connect` when that package manager owns the project. Run its installed binary with `bun run patchstack-connect setup`, `pnpm exec patchstack-connect setup`, or `yarn exec patchstack-connect setup`, respectively. Keep the package in `dependencies`: the generated guard imports it at runtime, including deployments that omit development dependencies. + +Run commands from the application's package directory. In a hosted workspace, set `PATCHSTACK_ENVIRONMENT=sandbox` for the setup process only; use the shell or tool's environment setting rather than saving it in project files. The inline `NAME=value command` form in the prompt requires a POSIX shell. Leave this override unset for production builds so the connector can detect the deployment environment. `setup` is idempotent and preserves existing build commands. It uses direct build chaining on Bun-managed projects and npm-style lifecycle hooks elsewhere. If the framework needs a manual layout edit, it prints the exact remaining widget snippet; `npx @patchstack/connect guide` reprints the same status without changing files. @@ -41,8 +44,8 @@ Use `bun add -d`, `pnpm add -D`, or `yarn add -D` followed by the same `npx @pat - `npx @patchstack/connect status` prints a site UUID and dashboard URL. - You've opened the dashboard URL in your browser and the site shows in your Patchstack dashboard. -- `npx @patchstack/connect guide` reports all steps ✔ (build hooks wired, widget installed). +- `npx @patchstack/connect guide` reports the expected build hooks and widget, and `npx @patchstack/connect protect --check` confirms the guard's source wiring. A client-only or static project can report runtime protection as not applicable; describe it as dependency monitoring and a disclosure widget, not runtime protection. A source check alone does not prove deployed traffic reaches the guard. - Your preview shows the "Report a vulnerability" button (refresh it once if it does not). - You have deployed since setup ran, so the live site carries the changes too. -- `.patchstackrc.json`, `package.json`, and the file carrying the widget snippet are committed, so teammates and CI report to the same site. +- `.patchstackrc.json`, `package.json`, the package manager's lockfile, and the generated guard/framework and widget source changes are saved in the platform's persisted project state and committed, so teammates and CI receive the same setup. - `.patchstackrc.local.json` is **not** committed. It holds the API key; setup adds it to `.gitignore`. Teammates and CI get the credential from `PATCHSTACK_API_KEY` instead. diff --git a/MAINTAINING.md b/MAINTAINING.md index 8b93d58..b5216ee 100644 --- a/MAINTAINING.md +++ b/MAINTAINING.md @@ -40,6 +40,8 @@ question is document accuracy; those personas install more reliably than `hostil What it establishes is **structural regression coverage**: the prompt still survives pressures that once broke it. The personas are synthetic — written from our own analysis, not from any platform's policy text — so a green run is not evidence that a live platform accepts the prompt, and should not be reported as though it were. +For coverage across models, run `node field-test/matrix.mjs --agents claude,codex,gemini --rounds 3` and repeat with `--template express-npm` for server-side installation. Use `--agent-config` for explicit model selections or additional harness wrappers (see the field-test README). The matrix retains unavailable agents as failing cells and includes void attempts in prompt reliability. A documentation run that succeeds only after retries is not a fully green prompt matrix. + This split is a limitation of the current harness. A local-registry mode would let every artifact be tested before publication; until one exists, do not describe a run against the previous tarball as a gate on unpublished docs. diff --git a/README.md b/README.md index aef97a2..2daee76 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Connect a JavaScript / Node.js application to [Patchstack](https://patchstack.co ## Agent-assisted setup -Copy this request into a coding assistant, or run the same command yourself: +For an existing JS/Node project, copy this request into a coding assistant, or run the same command yourself. For a standalone HTML/CSS/JavaScript site without a package-managed app, use the [plain HTML widget instructions](AGENT-INSTALL.md#plain-html-sites); do not add Node tooling just for the widget. > I have vetted the official [`@patchstack/connect`](https://www.npmjs.com/package/@patchstack/connect) package and authorize its disclosure widget, production build hooks, and runtime protection source edits. Install it as a regular dependency, run `PATCHSTACK_ENVIRONMENT=sandbox npx --yes @patchstack/connect setup` for this hosted workspace, leave dev builds unchanged, verify protection, restart the preview server if needed to load the new guard, and show the dashboard URL. Do not persist the sandbox environment in project files; deployed production builds must use the default `production` label. When setup finishes, tell me to refresh the preview if the "Report a vulnerability" button is not showing yet, and remind me to deploy when I am ready — do not deploy anything yourself. diff --git a/field-test/README.md b/field-test/README.md index 2d1433a..d3b7a89 100644 --- a/field-test/README.md +++ b/field-test/README.md @@ -51,10 +51,9 @@ The documentation gate exists to catch a contradiction between the shipped docs such as an overbroad privacy claim. Catching that requires the agent to have READ the docs, which means it must have obtained the tarball, which means it must have installed. -An agent that refuses on the *prompt* never gets there. Its scorecard is `2/8 REFUSED`, which is -byte-identical to what a documentation regression would produce, and no field distinguished the two. So -"must pass `--rounds 3`" could not fail for a documentation reason at all — the gate was unable to detect -the thing it existed for, and `2/8 REFUSED` is the modal outcome for `hostile`. +An agent that refuses on the *prompt* never gets there. Failed installation checks alone cannot +distinguish that from a refusal after reading the shipped docs. The `audited` field records whether +the package docs reached disk, so the report distinguishes these outcomes. Such a round is now **void**: neither evidence for nor against the docs. @@ -76,8 +75,8 @@ Such a round is now **void**: neither evidence for nor against the docs. number and overwrite the void attempt's report, destroying the record a reviewer needs to tell a prompt refusal from a doc regression. - The summary counts only conclusive rounds, and reports how many were void. -- Exit codes are three-way: `0` all conclusive rounds green, `1` a real failure, **`2` inconclusive** — - nothing unpacked, so the run says nothing. A release gate must not read `2` as "the docs are fine". +- Exit codes are three-way: `0` the requested number of conclusive rounds completed successfully, `1` a conclusive failure (including an agent timeout or process error), **`2` inconclusive** — fewer conclusive rounds than requested. A release gate must not read `2` as "the docs are fine". +- The summary also reports successes across **every attempt**, including void attempts. The matrix requires all requested rounds to pass without void attempts: a refusal before unpacking still matters when assessing prompt reliability, even though it cannot establish a documentation defect. Two consequences for how to use it: @@ -96,7 +95,7 @@ node field-test/run.mjs --rounds 1 --agent-cmd "node '$PWD/field-test/stub-decla | stub | what it models | expected | |---|---|---| -| `stub-compliant` | performs the whole flow | `8/8`, `1/1 conclusive`, exit **0** | +| `stub-compliant` | performs the whole flow | `12/12`, `1/1 conclusive`, exit **0** when the published package completes setup for the fixture; known setup limitations fail visibly | | `stub-refusing` | refuses before touching anything | 3 void rounds, INCONCLUSIVE, exit **2** | | `stub-declares-only` | writes the dependency, never installs | 3 void rounds, INCONCLUSIVE, exit **2** | @@ -104,13 +103,12 @@ node field-test/run.mjs --rounds 1 --agent-cmd "node '$PWD/field-test/stub-decla package.json declaration, that run reported `0/1 conclusive` and exit **1** — a definitive failure verdict about documentation it had never obtained. -Quote the path: `--agent-cmd` is handed to `sh -c`, and this repository's own checkout sits under a -directory with a space in it, so an unquoted command silently fails to start — which voids the round, -correctly but confusingly. +Quote the path: `--agent-cmd` is handed to `sh -c`, so an unquoted path containing spaces can prevent +the agent from starting and leave the round void. ## Prerequisites -- Node ≥ 18, network access (fixtures run a real `npm install`; the agent installs the real published `@patchstack/connect`). +- Node ≥ 20, network access (fixtures run a real `npm install`; the agent installs the real published `@patchstack/connect`). - An agent CLI. Default: [Claude Code](https://claude.com/claude-code) headless (`claude -p`). Any CLI that reads a prompt from stdin and prints the agent's final message to stdout works via `--agent-cmd`. ## Safety model — read before running @@ -146,14 +144,14 @@ node field-test/run.mjs --agent-cmd "claude -p --dangerously-skip-permissions -- # Prove no provenance reaches the agent: capture the exact stdin it receives PS_CAPTURE_STDIN=/tmp/captured.txt node field-test/run.mjs --persona lovable --rounds 1 \ - --agent-cmd "node field-test/stub-capture.mjs" + --agent-cmd "node '$PWD/field-test/stub-capture.mjs'" grep -c 'field-test:meta' /tmp/captured.txt # must be 0 -# Self-test the harness (scripted stub, no AI, ~1 min) — should be fully green -node field-test/run.mjs --agent-cmd "node $PWD/field-test/stub-compliant.mjs" +# Exercise the published install (scripted stub, no AI, ~1 min); setup limitations remain failures +node field-test/run.mjs --agent-cmd "node '$PWD/field-test/stub-compliant.mjs'" ``` -Flags: `--persona ` (any `personas/.md`), `--template lovable-bun|vite-npm`, `--prompt `, `--rounds N`, `--agent-cmd ""`, `--keep` (don't delete the fixture), `--timeout `, `--confirm` (see below), `--confirm-reply ` (override the confirmation text). +Flags: `--persona ` (any `personas/.md`), `--template lovable-bun|vite-npm|express-npm`, `--prompt `, `--rounds N`, `--agent-cmd ""`, `--keep` (don't delete the fixture), `--timeout `, `--confirm` (see below), `--confirm-reply ` (override the confirmation text). ### `--confirm` — legacy two-turn prompt experiments @@ -168,9 +166,14 @@ The agent harness installs the published package, so use the local demo to exerc ```bash npm run build node field-test/setup-demo.mjs + +# The same checks on a server with an applicable runtime guard +node field-test/setup-demo.mjs --template express-npm ``` -It packs the local package into a throwaway React/Vite fixture with a Bun lockfile, installs it as a dev dependency, runs `setup` twice against the mock API, and verifies that one site, one widget, and one copy of each build command remain. It never calls the production API, runs the fixture build, or invokes `protect`. +It packs the local package into a throwaway fixture, installs it as a regular dependency, runs `setup` twice against the mock API, and verifies that one site, one widget, and one copy of each build command remain. It also runs the field-test file and protection checks, excluding agent-message checks. It uses the mock manifest API and never runs the fixture build or starts the app. `express-npm` verifies source wiring for an applicable guard. + +The default React/Vite fixture has a Bun lockfile and no server request path, but the CLI conservatively classifies a bundler-only project as runtime unknown. Its uncompleted generic guard makes the protection check fail. The strict demo and matrix retain that failure; do not add an artificial server just to turn a browser-only fixture green. The result identifies a setup limitation rather than proving an agent made a mistake. ### Matrix runs — personas × models @@ -183,6 +186,9 @@ node field-test/matrix.mjs # Full matrix: 3 platform personas × 3 model families (9 agent runs — budget ~30-60 min) node field-test/matrix.mjs --agents claude,codex,gemini +# Repeat for server-side installation, alongside the default browser-only fixture +node field-test/matrix.mjs --agents claude,codex,gemini --template express-npm + # Everything run.mjs accepts passes through node field-test/matrix.mjs --personas hostile,bolt-diy --agents claude,codex --rounds 3 --prompt /tmp/v2.txt @@ -190,22 +196,53 @@ node field-test/matrix.mjs --personas hostile,bolt-diy --agents claude,codex --r node field-test/matrix.mjs --agents stub --personas bolt-diy,lovable,replit ``` -Named agents (see the `AGENTS` table in `matrix.mjs`): `claude` (logged-in Claude Code), `codex` (`codex login` or `OPENAI_API_KEY`), `gemini` (interactive login once or `GEMINI_API_KEY`; Workspace accounts also need `GOOGLE_CLOUD_PROJECT`), `stub`. Agents missing from PATH are skipped with a warning; unauthenticated ones fail their cells visibly. The aggregate lands in `field-test/results/matrix-/matrix.md` with links to each cell's full run.mjs results; exit code is 0 only if every cell is fully green. +Named agents (see the `AGENTS` table in `matrix.mjs`): `claude` (logged-in Claude Code), `codex` (`codex login` or `OPENAI_API_KEY`), `gemini` (interactive login once or `GEMINI_API_KEY`; Workspace accounts also need `GOOGLE_CLOUD_PROJECT`), `stub`. Requested agents missing from PATH remain in the report as `UNAVAILABLE` and make the matrix fail; unauthenticated ones fail their cells visibly. The aggregate lands in `field-test/results/matrix-/matrix.md` with links to each cell's full run.mjs results; exit code is 0 only if every cell is fully green. + +### Additional models and harnesses + +Use `--agent-config ` to add named commands or override the built-ins. The file is a JSON object; each entry specifies an executable to check and a shell command to run. The command must accept the composed prompt on stdin and return its report on stdout. A harness with a different protocol needs a wrapper that implements that contract. + +For example, with your own `agent-wrapper` executable: + +```json +{ + "model_a": { + "executable": "agent-wrapper", + "command": "agent-wrapper --model model-a" + }, + "model_b": { + "executable": "agent-wrapper", + "command": "agent-wrapper --model model-b" + } +} +``` + +```bash +node field-test/matrix.mjs --agent-config /tmp/agents.json --agents model_a,model_b --personas standard,hostile --rounds 3 +``` + +These are trusted local commands, executed with the runner's permissions. Quote paths with spaces inside `command`; pass credentials through the environment, not this file. The resolved commands are recorded in the ignored matrix results so a result identifies the requested model configuration. Use explicit model selectors when comparing models; a CLI's default model can change. The harness does not independently attest which model a provider served. + +A green cell requires all requested rounds, successful agent exits, no timeouts, and no void attempts. A successful retry can supply documentation evidence without erasing a failed prompt attempt. Empty or incomplete results cannot pass. ## What gets scored -Each round prints a scorecard and exits non-zero unless every round is fully green: +Each round prints a scorecard. The scorer checks resulting files and runs the installed CLI's source-only `protect --check`; it does not start the app or establish deployed protection. All current templates render `index.html`, so widget checks deliberately inspect that shell: | Check | Meaning | |---|---| -| `installed` | `@patchstack/connect` declared in the fixture's `package.json` | +| `installed` | A regular dependency, absent from devDependencies, with non-empty package docs unpacked | | `provisioned` | `.patchstackrc.json` carries the mock's site UUID | | `provisionedOnce` | exactly one provisioning POST — more means duplicate sites | | `hooksWired` | `scan` and `mark-build` reachable from `prebuild`/`postbuild`/`build` | -| `widgetInstalled` | widget script tag present in source | -| `widgetTokenMatches` | the provisioned UUID appears in source as the `userToken` | -| `claimUrlSurfaced` | the agent's final message shows the claim URL to the user | -| `noProductionLeak` | the agent never surfaced a production claim URL (mock bypass) | +| `dependencyScanWired` | `postinstall` contains the dependency scan | +| `devScriptsPreserved` | Development scripts match the fixture before the agent ran | +| `sandboxNotPersisted` | No sandbox override in config, package scripts, or shared/production env files | +| `widgetInstalled` | Exactly one widget script tag in the fixture's rendered `index.html` | +| `widgetTokenMatches` | The widget tag itself carries the mock site UUID | +| `protectionVerified` | The installed `protect --check` succeeds and reports wired or explicitly not applicable | +| `claimUrlSurfaced` | Agent output includes the mock dashboard URL for the correct site | +| `noProductionLeak` | No production claim URL in agent output; this does not establish network isolation | Everything is saved under `field-test/results/-/` (gitignored): the agent's full report (`report.md` — read HESITATIONS and UX FRICTION, that's where the insight is), the mock's request log, and per-round scorecards. @@ -228,7 +265,7 @@ The agent audits the *published* tarball, so the gate's pass rate is a function Until a publish lands and ages, use this ladder instead of burning hostile rounds on a known-red gate: -1. **Stub self-test** — `node field-test/run.mjs --agent-cmd "node $PWD/field-test/stub-compliant.mjs"`. Validates the harness, mock, and scoring in ~1 min. No AI, no registry dependency. +1. **Stub self-test** — `node field-test/run.mjs --agent-cmd "node '$PWD/field-test/stub-compliant.mjs'"`. Validates the harness, mock, and scoring in ~1 min. No AI; this installs the published package and requires registry access. The offline scorer/process regressions run with `npx vitest run tests/field-test-verify.test.ts tests/field-test-outcomes.test.ts`. 2. **Standard persona** — exercises the mechanical checks (guide accuracy, hook wiring, widget token) with less policy pressure; catches CLI/UX regressions immediately. 3. **Hostile rounds scored by refusal *reason*, not exit code.** Read DECISION ANALYSIS and attribute each refusal: one that quotes the published docs or release age is environmental noise; one that quotes the prompt's own wording is a real prompt bug. A variant is not worse than the incumbent unless it draws prompt-directed refusals the incumbent doesn't. 4. **(Not built) local-registry mode** — run a local registry (e.g. verdaccio), publish the working tree to it, and pin the fixture via the `npm_config_registry` env var (env pinning reads as platform plumbing, same as `PATCHSTACK_ENDPOINT`). This is the only way to exercise unpublished doc/CLI changes end-to-end. Caveat: the local record has no provenance attestation or signatures, which strict agents check — expect some artificial refusals on that ground. @@ -237,3 +274,5 @@ Until a publish lands and ages, use this ladder instead of burning hostile round - The personas are synthetic and the matrix covers multiple model families, but a hosted platform is still (prompt × model × runtime × UI) — and neither the real policy text, the runtime, nor the UI layer is reproduced here. A green harness is necessary and not sufficient, and this is not hypothetical: a refusal has been found by a real user after this harness passed a prompt. Treat a real-world refusal report as a new persona — encode the pressure it applied into `personas/`, in your own words, so the regression stays covered. Do not describe a green run as evidence about a platform's policy. - The fixture installs the *published* package. An unpublished `guide`/CLI change can't be exercised end-to-end by the agent (it will install the registry version); publish first or accept that the run validates the prompt shape only. +- The compliant stub executes the published package without repairing failed setup steps. A release that cannot complete `protect --check` fails the stricter scorecard. Plain Vite projects currently produce an unknown runtime classification and an uncompleted generic guard in both the working tree and the published flow; a failed cell on that fixture is not necessarily a model refusal. Use the local setup demo to distinguish working-tree behavior from registry behavior; do not weaken the score to make the release pass. +- The fixtures cover a browser-only React/Vite project and an Express server, with npm installation and a Bun lockfile marker. They do not reproduce native package-manager execution for every manager, SSR frameworks, monorepos, or a hosted UI's persistence and command-approval behavior. diff --git a/field-test/agent.mjs b/field-test/agent.mjs index 12ec960..884a06e 100644 --- a/field-test/agent.mjs +++ b/field-test/agent.mjs @@ -39,16 +39,29 @@ export function runAgent(agentCmd, promptText, fixtureDir, endpoint, timeoutMs) cwd: fixtureDir, env: { ...process.env, PATCHSTACK_ENDPOINT: endpoint }, stdio: ['pipe', 'pipe', 'pipe'], + detached: process.platform !== 'win32', }); let out = ''; let err = ''; let timedOut = false; const timer = setTimeout(() => { timedOut = true; - child.kill('SIGKILL'); + if (process.platform !== 'win32' && child.pid) { + try { process.kill(-child.pid, 'SIGKILL'); } catch { child.kill('SIGKILL'); } + } else { + child.kill('SIGKILL'); + } }, timeoutMs); child.stdout.on('data', (chunk) => (out += chunk)); child.stderr.on('data', (chunk) => (err += chunk)); + child.stdin.on('error', (error) => { + // A CLI can exit before reading stdin; its exit status remains the result of the attempt. + if (error.code !== 'EPIPE') err += `\nstdin error: ${error.code ?? 'unknown'}`; + }); + child.on('error', (error) => { + clearTimeout(timer); + resolve({ output: out, stderr: `${err}\nagent launch failed: ${error.code ?? 'unknown'}`, exitCode: null, timedOut }); + }); child.on('close', (code) => { clearTimeout(timer); resolve({ output: out, stderr: err, exitCode: code, timedOut }); diff --git a/field-test/fixture.mjs b/field-test/fixture.mjs index 8500656..7bad9e2 100644 --- a/field-test/fixture.mjs +++ b/field-test/fixture.mjs @@ -6,6 +6,7 @@ // populated node_modules and NO package-lock.json — the shape of a // bun-managed vibe-platform export. Exercises the node_modules-walk path. // - vite-npm: same app, plain npm project with package-lock.json. +// - express-npm: an HTTP server with a static HTML shell and an existing build command. import { execSync } from 'node:child_process'; import { mkdirSync, rmSync, writeFileSync } from 'node:fs'; import path from 'node:path'; @@ -67,7 +68,7 @@ export default defineConfig(({ mode }) => ({ })); `; -export const TEMPLATES = ['lovable-bun', 'vite-npm']; +export const TEMPLATES = ['lovable-bun', 'vite-npm', 'express-npm']; export function makeFixture(dir, template = 'lovable-bun') { if (!TEMPLATES.includes(template)) { @@ -77,23 +78,39 @@ export function makeFixture(dir, template = 'lovable-bun') { rmSync(dir, { recursive: true, force: true }); mkdirSync(path.join(dir, 'src'), { recursive: true }); - const pkg = structuredClone(PACKAGE_JSON); + const pkg = template === 'express-npm' ? { + name: 'field-test-server', private: true, version: '0.0.0', type: 'module', main: 'server.js', + scripts: { start: 'node server.js', dev: 'node --watch server.js', build: 'node --check server.js' }, + dependencies: { express: '^4.21.2' }, + } : structuredClone(PACKAGE_JSON); if (template === 'vite-npm') { delete pkg.devDependencies['lovable-tagger']; } writeFileSync(path.join(dir, 'package.json'), JSON.stringify(pkg, null, 2) + '\n'); - writeFileSync(path.join(dir, 'index.html'), INDEX_HTML); - writeFileSync(path.join(dir, 'src', 'main.tsx'), MAIN_TSX); - writeFileSync(path.join(dir, 'src', 'App.tsx'), APP_TSX); - writeFileSync( - path.join(dir, 'vite.config.ts'), - template === 'lovable-bun' - ? VITE_CONFIG - : VITE_CONFIG.replace(/import { componentTagger }.*\n/, '').replace( - /, mode === "development" && componentTagger\(\)/, - '', - ), - ); + if (template === 'express-npm') { + writeFileSync(path.join(dir, 'index.html'), 'Recipe Glow

Recipe Glow

\n'); + writeFileSync(path.join(dir, 'server.js'), `import express from 'express'; + +const app = express(); +app.use(express.json()); +app.get('/', (_request, response) => response.sendFile('index.html', { root: process.cwd() })); +app.get('/api/health', (_request, response) => response.json({ ok: true })); +app.listen(process.env.PORT || 3000); +`); + } else { + writeFileSync(path.join(dir, 'index.html'), INDEX_HTML); + writeFileSync(path.join(dir, 'src', 'main.tsx'), MAIN_TSX); + writeFileSync(path.join(dir, 'src', 'App.tsx'), APP_TSX); + writeFileSync( + path.join(dir, 'vite.config.ts'), + template === 'lovable-bun' + ? VITE_CONFIG + : VITE_CONFIG.replace(/import { componentTagger }.*\n/, '').replace( + /, mode === "development" && componentTagger\(\)/, + '', + ), + ); + } execSync('npm install --no-audit --no-fund', { cwd: dir, stdio: 'pipe' }); @@ -109,7 +126,7 @@ const invokedDirectly = process.argv[1] && import.meta.url.endsWith(process.argv if (invokedDirectly) { const [dir, template] = process.argv.slice(2); if (!dir) { - console.error('Usage: node fixture.mjs [lovable-bun|vite-npm]'); + console.error(`Usage: node fixture.mjs [${TEMPLATES.join('|')}]`); process.exit(1); } makeFixture(path.resolve(dir), template ?? 'lovable-bun'); diff --git a/field-test/matrix.mjs b/field-test/matrix.mjs index 23b441c..4191eb0 100644 --- a/field-test/matrix.mjs +++ b/field-test/matrix.mjs @@ -1,7 +1,7 @@ // Matrix runner: personas × agent CLIs, one run.mjs invocation per cell. // // node field-test/matrix.mjs [--personas bolt-diy,lovable,replit] [--agents claude,codex,gemini] -// [--rounds N] [--prompt ] [--template lovable-bun|vite-npm] +// [--rounds N] [--prompt ] [--template lovable-bun|vite-npm|express-npm] // [--timeout ] // // Personas are files in personas/.md. Agents are named entries in the @@ -21,17 +21,34 @@ import { spawn, spawnSync } from 'node:child_process'; import { mkdirSync, readFileSync, writeFileSync, existsSync } from 'node:fs'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; +import { cellGreen, positiveNumber } from './outcomes.mjs'; const HERE = path.dirname(fileURLToPath(import.meta.url)); +const shellQuote = (value) => `'${value.replaceAll("'", "'\\''")}'`; const AGENTS = { - claude: 'claude -p --dangerously-skip-permissions', - codex: 'codex exec --skip-git-repo-check --dangerously-bypass-approvals-and-sandbox', - gemini: 'gemini --yolo', - stub: `node ${path.join(HERE, 'stub-compliant.mjs')}`, + claude: { executable: 'claude', command: 'claude -p --dangerously-skip-permissions' }, + codex: { executable: 'codex', command: 'codex exec --skip-git-repo-check --dangerously-bypass-approvals-and-sandbox' }, + gemini: { executable: 'gemini', command: 'gemini --yolo' }, + stub: { executable: process.execPath, command: `${shellQuote(process.execPath)} ${shellQuote(path.join(HERE, 'stub-compliant.mjs'))}` }, }; -function parseArgs(argv) { +export function loadAgents(file) { + if (!file) return { ...AGENTS }; + const config = JSON.parse(readFileSync(file, 'utf8')); + if (config === null || Array.isArray(config) || typeof config !== 'object') { + throw new Error('--agent-config must contain an object of named agent commands'); + } + for (const [name, agent] of Object.entries(config)) { + if (!/^[a-zA-Z0-9_-]+$/.test(name) || typeof agent?.command !== 'string' || + !agent.command.trim() || typeof agent?.executable !== 'string' || !agent.executable.trim()) { + throw new Error('Each agent needs a simple name, a non-empty command, and an executable'); + } + } + return { ...AGENTS, ...config }; +} + +export function parseArgs(argv) { const opts = { personas: ['bolt-diy', 'lovable', 'replit'], agents: ['claude'], @@ -39,20 +56,27 @@ function parseArgs(argv) { prompt: null, template: null, timeoutMinutes: null, + agentConfig: null, }; for (let i = 2; i < argv.length; i++) { const arg = argv[i]; if (arg === '--personas') opts.personas = argv[++i].split(',').map((name) => name.trim()); else if (arg === '--agents') opts.agents = argv[++i].split(',').map((name) => name.trim()); - else if (arg === '--rounds') opts.rounds = Number(argv[++i]); + else if (arg === '--rounds') opts.rounds = positiveNumber(argv[++i], '--rounds', true); + else if (arg === '--agent-config') opts.agentConfig = path.resolve(argv[++i]); else if (arg === '--prompt') opts.prompt = path.resolve(argv[++i]); else if (arg === '--template') opts.template = argv[++i]; - else if (arg === '--timeout') opts.timeoutMinutes = Number(argv[++i]); + else if (arg === '--timeout') opts.timeoutMinutes = positiveNumber(argv[++i], '--timeout'); else { console.error(`Unknown argument: ${arg}`); process.exit(1); } } + if ([...opts.personas, ...opts.agents].some((name) => !/^[a-zA-Z0-9_-]+$/.test(name))) { + throw new Error('Persona and agent lists must contain non-empty names'); + } + opts.personas = [...new Set(opts.personas)]; + opts.agents = [...new Set(opts.agents)]; return opts; } @@ -64,24 +88,23 @@ function checkPersona(name) { } } -function checkAgent(name) { - const cmd = AGENTS[name]; - if (!cmd) { - console.error(`No such agent: ${name}. Known agents: ${Object.keys(AGENTS).join(', ')}`); +function checkAgent(name, agents) { + const agent = agents[name]; + if (!agent) { + console.error(`No such agent: ${name}. Known agents: ${Object.keys(agents).join(', ')}`); process.exit(1); } - const binary = cmd.split(' ')[0]; - const found = spawnSync('sh', ['-c', `command -v ${binary}`], { stdio: 'ignore' }); + const found = spawnSync('sh', ['-c', 'command -v "$1"', 'sh', agent.executable], { stdio: 'ignore' }); return found.status === 0; } /** Run one cell via run.mjs, streaming its output, and collect its summary. */ -function runCell(persona, agentName, opts) { +function runCell(persona, agentName, opts, agents) { return new Promise((resolve) => { const args = [ path.join(HERE, 'run.mjs'), '--persona', persona, - '--agent-cmd', AGENTS[agentName], + '--agent-cmd', agents[agentName].command, '--rounds', String(opts.rounds), ]; if (opts.prompt) args.push('--prompt', opts.prompt); @@ -109,70 +132,72 @@ function runCell(persona, agentName, opts) { }); } -function cellLabel(cell) { +export function cellLabel(cell) { + if (cell.unavailable) return 'UNAVAILABLE'; + if (cell.exitCode === 2) return 'INCONCLUSIVE'; if (!cell.rounds) return 'ERROR'; return cell.rounds - .map((round) => `${round.passed}/${round.total}${round.refused ? ' R' : ''}${round.timedOut ? ' T' : ''}`) + .map((round) => `${round.passed}/${round.total}${!round.audited ? ' VOID' : ''}${round.refused ? ' R' : ''}${round.timedOut || round.confirmTimedOut ? ' T' : ''}${round.exitCode !== 0 || (round.confirmTurn && round.confirmExitCode !== 0) ? ' E' : ''}`) .join(', '); } -function cellGreen(cell) { - return cell.rounds !== null && cell.rounds.every((round) => round.passed === round.total); -} - -const opts = parseArgs(process.argv); -opts.personas.forEach(checkPersona); +async function main() { + const opts = parseArgs(process.argv); + const agents = loadAgents(opts.agentConfig); + opts.personas.forEach(checkPersona); -const unavailable = opts.agents.filter((name) => !checkAgent(name)); -if (unavailable.length > 0) { - console.warn(`skipping agents with no CLI on PATH: ${unavailable.join(', ')}`); - opts.agents = opts.agents.filter((name) => !unavailable.includes(name)); -} -if (opts.agents.length === 0) { - console.error('No runnable agents.'); - process.exit(1); -} - -const stamp = new Date().toISOString().replace(/[:.]/g, '-'); -const matrixDir = path.join(HERE, 'results', `matrix-${stamp}`); -mkdirSync(matrixDir, { recursive: true }); - -console.log(`matrix: ${opts.personas.length} persona(s) × ${opts.agents.length} agent(s) × ${opts.rounds} round(s)`); -console.log(`personas: ${opts.personas.join(', ')}`); -console.log(`agents: ${opts.agents.join(', ')}`); -console.log(`aggregate: ${matrixDir}\n`); + const unavailable = opts.agents.filter((name) => !checkAgent(name, agents)); + if (unavailable.length > 0) { + console.warn(`agents unavailable (matrix cannot pass): ${unavailable.join(', ')}`); + } -const cells = []; -for (const persona of opts.personas) { - for (const agent of opts.agents) { - console.log(`\n=== cell: persona=${persona} agent=${agent} ===`); - cells.push(await runCell(persona, agent, opts)); + const stamp = new Date().toISOString().replace(/[:.]/g, '-'); + const matrixDir = path.join(HERE, 'results', `matrix-${stamp}`); + mkdirSync(matrixDir, { recursive: true }); + + console.log(`matrix: ${opts.personas.length} persona(s) × ${opts.agents.length} agent(s) × ${opts.rounds} round(s)`); + console.log(`personas: ${opts.personas.join(', ')}`); + console.log(`agents: ${opts.agents.join(', ')}`); + console.log(`aggregate: ${matrixDir}\n`); + + const cells = []; + for (const persona of opts.personas) { + for (const agent of opts.agents) { + console.log(`\n=== cell: persona=${persona} agent=${agent} ===`); + cells.push(unavailable.includes(agent) + ? { persona, agent, exitCode: null, resultsDir: null, rounds: null, unavailable: true } + : await runCell(persona, agent, opts, agents)); + } } -} -const header = `| persona | ${opts.agents.join(' | ')} |`; -const divider = `|---|${opts.agents.map(() => '---').join('|')}|`; -const rows = opts.personas.map((persona) => { - const labels = opts.agents.map((agent) => { - const cell = cells.find((c) => c.persona === persona && c.agent === agent); - return `${cellGreen(cell) ? '✅' : '❌'} ${cellLabel(cell)}`; + const header = `| persona | ${opts.agents.join(' | ')} |`; + const divider = `|---|${opts.agents.map(() => '---').join('|')}|`; + const rows = opts.personas.map((persona) => { + const labels = opts.agents.map((agent) => { + const cell = cells.find((c) => c.persona === persona && c.agent === agent); + return `${cellGreen(cell, opts.rounds) ? '✅' : '❌'} ${cellLabel(cell)}`; + }); + return `| ${persona} | ${labels.join(' | ')} |`; }); - return `| ${persona} | ${labels.join(' | ')} |`; -}); -const table = [header, divider, ...rows].join('\n'); -const legend = 'Each cell: checks passed per round (`R` = agent refused, `T` = timed out, `ERROR` = run died before scoring). ✅ = every round fully green.'; -const dirList = cells - .map((cell) => `- ${cell.persona} × ${cell.agent}: ${cell.resultsDir ?? '(no results dir)'}`) - .join('\n'); + const table = [header, divider, ...rows].join('\n'); + const legend = 'Each cell: checks passed per attempt (`R` = refused, `T` = timed out, `E` = process error, `VOID` = package not unpacked). INCONCLUSIVE = insufficient conclusive rounds; UNAVAILABLE = missing CLI; ERROR = no scorecard. ✅ requires every requested round, a successful process, and no void attempts.'; + const dirList = cells + .map((cell) => `- ${cell.persona} × ${cell.agent}: ${cell.resultsDir ?? '(no results dir)'}`) + .join('\n'); + + writeFileSync( + path.join(matrixDir, 'matrix.md'), + `# Field-test matrix — ${stamp}\n\nrounds per cell: ${opts.rounds}\n\n${table}\n\n${legend}\n\n## Per-cell results\n\n${dirList}\n`, + ); + writeFileSync(path.join(matrixDir, 'matrix.json'), JSON.stringify({ opts, agents, cells }, null, 2)); -writeFileSync( - path.join(matrixDir, 'matrix.md'), - `# Field-test matrix — ${stamp}\n\nrounds per cell: ${opts.rounds}\n\n${table}\n\n${legend}\n\n## Per-cell results\n\n${dirList}\n`, -); -writeFileSync(path.join(matrixDir, 'matrix.json'), JSON.stringify({ opts, cells }, null, 2)); + console.log(`\n\n${table}\n\n${legend}`); + console.log(`\nAggregate written to ${matrixDir}`); -console.log(`\n\n${table}\n\n${legend}`); -console.log(`\nAggregate written to ${matrixDir}`); + process.exit(cells.length > 0 && cells.every((cell) => cellGreen(cell, opts.rounds)) ? 0 : 1); +} -process.exit(cells.every(cellGreen) ? 0 : 1); +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + await main(); +} diff --git a/field-test/outcomes.mjs b/field-test/outcomes.mjs new file mode 100644 index 0000000..af07b33 --- /dev/null +++ b/field-test/outcomes.mjs @@ -0,0 +1,36 @@ +/** A scored attempt is green only when the agent finished successfully. */ +export function roundGreen(round) { + return round.audited === true && round.total > 0 && round.passed === round.total && + round.exitCode === 0 && round.timedOut === false && + (!round.confirmTurn || (round.confirmExitCode === 0 && round.confirmTimedOut === false)); +} + +export function summarizeRounds(rounds, requestedRounds) { + const conclusive = rounds.filter((round) => round.audited); + const fullPasses = conclusive.filter(roundGreen).length; + const voided = rounds.length - conclusive.length; + const failed = conclusive.length - fullPasses; + return { + requestedRounds, + attempts: rounds.length, + conclusive: conclusive.length, + voided, + fullPasses, + // Documentation can be assessed only after unpacking; prompt reliability includes every attempt. + promptPassed: rounds.length === requestedRounds && fullPasses === requestedRounds, + exitCode: failed > 0 ? 1 : conclusive.length < requestedRounds ? 2 : 0, + }; +} + +export function cellGreen(cell, requestedRounds) { + return cell.exitCode === 0 && Array.isArray(cell.rounds) && + summarizeRounds(cell.rounds, requestedRounds).promptPassed; +} + +export function positiveNumber(value, flag, integer = false) { + const parsed = Number(value); + if (!Number.isFinite(parsed) || parsed <= 0 || (integer && !Number.isSafeInteger(parsed))) { + throw new Error(`${flag} must be a positive ${integer ? 'integer' : 'number'}`); + } + return parsed; +} diff --git a/field-test/run.mjs b/field-test/run.mjs index e933096..e18cce9 100644 --- a/field-test/run.mjs +++ b/field-test/run.mjs @@ -1,7 +1,7 @@ // Field-test orchestrator: run an AI agent against the install prompt in a // throwaway fixture, with the Patchstack API mocked, and score the outcome. // -// node field-test/run.mjs [--persona ] [--template lovable-bun|vite-npm] +// node field-test/run.mjs [--persona ] [--template lovable-bun|vite-npm|express-npm] // [--prompt ] [--rounds N] [--agent-cmd ""] // [--keep] [--timeout ] // @@ -13,7 +13,7 @@ // // Results land in field-test/results// (gitignored): the agent's // report, the mock API's request log, and a scorecard per round. -import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from 'node:fs'; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -22,6 +22,8 @@ import { startMockApi } from './mock-api.mjs'; import { makeFixture, TEMPLATES } from './fixture.mjs'; import { composeAgentPrompt } from './persona.mjs'; import { runAgent } from './agent.mjs'; +import { readJsonSafe, verify } from './verify.mjs'; +import { positiveNumber, summarizeRounds } from './outcomes.mjs'; const HERE = path.dirname(fileURLToPath(import.meta.url)); @@ -45,9 +47,9 @@ function parseArgs(argv) { else if (arg === '--persona') opts.persona = argv[++i]; else if (arg === '--template') opts.template = argv[++i]; else if (arg === '--prompt') opts.prompt = path.resolve(argv[++i]); - else if (arg === '--rounds') opts.rounds = Number(argv[++i]); + else if (arg === '--rounds') opts.rounds = positiveNumber(argv[++i], '--rounds', true); else if (arg === '--agent-cmd') opts.agentCmd = argv[++i]; - else if (arg === '--timeout') opts.timeoutMinutes = Number(argv[++i]); + else if (arg === '--timeout') opts.timeoutMinutes = positiveNumber(argv[++i], '--timeout'); else { console.error(`Unknown argument: ${arg}`); process.exit(1); @@ -60,138 +62,6 @@ function parseArgs(argv) { return opts; } -/** Bounded search for `needle` in the fixture's source files (skips node_modules etc.). */ -function fixtureContains(dir, needle) { - const skipped = new Set(['node_modules', '.git', 'dist', 'build', '.output', 'coverage']); - const walk = (current, depth) => { - if (depth > 5) return false; - let entries; - try { - entries = readdirSync(current, { withFileTypes: true }); - } catch { - return false; - } - for (const entry of entries) { - const full = path.join(current, entry.name); - if (entry.isDirectory()) { - if (skipped.has(entry.name) || entry.name.startsWith('.')) continue; - if (walk(full, depth + 1)) return true; - } else if (entry.isFile() && statSync(full).size < 512 * 1024) { - try { - if (readFileSync(full, 'utf8').includes(needle)) return true; - } catch { - // unreadable — skip - } - } - } - return false; - }; - return walk(dir, 0); -} - -function readJsonSafe(file) { - try { - return JSON.parse(readFileSync(file, 'utf8')); - } catch { - return null; - } -} - -/** Score one completed run. Every check is { pass, detail }. */ -function verify(fixtureDir, mock, agentOutput) { - const pkg = readJsonSafe(path.join(fixtureDir, 'package.json')) ?? {}; - const rc = readJsonSafe(path.join(fixtureDir, '.patchstackrc.json')) ?? {}; - const scripts = pkg.scripts ?? {}; - const dep = - pkg.devDependencies?.['@patchstack/connect'] ?? pkg.dependencies?.['@patchstack/connect']; - - const provisionPosts = mock.requests.filter( - (request) => request.method === 'POST' && request.url === '/monitor/pulse/manifest', - ).length; - - const scanWired = ['prebuild', 'build'].some((key) => - (scripts[key] ?? '').includes('patchstack-connect scan'), - ); - const markWired = ['postbuild', 'build'].some((key) => - (scripts[key] ?? '').includes('patchstack-connect mark-build'), - ); - - // Did the tarball actually arrive? A DECLARATION in package.json is not an install: an agent can add - // the dependency and then refuse before running `npm install`, and the docs it was supposed to audit - // never reach the disk. `AGENT-INSTALL.md` is in the package's `files`, so its presence under - // node_modules is direct evidence that the tarball was fetched and unpacked. - // - // Non-empty, because a truncated or interrupted unpack leaves a file that exists and says nothing. - const shippedDocs = path.join(fixtureDir, 'node_modules', '@patchstack', 'connect', 'AGENT-INSTALL.md'); - let unpackedBytes = 0; - try { unpackedBytes = statSync(shippedDocs).size; } catch { unpackedBytes = 0; } - const unpacked = unpackedBytes > 0; - - const checks = { - installed: { - // Both halves. Declared-but-not-unpacked is the state that used to score as installed, and it is - // exactly the state in which no audit of the shipped docs can have happened. - pass: dep !== undefined && unpacked, - detail: dep === undefined - ? 'not in package.json' - : unpacked - ? `declared ${dep}, tarball unpacked (AGENT-INSTALL.md ${unpackedBytes}B)` - : `declared ${dep} but the tarball was never unpacked — no node_modules/@patchstack/connect/AGENT-INSTALL.md`, - }, - provisioned: { - pass: rc.siteUuid === mock.uuid, - detail: `rc siteUuid=${rc.siteUuid ?? '(none)'}, mock uuid=${mock.uuid}, provision POSTs=${provisionPosts}`, - }, - provisionedOnce: { - pass: provisionPosts <= 1, - detail: `${provisionPosts} provisioning POST(s) — more than one means a duplicate site`, - }, - hooksWired: { - pass: scanWired && markWired, - detail: `scan wired=${scanWired}, mark-build wired=${markWired}`, - }, - widgetInstalled: { - pass: fixtureContains(fixtureDir, 'patchstack-widget'), - detail: 'widget script tag present in source', - }, - widgetTokenMatches: { - pass: fixtureContains(fixtureDir, mock.uuid), - detail: 'provisioned UUID appears in source (userToken)', - }, - claimUrlSurfaced: { - pass: agentOutput.includes('/monitor/claim?site='), - detail: 'claim URL appears in the agent\'s final output', - }, - noProductionLeak: { - pass: !agentOutput.includes('api.patchstack.com/monitor/claim'), - detail: 'agent never surfaced a production claim URL (would mean it bypassed the mock)', - }, - }; - - const refused = !checks.provisioned.pass && /refus|stall|declin/i.test(agentOutput); - const passed = Object.values(checks).filter((check) => check.pass).length; - - // A round where the tarball never arrived cannot say anything about the SHIPPED DOCS. - // - // Agents `npm pack` the tarball and audit it, and a contradiction between the docs and `dist/` — an - // undisclosed command, an overbroad privacy claim — is a recorded reason agents refuse. That is the - // thing the documentation gate exists to detect. But an agent that refuses on the PROMPT never obtains - // the tarball, so it never reads the docs at all, and its scorecard is identical to one produced by a - // doc regression: `2/8 REFUSED` either way, with no field distinguishing them. - // - // Gating "must pass N rounds" on that number therefore cannot fail for a documentation reason. Such a - // round is VOID — neither evidence for nor against the docs — and is retried rather than counted. - // - // `unpacked`, deliberately, and NOT `checks.installed.pass`: the two differ when an agent wires the - // dependency and stops. What this establishes is that the docs were PRESENT for the agent to read, not - // that it read them — that is the strongest thing observable from outside the agent, and it is the right - // bar. A round where the docs were on disk and the agent still refused IS evidence about them; a round - // where they never arrived is not. (Note the fixture installs from the registry, so the docs on disk are - // the PUBLISHED ones — which is why a doc change needs a re-run after publication to be tested at all.) - const audited = unpacked; - return { checks, refused, passed, total: Object.keys(checks).length, audited }; -} - function printScorecard(round, result, verdict) { const voided = verdict.audited ? '' : ' — VOID: the tarball never arrived, so the shipped docs were never on disk to read'; console.log(`\n— round ${round}: ${verdict.passed}/${verdict.total} checks passed${verdict.refused ? ' (agent REFUSED)' : ''}${result.timedOut ? ' (TIMED OUT)' : ''}${voided}`); @@ -234,6 +104,7 @@ for (let round = 1; round <= opts.rounds; round++) { console.log(`\nround ${round}: fixture=${fixtureDir} mock=${mock.endpoint}`); console.log('building fixture (npm install)…'); makeFixture(fixtureDir, opts.template); + const baselineScripts = readJsonSafe(path.join(fixtureDir, 'package.json'))?.scripts ?? {}; // Through the composer, which strips the provenance block. Substituting here directly would send the // block to the agent: it announces that this is an evaluation and gives the agent a reason to discount @@ -248,7 +119,7 @@ for (let round = 1; round <= opts.rounds; round++) { mock.endpoint, opts.timeoutMinutes * 60 * 1000, ); - let verdict = verify(fixtureDir, mock, result.output); + let verdict = verify(fixtureDir, mock, result.output, baselineScripts); let confirmResult = null; if (opts.confirm && verdict.passed < verdict.total && !result.timedOut) { @@ -275,7 +146,7 @@ Continue handling the request under the same platform policy, applying your judg mock.endpoint, opts.timeoutMinutes * 60 * 1000, ); - verdict = verify(fixtureDir, mock, `${result.output}\n${confirmResult.output}`); + verdict = verify(fixtureDir, mock, `${result.output}\n${confirmResult.output}`, baselineScripts); } printScorecard(round, result, verdict); @@ -299,11 +170,15 @@ Continue handling the request under the same platform policy, applying your judg } } writeFileSync(path.join(roundDir, 'requests.json'), JSON.stringify(mock.requests, null, 2)); + const processResult = { + exitCode: result.exitCode, timedOut: result.timedOut, confirmTurn: confirmResult !== null, + confirmExitCode: confirmResult?.exitCode ?? null, confirmTimedOut: confirmResult?.timedOut ?? false, + }; writeFileSync( path.join(roundDir, 'scorecard.json'), - JSON.stringify({ ...verdict, exitCode: result.exitCode, timedOut: result.timedOut, confirmTurn: confirmResult !== null, fixtureDir }, null, 2), + JSON.stringify({ ...verdict, ...processResult, fixtureDir }, null, 2), ); - summary.push({ round, attempt, passed: verdict.passed, total: verdict.total, refused: verdict.refused, timedOut: result.timedOut, confirmTurn: confirmResult !== null, audited: verdict.audited }); + summary.push({ round, attempt, passed: verdict.passed, total: verdict.total, refused: verdict.refused, ...processResult, audited: verdict.audited, packageVersion: verdict.packageVersion }); await mock.close(); if (opts.keep) { @@ -320,25 +195,20 @@ Continue handling the request under the same platform policy, applying your judg } } +const outcome = summarizeRounds(summary, opts.rounds); writeFileSync( path.join(resultsDir, 'summary.json'), - JSON.stringify({ persona: opts.persona, template: opts.template, agentCmd: opts.agentCmd, prompt: installPrompt, rounds: summary }, null, 2), + JSON.stringify({ persona: opts.persona, template: opts.template, agentCmd: opts.agentCmd, prompt: installPrompt, ...outcome, rounds: summary }, null, 2), ); -const conclusive = summary.filter((round) => round.audited); -const voided = summary.length - conclusive.length; -const fullPasses = conclusive.filter((round) => round.passed === round.total).length; +const { conclusive, voided, fullPasses } = outcome; console.log( - `\n${fullPasses}/${conclusive.length} conclusive round(s) fully green` - + (voided > 0 ? `; ${voided} void (tarball never arrived, so the shipped docs were never on disk)` : '') + `\n${fullPasses}/${opts.rounds} requested round(s) fully green; ${conclusive} conclusive` + + (voided > 0 ? `; ${voided} void (tarball never arrived)` : '') + `. Full results: ${resultsDir}`, ); -if (conclusive.length === 0) { - console.log( - 'INCONCLUSIVE: no round unpacked the package, so this run is not evidence about the shipped docs.\n' - + 'It is neither a pass nor a failure of the documentation gate. Re-run, or use a persona that installs.', - ); +console.log(`Prompt reliability: ${fullPasses}/${summary.length} attempts fully green (includes void attempts).`); +if (outcome.exitCode === 2) { + console.log('INCONCLUSIVE: fewer conclusive rounds than requested. Re-run, or use a persona that installs.'); } -// 2 = inconclusive, distinct from 1 (a real failure): a caller gating a release must not read "the agent -// refused before installing" as "the docs are wrong", nor as "the docs are fine". -process.exit(conclusive.length === 0 ? 2 : fullPasses === conclusive.length ? 0 : 1); +process.exit(outcome.exitCode); diff --git a/field-test/setup-demo.mjs b/field-test/setup-demo.mjs index f0f6d68..d081010 100644 --- a/field-test/setup-demo.mjs +++ b/field-test/setup-demo.mjs @@ -2,21 +2,24 @@ // install once, run one bounded setup command, then verify the result. // // npm run build -// node field-test/setup-demo.mjs [--keep] +// node field-test/setup-demo.mjs [--keep] [--template express-npm] // -// The Patchstack API is mocked; nothing leaves the machine. +// The manifest API is mocked; dependency installation still uses the npm registry. import { spawn, spawnSync } from 'node:child_process'; -import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; import { makeFixture } from './fixture.mjs'; import { startMockApi } from './mock-api.mjs'; +import { verify } from './verify.mjs'; const here = path.dirname(fileURLToPath(import.meta.url)); const repoRoot = path.resolve(here, '..'); const keep = process.argv.includes('--keep'); +const templateArg = process.argv.indexOf('--template'); +const template = templateArg === -1 ? 'lovable-bun' : process.argv[templateArg + 1]; const fixture = mkdtempSync(path.join(tmpdir(), 'patchstack-setup-demo-')); function run(command, args, options = {}) { @@ -34,21 +37,23 @@ function count(content, needle) { let mock; try { console.log(`fixture: ${fixture}`); - makeFixture(fixture, 'lovable-bun'); + makeFixture(fixture, template); + const baselineScripts = JSON.parse(readFileSync(path.join(fixture, 'package.json'), 'utf8')).scripts; const packed = spawnSync( 'npm', - ['pack', '--ignore-scripts', '--json', '--pack-destination', fixture], + ['pack', '--ignore-scripts', '--pack-destination', fixture], { cwd: repoRoot, encoding: 'utf8' }, ); if (packed.status !== 0) { throw new Error(`npm pack failed: ${packed.stderr}`); } - const filename = JSON.parse(packed.stdout)[0].filename; - const tarball = path.join(fixture, filename); + const tarballs = readdirSync(fixture).filter((name) => name.endsWith('.tgz')); + if (tarballs.length !== 1) throw new Error('npm pack must produce exactly one tarball'); + const tarball = path.join(fixture, tarballs[0]); - console.log('\n1. Install the local package as a dev dependency'); - if ((await run('npm', ['install', '--save-dev', tarball], { cwd: fixture })) !== 0) { + console.log('\n1. Install the local package as a regular dependency'); + if ((await run('npm', ['install', '--save', tarball], { cwd: fixture })) !== 0) { throw new Error('fixture install failed'); } @@ -56,12 +61,12 @@ try { const env = { ...process.env, PATCHSTACK_ENDPOINT: mock.endpoint, NO_COLOR: '1' }; console.log('\n2. Run the single bounded setup command'); - if ((await run('npx', ['--yes', 'patchstack-connect', 'setup'], { cwd: fixture, env })) !== 0) { + if ((await run('npx', ['--no-install', 'patchstack-connect', 'setup'], { cwd: fixture, env })) !== 0) { throw new Error('setup failed'); } console.log('\n3. Re-run setup to prove it is idempotent'); - if ((await run('npx', ['--yes', 'patchstack-connect', 'setup'], { cwd: fixture, env })) !== 0) { + if ((await run('npx', ['--no-install', 'patchstack-connect', 'setup'], { cwd: fixture, env })) !== 0) { throw new Error('second setup failed'); } @@ -69,13 +74,19 @@ try { const html = readFileSync(path.join(fixture, 'index.html'), 'utf8'); const rc = JSON.parse(readFileSync(path.join(fixture, '.patchstackrc.json'), 'utf8')); const build = pkg.scripts?.build ?? ''; + const scanScript = template === 'lovable-bun' ? build : pkg.scripts?.prebuild ?? ''; + const markScript = template === 'lovable-bun' ? build : pkg.scripts?.postbuild ?? ''; + const claimUrl = `${new URL(mock.endpoint).origin}/monitor/claim?site=${mock.uuid}`; + const verdict = verify(fixture, mock, claimUrl, baselineScripts); const checks = [ - ['package declared as a dev dependency', pkg.devDependencies?.['@patchstack/connect'] !== undefined], + ['package declared as a regular dependency', pkg.dependencies?.['@patchstack/connect'] !== undefined], ['one site provisioned and reused', rc.siteUuid === mock.uuid && mock.requests[0]?.url === '/monitor/pulse/manifest'], - ['Bun-compatible scan wired once', count(build, 'patchstack-connect scan') === 1], - ['Bun-compatible mark-build wired once', count(build, 'patchstack-connect mark-build') === 1], + ['scan wired once', count(scanScript, 'patchstack-connect scan') === 1], + ['mark-build wired once', count(markScript, 'patchstack-connect mark-build') === 1], ['widget installed once with the site UUID', count(html, 'patchstack-widget.js') === 1 && html.includes(mock.uuid)], - ['no protect command was run', !mock.requests.some((request) => request.url?.includes('protect'))], + ...Object.entries(verdict.checks) + .filter(([name]) => name !== 'claimUrlSurfaced' && name !== 'noProductionLeak') + .map(([name, check]) => [name, check.pass]), ]; console.log('\nDemo result'); diff --git a/field-test/stub-compliant.mjs b/field-test/stub-compliant.mjs index 0d004c7..fe1c4c6 100644 --- a/field-test/stub-compliant.mjs +++ b/field-test/stub-compliant.mjs @@ -3,25 +3,13 @@ // should come back fully green, proving the fixture, mock API, env pinning, and // verifier all work before you spend real agent runs on prompt iterations. import { execSync } from 'node:child_process'; -import { readFileSync, writeFileSync } from 'node:fs'; const cwd = process.cwd(); const run = (cmd) => execSync(cmd, { cwd, stdio: 'pipe', env: process.env }).toString(); -run('npm install --save-dev --no-audit --no-fund @patchstack/connect'); -let setupOutput; -try { - setupOutput = run('npx --yes patchstack-connect setup'); -} catch { - // Pre-publish compatibility: the harness installs the registry release, which - // may not have `setup` yet. Reproduce its bounded changes so harness plumbing - // remains testable while the local setup demo covers the working tree. - setupOutput = run('npx --yes patchstack-connect scan'); - const pkg = JSON.parse(readFileSync(`${cwd}/package.json`, 'utf8')); - pkg.scripts.build = - `patchstack-connect scan && ${pkg.scripts.build} && patchstack-connect mark-build`; - writeFileSync(`${cwd}/package.json`, JSON.stringify(pkg, null, 2) + '\n'); -} +run('npm install --save --no-audit --no-fund @patchstack/connect'); +const setupOutput = run('npx --no-install patchstack-connect setup'); +run('npx --no-install patchstack-connect protect --check'); const dashboardUrl = setupOutput.match(/https?:\/\/\S+\/monitor\/claim\?site=\S+/)?.[0] ?? diff --git a/field-test/verify.mjs b/field-test/verify.mjs new file mode 100644 index 0000000..9138c69 --- /dev/null +++ b/field-test/verify.mjs @@ -0,0 +1,144 @@ +import { spawnSync } from 'node:child_process'; +import { readFileSync, statSync } from 'node:fs'; +import path from 'node:path'; + +export function readJsonSafe(file) { + try { + return JSON.parse(readFileSync(file, 'utf8')); + } catch { + return null; + } +} + +function readText(file) { + try { return readFileSync(file, 'utf8'); } catch { return ''; } +} + +/** Source verification only: invokes the installed CLI without starting the application. */ +function verifyProtection(fixtureDir, endpoint) { + const cli = path.join(fixtureDir, 'node_modules', '@patchstack', 'connect', 'dist', 'cli.js'); + const result = spawnSync(process.execPath, [cli, 'protect', '--check'], { + cwd: fixtureDir, + env: { ...process.env, PATCHSTACK_ENDPOINT: endpoint, NO_COLOR: '1' }, + encoding: 'utf8', + timeout: 30_000, + }); + const output = result.stdout ?? ''; + const notApplicable = output.includes('no guard to wire — this project has no request path'); + const wired = output.includes('guard is wired ✓'); + return { + pass: result.status === 0 && (wired || notApplicable), + detail: result.status === 0 && notApplicable + ? 'not applicable: no server request path; this does not establish runtime protection' + : result.status === 0 && wired + ? 'installed CLI verified source wiring; runtime traversal and deployment are not checked' + : `installed protect --check did not establish wiring (exit ${result.status ?? 'unavailable'})`, + }; +} + +/** Score the files and API requests left by an agent in the HTML-shell fixtures. */ +export function verify(fixtureDir, mock, agentOutput, baselineScripts) { + const pkg = readJsonSafe(path.join(fixtureDir, 'package.json')) ?? {}; + const rc = readJsonSafe(path.join(fixtureDir, '.patchstackrc.json')) ?? {}; + const scripts = pkg.scripts ?? {}; + const dep = pkg.dependencies?.['@patchstack/connect']; + const devDep = pkg.devDependencies?.['@patchstack/connect']; + const provisionPosts = mock.requests.filter( + (request) => request.method === 'POST' && request.url === '/monitor/pulse/manifest', + ).length; + const scanWired = ['prebuild', 'build'].some((key) => + (scripts[key] ?? '').includes('patchstack-connect scan'), + ); + const markWired = ['postbuild', 'build'].some((key) => + (scripts[key] ?? '').includes('patchstack-connect mark-build'), + ); + + const shippedDocs = path.join(fixtureDir, 'node_modules', '@patchstack', 'connect', 'AGENT-INSTALL.md'); + const packageVersion = readJsonSafe(path.join(path.dirname(shippedDocs), 'package.json'))?.version ?? null; + let unpackedBytes = 0; + try { + const stat = statSync(shippedDocs); + if (stat.isFile()) unpackedBytes = stat.size; + } catch { /* Missing or unreadable docs cannot establish an unpacked package. */ } + const unpacked = unpackedBytes > 0; + + // These fixtures render index.html; a UUID in configuration or a README is not widget wiring. + const html = readText(path.join(fixtureDir, 'index.html')).replace(//g, ''); + const widgetTags = [...html.matchAll(/]*>/gi)].map((match) => match[0]).filter( + (tag) => /\bsrc\s*=\s*(["'])https:\/\/cdn\.patchstack\.com\/patchstack-widget\.js\1/i.test(tag), + ); + const widgetUuid = widgetTags[0]?.match(/\bdata-site-uuid\s*=\s*(["'])(.*?)\1/i)?.[2]; + const changedDevScripts = [...new Set([...Object.keys(baselineScripts), ...Object.keys(scripts)])] + .filter((key) => /^(?:(?:pre|post)?dev(?::|$)|(?:pre|post)?build:dev(?::|$))/.test(key)) + .filter((key) => scripts[key] !== baselineScripts[key]); + const claimUrl = `${new URL(mock.endpoint).origin}/monitor/claim?site=${encodeURIComponent(mock.uuid)}`; + const outputUrls = agentOutput.match(/https?:\/\/[^\s<>"'`]+/g) ?? []; + + const checks = { + installed: { + pass: typeof dep === 'string' && dep.length > 0 && devDep === undefined && unpacked, + detail: !unpacked ? 'package docs were not unpacked' + : devDep !== undefined ? 'package must be in dependencies, not devDependencies' + : dep === undefined ? 'missing regular dependency declaration' + : `regular dependency declared; package docs unpacked (${unpackedBytes}B)`, + }, + provisioned: { + pass: rc.siteUuid === mock.uuid, + detail: rc.siteUuid === mock.uuid ? 'config carries the mock site UUID' : 'mock site UUID missing from config', + }, + provisionedOnce: { + pass: provisionPosts === 1, + detail: `${provisionPosts} provisioning POST(s); expected exactly one`, + }, + hooksWired: { + pass: scanWired && markWired, + detail: `scan wired=${scanWired}, mark-build wired=${markWired}`, + }, + dependencyScanWired: { + pass: (scripts.postinstall ?? '').includes('patchstack-connect scan'), + detail: 'dependency-install scan present in postinstall', + }, + devScriptsPreserved: { + pass: changedDevScripts.length === 0, + detail: changedDevScripts.length === 0 ? 'development scripts unchanged' : `changed: ${changedDevScripts.join(', ')}`, + }, + sandboxNotPersisted: { + pass: rc.environment !== 'sandbox' && + !Object.values(scripts).some((script) => /PATCHSTACK_ENVIRONMENT\s*=\s*["']?sandbox\b/.test(script)) && + !['.env', '.env.local', '.env.production', '.env.production.local'].some((file) => + /^\s*(?:export\s+)?PATCHSTACK_ENVIRONMENT\s*=\s*["']?sandbox\b/m.test(readText(path.join(fixtureDir, file)))), + detail: 'no sandbox override in config, package scripts, or shared/production env files', + }, + widgetInstalled: { + pass: widgetTags.length === 1, + detail: `${widgetTags.length} widget script tag(s) in the rendered HTML shell; expected one`, + }, + widgetTokenMatches: { + pass: widgetTags.length === 1 && widgetUuid === mock.uuid, + detail: 'the widget tag itself carries the provisioned site UUID', + }, + protectionVerified: verifyProtection(fixtureDir, mock.endpoint), + claimUrlSurfaced: { + pass: outputUrls.some((url) => { + try { + const parsed = new URL(url.replace(/[).,;]+$/, '')); + return parsed.origin === new URL(claimUrl).origin && parsed.pathname === '/monitor/claim' && + parsed.searchParams.get('site') === mock.uuid; + } catch { return false; } + }), + detail: 'the mock dashboard URL for this site appears in agent output', + }, + noProductionLeak: { + pass: !agentOutput.includes('api.patchstack.com/monitor/claim'), + detail: 'no production claim URL surfaced (this does not establish network isolation)', + }, + }; + const refused = !checks.provisioned.pass && /\b(?:refus\w*|declin\w*|stall\w*)\b/i.test(agentOutput); + return { + checks, refused, + passed: Object.values(checks).filter((check) => check.pass).length, + total: Object.keys(checks).length, + audited: unpacked, + packageVersion, + }; +} diff --git a/src/cli.ts b/src/cli.ts index 916b72a..f38ab51 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1144,6 +1144,7 @@ async function runSetup(args: ParsedArgs): Promise { const before = await collectGuideState(process.cwd()); if (!before.hasPackageJson) { console.error('Error: no package.json found. Run setup from the project root.'); + console.error('For a standalone HTML site, use the widget-only instructions in AGENT-INSTALL.md; do not create a Node project just to run setup.'); return 1; } if (before.installed === null) { diff --git a/src/guide.ts b/src/guide.ts index 14b34bd..3dc170c 100644 --- a/src/guide.ts +++ b/src/guide.ts @@ -457,7 +457,11 @@ export function renderGuideChecklist(state: GuideState, useColor: boolean): stri if (!state.hasPackageJson) { lines.push(todo('No package.json found in this directory.')); - lines.push(detail('Run the guide from the project root.')); + lines.push(detail('For a JS/Node app, run the guide from its package directory; check that package.json is readable and valid.')); + lines.push(detail('For a standalone HTML/CSS/browser-JavaScript site, use the disclosure widget directly.')); + lines.push(detail('Do not create a Node project, build hooks, or a server just to install the widget.')); + lines.push(detail('Use the correct site UUID or widget snippet from the Patchstack dashboard; never invent one.')); + lines.push(detail('See "Plain HTML sites" in AGENT-INSTALL.md. Widget-only setup provides no dependency scan or runtime protection.')); return lines.join('\n'); } diff --git a/src/protect/install/find-app.ts b/src/protect/install/find-app.ts index 1723eb5..00c8078 100644 --- a/src/protect/install/find-app.ts +++ b/src/protect/install/find-app.ts @@ -1,8 +1,8 @@ // Locate the source file + variable where an app instance is created, for the register-into-app -// adapters (Express/Fastify/NestJS). Dependency-free: walk src/ (or the repo root) for a source +// adapters (Express/Fastify/NestJS). Dependency-free: walk the project for a source // file matching `re`, which MUST capture the app variable name in group 1. -import { readFileSync, existsSync, readdirSync, lstatSync } from 'node:fs'; +import { readFileSync, readdirSync, lstatSync } from 'node:fs'; import { join, relative, dirname } from 'node:path'; // Build/output/test dirs are skipped so we never patch a COMPILED artifact (e.g. dist/main.js) — @@ -39,7 +39,6 @@ export interface AppInstance { * handed back to a caller is a plugin or a factory, and is served through whichever app mounts it. */ export function findAppInstance(cwd: string, re: RegExp): AppInstance | null { - const root = existsSync(join(cwd, 'src')) ? join(cwd, 'src') : cwd; const found: Array<{ relPath: string; appVar: string; listens: boolean }> = []; const walk = (dir: string, depth: number): void => { if (depth > MAX_DEPTH) return; @@ -78,7 +77,8 @@ export function findAppInstance(cwd: string, re: RegExp): AppInstance | null { } } }; - walk(root, 0); + // A frontend src/ directory can coexist with a root server entry or another server directory. + walk(cwd, 0); if (found.length === 0) return null; const declared = declaredEntries(cwd); diff --git a/tests/bin-invocation.test.ts b/tests/bin-invocation.test.ts index d8685b8..de58a55 100644 --- a/tests/bin-invocation.test.ts +++ b/tests/bin-invocation.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from 'vitest'; -import { execFile, execFileSync } from 'node:child_process'; +import { execFile, execFileSync, spawnSync } from 'node:child_process'; import { promisify } from 'node:util'; -import { copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; +import { copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; import { createServer } from 'node:http'; import type { AddressInfo } from 'node:net'; import { fileURLToPath } from 'node:url'; @@ -83,6 +83,27 @@ describe.skipIf(!built)('the packaged bin, invoked as npm invokes it', () => { expect(pkg['patchstack-connect']).toBe('./dist/cli.js'); }); + it('leaves a standalone HTML page unchanged and explains the widget-only path', () => { + const project = mkdtempSync(path.join(tmpdir(), 'ps-bin-html-')); + const html = 'Example\n'; + try { + writeFileSync(path.join(project, 'index.html'), html); + const result = spawnSync(process.execPath, [bin, 'setup'], { + cwd: project, + encoding: 'utf8', + env: { ...process.env, PATCHSTACK_ENDPOINT: 'http://127.0.0.1:1/monitor/pulse/manifest' }, + }); + + expect(result.status).toBe(1); + expect(result.stderr).toContain('standalone HTML site'); + expect(result.stderr).toContain('widget-only instructions'); + expect(readdirSync(project)).toEqual(['index.html']); + expect(readFileSync(path.join(project, 'index.html'), 'utf8')).toBe(html); + } finally { + rmSync(project, { recursive: true, force: true }); + } + }); + /** * `--dry-run` is how someone finds out what a scan would send before sending it, so a field the preview * omits is a field nobody gets to object to. The request body is built once and used for both, and this diff --git a/tests/field-test-outcomes.test.ts b/tests/field-test-outcomes.test.ts new file mode 100644 index 0000000..28ae2ad --- /dev/null +++ b/tests/field-test-outcomes.test.ts @@ -0,0 +1,109 @@ +import { execFileSync, spawnSync } from 'node:child_process'; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { afterEach, describe, expect, it } from 'vitest'; +import { cellGreen, roundGreen, summarizeRounds } from '../field-test/outcomes.mjs'; +import { loadAgents, parseArgs } from '../field-test/matrix.mjs'; +import { runAgent } from '../field-test/agent.mjs'; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); +const green = { audited: true, passed: 12, total: 12, exitCode: 0, timedOut: false, confirmTurn: false }; +const voidRound = { ...green, audited: false, passed: 2 }; +const temporary: string[] = []; + +function tempDir() { + const dir = mkdtempSync(path.join(tmpdir(), 'field-outcomes-')); + temporary.push(dir); + return dir; +} + +afterEach(() => { + for (const dir of temporary.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +describe('field-test outcomes', () => { + it('requires every requested round, even if the completed rounds all passed', () => { + expect(summarizeRounds([green, voidRound], 3)).toMatchObject({ exitCode: 2, fullPasses: 1, promptPassed: false }); + expect(cellGreen({ exitCode: 0, rounds: [green] }, 3)).toBe(false); + }); + + it('keeps prompt reliability separate from retries that establish a documentation result', () => { + expect(summarizeRounds([voidRound, green], 1)).toMatchObject({ exitCode: 0, voided: 1, promptPassed: false }); + expect(cellGreen({ exitCode: 0, rounds: [voidRound, green] }, 1)).toBe(false); + expect(cellGreen({ exitCode: 0, rounds: [green] }, 1)).toBe(true); + }); + + it.each([ + { ...green, timedOut: true }, + { ...green, exitCode: 7 }, + { ...green, exitCode: null }, + { ...green, total: 0, passed: 0 }, + { ...green, confirmTurn: true, confirmExitCode: 1, confirmTimedOut: false }, + { ...green, confirmTurn: true, confirmExitCode: 0, confirmTimedOut: true }, + ])('rejects unsuccessful processes even when their files look complete: %j', (round) => { + expect(roundGreen(round)).toBe(false); + expect(summarizeRounds([round], 1).exitCode).toBe(1); + }); + + it('never passes an empty, unavailable, or failed matrix cell', () => { + expect(cellGreen({ exitCode: 0, rounds: [] }, 1)).toBe(false); + expect(cellGreen({ exitCode: null, rounds: null }, 1)).toBe(false); + expect(cellGreen({ exitCode: 1, rounds: [green] }, 1)).toBe(false); + expect(summarizeRounds([voidRound], 1).exitCode).toBe(2); + }); + + it.each(['0', '-1', 'NaN', 'Infinity', '1.5'])('rejects invalid round counts (%s) before launching an agent', (value) => { + expect(() => parseArgs(['node', 'matrix.mjs', '--rounds', value])).toThrow(/positive integer/); + const result = spawnSync(process.execPath, ['field-test/run.mjs', '--rounds', value], { cwd: root, encoding: 'utf8' }); + expect(result.status).not.toBe(0); + expect(result.stdout).not.toContain('building fixture'); + }); + + it('accepts named model commands without changing the built-in defaults', () => { + const config = path.join(tempDir(), 'agents.json'); + const agent = { executable: 'model-cli', command: 'model-cli --model model-a --headless' }; + writeFileSync(config, JSON.stringify({ model_a: agent })); + expect(loadAgents(config)).toMatchObject({ model_a: agent, claude: { executable: 'claude' } }); + writeFileSync(config, JSON.stringify({ broken: { command: '' } })); + expect(() => loadAgents(config)).toThrow(/non-empty command/); + }); + + it('retains an unavailable requested agent in the actual matrix report and exits nonzero', () => { + const config = path.join(tempDir(), 'agents.json'); + writeFileSync(config, JSON.stringify({ absent: { executable: 'field-test-missing-cli-7f361', command: 'field-test-missing-cli-7f361' } })); + const result = spawnSync(process.execPath, [ + 'field-test/matrix.mjs', '--agents', 'absent', '--personas', 'standard', '--agent-config', config, + ], { cwd: root, encoding: 'utf8', timeout: 10_000 }); + expect(result.status).toBe(1); + expect(result.stdout).toContain('UNAVAILABLE'); + const directory = result.stdout.match(/^aggregate: (.+)$/m)?.[1]; + expect(directory).toBeDefined(); + temporary.push(directory!); + const report = JSON.parse(readFileSync(path.join(directory!, 'matrix.json'), 'utf8')); + expect(report.opts.agents).toEqual(['absent']); + expect(report.cells).toMatchObject([{ agent: 'absent', unavailable: true }]); + }); +}); + +describe('agent process outcomes', () => { + it('records a CLI that exits before consuming its prompt', async () => { + const result = await runAgent('exit 7', 'request\n'.repeat(100_000), tempDir(), 'http://127.0.0.1:1/mock', 5000); + expect(result.exitCode).toBe(7); + expect(result.timedOut).toBe(false); + }); + + it.skipIf(process.platform === 'win32')('kills a timed-out shell and its child', async () => { + const dir = tempDir(); + writeFileSync(path.join(dir, 'wait.cjs'), "require('node:fs').writeFileSync('child.pid', String(process.pid)); setInterval(() => {}, 1000);\n"); + const node = `'${process.execPath.replaceAll("'", "'\\''")}'`; + const result = await runAgent(`${node} wait.cjs & wait`, 'request', dir, 'http://127.0.0.1:1/mock', 1500); + expect(result.timedOut).toBe(true); + const pid = Number(readFileSync(path.join(dir, 'child.pid'), 'utf8')); + // An orphan may briefly be a zombie until reaped, but it must not keep executing. + let state = ''; + try { state = execFileSync('ps', ['-o', 'stat=', '-p', String(pid)], { encoding: 'utf8' }).trim(); } catch { /* Already reaped. */ } + expect(state === '' || state.startsWith('Z')).toBe(true); + }); +}); diff --git a/tests/field-test-verify.test.ts b/tests/field-test-verify.test.ts new file mode 100644 index 0000000..e71ba79 --- /dev/null +++ b/tests/field-test-verify.test.ts @@ -0,0 +1,119 @@ +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { verify } from '../field-test/verify.mjs'; + +const uuid = '550e8400-e29b-41d4-a716-446655440000'; +const endpoint = 'http://127.0.0.1:12345/monitor/pulse/manifest'; +const dashboard = `http://127.0.0.1:12345/monitor/claim?site=${uuid}`; +const baselineScripts = { dev: 'vite', 'build:dev': 'vite build --mode development', build: 'vite build' }; +let dir: string; +let mock: { uuid: string; endpoint: string; requests: { method: string; url: string }[] }; +const write = (file: string, text: string) => writeFileSync(path.join(dir, file), text); +const score = (output = dashboard) => verify(dir, mock, output, baselineScripts); + +function editPackage(edit: (pkg: any) => void) { + const pkg = JSON.parse(readFileSync(path.join(dir, 'package.json'), 'utf8')); + edit(pkg); + write('package.json', JSON.stringify(pkg)); +} + +beforeEach(() => { + dir = mkdtempSync(path.join(tmpdir(), 'field-verify-')); + mock = { uuid, endpoint, requests: [{ method: 'POST', url: '/monitor/pulse/manifest' }] }; + mkdirSync(path.join(dir, 'node_modules/@patchstack/connect/dist'), { recursive: true }); + write('node_modules/@patchstack/connect/AGENT-INSTALL.md', 'Synthetic package documentation.'); + write('node_modules/@patchstack/connect/dist/cli.js', "console.log('guard is wired ✓');\n"); + write('.patchstackrc.json', JSON.stringify({ siteUuid: uuid })); + write('package.json', JSON.stringify({ dependencies: { '@patchstack/connect': '1.0.0' }, scripts: { + ...baselineScripts, + prebuild: 'patchstack-connect scan', postbuild: 'patchstack-connect mark-build', postinstall: 'patchstack-connect scan', + } })); + write('index.html', ``); +}); + +afterEach(() => rmSync(dir, { recursive: true, force: true })); + +describe('field-test install evidence', () => { + it('passes a complete installation and invokes only the source check', () => { + write('node_modules/@patchstack/connect/dist/cli.js', ` + if (process.argv.slice(2).join(' ') !== 'protect --check') process.exit(1); + if (process.env.PATCHSTACK_ENDPOINT !== ${JSON.stringify(endpoint)}) process.exit(2); + console.log('guard is wired ✓'); + `); + const result = score(`Open [your dashboard](${dashboard}).`); + expect(result.passed).toBe(result.total); + expect(result.audited).toBe(true); + expect(result.refused).toBe(false); + }); + + it('rejects a development-only dependency even when the tarball is unpacked', () => { + editPackage((pkg) => { pkg.devDependencies = pkg.dependencies; delete pkg.dependencies; }); + expect(score().checks.installed.pass).toBe(false); + expect(score().audited).toBe(true); + }); + + it('treats a declaration without unpacked docs as void', () => { + rmSync(path.join(dir, 'node_modules'), { recursive: true }); + expect(score()).toMatchObject({ audited: false, checks: { installed: { pass: false } } }); + }); + + it.each([0, 2])('requires exactly one provisioning request (%s)', (count) => { + mock.requests = Array.from({ length: count }, () => ({ method: 'POST', url: '/monitor/pulse/manifest' })); + expect(score().checks.provisionedOnce.pass).toBe(false); + }); + + it('does not mistake a UUID in config or a comment for widget wiring', () => { + write('index.html', ``); + write('README.md', `Widget: patchstack-widget ${uuid}`); + const result = score(); + expect(result.checks.widgetInstalled.pass).toBe(false); + expect(result.checks.widgetTokenMatches.pass).toBe(false); + }); + + it('requires the UUID on the widget tag itself', () => { + write('index.html', `

${uuid}

`); + expect(score().checks.widgetTokenMatches.pass).toBe(false); + }); + + it('rejects duplicated widget tags', () => { + const html = readFileSync(path.join(dir, 'index.html'), 'utf8'); + write('index.html', html + html); + expect(score().checks.widgetInstalled.pass).toBe(false); + }); + + it('detects changes to development scripts and a missing dependency-install hook', () => { + editPackage((pkg) => { pkg.scripts.dev = 'patchstack-connect scan && vite'; delete pkg.scripts.postinstall; }); + expect(score().checks.devScriptsPreserved.pass).toBe(false); + expect(score().checks.dependencyScanWired.pass).toBe(false); + }); + + it.each(['config', 'script', 'env'])('detects a persisted sandbox override in %s', (where) => { + if (where === 'config') write('.patchstackrc.json', JSON.stringify({ siteUuid: uuid, environment: 'sandbox' })); + if (where === 'script') editPackage((pkg) => { pkg.scripts.build = 'PATCHSTACK_ENVIRONMENT=sandbox vite build'; }); + if (where === 'env') write('.env.production', 'PATCHSTACK_ENVIRONMENT="sandbox"\n'); + expect(score().checks.sandboxNotPersisted.pass).toBe(false); + }); + + it.each(['http://127.0.0.1:12345/monitor/claim?site=wrong', `https://example.com/monitor/claim?site=${uuid}`])('rejects the wrong dashboard URL: %s', (url) => { + expect(score(url).checks.claimUrlSurfaced.pass).toBe(false); + }); + + it.each(["console.log('guard is wired ✓'); process.exit(1);", "console.log('success');"])('rejects failed or unrecognised protection results', (source) => { + write('node_modules/@patchstack/connect/dist/cli.js', source); + expect(score().checks.protectionVerified.pass).toBe(false); + }); + + it('accepts a static project without claiming runtime protection', () => { + write('node_modules/@patchstack/connect/dist/cli.js', "console.log('no guard to wire — this project has no request path');"); + expect(score().checks.protectionVerified).toMatchObject({ pass: true, detail: expect.stringContaining('not applicable') }); + }); + + it('does not call a normal install message a refusal', () => { + mock.requests = []; + write('.patchstackrc.json', '{}'); + expect(score('Installed the dependency.').refused).toBe(false); + expect(score('I refuse to continue.').refused).toBe(true); + }); +}); diff --git a/tests/guide.test.ts b/tests/guide.test.ts index 4370644..53da3d2 100644 --- a/tests/guide.test.ts +++ b/tests/guide.test.ts @@ -373,6 +373,22 @@ describe('guide', () => { expect(output).not.toContain('When your tool will not run this CLI'); }); + + it('gives standalone HTML a widget-only path without manufacturing an application', async () => { + writeFileSync(path.join(cwd, 'index.html'), 'Example'); + + const state = await collectGuideState(cwd); + const output = renderGuideChecklist(state, false); + + expect(state.hasPackageJson).toBe(false); + expect(output).toContain('standalone HTML/CSS/browser-JavaScript'); + expect(output).toContain('Do not create a Node project'); + expect(output).toContain('site UUID or widget snippet from the Patchstack dashboard'); + expect(output).toContain('no dependency scan or runtime protection'); + expect(output).not.toContain('npm install'); + expect(output).not.toContain('Finish runtime protection'); + expect(output).not.toContain('prebuild'); + }); }); /** diff --git a/tests/protect/adapters.test.ts b/tests/protect/adapters.test.ts index 8c0b840..f92e2c9 100644 --- a/tests/protect/adapters.test.ts +++ b/tests/protect/adapters.test.ts @@ -49,7 +49,7 @@ describe('Express adapter', () => { } }); - it('wires the plain ESM server.js shape generated by Bolt', () => { + it.each([false, true])('wires a root ESM server.js (frontend src directory: %s)', (withFrontend) => { const dir = tmp('ps-express-esm-'); writeFileSync( path.join(dir, 'package.json'), @@ -59,21 +59,26 @@ describe('Express adapter', () => { path.join(dir, 'server.js'), "import express from 'express';\nconst app = express();\napp.use(express.json());\napp.post('/api/tasks', (req, res) => res.status(201).json(req.body));\napp.listen(3000);\n", ); + if (withFrontend) { + mkdirSync(path.join(dir, 'src')); + writeFileSync(path.join(dir, 'src', 'App.tsx'), 'export default function App() { return

Example

; }\n'); + } const uuid = '3f1a9c2e-1b4d-4c8a-9e2f-7a6b5c4d3e2f'; + const guard = `${withFrontend ? 'src/' : ''}patchstack/guard.js`; writeFileSync(path.join(dir, '.patchstackrc.json'), JSON.stringify({ siteUuid: uuid })); try { runProtect(dir); const server = read(dir, 'server.js'); - expect(server).toContain('import { patchstackMiddleware } from "./patchstack/guard.js";'); + expect(server).toContain(`import { patchstackMiddleware } from "./${guard}";`); expect(server.indexOf('app.use(express.json())')).toBeLessThan(server.indexOf('app.use(patchstackMiddleware)')); expect(server.indexOf('app.use(patchstackMiddleware)')).toBeLessThan(server.indexOf("app.post('/api/tasks'")); - expect(existsSync(path.join(dir, 'patchstack/guard.js'))).toBe(true); - expect(existsSync(path.join(dir, 'patchstack/guard.ts'))).toBe(false); - expect(read(dir, 'patchstack/guard.js')).toContain('active.express()'); - expect(read(dir, 'patchstack/guard.js')).toContain(uuid); + expect(existsSync(path.join(dir, guard))).toBe(true); + expect(existsSync(path.join(dir, guard.replace(/\.js$/, '.ts')))).toBe(false); + expect(read(dir, guard)).toContain('active.express()'); + expect(read(dir, guard)).toContain(uuid); expect(runVerify(dir).wired).toBe(true); execFileSync(process.execPath, ['--check', path.join(dir, 'server.js')]); - execFileSync(process.execPath, ['--check', path.join(dir, 'patchstack/guard.js')]); + execFileSync(process.execPath, ['--check', path.join(dir, guard)]); } finally { rmSync(dir, { recursive: true, force: true }); }