Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,8 @@ reads exactly like passing.

Changing onboarding, the install prompt or the setup guide? Read `MAINTAINING.md` first — it maps which
files are load-bearing — and note that the install prompt is a tested artifact with its own gate in
`field-test/`.
`field-test/`. That gate is structural regression coverage against synthetic personas, not evidence about
any live platform's policy; `field-test/README.md` says what it does and does not establish.

## Two packaging decisions, and why they are what they are

Expand Down
2 changes: 2 additions & 0 deletions MAINTAINING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ node field-test/run.mjs --persona hostile --rounds 3

It runs a real AI agent through the full install in a throwaway fixture against a mocked API and scores the outcome. Read each round's `report.md` (the HESITATIONS / DECISION ANALYSIS sections) — anything an agent pauses on is a future refusal. See [`field-test/README.md`](field-test/README.md) for the improve-and-retest loop, the safety model, and what to do when the gate is red for environmental reasons (stale published docs, release freshness).

What it establishes is **structural regression coverage**: the prompt still survives pressures that once broke it. The personas are synthetic — written from our own analysis, not from any platform's policy text — so a green run is not evidence that a live platform accepts the prompt, and should not be reported as though it were.

Caveat: the fixture installs the **published** package, so an unpublished `guide`/CLI change can't be exercised end-to-end — publish first, or accept that the run validates only the prompt shape.

## Don'ts (these are refusal triggers, not style nits)
Expand Down
27 changes: 17 additions & 10 deletions field-test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,29 @@ Runs a real AI coding agent against the [README install prompt](../README.md#ins

Dev-only: nothing in this directory ships in the npm package.

## The personas are synthetic
## What this harness is, and what it is not

Each persona reproduces the *pressures* that a class of platform applies to a request like this one —
It is **structural regression coverage** for the install prompt. It does not establish that any real
platform will accept the prompt, and no result from it should be reported as though it did.

Each persona reproduces the *pressures* a class of platform applies to a request like this one — a
non-technical user, supply-chain caution, a runtime that stages commands rather than running them — written
from our own analysis of observed behaviour.
from our own analysis of observed behaviour. They are not the platforms' system prompts and not paraphrases
of them: third-party prompts are not ours to publish, whatever the tarball excludes.

That trade is worth stating plainly rather than working around. A synthetic persona is weaker evidence than
the policy a product actually runs, so a green run means the prompt still survives the SHAPE of a pressure
that once broke it. It is a regression check, and its value is that the pressures encoded here are ones
that produced real refusals — every clause in the prompt exists because a run like this failed without it.

They are not the platforms' system prompts and not paraphrases of them: third-party prompts are not ours to
publish, whatever the tarball excludes. That is a real trade. A synthetic persona is weaker evidence than
the policy a product actually runs, so what this harness proves is that the prompt survives the SHAPE of
the pressure. Higher-fidelity evaluation, and the detailed record of which refusals came from where,
belong in a private evaluation repository.
What it cannot tell you is whether a live platform's current policy accepts the prompt. A hosted product is
(prompt × model × runtime × UI), and this reproduces one of the four. Treat a refusal reported from the
field as new coverage to add, not as a failure of this harness to have predicted it.

`{{FIXTURE_DIR}}` and `{{INSTALL_PROMPT}}` are the only contract a persona has to satisfy; a new one is a
markdown file using both.

Provenance — that a policy is synthetic, why, and what the harness is proving — goes in a
Provenance — that a policy is synthetic, why, and what the persona is exercising — goes in a
`<!-- field-test:meta ... -->` block. The runner strips those blocks before the prompt reaches the agent,
and `runAgent` REFUSES to write a prompt still containing one. The refusal sits at the seam —
`child.stdin.write` — rather than in the runner, so it covers every call site including ones not written
Expand Down Expand Up @@ -224,5 +231,5 @@ Until a publish lands and ages, use this ladder instead of burning hostile round

## Known limitations

- The personas are synthetic and the matrix covers multiple model families, but a hosted platform is still (prompt × model × runtime × UI) — and neither the real policy text, the runtime, nor the UI layer is reproduced here. A green harness is necessary and not sufficient: a refusal has been found by a real user after this harness passed a prompt. Treat a real-world refusal report as a new persona — encode the pressure it applied into `personas/`, in your own words, so the regression stays covered.
- The personas are synthetic and the matrix covers multiple model families, but a hosted platform is still (prompt × model × runtime × UI) — and neither the real policy text, the runtime, nor the UI layer is reproduced here. A green harness is necessary and not sufficient, and this is not hypothetical: a refusal has been found by a real user after this harness passed a prompt. Treat a real-world refusal report as a new persona — encode the pressure it applied into `personas/`, in your own words, so the regression stays covered. Do not describe a green run as evidence about a platform's policy.
- The fixture installs the *published* package. An unpublished `guide`/CLI change can't be exercised end-to-end by the agent (it will install the registry version); publish first or accept that the run validates the prompt shape only.
Loading