Skip to content

Cover every shipped policy with a matching and a non-matching sample - #213

Merged
patchstackdave merged 1 commit into
mainfrom
test/default-policy-fixtures
Sep 3, 2026
Merged

patchstackdave merged 1 commit into
mainfrom
test/default-policy-fixtures

Conversation

@patchstackdave

@patchstackdave patchstackdave commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Every policy the guard ships has two samples: one it must mask, and the nearest thing it must leave alone.

Policy Must mask Must leave alone
private key a PEM block prose about a private key
AWS access key id the upper-case id a lower-case run
Google API key the full-length key a shorter run
vendor token a live key a test key
Supabase secret key sb_secret_… sb_publishable_…
Supabase service_role a service_role JWT an anon JWT
database URL one carrying credentials one without
Node stack trace a real frame " at " in prose
SQL error SQLSTATE[…] an ordinary error message
exception trace a Python traceback a class name that is not an exception
egress internal address loopback, metadata a public address literal

Why the second column is the point

A pattern that masks a secret is easy to write and easy to write too broadly, and a policy that masks legitimate content breaks the response it was meant to protect. The anon key is public by design; a publishable key is meant for public clients; a test key belongs in a response body.

What makes each case prove something about its own policy

One policy per case. Each body is screened through the named policy alone. Screened through the whole shipped set, "the sample was masked" says only that something masked it, and a case can pass while the policy it names matches nothing.

The policy's own sample as the control. Its matching sample travels in the same response as its benign one, because "the benign value survived" is also what an unreached policy looks like. The same rule, in one body, masks one and leaves the other.

Prefilter reachability. A prefilter is a gate — the pattern runs only when one of its anchors is in the body — so each policy is checked to have an anchor present in its own matching sample. An anchor absent from the content the policy is for is a policy that cannot fire, and the body comes back unmasked for a reason that looks like the pattern simply not matching.

The case list is asserted against the shipped set, so a policy without a sample fails here.

Verification

2348 tests pass; typecheck clean.

Note on the fixtures

Prefixed keys are assembled from parts rather than written out: a literal in a provider's live-key shape is what secret scanning exists to find, and it cannot tell a synthetic one from a real one. Every value in the file is synthetic.

@coderbuds

coderbuds Bot commented Sep 3, 2026 •

Copy link
Copy Markdown

Comprehensive synthetic tests thoroughly validate every default response and egress policy.

🎯 Quality: 98% Elite · 📦 Size: Medium

📈 This month: Your 178th PR — above team average · Averaging Good

See how your team is trending →

@patchstackdave

Copy link
Copy Markdown
Contributor Author

/review

Each shipped policy has a sample it must mask and the nearest thing it must leave
alone: a test key beside a live one, a publishable key beside a secret one, the
public `anon` role beside `service_role`, a connection string without credentials
beside one with them, prose containing " at " beside a stack frame.

The non-matching half is what the file is for. A pattern that masks a secret is
easy to write and easy to write too broadly, and a policy that masks legitimate
content breaks a response it was meant to protect.

Each case runs through its own policy alone. Screened through the whole shipped
set, "the sample was masked" says only that something masked it, and a case can
pass while the policy it names matches nothing.

The policy's own matching sample travels in the same response as its benign one,
because "the benign value survived" is also what an unreached policy looks like.
The same rule, in one body, masks one and leaves the other.

A prefilter is a gate — the pattern runs only when one of its anchors is in the
body — so each policy is checked to have an anchor present in its own matching
sample. An anchor absent from the content the policy is for is a policy that
cannot fire, and the body comes back unmasked for a reason that looks like the
pattern not matching.

The case list is asserted against the shipped set, so a policy without a sample
fails here.

Prefixed keys are assembled from parts rather than written out: a literal in a
provider's live-key shape is what secret scanning exists to find, and it cannot
tell a synthetic one from a real one. Every value in the file is synthetic.
@patchstackdave
patchstackdave force-pushed the test/default-policy-fixtures branch from 3bc2fe2 to 3e39ed6 Compare September 3, 2026 12:51
@patchstackdave
patchstackdave merged commit 22620cd into main Sep 3, 2026
14 checks passed
@patchstackdave
patchstackdave deleted the test/default-policy-fixtures branch September 3, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants