Cover every shipped policy with a matching and a non-matching sample - #213
Merged
Merged
Conversation
|
Comprehensive synthetic tests thoroughly validate every default response and egress policy. 🎯 Quality: 98% Elite · 📦 Size: Medium 📈 This month: Your 178th PR — above team average · Averaging Good |
Contributor
Author
|
/review |
daniloradovic
approved these changes
Sep 3, 2026
Each shipped policy has a sample it must mask and the nearest thing it must leave alone: a test key beside a live one, a publishable key beside a secret one, the public `anon` role beside `service_role`, a connection string without credentials beside one with them, prose containing " at " beside a stack frame. The non-matching half is what the file is for. A pattern that masks a secret is easy to write and easy to write too broadly, and a policy that masks legitimate content breaks a response it was meant to protect. Each case runs through its own policy alone. Screened through the whole shipped set, "the sample was masked" says only that something masked it, and a case can pass while the policy it names matches nothing. The policy's own matching sample travels in the same response as its benign one, because "the benign value survived" is also what an unreached policy looks like. The same rule, in one body, masks one and leaves the other. A prefilter is a gate — the pattern runs only when one of its anchors is in the body — so each policy is checked to have an anchor present in its own matching sample. An anchor absent from the content the policy is for is a policy that cannot fire, and the body comes back unmasked for a reason that looks like the pattern not matching. The case list is asserted against the shipped set, so a policy without a sample fails here. Prefixed keys are assembled from parts rather than written out: a literal in a provider's live-key shape is what secret scanning exists to find, and it cannot tell a synthetic one from a real one. Every value in the file is synthetic.
patchstackdave
force-pushed
the
test/default-policy-fixtures
branch
from
September 3, 2026 12:51
3bc2fe2 to
3e39ed6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every policy the guard ships has two samples: one it must mask, and the nearest thing it must leave alone.
sb_secret_…sb_publishable_…service_roleJWTanonJWTSQLSTATE[…]Why the second column is the point
A pattern that masks a secret is easy to write and easy to write too broadly, and a policy that masks legitimate content breaks the response it was meant to protect. The
anonkey is public by design; a publishable key is meant for public clients; a test key belongs in a response body.What makes each case prove something about its own policy
One policy per case. Each body is screened through the named policy alone. Screened through the whole shipped set, "the sample was masked" says only that something masked it, and a case can pass while the policy it names matches nothing.
The policy's own sample as the control. Its matching sample travels in the same response as its benign one, because "the benign value survived" is also what an unreached policy looks like. The same rule, in one body, masks one and leaves the other.
Prefilter reachability. A prefilter is a gate — the pattern runs only when one of its anchors is in the body — so each policy is checked to have an anchor present in its own matching sample. An anchor absent from the content the policy is for is a policy that cannot fire, and the body comes back unmasked for a reason that looks like the pattern simply not matching.
The case list is asserted against the shipped set, so a policy without a sample fails here.
Verification
2348 tests pass; typecheck clean.
Note on the fixtures
Prefixed keys are assembled from parts rather than written out: a literal in a provider's live-key shape is what secret scanning exists to find, and it cannot tell a synthetic one from a real one. Every value in the file is synthetic.