Withhold what masking cannot bound - #216
Merged
Merged
Conversation
|
Well-documented response rules with focused regex improvements and comprehensive tests. 🎯 Quality: 100% Elite · 📦 Size: Large — consider splitting if possible 🛡️ Standards: no pre-flight fit check ran for this change — wire 📈 This month: Your 182nd PR — above team average · Averaging Excellent |
patchstackdave
force-pushed
the
fix/withhold-what-redaction-cannot-bound
branch
3 times, most recently
from
September 4, 2026 12:06
4c6a85a to
c1d41b7
Compare
Contributor
Author
|
/review |
daniloradovic
approved these changes
Sep 4, 2026
patchstackdave
force-pushed
the
fix/withhold-what-redaction-cannot-bound
branch
2 times, most recently
from
September 4, 2026 12:52
f9bf75d to
f233e4b
Compare
`redact` masks the span a pattern matched and serves the rest of the page. It is the right action only
where the match IS the whole disclosure, which holds for a credential with a grammar: a provider token
has a prefix, an alphabet and a length, so matching it matches all of it. Five of the shipped response
policies are of that kind and mask.
The other five identify a disclosure they cannot delimit, and they withhold the response instead:
- a private key, whose material follows the BEGIN line;
- a Node stack frame, a database error signature and an exception header, each of which opens a
disclosure whose file names, line numbers, query text and remaining frames come after it;
- a connection URI carrying credentials, where the host, port, database name and query are the rest
of the disclosure and a URI's own grammar admits commas, parentheses and semicolons — so no
end-of-URI character class delimits one in free text without either stopping inside a real URI or
consuming the punctuation around it.
That URI's username is the userinfo grammar without `:`, and its password is the same grammar with it.
Every other character either class admits may appear raw in a real credential, so a narrower one turns
a live credential into a rule that says nothing — `postgres://user:p;ss@host` is an ordinary DSN.
The first raw colon separates username from password. Excluding it from the username makes the
separator unambiguous and keeps matching linear; the password continues to admit raw colons, and a
username that contains one carries it as `%3A`.
Both classes admitting `:` would leave every colon available as the separator, so a candidate run with
no `@` is re-split at every position. The screening cap does not bound that, since `max_bytes` raises
it and `bypass_limit` removes it. Two tests hold the property: one reads the username class and fails
if it admits a colon, the other screens a cap-sized adversarial candidate and fails outside the bound.
The structural one is necessary because the engine's expression guard accepts either form.
What terminates a candidate is everything the grammar excludes: whitespace, `/`, `?`, `#`, `@`, quotes,
backslashes, angle and square brackets and braces. That keeps a candidate inside one value. Expressed
as "anything but `:`, `@`, `/` and whitespace" it crosses structure instead, reaching the punctuation of
an unrelated field, so a body carrying a documentation URL in one property and an address in another is
withheld while disclosing nothing.
The consequence is deliberate. Punctuation-joined text that parses as a credential URI is treated as
one: `postgres://db.internal;contact:admin@example.com` has username `db.internal;contact`, password
`admin` and host `example.com`, which nothing distinguishes from a leak. Ordinary prose separates with
whitespace, which terminates the candidate.
The private-key pattern matches a PEM BEGIN line whose label contains `PRIVATE KEY`, with up to 32
further label characters — uppercase letters, digits, spaces and hyphens — on either side of it. That
covers the enumerated types, a label carrying words after `PRIVATE KEY` such as `PGP PRIVATE KEY
BLOCK`, a hyphenated or unlisted type, and a bare `-----BEGIN PRIVATE KEY-----`. `PUBLIC KEY` and
`CERTIFICATE` do not match, nor does prose mentioning a private key. No footer is required: a
truncated response or an absent END marker does not make the material above it less of a key. Two
bounded character classes rather than a repeated group, which the engine's expression guard refuses as
a backtracking risk — a refused pattern is a rule that never fires. A lowercase label, or one longer
than 32 characters on either side, is not matched.
The stack-frame pattern accepts a real newline and a JSON-escaped one, since most traces reach a client
inside a JSON error body where the newline is two characters.
`tests/protect/response-policy-wire-behaviour.test.ts` states each contract as the response a client
receives. For a masked credential: no eight-character prefix or tail of the value survives, the rest of
the body is unchanged, and JSON still parses. For a withheld one: the status, the exact withheld body,
and the absence of every field the disclosure carried. The private-key cases cover a complete PEM, CRLF
endings, a missing footer, a PGP key block, a hyphenated label, a label at the accepted boundary and a
bare marker, against four near-misses. The URI cases cover plain text, JSON, parentheses, a trailing
comma, `mongodb+srv`, two occurrences in one body, credentials carrying percent-encoded characters, and
passwords holding one and several raw colons, a semicolon, a comma, parentheses and sub-delimiters, and
a username carrying `%3A`. The negatives are the
bodies whose separator is outside the grammar — a quote or whitespace between the scheme and an
unrelated `:` and `@` — each asserted byte-identical at 200, alongside a credential-free URL whose
surrounding sentence is returned unchanged.
`default-policy-fixtures.test.ts` asserts that the named policy fired, rather than that its sample is
absent from the body: a withheld response removes the sample, and so does JSON escaping a sample that
contains a newline. Its benign near-miss is screened on its own request, because a withheld response
would otherwise take the control value with it.
patchstackdave
force-pushed
the
fix/withhold-what-redaction-cannot-bound
branch
from
September 4, 2026 13:04
f233e4b to
620ff99
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The rule
redactmasks the span a pattern matched and serves the rest of the page. It is correct only where thematch is the whole disclosure.
blockwithholds the response and replaces it with a generic error,and is used where a pattern can identify a disclosure but not delimit it.
The shipped response set splits on that line: five mask, five withhold.
redact[REDACTED], rest of the body unchangedredactredactredactservice_rolekeyredactblock{"error":"Response withheld by Patchstack (sensitive data detected)"}blockblockblockblockThe five that withhold each identify the start of a disclosure: the key material follows the BEGIN
line; the file names, line numbers, query text and remaining frames follow a frame, an error signature
or an exception header; and for a connection URI the host, port, database name and query are the rest of
the disclosure. Masking any of them leaves nearly all of it in a response that reports itself protected.
Why the connection URI withholds rather than masks
A URI's own grammar admits commas, parentheses and semicolons, so no end-of-URI character class
delimits one in free text. A class that excludes them stops inside a real URI and leaves the host and
database name; one that permits them consumes the punctuation around it —
connect via (postgres://…/app) todaycomes back as([REDACTED] today, with the sentence broken andthe disclosure only partly removed. Withholding is the only contract that is both complete and
non-destructive, and it is consistent with the four signature-identified policies.
The pattern requires credentials, so a URL without them is not matched: a sentence containing
postgres://db.internal:5432/appis returned byte for byte.Both userinfo components are the complete representable userinfo grammar, with the separator
disambiguated: the username is that grammar without raw
:, the password is the same grammarwith it.
That asymmetry is the grammar, not a concession. In
user:password@the first raw colon is theseparator, so a raw colon cannot belong to the username — a username holding one carries it as
%3A.A password may hold as many raw colons as it likes. Every other character either class admits appears
raw in real credentials, so a narrower class turns a live credential into a rule that says nothing.
What terminates a candidate is everything the grammar excludes — whitespace,
/,?,#,@,quotes, backslashes, angle and square brackets, braces — and that is what keeps a candidate inside one
value. Written as "anything but
:,@,/and whitespace" it crosses structure instead, so{"docs":"postgres://db.internal","contact":"user@example.com"}matches across the two properties and withholds a response that discloses nothing.
The separator is what keeps matching linear. The first raw colon separates username from
password, so excluding it from the username makes the separator unambiguous; the password continues to
admit raw colons, and a username holding one carries it as
%3A. Both classes admitting:wouldleave every colon available as the separator, so a candidate run with no
@is re-split at everyposition — and the screening cap does not bound that, since
max_bytesraises it andbypass_limitremoves it.
Two tests hold the property: one reads the username class and fails if it admits a colon, the other
screens a cap-sized adversarial candidate and fails outside the bound. The structural one is necessary
because the engine's expression guard accepts either form.
The remaining trade-off is explicit. Punctuation-joined text that parses as a credential URI is
treated as one:
postgres://db.internal;contact:admin@example.comhas usernamedb.internal;contact,password
adminand hostexample.com, which nothing distinguishes from a leak, so it is withheld.Ordinary prose separates with whitespace, which terminates the candidate. The negative controls are
only bodies whose separator is outside the grammar — a quote between two properties, two array
elements, whitespace in prose. Pinning a semicolon- or comma-joined body as safe would tell the rule
to ignore a real disclosure.
The private-key grammar
Matches a PEM BEGIN line whose label contains
PRIVATE KEY, with up to 32 further label characters— uppercase letters, digits, spaces and hyphens — on either side of it:
Covered: the enumerated types (
RSA,EC,OPENSSH,DSA,ENCRYPTED); a label carrying wordsafter
PRIVATE KEY, which is the real ASCII-armored form-----BEGIN PGP PRIVATE KEY BLOCK-----;a hyphenated or otherwise unlisted type; and a bare
-----BEGIN PRIVATE KEY-----. No footer isrequired — a truncated response or an absent END marker does not make the material above it less of a
key.
Not matched:
PUBLIC KEY,CERTIFICATE, prose that mentions a private key without a BEGIN line, alowercase label, and a label longer than 32 characters on either side. The last is a limitation, not
a guarantee, and is asserted as such.
Two bounded character classes rather than a repeated group: the engine's expression guard refuses a
quantifier inside a quantified group, and a refused pattern is a rule that never fires.
The stack-frame pattern
Accepts a real newline and a JSON-escaped one, since most traces reach a client inside a JSON error
body where the newline is the two characters
\andn.Tests
tests/protect/response-policy-wire-behaviour.test.ts(33 cases) states every contract as the responsea client receives, not as an action name:
unchanged, and JSON still parses;
carried (
index.js,42:15,users_email_unique,SQLSTATE,app.py,line 42, key material,db.internal,5432,sslmode);label at the accepted boundary, bare marker; against prose, a heading, a public key and a
certificate;
mongodb+srv, twice inone body; plus a credential-free URL whose surrounding sentence is returned byte for byte.
default-policy-fixtures.test.tsasserts that the named policy fired, rather than that its sampleis absent: a withheld response removes the sample, and so does JSON escaping a sample containing a
newline. Its benign near-miss is screened on its own request, since a withheld response would otherwise
take the control value with it.
Known limitations
PRIVATE KEY, or a lowercase label,does not fire this policy.
form that serves the page.
Verification
2419 tests pass, 154 files. Typecheck and the template typecheck harness clean.
wire()releases each protection instance through the publicstop()in afinally, so the testshold the documented lifecycle even if these guards later start something that needs releasing.