diff --git a/AGENT-INSTALL.md b/AGENT-INSTALL.md index 94be61d9..163df54d 100644 --- a/AGENT-INSTALL.md +++ b/AGENT-INSTALL.md @@ -144,18 +144,112 @@ would have stopped while it is still in dry-run. Two separate paths, with differ token; the `apiKey` itself is not sent to the log endpoint. Disable with `PATCHSTACK_TELEMETRY=off`, or `reportFirewallLog: false` in `createProtection`. - **Every rule that matched** goes to `monitor/pulse/detections/` — including matches that - blocked, which are reported on both paths. This is **off unless you pass `reportDetections: true`** to - `createProtection`; the scaffolded guard does not pass it. It also requires a provisioned site UUID, a - resolvable credential, and is disabled by `PATCHSTACK_TELEMETRY=off`. It exists because a rule carrying - `dry-run` blocks nothing, so without it nothing distinguishes a rule that is protecting from one that is - quietly wrong. - -What a detection report contains, per matched rule: the rule id, the request path **with any query string -removed**, the parameter names that rule reads (from the rule's own definition), which phase matched, -whether it was enforced, the identifier of the rule bundle in use, the revision of the rule itself when the -bundle carried one, and a timestamp. Each batch also -carries a count of reports dropped when traffic outran the flush, so a partial sample is not read as a -complete one. + blocked, which are reported on both paths. It exists because a rule carrying `dry-run` blocks nothing, so + without it nothing distinguishes a rule that is protecting from one that is quietly wrong. + + **This is on by default for a site enrolled with Patchstack that is running Patchstack-delivered rules**, + and off otherwise. Specifically, it requires all of: a provisioned site UUID, rules that came from + Patchstack rather than from a local bundle, and a resolvable credential. A local install, or a guard + running its own `rules`, sends nothing. + + Switch it off with **`PATCHSTACK_REPORT_DETECTIONS=0`**, or `reportDetections: false` in + `createProtection`, or `PATCHSTACK_TELEMETRY=off` which covers all telemetry. `reportDetections` is an + opt-out only — passing `true` cannot switch reporting on for a site that is not enrolled. + + `protection.detectionReporting` names the current state locally, so a guard that is not reporting says + which reason applies: `on`, `disabled-by-config`, `disabled-by-telemetry-opt-out`, `not-enrolled`, + `no-managed-rules`, or `unavailable-no-credential`. + + **How the state reaches Patchstack, and what that costs on the network.** The state travels as a header + on the rules request the guard already makes — no extra request for it. Two of the six never travel: + `not-enrolled` makes no site-addressed request at all, and `unavailable-no-credential` cannot produce an + authenticated one, and the header is withheld from unauthenticated requests. Those two are local + diagnostics only. + + There is one case that does add a request. The header is set before the rules request finishes, so a + guard booting with no cached rules declares `no-managed-rules` and then receives managed rules on that + same response. When that happens it sends **one immediate POST to the detections endpoint containing the + corrected state and no detections at all** — an empty `detections` array plus `reporting_state`. It is + sent once per process, only when the state changed, and never when the guard already had cached rules. + Without it, a guard with rule refreshing switched off would leave Patchstack holding the pre-resolution + answer for the life of the process. + +What a detection report contains, per matched rule — on every phase, whatever fired it: the rule id, the revision of the rule when the +bundle carried one, the identifier of the rule bundle in use, which phase matched, +whether it was enforced, the request path **with the query string's values removed**, +that query's parameter names, the method, and a timestamp. Each batch also carries a count of reports dropped when traffic outran the flush, +so a partial sample is not read as a complete one. + +Two fields depend on the phase, because one kind of detection has a client and the other does not. A +**request or response** detection also carries the user agent, +and the client address together with where that address came from. +An **egress** detection — a rule that fired on a call your application made outbound — carries neither: +the call was your application's own, so there is no visitor to attribute it to, and those fields read +`null` and `unavailable` rather than being guessed at. "What values a report can contain" below says the +same thing about captured evidence. + +Every field is bounded in size, and an event that had to be shortened says so. +`truncated` lists **which fields were shortened**. +`parameters_total` records **how many parameters the rule reads**, when a rule reads more than the event +names. +`query_keys_total` records **how many query parameters the request carried**, counted as DISTINCT names, +when it carried more than the event lists — a parameter repeated three times is one name to look up. The +names themselves are the ones the guard addresses a parameter by, so `?first+name=x` is reported as +`first name`. Both appear only when something really was shortened, so their absence is not a claim of its own — +and a shortened route or rule id is marked rather than passed off as complete, because a reader must not +use one as a key believing it names the whole thing. + +Delivery is retried, up to four attempts per batch, with exponential backoff and jitter, honouring a +`Retry-After` header when the endpoint sets one. Only failures worth retrying are retried — unreachable, +rate-limited, or a server error; a batch that was refused on its merits is not sent again. Every attempt +of one batch carries the same `Idempotency-Key` header, and a different batch carries a different one, so +a redelivery is identifiable as the same batch rather than a new one — an acknowledgement can be lost +after the server has already taken a batch. One request is in flight at a time, so a slow endpoint slows +the queue rather than opening more sockets; each attempt is abandoned after 10 seconds, so a request that +never settles cannot hold that slot; and a batch that exhausts its attempts is dropped and counted rather +than retried forever. Stopping a guard makes one last attempt at whatever is outstanding and counts +anything it could not send. `stop()` returns a promise that settles once the reporters have finished or +been given up on, so a shutdown handler can `await protection.stop()` instead of racing the last batch +against process exit. Each reporter has its own budget, and when it runs out that reporter is ended: its +requests are aborted, it starts nothing further, and it discards what it was holding. An abort is a +request to stop, not a guarantee — a transport that ignores it is detached rather than completed, so +"resolved" means the reporter is finished with it, and a runtime that kills the process still wins +regardless. Every detection event ends up delivered, refused or dropped and is reported in the health +counts; block-log records have no counters, so one lost to a failed send or an expired shutdown is +reported nowhere. + +The client address is reported with its **provenance**, because an address is only as trustworthy as +whatever supplied it. `client_ip_source` is one of `runtime` (the address the transport observed), +`trusted-proxy` (read from a forwarded header, through peers you declared via `trustedProxy`), or +`unavailable`. When it is `unavailable` the `client_ip` field is **omitted entirely** rather than sent +empty, so a missing address cannot read as a failed lookup of a real one. A forwarded header is never +trusted implicitly: with no `trustedProxy` policy the address is whatever the transport observed, and in a +runtime that exposes no transport peer there is no address to report at all. + +### Behaviour change: how the client address is determined + +The guard resolves the client address itself, once per request, and shares that one answer with rule +matching, block logging and detection reports — so those cannot disagree about who a request came from. +Two consequences if you are upgrading: + +- **Express and Node: forwarded headers are no longer read implicitly.** Earlier versions took the + address from `X-Forwarded-For`, `CF-Connecting-IP` or `X-Real-IP` (the Node guard), or from `req.ip` + (the Express guard, where it reflects Express's own `trust proxy` setting). Neither source can be + verified by the guard, and any client can send those headers, so both guards now read the transport + peer. **If your app runs behind a proxy or load balancer, addresses will now show as the proxy's** + until you declare your proxies with `trustedProxy` (below) — which affects attribution in reports and + any rule matching on `server.ip` or `REMOTE_ADDR`. +- **Fetch runtimes report no address at all.** A WHATWG `Request` exposes no transport peer, so a Fetch + guard (Workers, Deno, Bun, edge) has nothing to observe, and no forwarded header is accepted in its + place under any `trustedProxy` policy: `client_ip_source` is `unavailable` and no address is sent. + Earlier versions reported the forwarded header here, so an address-scoped rule that appeared to work on + such a runtime was matching a client-supplied value. + +`trustedProxy` is the only way to make a forwarded header count. It takes the proxies you actually run — +`{ peers: ['10.0.0.0/8'] }`, or `{ hops: 1 }` to trust that many hops in from the peer, plus optional +`header` and `isTrusted` — and the chain is then read from your application inward, stopping at the first +hop you have not declared. There are no built-in provider presets: a header a provider sets is +indistinguishable from one a client sent unless you say which peers may set it. The parameter names are **identifiers, and they name the request region they refer to** — `post.title`, `get.redirect_to`, `cookie.session`, `server.HTTP_AUTHORIZATION`. So a rule that inspects a cookie or an @@ -163,16 +257,79 @@ The parameter names are **identifiers, and they name the request region they ref definition, not from your traffic, so they describe what is being screened rather than what any request contained. -What it does not contain: **no values of any kind.** Not the value that matched, not the request body, -and not the value of any header, cookie or query-string parameter — including those of the parameters -named above. Reports are batched, capped in memory, and dropped rather than retried if Patchstack cannot +### What values a report can contain + +**A request or response detection** carries the request's method and path, +the query string's parameter **names**, +the user agent, the client address with its provenance, and a timestamp. + +**An egress detection** — a rule that fired on a request your application made outbound — carries the +outbound method and path, the query's parameter names, and a timestamp. It carries **no user agent and no +client address**: the call was your application's own, so there is no visitor to attribute it to, and +those fields read `null` and `unavailable` rather than being guessed at. + +Beyond that baseline, either can include the **values of the parameters a rule names** — and nothing +else. Counting that a +rule fired is not enough to act on it: whoever triages a detection still has to decide whether the request +was really an attack, and for that they need to see what the rule saw. + +**A rule earns each permission by naming what it reads.** What may be captured is derived from the rule +itself, never configured per site: + +- a rule naming a parameter (`post.title`, `cookie.session`, `server.HTTP_AUTHORIZATION`) permits **that + parameter's value**, because the rule was written to inspect it; +- a prefix (`post.field_*`) permits the values of keys that match, and no others; +- a rule reading `raw` or `all` — the whole request — permits **nothing at all**, so the broadest rules + grant the narrowest capture; +- **response** values are never captured: the phase that reads them exists to redact secrets, and + capturing them would collect the very values that redaction stops leaving; +- **raw request bytes** need an explicit, reviewed opt-in on the individual rule, and are then limited to + a short prefix of the body. + +**Everything is bounded, and the bounds report themselves.** +At most 10 values per detection, at most 512 characters each, and at most 5 values from any one prefix. A value shortened to fit is marked; values a bound +left out are counted; a value refused because it was not a plain string, number or boolean is counted +separately; and a read that failed is counted as a failure rather than as absence — so a short list is +never mistaken for a complete one. + +**A `capture.plan` identifies the permissions, not the rule.** Each report carries a `capture.plan` reference derived +from the permissions themselves — which parameters, which prefixes, which bounds — so what a given report +was permitted to include can be established after the fact, without the rule in front of you. It +identifies the PERMISSIONS, not the rule: two different rules that read the same parameters share one +reference, and the rule document is identified by `rule_id` with `rule_revision`. + +**Capture is the union of everything the rule reads, not only the condition that matched.** The engine +reports which rule fired, not which of its conditions did, so a rule reading `post.title` and +`cookie.session` permits both values whichever one triggered the detection. A rule scoped to one parameter +captures one; a broad rule captures what it is broad about. + +**One header value always travels, whatever the rule names: the User-Agent** — on a request or response +detection, where there is a client to attribute. It is part of the baseline above, because attribution is +what this channel is for and a detection without it cannot be told from another client's. It is the only +exception to the rule-scoped policy, and the only header value sent without a rule naming it. + +**What a report never contains:** the value of any parameter the matched rule does not name — the +User-Agent above excepted; any response body, header or status value; +the request body, other than the reviewed raw prefix above; +and anything at all from a rule that reads the whole request without that opt-in. + +**One qualification on the query string.** The exclusion above is about baseline URL metadata: `route` and +`query_keys` describe a URL without disclosing what was in it. It is not a promise about captured +evidence. A rule that names `egress.url` reads the outbound URL, so its capture carries that URL as the +rule read it — query values included. That is the rule-scoped policy working as described, not an +exception to it: the rule named the parameter, so the parameter's value travels. The value recorded is the request as the +engine resolved it — URL- and entity-decoded — and not the result of a rule's own further mutations. + +Reports are batched, capped in memory, retried a bounded number of times, and dropped if Patchstack cannot be reached — a reporting failure never delays or fails a request. -The endpoint needs a credential, so `reportDetections: true` with none resolved starts nothing: the guard -warns once at boot and `protection.detectionReporting` reads `unavailable-no-credential` instead of `on`. +The endpoint needs a credential, so an enrolled site with none resolved starts nothing: the guard warns +once at boot and `protection.detectionReporting` reads `unavailable-no-credential` instead of `on`. When reporting is on, `protection.detectionHealth()` returns local counts — detections attempted, acknowledged, refused or unreachable, dropped for queue pressure — and the time of the last -acknowledgement. Those counts stay in your process; nothing extra is sent to report them. +acknowledgement. Counts for the state POST described above are kept separately under `capability`, since it +carries no detections and would otherwise read as one. Those counts stay in your process; nothing extra is +sent to report them. `protection.stop()` stops everything the guard has running in the background — the rule-refresh loop, the block-log reporter, the detection reporter — and flushes what is buffered. `protection.stopRefresh()` is diff --git a/RELEASING.md b/RELEASING.md index 378f0299..9d84e768 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -68,6 +68,25 @@ You can also publish an existing tag directly: gh workflow run publish.yml -f version=0.3.3 ``` +## Before publishing detection reporting + +Detection reporting depends on two server-side behaviours. Check both before cutting a +release that includes it, because a published version cannot be withdrawn from anyone +who has already installed it: + +- **The detections endpoint deduplicates on `Idempotency-Key`.** Connect sends a stable + key for every attempt at a batch and a fresh one per batch, so a redelivery is + identifiable — but whether it is counted once is the endpoint's to decide. Published + ahead of that, a retry after a lost acknowledgement inflates the counts these reports + are read for. +- **Ingest accepts and stores the current payload:** the `capture` object, the baseline + fields `method`, `user_agent`, `query_keys` and `query_keys_total`, and + `reporting_state` on the detections body. An endpoint that rejects or silently drops + them turns every report into a delivery failure, or into a record missing the evidence + it was sent to carry. + +Delete this section once both have shipped. + ## Notes - Tags must be `vX.Y.Z` (the leading `v` is stripped to get the npm version). diff --git a/rule-contract.json b/rule-contract.json index 6e629f44..25aac94e 100644 --- a/rule-contract.json +++ b/rule-contract.json @@ -1,6 +1,6 @@ { "$comment": "Generated from src/protect/rules/contract.js by scripts/emit-rule-contract.mjs. Do not edit.", - "version": "2.7", + "version": "2.8", "sources": { "raw": { "keyed": false @@ -359,6 +359,9 @@ "method" ], "null_valued_properties": "refused", + "null_exempt_properties": [ + "capture" + ], "rule_property_shapes": { "max_bytes": "positive-number", "bypass_limit": "boolean", @@ -386,8 +389,31 @@ "set_headers", "remove_headers", "cookie_flags", - "ensure" + "ensure", + "capture" ], + "capture": { + "version": 1, + "required": [ + "version", + "raw_chars" + ], + "additional_properties": false, + "properties": { + "version": { + "type": "integer", + "const": 1 + }, + "raw_chars": { + "type": "integer", + "minimum": 1 + } + }, + "raw_chars_effective_maximum": 512, + "unknown_version": "grants no capture; the rule still applies", + "unreadable": "grants no capture; the rule still applies", + "raw_chars_note": "a request for a bounded PREFIX of the body; more than the effective maximum yields the maximum" + }, "limits": { "maxRules": 5000, "maxWhitelists": 2000, diff --git a/src/protect/capture-plan.js b/src/protect/capture-plan.js new file mode 100644 index 00000000..fa80f9c0 --- /dev/null +++ b/src/protect/capture-plan.js @@ -0,0 +1,447 @@ +/** + * What a rule permits to be captured as evidence — derived from the rule, never from the traffic. + * + * A detection says a rule matched. On its own that is enough to count hits, and not enough to act on: + * whoever triages it still has to decide whether the request was really an attack, and for that they need + * to see what the rule saw. Capturing that is the difference between a counter and evidence. + * + * It is also the point where a security channel could quietly become a copy of an application's traffic. + * So capture is not a switch. What may be captured is DERIVED FROM THE RULE, and a rule earns each + * permission by naming what it reads: + * + * - A rule naming a parameter (`post.title`, `cookie.session`, `server.HTTP_AUTHORIZATION`) permits that + * parameter's value. The rule was written to inspect it, so its value is what the finding is about. + * - A prefix (`post.field_*`) permits the values of keys that match, and no others — bounded by a count, + * because a prefix can match an unbounded number of keys and "a rule that reads a prefix" must not + * become "a rule that reads the whole body". + * - `raw` and `all` permit NOTHING. They read the entire request, so deriving a permission from them + * would derive permission for everything — the broadest rules granting the broadest capture, which is + * exactly backwards. + * - Raw request bytes are capturable only when the rule carries an explicit, reviewed `capture` opt-in. + * That is a decision someone makes about one rule, not a consequence of how the rule happens to match. + * + * Two properties matter as much as the policy itself. + * + * **It is the rule's, not the clause's.** The engine reports which RULE matched, not which of its + * conditions did, so a plan covers the union of everything the rule reads. Narrowing to the matching + * clause would mean changing evaluation to report one, and a plan that claimed clause-level precision it + * did not have would be worse than one that says what it is. + * + * **It is fixed to a revision.** The plan is derived from the immutable rule revision and cached against + * it, and the event records which plan governed it. A rule that changes gets a new revision and a new + * plan, so a captured value can always be traced to the policy that permitted it — rather than to + * whatever the rule says by the time anyone looks. + */ + +import { + CAPTURE_RAW_CHARS_MAX, + SOURCES, + captureProblem, + parameterProblem, +} from './rules/contract.js'; +import { enforceableRuleProblem } from './rules/validate.js'; + +/** + * `response.*` is the one keyed source a rule can never capture from. + * + * Those read what the application is about to send, so they are the app's own output rather than + * something a client submitted — the phase that inspects them exists to REDACT secrets, and a channel + * that captured them would collect the very values the redaction is there to stop leaving. + */ +const NEVER_CAPTURABLE = new Set(['response']); + +/** + * Bounds every plan carries, so a permission cannot become an unbounded one. + * + * Named for what they count. `capturedValues` is the budget for named and prefix captures together; raw + * is NOT drawn from it, because raw is separately opted into and separately bounded by its own + * `raw.chars`, and making it consume a value slot would have a raw opt-in silently reduce the named + * evidence a reviewer needs. `prefixValues` counts resolved VALUES rather than matched keys: the + * resolver answers a wildcard with values, so keys are not available to count and claiming otherwise + * would describe a bound this cannot enforce. + */ +export const CAPTURE_LIMITS = Object.freeze({ + /** Named and prefix values in one event, together. Raw has its own allowance. */ + capturedValues: 10, + /** Characters of any single captured value. */ + valueChars: 512, + /** Resolved values one prefix permission may contribute. */ + prefixValues: 5, +}); + +/** The parameters a rule reads, as the union over its conditions and nested groups. */ +function parametersOf(rule) { + const out = new Set(); + // A condition may name one parameter or a list of them, and the engine reads every member. A walker + // that saw only the string form would derive an empty plan from a rule that reads a dozen fields. + const add = (parameter) => { + if (typeof parameter === 'string' && parameter !== 'rules') out.add(parameter); + }; + const collect = (parameter) => { + // One level, because the engine expands one level. A nested list resolves to nothing there, so + // flattening it here would grant a permission for a parameter no match can read. + if (Array.isArray(parameter)) { + for (const member of parameter) add(member); + + return; + } + add(parameter); + }; + const walk = (conditions, depth) => { + if (!Array.isArray(conditions) || depth > 20) return; + for (const condition of conditions) { + if (!condition || typeof condition !== 'object') continue; + collect(condition.parameter); + if (Array.isArray(condition.rules)) walk(condition.rules, depth + 1); + } + }; + walk(rule?.rule_v2, 0); + + return [...out]; +} + +/** + * The raw-body opt-in a rule carries, if it carries a valid one. + * + * Only ever a prefix of the body, never the whole of it, and only up to a reviewed number of characters. + * A rule asking for more than the cap gets the cap rather than what it asked for: the opt-in says a + * reviewer agreed raw bytes are needed here, not that this rule sets its own bounds. + */ +function rawOptIn(rule) { + // The rule's OWN property, validated by the contract. A `capture` reachable through a polluted + // prototype belongs to no rule, and would grant raw capture to every rule at once; a value coerced + // into a number — a string, a boolean, a one-element array — is not an opt-in anyone reviewed. + if (!rule || typeof rule !== 'object' || !Object.hasOwn(rule, 'capture')) return null; + + const capture = rule.capture; + // `captureProblem` reads an absent capture as "no opt-in", which is the default and not a problem — so + // the shape is confirmed here before anything is read off it. + if (capture === null || typeof capture !== 'object') return null; + if (captureProblem(capture) !== null) return null; + + return { chars: Math.min(capture.raw_chars, CAPTURE_RAW_CHARS_MAX) }; +} + +/** + * Derive what may be captured for a rule. + * + * Pure, and total: an unreadable rule yields a plan that permits nothing, because the failure to + * understand a rule must never be the reason something gets captured. + */ +const NOTHING = Object.freeze({ + named: Object.freeze([]), + prefixes: Object.freeze([]), + raw: null, + limits: CAPTURE_LIMITS, +}); + +export function derivePlan(rule) { + /** @type {Set} */ + const named = new Set(); + /** @type {Set} */ + const prefixes = new Set(); + + // A permission exists to explain a detection, and a rule the guard would not run produces none. So the + // question is whether this rule is one the validator accepts — the same judgement that decides whether + // it protects anything — rather than whether its parameters happen to be spelled correctly. A rule with + // a parameter and no match is refused there and authorises nothing here, and a rule matching on the + // whole request carries no parameter at all yet is perfectly able to fire. + // + // Capture validity is deliberately not part of that judgement: it governs collection, never protection. + if (enforceableRuleProblem(rule) !== null) return NOTHING; + + // The contract decides what is a parameter at all. Judging that here would be a second grammar to keep + // in step with the engine's, and the two drifting apart means authorising capture of something no rule + // can even read — `server.HTTP_*` and `egress.anything` are refused there, not here. + const parameters = parametersOf(rule).filter((parameter) => parameterProblem(parameter) === null); + + for (const parameter of parameters) { + const dot = parameter.indexOf('.'); + // A keyless source reads the whole request, so it names nothing to capture. + if (dot === -1) continue; + + const source = parameter.slice(0, dot); + const key = parameter.slice(dot + 1); + if (SOURCES[source]?.keyed !== true || NEVER_CAPTURABLE.has(source)) continue; + + if (key.endsWith('*')) { + const prefix = key.slice(0, -1); + // A bare `source.*` is `all` wearing a different hat: it names nothing, so it permits nothing. + if (prefix !== '') prefixes.add(`${source}.${prefix}`); + continue; + } + named.add(`${source}.${key}`); + } + + const raw = rawOptIn(rule); + + // Frozen: the reference below identifies a set of permissions, so a plan that could be edited after + // its reference was computed would leave the reference naming permissions that no longer apply. + return Object.freeze({ + named: Object.freeze([...named].sort()), + prefixes: Object.freeze([...prefixes].sort()), + raw: raw === null ? null : Object.freeze(raw), + limits: CAPTURE_LIMITS, + }); +} + +/** Whether a plan permits anything at all — the common case is that it does not. */ +export function permitsAnything(plan) { + return plan.named.length > 0 || plan.prefixes.length > 0 || plan.raw !== null; +} + +/** + * A stable reference for a plan, recorded on every event the plan governed. + * + * Content-derived rather than a counter, so the same permissions have the same reference across + * processes and releases, and two events carrying one reference really were governed by the same + * permissions. A reader can then ask what a capture was allowed to include without the rule in front of + * them. + * + * It covers the WHOLE plan, limits included. Two plans naming the same parameters but allowing 512 and + * 4096 characters are different permissions, and a reference that could not tell them apart would be + * making exactly the claim it exists to support. + * + * Not a security boundary. Nothing is authenticated by it, and it is not built to resist anyone trying + * to collide it: four FNV-1a lanes with distinct bases give 128 output bits from a non-cryptographic + * function, chosen because it needs no dependency and no runtime API an edge target may lack. What it is + * built for is accidental collision between the small number of plans a bundle produces, which that is + * comfortably wide enough for. + * + * The algorithm and the canonical form are part of what the prefix means. Changing either changes what + * every existing reference refers to, so it takes a new prefix rather than a new implementation under the + * old one — a pinned vector in the tests is what makes that a decision instead of an accident. Renaming a + * limit changes the canonical form, which is why this reads `cp2-`. + */ +const LANES = Object.freeze([0x811c9dc5, 0x01000193, 0x9e3779b9, 0x85ebca6b]); + +export function planReference(plan) { + // Sorted here as well as in `derivePlan`: this is the value that ties a captured value to the policy + // that permitted it, so the tie must not rest on an ordering established somewhere else. + const limits = plan?.limits ?? {}; + const canonical = JSON.stringify([ + [...(plan?.named ?? [])].sort(), + [...(plan?.prefixes ?? [])].sort(), + plan?.raw?.chars ?? 0, + Object.entries(limits) + .map(([key, value]) => [key, value]) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)), + ]); + + const digest = LANES.map((base) => { + let hash = base >>> 0; + for (let i = 0; i < canonical.length; i++) { + hash ^= canonical.charCodeAt(i); + hash = Math.imul(hash, 0x01000193) >>> 0; + } + + return hash.toString(16).padStart(8, '0'); + }).join(''); + + return `cp2-${digest}`; +} + +/** + * Plans for the rules in play, derived once and reused. + * + * Keyed on the rule OBJECT, not on a revision string. A revision identifies a version of one rule, not a + * rule — two rules can carry the same revision, and a cache keyed on that alone would answer for the + * second with the first's permissions, capturing a field the second rule never authorised. Object + * identity cannot make that mistake, and a bundle that is re-fetched brings new objects, so a changed + * rule is derived again rather than answered from a stale entry. + * + * A `WeakMap` also means an entry lives exactly as long as the rule it describes. + */ +export function createPlanCache() { + const byRule = new WeakMap(); + let derivations = 0; + + return { + for(rule) { + if (rule === null || typeof rule !== 'object') { + // Nothing to key on, and nothing to cache: derive an empty plan and let the caller carry on. + const plan = derivePlan(rule); + + derivations += 1; + + return Object.freeze({ plan, reference: planReference(plan) }); + } + + const hit = byRule.get(rule); + if (hit) return hit; + + const plan = derivePlan(rule); + derivations += 1; + // Frozen with the plan, so the reference and the permissions it names cannot come apart. + const entry = Object.freeze({ plan, reference: planReference(plan) }); + byRule.set(rule, entry); + + return entry; + }, + /** How many plans have been derived — the observable difference between a hit and a miss. */ + get derivations() { + return derivations; + }, + }; +} + +/** + * The evidence a plan permits, read from one request. + * + * Reads only what the plan names, in the plan's order, and through THE resolver the match was decided by + * — handed in, never built here. Building one would read the request a second time, and a request can + * answer differently twice: a getter, a stream, anything lazy. Evidence that disagrees with the match it + * belongs to is worse than no evidence, so the only reading available here is the one that already + * happened. + * + * **It is the resolved value, which is neither the bytes as sent nor the string the matcher compared.** + * Three forms of a parameter exist. The engine normalises a request first — URL-decoding, HTML-entity + * decoding, stripping comments and control characters, collapsing whitespace — and then applies whatever + * mutations a rule's own condition asks for. This records the middle one, because that is what the + * resolver answers with and reading either of the others would be a second interpretation of the request. + * + * So a percent-encoded payload arrives here decoded, while a rule's `base64_decode` is not applied and its + * subject arrives still encoded. Reproducing the matched subject would need condition-level evidence: the + * engine reports which RULE fired, not which condition, and this plan deliberately does not model one. + * + * Bounds apply, and each reports what it left out: + * + * - a total across named and prefix values, so one detection cannot carry an unbounded amount of an + * application's data + * - a length per value, so one field cannot + * - a number of resolved values per prefix, so a prefix permission stays narrower than the body it sits in + * + * Raw is bounded separately by its own opt-in and does not draw on the value total. + * + * Absence and failure are different answers, and the result distinguishes them. A field that was not + * there, a value refused for its type, a parameter whose read threw, and a request nothing could be read + * from would otherwise all arrive as "no evidence" — letting a reviewer read incomplete evidence as + * complete. None of it is content: they are counts of what did not make it, and why. + * + * `unavailable` means no read completed at all: there was no resolver to read with, or every permitted + * read threw. It is the difference between "this request carried none of what the rule reads" and "this + * request could not be read", which are opposite conclusions from the same empty list. + */ +export function captureValues(plan, resolver) { + const nothing = { values: [], omitted: 0, unsupported: 0, failed: 0, unavailable: false, raw: null }; + if (!plan || !permitsAnything(plan)) return nothing; + if (!resolver || typeof resolver.resolve !== 'function') return { ...nothing, unavailable: true }; + + const limits = plan.limits ?? CAPTURE_LIMITS; + const values = []; + let omitted = 0; + let unsupported = 0; + let failed = 0; + let attempted = 0; + + const read = (parameter) => { + attempted += 1; + try { + const resolved = resolver.resolve(parameter); + + return Array.isArray(resolved) ? resolved : []; + } catch { + // Fail-open for the application: a capture that cannot be taken is not taken. Counted, so that + // "nothing was captured" is not mistaken for "there was nothing to capture". + failed += 1; + + return null; + } + }; + + /** + * What one resolved value is, before anything is decided about room for it. + * + * Classified here rather than inside the recording step, so that "refused for its type" and "left out + * by a bound" cannot swap places depending on where a value happens to fall in a request. + */ + const classify = (value) => { + const text = asText(value); + if (text === null) { + unsupported += 1; + + return null; + } + + return text; + }; + + /** @returns {boolean} whether there was room for it */ + const record = (parameter, text) => { + if (values.length >= limits.capturedValues) { + omitted += 1; + + return false; + } + + const capped = text.length > limits.valueChars; + values.push({ + parameter, + value: capped ? text.slice(0, limits.valueChars) : text, + ...(capped ? { truncated: true } : {}), + }); + + return true; + }; + + for (const parameter of plan.named) { + // Every resolved value is offered, not just up to the first refusal: a parameter can resolve to many + // values, and stopping at the first excess would report one omission where there were several. + for (const value of read(parameter) ?? []) { + const text = classify(value); + if (text !== null) record(parameter, text); + } + } + + for (const prefix of plan.prefixes) { + // The pattern the rule wrote, put back together: the plan holds the prefix, the resolver reads the + // wildcard. Labelled by the pattern rather than by the key it matched, because the resolver answers + // with values and inventing a key here would mean enumerating the request a second way. + const pattern = `${prefix}*`; + let taken = 0; + for (const value of read(pattern) ?? []) { + // Classified before the prefix bound is consulted, for the same reason as above: what a value IS + // does not depend on how many came before it. + const text = classify(value); + if (text === null) continue; + if (taken >= limits.prefixValues) { + omitted += 1; + continue; + } + if (record(pattern, text)) taken += 1; + } + } + + let raw = null; + if (plan.raw !== null) { + const text = asText((read('raw') ?? [])[0]); + if (text !== null) { + const capped = text.length > plan.raw.chars; + raw = { value: capped ? text.slice(0, plan.raw.chars) : text, ...(capped ? { truncated: true } : {}) }; + } + } + + // Nothing was readable, as opposed to nothing being there to read. + const unavailable = attempted > 0 && failed === attempted; + + return { values, omitted, unsupported, failed, unavailable, raw }; +} + +/** + * A value as text, or null when its type is not one a request carries as a value. + * + * An empty string is text. A rule can be written so that its finding IS that a parameter is empty, and + * collapsing that into absence would erase the evidence for exactly those rules — an absent field + * resolves to no value at all, which is already a different answer. + * + * An object is refused rather than serialised, because serialising it would reach past the field the plan + * named into whatever it contains: a permission for `post.profile` is not a permission for everything + * under it. + */ +function asText(value) { + if (typeof value === 'string') return value; + if (typeof value === 'number' && Number.isFinite(value)) return String(value); + if (typeof value === 'boolean') return String(value); + + return null; +} diff --git a/src/protect/client-ip.js b/src/protect/client-ip.js new file mode 100644 index 00000000..d8a63d38 --- /dev/null +++ b/src/protect/client-ip.js @@ -0,0 +1,476 @@ +/** + * Resolve a client address, and say where it came from. + * + * A client address is only as trustworthy as the thing that supplied it. A transport peer address is + * observed by the runtime and cannot be set by the caller. A forwarded header is an ordinary request + * header: anyone can send one, and it means something only when the request is known to have arrived + * through a proxy that sets it and discards what the client sent. + * + * So provenance travels with the value, and nothing is trusted implicitly: + * + * `runtime` the transport peer address, observed rather than claimed + * `trusted-proxy` read from a forwarded chain, through peers a policy declares trustworthy + * `unavailable` no address this can stand behind + * + * `unavailable` is the default and a real answer. A runtime that cannot produce a verifiable address + * should say so rather than pass on a value the caller chose: an address recorded against a security + * event attributes that event, and one an attacker picked attributes it to whoever they name. + * + * ## What makes a header trustworthy + * + * Two things, together, and neither alone: + * + * 1. The PEER is a declared trusted proxy. A peer merely existing proves nothing — every direct + * connection has one — so a policy has to say which peers are the deployment's own front end. + * 2. The chain is walked from the APPLICATION side inward, skipping trusted hops, and the first + * untrusted address is the client. Taking the client-most entry instead trusts whatever the caller + * prepended, because a proxy appends rather than replaces. + * + * There are deliberately no provider shortcuts. A provider's name does not establish that the provider + * overwrote the header — several document that a client-supplied value survives unless the service is + * configured to replace it, and one that does overwrite it only does so for requests that actually + * traversed it. A shortcut worth having has to encode a policy that can be verified at run time, which + * needs a platform adapter that positively establishes the runtime; a header name on its own does not. + */ + +/** + * The zone identifiers this accepts: interface names and numeric scope ids. + * + * Deliberately narrow. A zone reaches logs and retained event payloads, so anything outside the forms + * real runtimes produce — whitespace, newlines, path separators, brackets, arbitrary Unicode — is treated + * as a malformed address rather than passed through as part of one. Covers `eth0`, `en0`, `eth0.100`, + * `br-abc123` and a bare scope number. + */ +const ZONE = /^[A-Za-z0-9._-]{1,64}$/; + +/** The fields a trust policy may declare. Anything else is a typo, not an extension. */ +const POLICY_FIELDS = Object.freeze(['peers', 'hops', 'header', 'isTrusted']); + +/** Provenance values. */ +export const IP_SOURCES = Object.freeze(['runtime', 'trusted-proxy', 'unavailable']); + +/** The header consulted when a policy does not name one. */ +const DEFAULT_FORWARDED_HEADER = 'x-forwarded-for'; + +/** + * Parse an IPv4 literal into its four octets, or null. + * + * Leading zeros are rejected: `010` is read as decimal 10 here and as octal 8 by some resolvers, and an + * address that means two things is not an identity. + */ +function parseIpv4(value) { + const parts = value.split('.'); + if (parts.length !== 4) return null; + + const octets = []; + for (const part of parts) { + if (!/^\d{1,3}$/.test(part)) return null; + if (part.length > 1 && part.startsWith('0')) return null; + const n = Number(part); + if (n > 255) return null; + octets.push(n); + } + + return octets; +} + +/** + * Parse an IPv6 literal into its eight 16-bit groups, or null. + * + * Handles `::` compression and a trailing embedded IPv4 (`::ffff:203.0.113.1`), which is the form Node + * reports for an IPv4 client on a dual-stack socket. + */ +function parseIpv6(value) { + if (value === '' || value.includes(':::')) return null; + + let head = value; + let embedded = null; + const lastColon = head.lastIndexOf(':'); + const tail = lastColon === -1 ? '' : head.slice(lastColon + 1); + if (tail.includes('.')) { + embedded = parseIpv4(tail); + if (embedded === null) return null; + head = head.slice(0, lastColon + 1) + '0:0'; + } + + const halves = head.split('::'); + if (halves.length > 2) return null; + + const readGroups = (text) => { + if (text === '') return []; + const groups = []; + for (const g of text.split(':')) { + if (!/^[0-9a-fA-F]{1,4}$/.test(g)) return null; + groups.push(Number.parseInt(g, 16)); + } + + return groups; + }; + + const left = readGroups(halves[0]); + const right = halves.length === 2 ? readGroups(halves[1]) : []; + if (left === null || right === null) return null; + + let groups; + if (halves.length === 2) { + const fill = 8 - (left.length + right.length); + if (fill < 1) return null; + groups = [...left, ...new Array(fill).fill(0), ...right]; + } else { + groups = left; + } + if (groups.length !== 8) return null; + + if (embedded !== null) { + groups[6] = (embedded[0] << 8) | embedded[1]; + groups[7] = (embedded[2] << 8) | embedded[3]; + } + + return groups; +} + +/** + * An address as a comparable big integer, its width, and its canonical spelling — or null when the text + * is not an address. + * + * Strict about the syntax around the address, not only the address itself: + * + * - The bracketed form is not accepted. A bracket is not a valid character in an address, so the group + * parser rejects `[::1`, `::1]` and `[2001:db8::1]:8080` without needing a rule of its own — and + * stripping the delimiters instead would accept the unbalanced spellings. A proxy emitting the + * bracketed form is therefore not understood, and the walk falls back to the observed peer. + * - A zone identifier (`%eth0`) is permitted only on IPv6, only once, and only in the conservative + * grammar below. An IPv4 literal has no zone, so `1.2.3.4%eth0` is malformed rather than an address + * with decoration, and `fe80::1%eth0%oops` is malformed rather than a zone containing a `%`. + * - A zone is KEPT in the canonical spelling. The zone is what makes a link-local address identify an + * interface, and dropping it would make two addresses on different interfaces compare equal. It plays + * no part in the numeric comparison, which is why a policy entry may not carry one. + * + * The canonical spelling is what callers should store and match on, so a value the validator accepted is + * the same value everywhere: IPv4 in dotted decimal, IPv6 lowercased with the longest zero run + * compressed, and an IPv4-mapped IPv6 address reduced to the IPv4 it carries. + */ +function toNumeric(value) { + const text = String(value).trim(); + if (text === '') return null; + + const zoneParts = text.split('%'); + if (zoneParts.length > 2) return null; + const addr = zoneParts[0]; + const zone = zoneParts.length === 2 ? zoneParts[1] : null; + // A zone must name something in the accepted grammar, and only IPv6 has one. + if (zone !== null && (!ZONE.test(zone) || !addr.includes(':'))) return null; + + const v4 = parseIpv4(addr); + if (v4 !== null) { + return { + bits: 32, + value: v4.reduce((acc, octet) => (acc << 8n) | BigInt(octet), 0n), + canonical: v4.join('.'), + }; + } + + if (!addr.includes(':')) return null; + const v6 = parseIpv6(addr); + if (v6 === null) return null; + + // `::ffff:a.b.c.d` is the same host as `a.b.c.d`, so it canonicalises to the IPv4 form and compares + // against IPv4 policies. + // + // A zone on one is refused rather than dropped. The canonical form of a mapped address is IPv4, and an + // IPv4 identity has no zone to carry — so keeping the address would mean discarding the scope, which is + // the one thing a zone must never do silently. Applies to both spellings of the mapped form. + const mapped = v6.slice(0, 5).every((g) => g === 0) && v6[5] === 0xffff; + if (mapped) { + if (zone !== null) return null; + const octets = [v6[6] >> 8, v6[6] & 0xff, v6[7] >> 8, v6[7] & 0xff]; + + return { bits: 32, value: (BigInt(v6[6]) << 16n) | BigInt(v6[7]), canonical: octets.join('.') }; + } + + return { + bits: 128, + value: v6.reduce((acc, group) => (acc << 16n) | BigInt(group), 0n), + // The zone survives: without it a link-local address does not identify an interface. + canonical: zone === null ? canonicalIpv6(v6) : `${canonicalIpv6(v6)}%${zone}`, + }; +} + +/** Lowercase hex groups with the longest run of two or more zero groups compressed to `::`. */ +function canonicalIpv6(groups) { + let bestStart = -1; + let bestLength = 0; + let runStart = -1; + + for (let i = 0; i <= groups.length; i++) { + if (i < groups.length && groups[i] === 0) { + if (runStart === -1) runStart = i; + } else if (runStart !== -1) { + const length = i - runStart; + if (length > bestLength) { + bestLength = length; + bestStart = runStart; + } + runStart = -1; + } + } + + const hex = groups.map((g) => g.toString(16)); + if (bestLength < 2) return hex.join(':'); + + const head = hex.slice(0, bestStart).join(':'); + const tail = hex.slice(bestStart + bestLength).join(':'); + + return `${head}::${tail}`; +} + +/** + * The canonical spelling of an address, or null when the text is not one. + * + * Exported because every surface that stores or matches an address should use the same spelling — two + * records of one client that differ only in how the address was written are two records. + */ +export function canonicalIp(value) { + return toNumeric(value)?.canonical ?? null; +} + +/** + * Whether `value` is a syntactically valid IP address. + * + * Applied to everything before it is used or reported. A forwarded chain can carry `unknown`, a hostname + * or arbitrary text, and the resolved value is matched by rules and recorded against retained events — so + * a string that is not an address is not an answer. + */ +export function isIpAddress(value) { + return canonicalIp(value) !== null; +} + +/** + * Parse `1.2.3.0/24` or a bare address into a matcher, or null. + * + * Exactly one optional slash. `10.0.0.0/8/typo` is a typo, and reading it as `/8` would silently install + * a policy the operator did not write. + */ +function parseCidr(entry) { + if (typeof entry !== 'string') return null; + // A zone plays no part in the numeric comparison, so an entry carrying one would trust every address + // with those bits on every interface — including the one it was written to exclude. + if (entry.includes('%')) return null; + + const parts = entry.trim().split('/'); + if (parts.length > 2) return null; + + const numeric = toNumeric(parts[0]); + if (numeric === null) return null; + + let length = numeric.bits; + if (parts.length === 2) { + if (!/^\d{1,3}$/.test(parts[1])) return null; + length = Number(parts[1]); + if (length > numeric.bits) return null; + } + + const shift = BigInt(numeric.bits - length); + + return { bits: numeric.bits, network: numeric.value >> shift, shift }; +} + +/** + * Read a trusted-proxy policy, or null when the configuration declares none. + * + * A policy needs at least one way to recognise the deployment's own front end — `peers`, `hops`, or + * `isTrusted`. A configuration carrying only a header name declares nothing: it says which header to + * read without saying when reading it is safe, and that is the case where request input silently becomes + * an identity. + */ +export function readTrustPolicy(trustedProxy) { + if (trustedProxy === null || typeof trustedProxy !== 'object' || Array.isArray(trustedProxy)) return null; + + // Own properties only, and no unknown ones. + // + // A field read through the prototype chain was not written by whoever configured this object, and a key + // this does not recognise is a typo rather than an extension — `heder: 'x-real-ip'` would otherwise be + // ignored and the default header used, which is the quiet substitution this whole function exists to + // avoid. Both directions matter on the configuration that decides whether request input becomes an + // identity. + const has = (field) => Object.hasOwn(trustedProxy, field); + + for (const key of Object.keys(trustedProxy)) { + if (!POLICY_FIELDS.includes(key)) return null; + } + + // Every field that is PRESENT has to be valid. Substituting a default for a malformed value, or + // dropping it, installs a policy the operator did not write and gives them no way to tell from the + // behaviour which part took effect. + if (has('header') && (typeof trustedProxy.header !== 'string' || trustedProxy.header.trim() === '')) { + return null; + } + if (has('hops') && (!Number.isInteger(trustedProxy.hops) || trustedProxy.hops < 1)) return null; + if (has('isTrusted') && typeof trustedProxy.isTrusted !== 'function') return null; + + const header = has('header') ? trustedProxy.header.trim().toLowerCase() : DEFAULT_FORWARDED_HEADER; + + // Trust configuration fails closed. One unparseable entry invalidates the whole policy rather than + // being dropped, and an EMPTY list is a declaration that no peer is trusted — not an absent field to + // be filled in by a hop count. + const cidrs = []; + if (has('peers')) { + if (!Array.isArray(trustedProxy.peers) || trustedProxy.peers.length === 0) return null; + for (const entry of trustedProxy.peers) { + const parsed = parseCidr(entry); + if (parsed === null) return null; + cidrs.push(parsed); + } + } + + const hops = has('hops') ? trustedProxy.hops : null; + const predicate = has('isTrusted') ? trustedProxy.isTrusted : null; + + if (cidrs.length === 0 && hops === null && predicate === null) return null; + + return { header, cidrs, hops, predicate }; +} + +/** Whether an address is one of the deployment's declared proxies. */ +function isTrustedPeer(policy, value) { + const numeric = toNumeric(value); + if (numeric === null) return false; + + for (const cidr of policy.cidrs) { + if (cidr.bits === numeric.bits && numeric.value >> cidr.shift === cidr.network) return true; + } + + if (policy.predicate !== null) { + try { + if (policy.predicate(String(value)) === true) return true; + } catch { + // A throwing predicate is not a grant of trust. + return false; + } + } + + return false; +} + +/** + * Addresses that are syntactically valid but identify nobody. + * + * The unspecified addresses mean "no particular host". A runtime reporting one has not told us who + * connected, and a chain carrying one names no client — so neither may be reported as an address, even + * though both parse. Kept separate from parsing, because they are perfectly well-formed. + */ +const UNSPECIFIED = new Set(['0.0.0.0', '::']); + +/** The canonical form of an address that identifies a host, or null. */ +function identifyingIp(value) { + const canonical = canonicalIp(value); + + return canonical === null || UNSPECIFIED.has(canonical) ? null : canonical; +} + +/** The forwarded chain, in wire order (client-most first), with only real addresses kept. */ +function chainFrom(headers, header) { + // Own property only. A header inherited through the prototype chain was not sent with this request, so + // prototype pollution elsewhere in an application must not be able to supply an attributed client + // address. + if (headers === null || typeof headers !== 'object' || !Object.hasOwn(headers, header)) return []; + + const raw = headers[header]; + const value = Array.isArray(raw) ? raw.join(',') : raw; + if (typeof value !== 'string' || value.trim() === '') return []; + + return value.split(',').map((entry) => entry.trim()).filter((entry) => entry !== ''); +} + +/** + * Resolve the client address for a request. + * + * @param {{ + * peer?: unknown, + * headers?: Record, + * trustedProxy?: unknown, + * }} input + * @returns {{ ip: string | null, source: 'runtime' | 'trusted-proxy' | 'unavailable' }} + */ +export function resolveClientIp(input) { + const headers = input.headers ?? {}; + // Canonical from here on, so every surface stores and matches the same spelling. An address that + // identifies nobody is treated as no address at all. + const peer = identifyingIp(input.peer); + const policy = readTrustPolicy(input.trustedProxy); + + // No peer means no transport-level anchor. A forwarded header here is indistinguishable from one the + // caller wrote, whatever it is called, so there is nothing to report. + if (peer === null) return { ip: null, source: 'unavailable' }; + + if (policy === null) return { ip: peer, source: 'runtime' }; + + // Which part of the policy gates the peer. + // + // A policy declaring `peers` or `isTrusted` names the deployment's own front end, so that verdict + // governs: a connection from anywhere else did not arrive through it. A policy declaring only `hops` + // makes the statement numerically instead — the peer IS hop one — so the count is itself the trust and + // has to be evaluated on its own. Requiring a CIDR match there would make a hops-only policy accepted + // by configuration and inert in practice. + const gatedByAddress = policy.cidrs.length > 0 || policy.predicate !== null; + const peerTrusted = gatedByAddress ? isTrustedPeer(policy, peer) : policy.hops !== null; + if (!peerTrusted) return { ip: peer, source: 'runtime' }; + + const chain = chainFrom(headers, policy.header); + if (chain.length === 0) return { ip: peer, source: 'runtime' }; + + if (policy.hops !== null) { + // `hops` counts trusted proxies starting AT THE PEER, matching the numeric form of Express's trust + // policy. So `hops: 1` means the peer is the only proxy and the client is the application-most entry + // in the chain; `hops: 2` means the peer plus one charted hop; and so on. Counting from the chain + // instead would be off by one against every deployment that copied its number from an Express + // configuration. + const index = chain.length - policy.hops; + const candidate = index >= 0 ? chain[index] : undefined; + + const canonical = identifyingIp(candidate); + + return canonical === null ? { ip: peer, source: 'runtime' } : { ip: canonical, source: 'trusted-proxy' }; + } + + // Walk inward from the application side, stepping over hops the policy trusts. The first address that + // is not one of ours is the client. An entry that is not an address at all stops the walk: the chain + // cannot be reasoned about past something that is not a hop. + for (let i = chain.length - 1; i >= 0; i--) { + const canonical = identifyingIp(chain[i]); + if (canonical === null) return { ip: peer, source: 'runtime' }; + if (!isTrustedPeer(policy, canonical)) return { ip: canonical, source: 'trusted-proxy' }; + } + + // Every hop was one of ours, which leaves no client in the chain to name. + return { ip: peer, source: 'runtime' }; +} + +/** + * The event fields for a resolved address. + * + * `client_ip` is omitted entirely when there is none, rather than sent as null or an empty string: a + * field that is present but empty reads as a failed lookup of a real address. The provenance is always + * present, because "this could not be established" is the part a reader needs. + */ +export function clientIpFields(resolved) { + // Three states are coherent, and nothing else is emitted: + // + // `runtime` or `trusted-proxy` WITH an address — something was established, and this is where from + // `unavailable` WITHOUT an address — nothing was established + // + // Every other combination contradicts itself. An address carrying `unavailable` attaches a value to a + // claim that denies it; `runtime` or `trusted-proxy` carrying no address asserts that an address was + // established and then declines to name one. Both are normalised to `unavailable`, because the one + // thing that can be said honestly about a self-contradicting pair is that nothing was established. + const established = resolved?.source === 'runtime' || resolved?.source === 'trusted-proxy'; + // Validated here rather than assumed. This is the function that states the payload invariant, so it + // checks the address itself: a caller that has not been through the resolver — a future adapter, or a + // record reconstructed from somewhere else — must not be able to put a hostname, a malformed literal or + // an address that identifies nobody into a retained event. + const address = identifyingIp(resolved?.ip); + + return established && address !== null + ? { client_ip: address, client_ip_source: resolved.source } + : { client_ip_source: 'unavailable' }; +} diff --git a/src/protect/detections.js b/src/protect/detections.js index cb6f73ab..2ad1004e 100644 --- a/src/protect/detections.js +++ b/src/protect/detections.js @@ -1,4 +1,5 @@ -import { pulseAuthHeader } from '../pulse-token.js'; +import { pulseFetch } from '../pulse-token.js'; +import { clientIpFields } from './client-ip.js'; import { isSafeOrigin } from './safe-origin.js'; /** @@ -18,13 +19,26 @@ import { isSafeOrigin } from './safe-origin.js'; * the bundle identity, and the rule's own revision where the bundle carried one. That is enough to count * hits per rule, compare them against traffic, and decide whether a rule is wrong. * - * What it never carries: **the matched value, the request body, headers, or query-string values**. A - * channel that counts detections is a different thing from a copy of an application's traffic, and once - * values are collected every question about retention, access and jurisdiction arrives with them. - * Anything value-level belongs behind its own explicit opt-in with its own controls, not as a side - * effect of counting. + * It also carries the values of the parameters the matched rule NAMES, under a plan derived from that + * rule — because counting that a rule fired is not enough to act on it. What may be captured is the + * rule's own doing: a rule reading the whole request permits nothing, response values are never + * captured, and raw request bytes need a reviewed opt-in on the rule. Every value is bounded in number + * and length, and what a bound leaves out is counted, so a short list is never mistaken for a complete + * one. * - * The route is the request PATH with any query string dropped, because `?token=…` is a value. + * The user agent is the one exception, and travels whether or not a rule names it. + * That is so on a request or response detection, where there is a client to attribute. It is part of that baseline, because a + * detection nobody can attribute is of little use. An EGRESS detection carries no user agent and no + * client address at all: the call was the application's own, so there is no visitor to attribute it to. + * + * What it never carries: **the value of any OTHER parameter the matched rule does not name**, any + * response value, or the query string's values AS BASELINE METADATA — `route` and `query_keys` describe + * a URL without disclosing what was in it, while a rule naming `egress.url` captures that URL as the rule + * read it, which is the rule-scoped policy rather than an exception to it. A channel that reports detections is a different thing from a + * copy of an application's traffic, and the plan is what keeps the difference. + * + * The route is the request PATH; the query travels as parameter NAMES only, because `?token=…` is a + * value and the rule that fired may never have named it. */ const DEFAULT_BASE_URL = 'https://api.patchstack.com/monitor/pulse'; @@ -33,6 +47,336 @@ const MAX_BATCH = 50; /** Bounded so a detection storm costs memory it cannot grow out of. Oldest go first. */ const MAX_QUEUE = 500; +/** + * Delivery bounds. + * + * A retry exists because the common failure is transient — a restart, a rate limit, a dropped + * connection — and losing evidence to a five-second outage is avoidable. It is bounded because the + * failure that is NOT transient must not turn into a loop: after `MAX_ATTEMPTS` the batch is dropped and + * counted, which is visible, where an unbounded retry would be an app quietly spending itself on a + * refusing endpoint. + * + * Only one send is ever in flight. That keeps memory bounded to the queue plus one batch, keeps the + * retry sequence unambiguous, and means a slow endpoint applies back pressure to the queue rather than + * to the number of open sockets. + */ +const MAX_ATTEMPTS = 4; +const RETRY_BASE_MS = 1000; +const RETRY_CAP_MS = 30_000; +/** + * A refusal that will refuse again is terminal; these are the ones worth trying later. + * + * Any 5xx counts, not a chosen few. A server error is the endpoint saying the fault is its own, and + * picking a subset would leave the rest abandoned on the first attempt while the documented contract + * says a server error is retried — a difference nothing in the output would reveal. + */ +const RETRYABLE_EXPLICIT = new Set([408, 425, 429]); +export function worthRetrying(status) { + if (status === null) return true; // Unreachable: nothing has said the endpoint is unwilling. + + return RETRYABLE_EXPLICIT.has(status) || (status >= 500 && status <= 599); +} + +/** + * How long one attempt may take before it is abandoned and retried. + * + * Only one send is in flight, so a request that never settles would hold that slot for the life of the + * process: the queue would fill, every later event would be dropped for pressure, and the health + * counters would show a single attempt that never failed. A hung connection has to look like a failure. + */ +const ATTEMPT_TIMEOUT_MS = 10_000; + + +/** + * How long `stop()` will wait for the drain before resolving anyway. + * + * A shutdown that waits without a bound is a shutdown that can hang, and a host handling SIGTERM has its + * own deadline. So the promise resolves either when nothing is outstanding or when this elapses — never + * later. Waiting is the caller's option, not an obligation: ignoring the promise leaves the old + * behaviour exactly as it was. + */ +const STOP_BUDGET_MS = 5_000; + +/** + * Size bounds, applied per event and per batch. + * + * Every field is an identifier rather than traffic, but an identifier can still be long: a route is + * whatever the application routes, and a broad rule can read many parameters. A capped field carries + * a `truncated` list naming what was shortened, so a reader can tell a shortened value from a complete + * one instead of drawing conclusions from a route that looks like a different route. + */ +const MAX_ROUTE_CHARS = 256; +const MAX_PARAMETERS = 25; +const MAX_PARAMETER_CHARS = 64; +/** + * Identifiers are capped too, and marked when capped. + * + * These come from the rule bundle rather than from traffic, so a long one is our own bug rather than an + * attack — but an event has to be bounded by every field it carries, not by most of them. Marked rather + * than silently shortened, because a shortened identifier no longer matches the rule it names and a + * reader must not use it as a key believing it does. + */ +const MAX_IDENTIFIER_CHARS = 256; +/** + * Bounds re-applied to captured evidence at the wire. + * + * Chosen so one worst-case event stays well inside `MAX_BODY_BYTES`: a batch always sends at least one + * event, so an event that cannot fit could never be delivered at all. + */ +const MAX_CAPTURED_VALUES = 10; +const MAX_CAPTURED_VALUE_CHARS = 512; +/** Query-string parameter NAMES from the request line. Names, never values — see `queryKeysOf`. */ +const MAX_QUERY_KEYS = 10; +const MAX_METHOD_CHARS = 16; +/** + * The body bound, set where a full batch can actually reach it. + * + * A bound above anything the other caps allow is not a bound, it is a comment: the count and the field + * caps together put a full batch of worst-case events over this, so the split is a path traffic reaches + * rather than a branch nothing can enter. It is also a modest request body, which is the point — an + * endpoint or proxy that refuses an oversized body would refuse every retry of it too. + */ +const MAX_BODY_BYTES = 64 * 1024; + +/** + * The query string's parameter names, without any of its values. + * + * A reviewer needs the shape of the URL that was requested, and the query is where a URL carries values. + * Sending it verbatim would put the values of parameters the matched rule never named onto the wire — + * exactly what the capture plan exists to prevent — so what travels is the path plus the NAMES of the + * query parameters, which describe the request without disclosing what was in it. + * + * `total` counts DISTINCT names, matching the deduplicated `keys` it accompanies. + * + * @returns {{ keys: string[], total: number }} + */ +export function queryKeysOf(path) { + if (typeof path !== 'string') return { keys: [], total: 0 }; + const start = path.indexOf('?'); + if (start === -1) return { keys: [], total: 0 }; + + let names; + try { + // `URLSearchParams`, because that is how the guard itself addresses a query parameter: `+` is a + // space, percent sequences are decoded, and an invalid one is left as written. Decoding by hand here + // would report `first+name` for a parameter every rule addresses as `first name` — a name that + // matches nothing a reviewer could look up. + names = [...new URLSearchParams(path.slice(start + 1)).keys()]; + } catch { + return { keys: [], total: 0 }; + } + + const seen = new Set(); + const keys = []; + for (const name of names) { + // A parameter with no name addresses nothing, so there is nothing to report. + if (name === '') continue; + if (seen.has(name)) continue; + seen.add(name); + // Counted past the cap as well as under it, so a reader knows the list is short rather than complete. + if (keys.length < MAX_QUERY_KEYS) keys.push(name); + } + + // DISTINCT names, matching the list above: a parameter repeated three times is one name to look up, and + // a total that counted repeats would not describe the list it accompanies. + return { keys, total: seen.size }; +} + +/** + * Capture, bounded for the wire. + * + * The extractor bounds what it takes, but a parameter NAME comes from the rule and rules carry no length + * limit — so a label alone can carry an event past the body bound, and a batch always sends at least one + * event. This is the last gate before the wire, so it re-applies every bound rather than trusting whatever + * produced the capture, and marks what it shortened. + */ +function boundCapture(capture) { + if (!capture || typeof capture !== 'object' || Array.isArray(capture)) return null; + + const truncated = []; + // Own properties only, each snapshotted once. + // + // A capture is an ordinary object, so a write to `Object.prototype` supplies any field it does not + // carry itself — and `Object.prototype.raw = { value: … }` would have a plan that permitted nothing + // transmit that value. Reading each field once also stops a getter answering differently between the + // check and the send. This guard shields applications against prototype pollution; its own reporting + // must not be the way one lands. + const own = (object, key) => (Object.hasOwn(object, key) ? object[key] : undefined); + + const declaredPlan = own(capture, 'plan'); + // Validated, not coerced. `String(x)` runs whatever `toString` an object carries, which turns a value + // this channel refuses into reportable content — and the refusal is the whole point of the type rule. + const plan = typeof declaredPlan === 'string' ? capText(declaredPlan, MAX_IDENTIFIER_CHARS) : null; + if (plan === null) return null; + if (plan.truncated) truncated.push('plan'); + + const out = { plan: plan.value }; + // Counted apart, because they mean different things: one says the event was full, the other says the + // value was not something this channel reports. + let overCap = 0; + let rejected = 0; + + const declaredValues = own(capture, 'values'); + if (Array.isArray(declaredValues) && declaredValues.length > 0) { + const values = []; + for (const entry of declaredValues) { + if (!entry || typeof entry !== 'object' || Array.isArray(entry)) { + rejected += 1; + continue; + } + const declaredParameter = own(entry, 'parameter'); + const text = scalarText(own(entry, 'value')); + if (typeof declaredParameter !== 'string' || text === null) { + rejected += 1; + continue; + } + if (values.length >= MAX_CAPTURED_VALUES) { + overCap += 1; + continue; + } + + const label = capText(declaredParameter, MAX_PARAMETER_CHARS); + const value = capText(text, MAX_CAPTURED_VALUE_CHARS); + if (label.truncated && !truncated.includes('parameter')) truncated.push('parameter'); + if (value.truncated && !truncated.includes('value')) truncated.push('value'); + values.push({ + parameter: label.value, + value: value.value, + ...(own(entry, 'truncated') === true || value.truncated ? { truncated: true } : {}), + }); + } + if (overCap > 0) truncated.push('values'); + if (values.length > 0) out.values = values; + } + + const declaredRaw = own(capture, 'raw'); + if (declaredRaw !== undefined && declaredRaw !== null) { + const rawValue = declaredRaw && typeof declaredRaw === 'object' ? own(declaredRaw, 'value') : undefined; + if (typeof rawValue === 'string') { + const raw = capText(rawValue, MAX_CAPTURED_VALUE_CHARS); + out.raw = { + value: raw.value, + ...(own(declaredRaw, 'truncated') === true || raw.truncated ? { truncated: true } : {}), + }; + } else { + // Raw evidence that is not raw evidence is accounted for the same way any refused value is. + rejected += 1; + } + } + + const declaredCount = (key) => { + const value = own(capture, key); + + return Number.isFinite(value) && value > 0 ? Math.floor(value) : 0; + }; + // What this gate excluded is added to what the producer already excluded, in the matching counter: the + // documented promise is that a bound's exclusions and a type's refusals are countable separately, and a + // reader cannot otherwise tell eleven values from two hundred, or a full event from a refused one. + const totals = { + omitted: declaredCount('omitted') + overCap, + unsupported: declaredCount('unsupported') + rejected, + failed: declaredCount('failed'), + }; + for (const [key, total] of Object.entries(totals)) { + if (total > 0) out[key] = total; + } + if (own(capture, 'unavailable') === true) out.unavailable = true; + if (truncated.length > 0) out.truncated = truncated; + + return out; +} + +/** A value as text, or null when its type is not one this channel reports. */ +function scalarText(value) { + if (typeof value === 'string') return value; + if (typeof value === 'number' && Number.isFinite(value)) return String(value); + if (typeof value === 'boolean') return String(value); + + return null; +} + +/** A per-reporter identity, so idempotency keys from two guards cannot collide. */ +function makeInstanceId() { + try { + const uuid = globalThis.crypto?.randomUUID?.(); + if (typeof uuid === 'string' && uuid !== '') return uuid; + } catch { + // A runtime without usable web crypto falls through to the counter below. + } + + return `${Date.now().toString(36)}-${Math.floor(Math.random() * 0xffffff).toString(36)}`; +} + +/** A field shortened to fit, and whether it had to be. */ +function capText(value, limit) { + const text = typeof value === 'string' ? value : ''; + + return text.length > limit ? { value: text.slice(0, limit), truncated: true } : { value: text, truncated: false }; +} + +/** + * How long to wait before attempting again. + * + * `Retry-After` is honoured when the endpoint sets one, because it is the endpoint saying what it can + * take — but capped, so a header cannot park a batch indefinitely. Otherwise exponential from + * `RETRY_BASE_MS` with jitter, so many guards retrying after one shared outage do not return in step. + */ +export function retryDelayMs(attempts, retryAfter, random = Math.random) { + const advertised = parseRetryAfter(retryAfter); + if (advertised !== null) return Math.min(advertised, RETRY_CAP_MS); + + const backoff = Math.min(RETRY_BASE_MS * 2 ** (attempts - 1), RETRY_CAP_MS); + + // ±25%, then capped again: jitter applied to a capped value can exceed the cap, so the cap goes last. + return Math.min(RETRY_CAP_MS, Math.round(backoff * (0.75 + random() * 0.5))); +} + +function parseRetryAfter(value) { + if (typeof value !== 'string' || value.trim() === '') return null; + const seconds = Number(value.trim()); + if (Number.isFinite(seconds)) return seconds >= 0 ? seconds * 1000 : null; + const at = Date.parse(value); + if (Number.isNaN(at)) return null; + + return Math.max(0, at - Date.now()); +} + +const encoder = new TextEncoder(); + +/** The size of a string on the wire. `length` counts UTF-16 code units, which is not that. */ +export function byteLength(text) { + return encoder.encode(text).length; +} + +/** + * As many events as fit the byte bound, and the rest. + * + * The bound is on the REQUEST, so `wrap` builds the body that will actually be sent — envelope, drop + * count and reporting state included — and it is measured in bytes rather than characters. Measuring the + * events alone, or measuring `length`, both understate the request: one leaves out the envelope, the + * other counts a multi-byte character as one. Either would let a body past the bound on the wire while + * the check reported it as fitting. + * + * At least one event always goes, since a batch of none makes no progress and would meet the same bound + * on every retry. + */ +export function splitToFit(events, maxBytes, wrap = (batch) => ({ detections: batch })) { + const batch = events.slice(); + const rest = []; + while (batch.length > 1 && byteLength(JSON.stringify(wrap(batch))) > maxBytes) { + rest.unshift(batch.pop()); + } + + return [batch, rest]; +} + +/** A timer that cannot hold a process open: a pending retry must never be why a command does not exit. */ +function unattended(timer) { + if (timer && typeof timer.unref === 'function') timer.unref(); + + return timer; +} + /** * The parameters a rule reads, from its own definition. * @@ -112,8 +456,11 @@ export function createDetectionReporter(opts) { // Nothing to report against. A no-op rather than a throw: reporting is never worth failing a boot. // It answers the whole interface, so a caller never has to know which kind it holds. return { - record() {}, flush() {}, stop() {}, setRulesEtag() {}, dropped: () => 0, - health: () => ({ sent: 0, delivered: 0, failed: 0, dropped: 0, lastDeliveredAt: null }), + record() {}, flush() {}, stop: () => Promise.resolve(), setRulesEtag() {}, announce() {}, dropped: () => 0, + health: () => ({ + sent: 0, delivered: 0, failed: 0, dropped: 0, retried: 0, reauthorized: 0, lastDeliveredAt: null, + capability: { announced: 0, acknowledged: 0, failed: 0, retried: 0, lastAcknowledgedAt: null }, + }), }; } @@ -141,59 +488,365 @@ export function createDetectionReporter(opts) { /** @type {string | null} */ let lastDeliveredAt = null; + // Capability accounting, kept apart from the event counters above. + // + // The event counters are measured in EVENTS. A state announcement carries none, so letting it advance + // `lastDeliveredAt` or `failed` produces readings that cannot describe any real delivery — `sent: 0` + // with `failed: 1` — and makes a capability acknowledgement indistinguishable from a delivered + // detection. They answer different questions and are counted separately. + let capabilityAnnounced = 0; + let capabilityAcknowledged = 0; + let capabilityFailed = 0; + /** @type {string | null} */ + let lastCapabilityAckAt = null; + /** @type {Array>} */ let queue = []; /** @type {ReturnType | null} */ let timer = null; + /** @type {ReturnType | null} */ + let retryTimer = null; let stopped = false; let dropped = 0; + let retried = 0; + let capabilityRetried = 0; + let flushRequested = false; + let draining = false; + /** @type {(() => void) | null} */ + let drainResolve = null; + /** @type {Promise | null} */ + let drainPromise = null; + /** @type {ReturnType | null} */ + let budgetTimer = null; + /** Bumped when a drain is terminated, so a late response cannot move a counter after the fact. */ + let epoch = 0; + let reauthorized = 0; - const flush = () => { - if (timer) { - clearTimeout(timer); - timer = null; - } - if (queue.length === 0 || typeof fetchImpl !== 'function') return; + // One send at a time, and one batch's worth of state while it runs. + let sending = false; + /** + * @type {{ + * key: string, events: Array>, dropped: number, + * state: string | null, attempts: number, + * } | null} + */ + let inFlight = null; + + /** + * The newest reporting state not yet declared, and only the newest. + * + * States supersede rather than accumulate: what the platform needs is the state this guard is in now, + * so a queue of them would deliver a history nobody asked for and end by declaring the same thing + * anyway. A state arriving while a send runs replaces whatever was waiting, and travels with the next + * send — attached to a batch of events when there is one, alone when there is not. + * + * @type {string | null} + */ + let pendingState = null; + const instanceId = makeInstanceId(); + let sequence = 0; - const batch = queue.splice(0, MAX_BATCH); + /** Events up to the batch and byte bounds, leaving the rest queued. */ + /** The exact request body for a batch: what is measured is what is sent. */ + const bodyFor = (events, droppedWith, state) => ({ + detections: events, // The count of what never made it, sent WITH the batch rather than inferred from a gap: a consumer // computing a false-positive rate needs to know its denominator is short, and silence about that // would make a truncated sample look like a complete one. + dropped: droppedWith, + ...(state !== null ? { reporting_state: state } : {}), + }); + + const takeBatch = (droppedWith, state) => { + const [batch, rest] = splitToFit(queue.splice(0, MAX_BATCH), MAX_BODY_BYTES, (events) => + bodyFor(events, droppedWith, state), + ); + if (rest.length > 0) queue.unshift(...rest); + + return batch; + }; + + /** + * The credential exchange's own bound: this reporter's, not the application's. + * + * It matches an attempt, because the exchange is a separate request that an attempt's abort signal does + * not reach — bounded any longer, a token call would hold the single send slot past the point the + * attempt was meant to end. It must also BE a number: the exchange builds its timeout from this value, + * and an absent one makes that construction throw, which the exchange reports as "no token" and every + * site-addressed endpoint then refuses. + * + * Deliberately not a knob. Nothing in the public options feeds a value here, so reading one would be a + * setting a caller cannot set — always undefined, always falling through to a default. + */ + const authConfig = { pulseAuth: opts.pulseAuth, endpoint: baseUrl, timeoutMs: ATTEMPT_TIMEOUT_MS }; + const detectionsUrl = `${baseUrl}/detections/${encodeURIComponent(siteUuid)}`; + + const post = async (body, key, signal, transport) => + // Through the shared Pulse path, which attaches the token and — on a 401 — discards it and retries + // once with a fresh one. A cached token can stop being valid before it expires, and the server's + // refusal is authoritative over our own clock; the batch's own headers, this key included, are + // carried through both sends, so the redelivery is still recognisable as the same batch. + pulseFetch( + authConfig, + detectionsUrl, + { + method: 'POST', + ...(signal ? { signal } : {}), + headers: { + 'Content-Type': 'application/json', + Accept: 'application/json', + 'User-Agent': '@patchstack/connect', + // The same key on every attempt of one batch: an acknowledgement can be lost after the server + // has already taken the batch, and a redelivery has to be identifiable as the same one. + 'Idempotency-Key': key, + }, + body: JSON.stringify(body), + }, + transport, + ); + + /** Give up on the batch in flight, counting what it carried. */ + const abandon = () => { + if (!inFlight) return; + failed += inFlight.events.length; + if (inFlight.state !== null) capabilityFailed += 1; + inFlight = null; + }; + + const settle = () => { + if (!inFlight) return; + if (inFlight.events.length > 0) { + delivered += inFlight.events.length; + lastDeliveredAt = new Date().toISOString(); + } + if (inFlight.state !== null) { + capabilityAcknowledged += 1; + lastCapabilityAckAt = new Date().toISOString(); + } + inFlight = null; + }; + + /** + * One attempt at the batch in flight, then either done, retried, or given up on. + * + * Fail-open throughout: no path here rejects, throws into the caller, or blocks a request. A delivery + * problem is counted and nothing more. + */ + const attempt = async () => { + if (!inFlight) return; + const mine = epoch; + sending = true; + inFlight.attempts += 1; + if (inFlight.attempts > 1) { + // Counted against whatever the request carries. A state-only request carries no events, so letting + // it advance the event counter would report retries of deliveries that never happened — the same + // conflation the delivered/acknowledged split exists to prevent. A request carrying both counts on + // both, because both were retried. + if (inFlight.events.length > 0) retried += 1; + if (inFlight.state !== null) capabilityRetried += 1; + } + + const body = bodyFor(inFlight.events, inFlight.dropped, inFlight.state); + + let status = null; + let retryAfter = null; + // A hung request must look like a failure rather than holding the only send slot forever. + const controller = typeof AbortController === 'function' ? new AbortController() : null; + inFlight.controller = controller; + const timeout = controller + ? unattended(setTimeout(() => controller.abort(), ATTEMPT_TIMEOUT_MS)) + : null; + // Owned by the batch, so termination can cancel it. A transport that ignores an abort would otherwise + // leave this scheduled past the shutdown that already reported itself finished. + inFlight.timeout = timeout; + // The credential path may send the same batch twice: once with a token the server refuses, once with + // a reissued one. That is a redelivery of this batch, so it is counted rather than invisible. + let sends = 0; + const counted = (url, init) => { + if (String(url) === detectionsUrl) { + sends += 1; + // Counted as the second send is made, and only while this attempt still owns the numbers. After + // `terminate()` the health has been reported as final, and a refresh completing later must not + // move it. + if (sends > 1 && mine === epoch) reauthorized += 1; + } + + return fetchImpl(url, init); + }; + try { + const res = await post(body, inFlight.key, controller?.signal, counted); + if (mine !== epoch) return; // the drain was terminated while this was open + if (res && res.ok) { + settle(); + if (timeout) clearTimeout(timeout); + finish(); + + return; + } + status = typeof res?.status === 'number' ? res.status : 0; + retryAfter = res?.headers?.get?.('retry-after') ?? null; + } catch { + // Unreachable, timed out, or aborted: worth another attempt, since nothing says the endpoint is + // unwilling. An abort from `stop()` is not retried, because `stopped` closes that path below. + status = null; + } finally { + if (timeout) clearTimeout(timeout); + // Only if it is still THIS attempt's. On the acknowledged path `settle()` and `finish()` run inside + // the block above, so by the time this executes `inFlight` can already be the NEXT batch — and + // clearing its controller would leave that batch with nothing to abort it by. + if (inFlight && inFlight.controller === controller) { + inFlight.controller = null; + inFlight.timeout = null; + } + } + + if (mine !== epoch) return; + + const retryable = worthRetrying(status); + // Not after `stop()`: the guard is going away, and a timer that outlives it would keep a process + // alive to deliver a report nobody is waiting for. + if (retryable && inFlight.attempts < MAX_ATTEMPTS && !stopped) { + const delay = retryDelayMs(inFlight.attempts, retryAfter); + sending = false; + retryTimer = unattended( + setTimeout(() => { + retryTimer = null; + void attempt(); + }, delay), + ); + + return; + } + + abandon(); + finish(); + }; + + /** Whatever accumulated while that send ran. */ + const finish = () => { + sending = false; + kick(); + }; + + const arm = () => { + if (!timer) { + timer = unattended( + setTimeout(() => { + timer = null; + flushRequested = true; + kick(); + }, flushMs), + ); + } + }; + + /** + * Whether there is reason to send NOW, as opposed to reason to send eventually. + * + * Without this the buffer would empty every time a send completed, because whatever accumulated during + * one request would immediately become the next — and the flush interval, which exists so a busy app + * makes one request instead of fifty, would apply only to the first batch of a guard's life. + */ + // While draining there is no "eventually": everything left goes now, or is accounted for. + const due = () => draining || pendingState !== null || queue.length >= MAX_BATCH || flushRequested; + + /** Start a send if one is due and nothing is already in flight; otherwise wait for the interval. */ + const kick = () => { + if (sending || inFlight) return; + if (typeof fetchImpl !== 'function') { + // Nothing can be sent, so a drain makes no progress and the queue is accounted for here. + if (draining) drained(); + + return; + } + // After `stop()` the only sends are the drain's own. `record` and `announce` also refuse once stopped, + // so this is the second of two independent refusals rather than the only one — deliberately, because + // the property it protects is that a torn-down guard opens no connections and arms no timers. + if (stopped && !draining) return; + if (queue.length === 0 && pendingState === null) { + flushRequested = false; + if (draining) drained(); + + return; + } + if (!due()) { + arm(); + + return; + } + flushRequested = false; + const droppedWith = dropped; dropped = 0; droppedTotal += droppedWith; - sent += batch.length; - - void (async () => { - try { - const res = await fetchImpl(`${baseUrl}/detections/${encodeURIComponent(siteUuid)}`, { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - Accept: 'application/json', - 'User-Agent': '@patchstack/connect', - // Same credential path as the rules fetch. The detections endpoint is site-addressed and - // requires a verified, site-bound token, so a batch sent without one is refused — which is - // why the runtime does not build a reporter when no credential resolves, rather than - // posting into a 401. - ...(await pulseAuthHeader({ pulseAuth: opts.pulseAuth, endpoint: baseUrl }, fetchImpl)), - }, - body: JSON.stringify({ detections: batch, dropped: droppedWith }), - }); - // Fail-open and no retry: a rejected or unreachable endpoint must not disturb the app, and a - // retry loop over a refusing endpoint is worse than the lost batch. The outcome is counted, so - // that a delivery path which refuses everything is distinguishable from an app where no rule - // fired — both are silence at the server otherwise. - if (res && res.ok) { - delivered += batch.length; - lastDeliveredAt = new Date().toISOString(); - } else { - failed += batch.length; - } - } catch { - failed += batch.length; - } - })(); + + const state = pendingState; + pendingState = null; + + const events = takeBatch(droppedWith, state); + sent += events.length; + // Counted where the declaration is committed to a request, so coalesced states count once — the + // number describes declarations this guard undertook to make, not calls received. A state still + // waiting when a shutdown ends counts here too, against a matching failure. + if (state !== null) capabilityAnnounced += 1; + + sequence += 1; + inFlight = { key: `${instanceId}-${sequence}`, events, dropped: droppedWith, state, attempts: 0 }; + void attempt(); + }; + + /** + * End the drain now, because the shutdown budget is spent. + * + * Resolving alone would have been the promise claiming a completion that had not happened: the request + * would still be open, the queue unaccounted, and later batches free to follow. So the attempt is + * abandoned and counted, the queue is accounted for, and `epoch` moves — which is what stops a response + * that lands afterwards from moving any counter, since by then the numbers have already been reported + * as final. + */ + const terminate = () => { + epoch += 1; + if (inFlight) { + inFlight.controller?.abort(); + if (inFlight.timeout) clearTimeout(inFlight.timeout); + failed += inFlight.events.length; + if (inFlight.state !== null) capabilityFailed += 1; + inFlight = null; + } + // A state that was waiting for a request it will now never get. Counted, because "the platform was + // not told my final state" is exactly what a reader of these numbers is trying to find out. + if (pendingState !== null) { + pendingState = null; + capabilityAnnounced += 1; + capabilityFailed += 1; + } + sending = false; + drained(); + }; + + /** Nothing left to send: whatever never left is counted rather than forgotten, and the wait ends. */ + const drained = () => { + draining = false; + if (queue.length > 0) { + droppedTotal += queue.length; + queue = []; + } + if (budgetTimer) { + clearTimeout(budgetTimer); + budgetTimer = null; + } + const resolve = drainResolve; + drainResolve = null; + if (resolve) resolve(); + }; + + const flush = () => { + if (timer) { + clearTimeout(timer); + timer = null; + } + flushRequested = true; + kick(); }; return { @@ -215,34 +868,169 @@ export function createDetectionReporter(opts) { dropped++; } + // Capped, with a note of what was capped. Every field is an identifier rather than traffic, but a + // route is whatever the application routes and a broad rule can read many parameters — and a reader + // who cannot tell a shortened route from a complete one will read it as a different route. + // `null` survives: there being no route is not the same as the route being empty, and a cap that + // turned one into the other would invent a known path where none was established. + const query = queryKeysOf(detection.path); + const method = typeof detection.method === 'string' ? capText(detection.method, MAX_METHOD_CHARS) : null; + const userAgent = + typeof detection.userAgent === 'string' && detection.userAgent !== '' + ? capText(detection.userAgent, MAX_IDENTIFIER_CHARS) + : null; + const queryKeys = query.keys.map((name) => capText(name, MAX_PARAMETER_CHARS)); + // Computed once: calling the gate twice would let a getter or a proxy answer differently between + // the check and the send. + const capture = boundCapture(detection.capture); + const rawRoute = routeOf(detection.path); + const route = typeof rawRoute === 'string' ? capText(rawRoute, MAX_ROUTE_CHARS) : { value: rawRoute, truncated: false }; + const allParameters = ruleParameters(detection.rule); + const parameters = allParameters.slice(0, MAX_PARAMETERS).map((name) => capText(name, MAX_PARAMETER_CHARS)); + const truncated = []; + if (route.truncated) truncated.push('route'); + if (method?.truncated) truncated.push('method'); + if (userAgent?.truncated) truncated.push('user_agent'); + if (query.total > queryKeys.length || queryKeys.some((entry) => entry.truncated)) { + truncated.push('query_keys'); + } + if (allParameters.length > MAX_PARAMETERS || parameters.some((entry) => entry.truncated)) { + truncated.push('parameters'); + } + + const id = capText(String(ruleId), MAX_IDENTIFIER_CHARS); + const revision = capText(revisionOf(detection.rule) ?? '', MAX_IDENTIFIER_CHARS); + const etag = capText(rulesEtag ?? '', MAX_IDENTIFIER_CHARS); + for (const [name, field] of [ + ['rule_id', id], + ['rule_revision', revision], + ['rules_etag', etag], + ]) { + if (field.truncated) truncated.push(name); + } + queue.push({ - rule_id: ruleId, - route: routeOf(detection.path), - parameters: ruleParameters(detection.rule), + rule_id: id.value, + route: route.value, + parameters: parameters.map((entry) => entry.value), + // Present only when something was shortened, so its absence is not a claim of its own. + ...(truncated.length > 0 ? { truncated } : {}), + // Only when parameters were actually left out. Reporting a total because some OTHER field was + // shortened states that parameters were omitted when none were. + ...(truncated.includes('parameters') ? { parameters_total: allParameters.length } : {}), + method: method === null ? null : method.value, + // The rest of the URL, as names only. `route` is the path; together they say what was requested + // without saying what was in it. + query_keys: queryKeys.map((entry) => entry.value), + // Present only when the list is short, so its absence is not a claim of its own. + ...(query.total > queryKeys.length ? { query_keys_total: query.total } : {}), + // Who asked. Capped, since it is client-supplied text and this is an event with a size bound. + user_agent: userAgent === null ? null : userAgent.value, phase: detection.phase ?? null, // The state this detection was handled under, which is the whole point: `false` is a rule that // saw traffic it would have stopped. enforced: detection.mode === 'block', - rules_etag: rulesEtag, + rules_etag: rulesEtag === null ? null : etag.value, // The revision of THIS rule, as the bundle delivered it. The bundle identity above answers "which // bundle", which changes whenever anything in it changes — so it cannot say whether the counts for // one rule describe the document that rule has now. Passed through untouched, and null when the // bundle carried none. - rule_revision: revisionOf(detection.rule), + rule_revision: revisionOf(detection.rule) === null ? null : revision.value, + // What the rule was permitted to show, and what it showed. Present only when a plan was derived, + // which is to say only for a phase that has a reading to derive from. + ...(capture ? { capture } : {}), + // The client address and where it came from. `client_ip` is omitted entirely when there is none, + // so a present-but-empty field cannot read as a failed lookup of a real address; the provenance is + // always present, because "this could not be established" is the part a reader needs. + ...clientIpFields({ ip: detection.ip ?? null, source: detection.clientIpSource ?? 'unavailable' }), detected_at: new Date().toISOString(), }); - if (queue.length >= MAX_BATCH) { - flush(); - - return; - } - if (!timer) timer = setTimeout(flush, flushMs); + // `kick` decides whether this is due now or waits for the interval, so the batch bound and the + // interval cannot disagree about when a full queue goes out. + kick(); }, flush, + /** + * Tell the platform which reporting state this guard settled on. + * + * The state also travels on the rules fetch, but that request is made BEFORE the fetch decides + * whether the rules are the platform's — so a site booting with an empty cache declares that it holds + * no managed rules, then receives them. Without this, the corrected state would wait for the next + * refresh, and a guard with refreshing switched off has none. + * + * Goes through the same transport as events, so it inherits the retry, the idempotency key and the + * single-flight bound. Only the newest state is kept: calling this twice before a send declares the + * second, because what the platform needs is the state now, not how it got there. A declaration that + * exhausts its retries is dropped rather than held — the state travels on every rules fetch too, so + * the next one corrects it. + * + * @param {string} state + */ + announce(state) { + if (stopped || typeof fetchImpl !== 'function' || typeof state !== 'string') return; + pendingState = state; + // Sent on a microtask, so declarations made in the same turn coalesce into one: by the time this + // runs, `pendingState` holds the last of them. Sending on the call would commit the first state + // before the second could supersede it, and the platform would be told a state this guard had + // already left. A second microtask finds nothing pending and does nothing. + queueMicrotask(() => { + if (!stopped) kick(); + }); + }, + /** + * Stop reporting, leaving nothing stranded and nothing scheduled. + * + * Three things can be outstanding, and each needs an answer: + * + * - a batch waiting on a retry timer — it holds the only send slot, so clearing the timer alone would + * leave it neither delivered nor counted. It gets one final attempt, with no retry behind it. + * - a request in flight — it is aborted, and its completion drains what is left rather than starting + * open-ended work. + * - events still queued — drained a batch at a time, each attempted once. + * + * Whatever remains unsent when the drain runs out is counted as dropped, so every recorded event ends + * up delivered, refused or dropped, and none simply disappears. + * + * The drain is asynchronous, so this returns a promise that settles when nothing is outstanding — + * which a host shutting down can await instead of racing the last batch against process exit. It is + * best-effort and bounded: a runtime that terminates the process regardless, or a drain slower than + * `STOP_BUDGET_MS`, still ends the wait. Ignoring the promise behaves exactly as before. + */ stop() { + if (stopped) return drainPromise ?? Promise.resolve(); stopped = true; - flush(); + drainPromise = new Promise((resolve) => { + drainResolve = resolve; + }); + budgetTimer = unattended(setTimeout(() => { + budgetTimer = null; + terminate(); + }, STOP_BUDGET_MS)); + if (timer) { + clearTimeout(timer); + timer = null; + } + draining = true; + + if (retryTimer) { + clearTimeout(retryTimer); + retryTimer = null; + void attempt(); + + return drainPromise; + } + if (sending) { + // Its completion continues the drain. Aborting turns a hung request into a failure now rather + // than a slot held until the process exits. + inFlight?.controller?.abort(); + + return drainPromise; + } + flushRequested = true; + kick(); + + return drainPromise; }, /** * Point later events at the bundle now running. Called after an ACCEPTED swap only — a rejected @@ -261,6 +1049,27 @@ export function createDetectionReporter(opts) { * Delivery health, counted in events: attempted, acknowledged, refused or unreachable, and dropped * for queue pressure — plus when a batch was last acknowledged. No request data of any kind. */ - health: () => ({ sent, delivered, failed, dropped: droppedTotal + dropped, lastDeliveredAt }), + health: () => ({ + sent, + delivered, + failed, + dropped: droppedTotal + dropped, + // Attempts beyond the first, counted in ATTEMPTS rather than events: a path that only ever + // succeeds on a second try is working, and is worth telling apart from one that never retries. + retried, + // Redeliveries the credential path made after a refused token, which are not backoff retries and + // would otherwise appear nowhere: a rotated or revoked credential is worth seeing as itself. + reauthorized, + lastDeliveredAt, + // Separate, because a capability announcement delivers no events. Reading zero here alongside a + // non-zero `delivered` is a normal state, and so is the reverse. + capability: { + announced: capabilityAnnounced, + acknowledged: capabilityAcknowledged, + failed: capabilityFailed, + retried: capabilityRetried, + lastAcknowledgedAt: lastCapabilityAckAt, + }, + }), }; } diff --git a/src/protect/engine/engine.js b/src/protect/engine/engine.js index b2a8af55..be9d75b4 100644 --- a/src/protect/engine/engine.js +++ b/src/protect/engine/engine.js @@ -829,7 +829,12 @@ export class RuleEngine { return { blocked: true, rule, - message: rule.message ?? `Blocked by Patchstack WAF rule: ${rule.title ?? rule.id}` + message: rule.message ?? `Blocked by Patchstack WAF rule: ${rule.title ?? rule.id}`, + // The resolver this match was decided by, for whatever needs to report on it. A consumer + // that re-read the request instead would be reading it a second time: a getter, a stream or + // anything else that answers once can give a different value, and evidence that disagrees + // with the match it belongs to is worse than none. + resolver }; } } catch (err) { diff --git a/src/protect/engine/fetch.js b/src/protect/engine/fetch.js index 11918882..9c461499 100644 --- a/src/protect/engine/fetch.js +++ b/src/protect/engine/fetch.js @@ -6,6 +6,7 @@ // The engine already consumes a runtime-neutral request shape; this adapter builds that // shape from a `Request`, so no engine changes are needed beyond keeping the hot path // free of Node-only APIs. +import { resolveClientIp } from '../client-ip.js'; import { RuleEngine } from './engine.js'; import { notify } from '../notify.js'; @@ -49,8 +50,10 @@ export async function fromFetchRequest(request, options = {}) { } const uri = url.pathname + url.search; - const forwarded = - headers['cf-connecting-ip'] || headers['x-forwarded-for'] || headers['x-real-ip'] || ''; + // The client address, resolved once for this request, with its provenance. A WHATWG Request exposes no + // transport peer, so a generic Fetch runtime has none to observe and the answer is `unavailable` unless + // the deployment declared a trust policy this runtime can satisfy. + const client = resolveClientIp({ peer: options.peer, headers, trustedProxy: options.trustedProxy }); return { method, @@ -60,7 +63,10 @@ export async function fromFetchRequest(request, options = {}) { body, files, headers, - ip: forwarded.split(',')[0].trim(), + // The resolved address, as an own property. Nothing downstream re-derives or falls back: a consumer + // that guessed differently would attribute one request to two addresses. + ip: client.ip ?? '', + _clientIp: client, cookies: parseCookies(headers.cookie), // Verbatim body text: preserves literal keys (e.g. `__proto__`) that JSON.stringify // drops, so prototype-pollution rules on `raw` are robust. diff --git a/src/protect/engine/middleware.js b/src/protect/engine/middleware.js index ab19a4a4..8d5f29ce 100644 --- a/src/protect/engine/middleware.js +++ b/src/protect/engine/middleware.js @@ -6,7 +6,11 @@ export function createMiddleware(rulesData, options = {}) { const engine = new RuleEngine(rulesData); const middleware = (req, res, next) => { - const result = engine.evaluate(req); + // The address resolved for this request, when the caller supplied a resolver. Without one the engine + // sees whatever own `ip` the request carries, and this record says the same. + const client = typeof options.resolveClient === 'function' ? options.resolveClient(req) : null; + const evaluated = client === null ? req : client.shaped; + const result = engine.evaluate(evaluated); if (result.blocked) { // Contained: a throw here would replace the 403 below with the callback's exception, which for @@ -17,7 +21,10 @@ export function createMiddleware(rulesData, options = {}) { request: { method: req.method, url: req.url, - ip: req.ip ?? req.socket?.remoteAddress + // Never `req.ip`: under Express's `trust proxy` that is header-derived by a policy this guard + // has not verified. The engine's own value is what this record reports, so the two agree. + ip: typeof evaluated.ip === 'string' ? evaluated.ip : null, + clientIpSource: client?.client?.source ?? null, } }, 'onBlock'); diff --git a/src/protect/engine/node.js b/src/protect/engine/node.js index 20ae129e..8b60c0ec 100644 --- a/src/protect/engine/node.js +++ b/src/protect/engine/node.js @@ -6,12 +6,13 @@ // This complements the Express `createMiddleware` (which assumes `req.body`/`req.query` // are already populated) and the Web-Fetch adapter (Workers/edge). Mount it FIRST, before // any body-parser — it consumes the stream and exposes the parsed body as `req.body`. +import { resolveClientIp } from '../client-ip.js'; import { RuleEngine } from './engine.js'; import { parseBody } from './fetch.js'; import { notify } from '../notify.js'; // Build the engine's request shape from a Node IncomingMessage + its raw body text. -export function fromNodeRequest(req, rawBody = '') { +export function fromNodeRequest(req, rawBody = '', options = {}) { const method = (req.method || 'GET').toUpperCase(); const headers = {}; @@ -54,8 +55,13 @@ export function fromNodeRequest(req, rawBody = '') { } const uri = url.pathname + url.search; - const forwarded = - headers['cf-connecting-ip'] || headers['x-forwarded-for'] || headers['x-real-ip'] || ''; + // Resolved once, from the socket peer the transport observed. `req.ip` is deliberately not consulted: + // under Express's `trust proxy` it is itself header-derived by a policy this guard has not verified. + const client = resolveClientIp({ + peer: req.socket?.remoteAddress, + headers, + trustedProxy: options.trustedProxy, + }); return { method, @@ -65,7 +71,8 @@ export function fromNodeRequest(req, rawBody = '') { body, files, headers, - ip: forwarded.split(',')[0].trim() || (req.socket && req.socket.remoteAddress) || '', + ip: client.ip ?? '', + _clientIp: client, cookies: parseCookies(headers.cookie), // Verbatim body text: preserves literal keys (e.g. `__proto__`) that JSON.stringify drops. _rawBody: rawBody @@ -131,7 +138,9 @@ export function createNodeMiddleware(rulesData, options = {}) { let shaped; try { const rawBody = overflow ? '' : Buffer.concat(chunks).toString('utf8'); - shaped = fromNodeRequest(req, rawBody); // shaping is inside the try too — never crash + // The caller's policy reaches the shaping, or this adapter would always report the socket peer + // even where its caller declared a trusted front end. + shaped = fromNodeRequest(req, rawBody, { trustedProxy: options.trustedProxy }); // never crash result = engine.evaluate(shaped); } catch (err) { notify(options.onError, err, 'onError'); diff --git a/src/protect/engine/normalizer.js b/src/protect/engine/normalizer.js index 692daa8e..762867c5 100644 --- a/src/protect/engine/normalizer.js +++ b/src/protect/engine/normalizer.js @@ -201,21 +201,68 @@ function serializeForRawDetection(body, visited = new Set(), isRoot = true) { return '{' + parts.join(',') + '}'; } +// The only fields a supported framework supplies through an inherited accessor: `headers` is a getter on +// `IncomingMessage.prototype`, and Express defines `query` on its request prototype. Every other field +// arrives as an own property — body parsers, cookie parsers and upload middleware all assign one, and the +// Node and Fetch adapters build their request shape as a literal. +const INHERITED_ACCESSORS = new Set(['headers', 'query']); + +/** + * A request field, taken only from the request itself. + * + * Evidence is what a request actually carried. A value reachable only through a prototype was carried by + * nothing, and materialising it would let one write stand as request data and fire every rule that + * matches it — arriving indistinguishable from a real body, query or header. + * + * An own property is evidence. Everything inherited is refused, with one exception: a getter on a + * prototype, for the two fields above. That exception exists because requiring an own property would + * discard how Node and Express really expose headers and the query string, and silently stop screening + * the sources rules read most — a worse failure than the pollution it prevents. + * + * The exception is deliberately narrow in both directions. An inherited DATA property is refused however + * it arrives, because a framework does not install request data that way and a write to a prototype does; + * and `Object.prototype` is refused even for an accessor, because that is where a pollution primitive + * lands. + */ +export function requestField(req, key) { + if (req === null || typeof req !== 'object') return undefined; + if (Object.hasOwn(req, key)) return req[key]; + if (!INHERITED_ACCESSORS.has(key)) return undefined; + + let holder = Object.getPrototypeOf(req); + while (holder !== null && !Object.hasOwn(holder, key)) holder = Object.getPrototypeOf(holder); + if (holder === null || holder === Object.prototype) return undefined; + + // An accessor, not a value parked on a prototype the request happens to inherit from. + const descriptor = Object.getOwnPropertyDescriptor(holder, key); + if (typeof descriptor?.get !== 'function') return undefined; + + // Read through `req` so the accessor runs with the receiver it expects. + return req[key]; +} + export function normalizeRequest(req, options = {}) { // Prefer a caller-provided verbatim body string (set by the fetch/node adapters): // it preserves literal keys like `__proto__` that JSON.stringify drops, which is // what makes prototype-pollution rules on `raw` robust. Fall back to a // reconstruction from the parsed body (the Express path, which has no raw text). - const rawBody = typeof req._rawBody === 'string' + // Own property only: a `_rawBody` reachable through a polluted prototype is not something this + // request carried, and accepting it here would let it stand as verbatim evidence on every path. + const ownRaw = Object.hasOwn(req ?? {}, '_rawBody') && typeof req._rawBody === 'string'; + // Every field below comes through the same gate, because the reconstruction fallback reads the parsed + // body: gating `_rawBody` alone would leave a polluted `body` serialised into raw evidence anyway. + const body = requestField(req, 'body'); + const url = requestField(req, 'url'); + const rawBody = ownRaw ? req._rawBody - : serializeForRawDetection(req.body ?? null); + : serializeForRawDetection(body ?? null); return { - query: normalizeObject(req.query || {}, options), - body: normalizeObject(req.body || {}, options), - headers: normalizeObject(req.headers || {}, options), - url: normalize(req.url || '', options), - originalUrl: normalize(req.originalUrl || req.url || '', options), + query: normalizeObject(requestField(req, 'query') || {}, options), + body: normalizeObject(body || {}, options), + headers: normalizeObject(requestField(req, 'headers') || {}, options), + url: normalize(url || '', options), + originalUrl: normalize(requestField(req, 'originalUrl') || url || '', options), _rawBody: rawBody }; } diff --git a/src/protect/engine/pulse-client.js b/src/protect/engine/pulse-client.js index 6374916d..e0efcf87 100644 --- a/src/protect/engine/pulse-client.js +++ b/src/protect/engine/pulse-client.js @@ -28,9 +28,9 @@ export class PulseRuleClient { #etag; #pulseAuth; - #reportsDetections; + #detectionState; - constructor({ siteUuid, baseUrl, cacheTtl, etag, timeoutMs, pulseAuth, reportsDetections } = {}) { + constructor({ siteUuid, baseUrl, cacheTtl, etag, timeoutMs, pulseAuth, detectionState } = {}) { // Bounded so app STARTUP can't hang on a slow API: hosted platforms fail a deploy whose health // check is slow, and we always have a cache/bundled fallback to boot from. this.#timeoutMs = Number(timeoutMs) > 0 ? Number(timeoutMs) : 30_000; @@ -49,7 +49,7 @@ export class PulseRuleClient { // // A capability, not a timestamp: the server records when IT saw this, because a client clock is a // value from outside and "alive as of" is exactly the claim a stale or wrong clock would fake. - this.#reportsDetections = reportsDetections === true; + this.#detectionState = typeof detectionState === 'string' ? detectionState : null; if (!this.#siteUuid) { throw new Error('Patchstack site UUID is required. Pass { siteUuid } or set PATCHSTACK_SITE_UUID.'); } @@ -78,8 +78,12 @@ export class PulseRuleClient { // // A courtesy, never the guarantee: a client-side gate only removes the accidental case. Anything // acting on this header has to require a verified token itself before believing it. - if (this.#reportsDetections && typeof auth.Authorization === 'string') { - headers['X-Patchstack-Detections'] = 'enabled'; + // The state itself, not a bit. "No events arrived" has several causes — nothing matched, an + // explicit opt-out, never enrolled, no credential — and a boolean collapses them into the + // reassuring reading. Sent on the fetch the guard already makes, so the platform learns the state + // without waiting for a rule to fire. + if (this.#detectionState !== null && typeof auth.Authorization === 'string') { + headers['X-Patchstack-Detections'] = this.#detectionState; } if (this.#etag) headers['If-None-Match'] = this.#etag; const response = await fetch(url, { method: 'GET', headers, signal: AbortSignal.timeout(this.#timeoutMs) }); diff --git a/src/protect/engine/request.js b/src/protect/engine/request.js index d6273240..5e55003c 100644 --- a/src/protect/engine/request.js +++ b/src/protect/engine/request.js @@ -288,7 +288,10 @@ export class RequestResolver { return req.headers?.host ? [req.headers.host] : []; case 'REMOTE_ADDR': case 'ip': - return [req.ip ?? req.socket?.remoteAddress ?? '']; + // The address the caller resolved, and nothing else. Falling back to the socket — or to a + // forwarded header — would let one request be attributed to two different addresses depending on + // which consumer asked, and would reintroduce a value the client can set. + return [typeof req.ip === 'string' ? req.ip : '']; case 'CONTENT_TYPE': return req.headers?.['content-type'] ? [req.headers['content-type']] : []; case 'CONTENT_LENGTH': diff --git a/src/protect/firewall-log.js b/src/protect/firewall-log.js index 7733a3be..f493d03a 100644 --- a/src/protect/firewall-log.js +++ b/src/protect/firewall-log.js @@ -5,6 +5,8 @@ import { isSafeOrigin } from './safe-origin.js'; // Opt out: PATCHSTACK_TELEMETRY=off. Never put api_key in the public widget. const DEFAULT_API_BASE = 'https://api.patchstack.com'; +/** The shutdown budget, matching the detection reporter's. */ +const STOP_BUDGET_MS = 5_000; const DEFAULT_FLUSH_MS = 1000; const MAX_BATCH = 50; const TOKEN_SKEW_MS = 60_000; @@ -72,7 +74,7 @@ export function resolveApiBase(pulseOrManifestUrl) { export function createFirewallLogReporter(opts) { const creds = parseApiKey(opts.apiKey); if (!creds) { - return { record() {}, flush() {}, stop() {} }; + return { record() {}, flush: () => Promise.resolve(), stop: () => Promise.resolve() }; } const apiBase = (opts.apiBase ?? DEFAULT_API_BASE).replace(/\/$/, ''); @@ -85,13 +87,36 @@ export function createFirewallLogReporter(opts) { /** @type {ReturnType | null} */ let timer = null; let stopped = false; + /** + * Every send that has been started and not finished. + * + * A flush that has already taken its batch leaves an empty queue behind it, so a shutdown looking only + * at the queue would see nothing to wait for while a token exchange or a post was still open. What is + * outstanding is the set of sends, not the contents of the queue. + * + * @type {Set>} + */ + const outstanding = new Set(); + /** @type {Promise | null} */ + let drainPromise = null; + /** + * One controller for every request this reporter makes, for its whole life. + * + * Not created at shutdown: by then the sends worth ending have already started, and a signal handed + * out afterwards reaches none of them. Not one per send either, because the token exchange is SHARED — + * a second send awaits the first send's exchange, so a signal belonging to the second would not reach + * the request it is waiting on. + */ + const lifetime = typeof AbortController === 'function' ? new AbortController() : null; + /** Set when a shutdown gives up waiting: nothing may start, continue, or be retained after it. */ + let ended = false; /** @type {{ token: string, expiresAt: number } | null} */ let cachedToken = null; /** @type {Promise | null} */ let tokenInflight = null; - const fetchAccessToken = async () => { + const fetchAccessToken = async (signal) => { if (cachedToken && Date.now() < cachedToken.expiresAt - TOKEN_SKEW_MS) { return cachedToken.token; } @@ -101,6 +126,7 @@ export function createFirewallLogReporter(opts) { try { const res = await fetchImpl(`${apiBase}/oauth/token`, { method: 'POST', + ...(signal ? { signal } : {}), headers: { 'Content-Type': 'application/json', Accept: 'application/json', @@ -135,18 +161,26 @@ export function createFirewallLogReporter(opts) { clearTimeout(timer); timer = null; } - if (queue.length === 0 || typeof fetchImpl !== 'function') return; + if (ended || queue.length === 0 || typeof fetchImpl !== 'function') return Promise.resolve(); const batch = queue.splice(0, MAX_BATCH); - void (async () => { - const token = await fetchAccessToken(); - if (!token) return; - - const body = new URLSearchParams(); - body.set('type', 'firewall'); - body.set('logs', JSON.stringify(batch)); + // Returned so a shutdown can wait for it, and tracked so a shutdown can find it even when the queue + // it came from is already empty. It never rejects: a caller that ignores it must not produce an + // unhandled rejection, and one that awaits it is waiting for the attempt to finish, not asking + // whether it succeeded. + /** @type {Promise} */ + let entry; + const send = (async () => { try { + const token = await fetchAccessToken(lifetime?.signal); + // Not after a shutdown gave up: it has already reported itself finished. + if (!token || ended) return; + + const body = new URLSearchParams(); + body.set('type', 'firewall'); + body.set('logs', JSON.stringify(batch)); + const p = fetchImpl(`${apiBase}/api/logs/log`, { method: 'POST', headers: { @@ -157,12 +191,23 @@ export function createFirewallLogReporter(opts) { ...(sourceHost ? { 'Source-Host': sourceHost } : {}), }, body, + // Both phases carry it, so a shutdown that runs out of time can end either one. + ...(lifetime ? { signal: lifetime.signal } : {}), }); - if (p && typeof p.then === 'function') p.catch(() => {}); + if (p && typeof p.then === 'function') await p.catch(() => {}); } catch { - /* ignore */ + /* A delivery problem is never worth disturbing the app over. */ + } finally { + // Here rather than in a `.then`: this runs before the promise settles, so a waiter that looks at + // the set the moment its wait resolves cannot see a send that has already finished. + outstanding.delete(entry); } })(); + + entry = send; + outstanding.add(send); + + return send; }; return { @@ -196,9 +241,79 @@ export function createFirewallLogReporter(opts) { if (!timer) timer = setTimeout(flush, flushMs); }, flush, + /** + * Stop, and hand back a wait for what was outstanding. + * + * One flush sends at most a batch, so the queue is drained a batch at a time. The loop stops as soon + * as a pass cannot shrink the queue — with no usable transport there is nothing to wait for, and + * spinning would be worse than leaving the records where they are. + */ stop() { + // The same wait every time. A second call must not hand back a resolved promise while the first + // drain is still running, and must not start a second drain behind it. + if (drainPromise) return drainPromise; stopped = true; - flush(); + // Before anything else. An armed interval would otherwise fire mid-drain, take the queued records + // for itself, and start a send the drain never learns about — leaving the drain to look at an empty + // queue, conclude it is finished, and resolve with that send still open. + if (timer) { + clearTimeout(timer); + timer = null; + } + + /** + * End the drain, rather than merely stop waiting for it. + * + * Racing the wait against a timer would leave the work alive: still blocked, still holding its + * entry, and free to run another flush if the transport answered later — after the shutdown had + * reported itself finished. So the reporter is marked ended, which closes `flush` and the loop + * below, the open requests are aborted, and what was never sent is discarded rather than retained + * by a reporter nobody will read again. + */ + const terminate = () => { + ended = true; + lifetime?.abort(); + queue = []; + outstanding.clear(); + }; + + /** @type {ReturnType | null} */ + let budget = null; + // Bounded like the detection reporter's, and for the same reason: a hung transport would otherwise + // keep a shutdown pending for as long as the process lived, which is not a bounded shutdown. + const spent = new Promise((resolve) => { + budget = setTimeout(() => { + terminate(); + resolve(); + }, STOP_BUDGET_MS); + if (typeof budget.unref === 'function') budget.unref(); + }); + + const work = (async () => { + // Two things can be outstanding and each can produce the other: a send holds records that have + // left the queue, and the queue holds records that will become a send. So this asks again after + // every wait rather than taking one snapshot — a snapshot resolves as soon as the sends it + // happened to capture are done, whatever appeared in the meantime. + // + // It ends when both are empty, when a pass cannot shrink the queue (with no usable transport + // there is nothing to wait for, and spinning would be worse), or when the budget ends it. + while (!ended) { + if (outstanding.size > 0) { + await Promise.all([...outstanding]); + continue; + } + if (queue.length === 0) break; + const before = queue.length; + await flush(); + if (queue.length >= before) break; + } + })(); + + drainPromise = Promise.race([work, spent]).then(() => { + if (budget) clearTimeout(budget); + }); + + return drainPromise; }, }; } diff --git a/src/protect/protect.d.ts b/src/protect/protect.d.ts index 3a06db7c..63abf3b8 100644 --- a/src/protect/protect.d.ts +++ b/src/protect/protect.d.ts @@ -34,27 +34,86 @@ export interface Protection { /** Present with a live source — re-fetch + hot-swap the rules once (used by the loop + push). * Resolves with the outcome of the attempt: `ok: false` means the rules in force came from the * cache or the bundled fallback, not from the source. It does not reject on a source failure. */ - refresh?: () => Promise<{ ok: boolean; reason?: string }>; + /** Refresh the rules now. `ok` is whether the resolution was clean; `origin` is which source supplied + * the rules now in force — `api` and `cache` are Patchstack-delivered, `bundled` is the caller's own + * `rules` option, `empty` is none. A fallback is `ok: false` with the origin it fell back to. */ + refresh?: () => Promise<{ + ok: boolean; + origin?: "api" | "cache" | "bundled" | "empty"; + reason?: string; + }>; /** Present with a live source — a fetch handler that runs `refresh()` when the request carries * the configured refresh secret (a push/zero-day trigger). No secret set → the handler 404s. */ refreshHandler?: () => (request: Request) => Promise; /** Stops everything with a timer or a buffer behind it: the refresh loop, the block log, the * detection reporter (flushing what it holds). Always present, and safe to call twice. */ - stop: () => void; + /** + * Stop everything holding a timer or a buffer. + * + * Resolves once the reporters this reaches — the detection reporter and the block log — have finished + * or been given up on, so a shutdown handler can await it instead of racing process exit. Each has its + * own budget, and when one elapses that reporter is ENDED: its requests are aborted, it starts nothing + * further, and it discards what it was holding. Ignoring the promise behaves as it always has. + * + * Two limits are worth knowing, because neither can be promised away: + * + * - A request is aborted, not guaranteed to stop. A transport that ignores its abort signal is + * DETACHED — this stops waiting on it and stops acting on its result — so "resolved" means the + * reporter is finished with it, not that the underlying request has ended. + * - A runtime that terminates the process regardless still wins, whatever this resolves. + * + * What is accounted for also differs by reporter. Every detection event ends up delivered, refused or + * dropped, and `detectionHealth()` reports each. Block-log records have no counters at all, so one lost + * to a failed token exchange, a failed post, or a shutdown that ran out of time is reported nowhere. + */ + stop: () => Promise; /** Alias of `stop`, under the name callers already have. */ - stopRefresh: () => void; - /** Whether detection reporting is running, requested but undeliverable, or not requested. - * `unavailable-no-credential` means `reportDetections` was set but no credential resolved, so - * nothing is being sent. */ - detectionReporting: "on" | "off" | "unavailable-no-credential"; + stopRefresh: () => Promise; + /** Whether this guard reports security events, and if not, why not. + * + * Reporting is on for a site enrolled in Patchstack-managed mitigation that is running managed rules + * with a credential, and off everywhere else. Each state is distinct so "no events arrived" can be + * told apart from "reporting is off" — and it follows refreshes, so a guard that starts on cached or + * bundled rules and later receives managed rules begins reporting without a restart. + * + * - `on` — events are being sent + * - `disabled-by-config` — `PATCHSTACK_REPORT_DETECTIONS` is false, or `reportDetections: false` + * - `disabled-by-telemetry-opt-out` — `PATCHSTACK_TELEMETRY` is false + * - `not-enrolled` — no site identity + * - `no-managed-rules` — the rules in force did not come from Patchstack + * - `unavailable-no-credential` — enrolled, but no credential resolved */ + detectionReporting: + | "on" + | "disabled-by-config" + | "disabled-by-telemetry-opt-out" + | "not-enrolled" + | "no-managed-rules" + | "unavailable-no-credential"; /** Present when detection reporting is on — delivery counts (in events) and the last acknowledgement. * Carries no request data. */ detectionHealth?: () => { + /** Events attempted, acknowledged, refused or unreachable, and dropped for queue pressure. */ sent: number; delivered: number; failed: number; dropped: number; + /** Backoff attempts beyond the first. A path that only ever succeeds on a retry is working, and is + * worth telling apart from one that never has to retry. */ + retried: number; + /** Redeliveries made after the endpoint refused a token, which are not backoff retries: a rotated or + * revoked credential is worth seeing as itself rather than as a delivery failure. */ + reauthorized: number; lastDeliveredAt: string | null; + /** Capability announcements, counted separately: these carry no events, so they never move the + * counters above. Zero here alongside delivered events is normal, and so is the reverse. */ + capability: { + announced: number; + acknowledged: number; + failed: number; + /** Retries of a declaration, counted apart from event retries for the same reason as the rest. */ + retried: number; + lastAcknowledgedAt: string | null; + }; }; } @@ -95,20 +154,41 @@ export interface CreateProtectionOptions { */ reportFirewallLog?: boolean; /** - * Report EVERY rule that fired — including one in `dry-run` that did not block — to the Pulse - * detections endpoint. Off unless explicitly `true`. + * Opt OUT of reporting every rule that fired — including one in `dry-run` that did not block — to the + * Pulse detections endpoint. + * + * Reporting is ON by default for a site enrolled with Patchstack that is running Patchstack-delivered + * rules and has a resolvable credential; it is off for a local install and for a guard running its own + * `rules`. This option can only switch it OFF: passing `true` cannot enable reporting for a site that is + * not enrolled, because whether a site is managed is Patchstack's answer and not a caller's to assert. + * `PATCHSTACK_REPORT_DETECTIONS=0` does the same thing from the environment. + * + * Why it exists: a rule that blocks nothing reports nothing, so a rule that is quietly wrong and a rule + * that is protecting look identical from the outside. + * + * What it sends on EVERY detection: the rule id and its revision, the request path, the query string's + * parameter NAMES, the method, the parameters the rule reads, the phase, whether it was enforced, the + * rule-bundle ETag, and a timestamp — plus the values of the parameters the matched rule names, under a + * capture plan derived from that rule. * - * Why it exists: a rule that blocks nothing reports nothing, so a rule that is quietly wrong and a - * rule that is protecting look identical from the outside. + * Two fields depend on the phase. A request or response detection also carries the user agent and the + * client address with its provenance. An egress detection carries neither: the call was the + * application's own, so there is no visitor to attribute it to, and both read `null`/`unavailable`. * - * What it sends, per detection: the rule id, the request PATH with the query string removed, the - * parameters the rule reads, the phase, whether it was enforced, the rule-bundle ETag, and a - * timestamp. It does NOT send the matched value, the request body, headers, or query-string values — - * this is a counting channel, not a copy of your traffic. + * A rule earns each captured value by naming what it reads. A rule reading the whole request (`raw`, + * `all`) permits nothing; response values are never captured; raw request bytes need an explicit, + * reviewed opt-in on the rule itself. Values are bounded in number and length, and what a bound left + * out is counted. The User-Agent is the one exception to the rule-scoped policy: it is part of the + * baseline and travels whether or not a rule names it, because a detection nobody can attribute is of + * little use. It does NOT send the value of any other parameter the matched rule does not name, any + * response value, or the query string's values as baseline metadata — `route` and `query_keys` + * describe a URL without disclosing what was in it, while a rule naming `egress.url` captures that URL + * as the rule read it. An egress detection has no user agent and no client address: the call was the + * application's own, so there is no visitor to attribute it to. * - * Off by default because switching it on adds an outbound request to every guard with a site UUID. - * Needs a resolvable API credential: the endpoint requires a verified, site-bound token, so with no - * credential no reporter is created and `detectionReporting` reads `unavailable-no-credential`. + * `AGENT-INSTALL.md` carries the full statement, and is the version to read before enabling this. + * + * `detectionReporting` names the state, including the reason when reporting is off. */ reportDetections?: boolean; /** How long to buffer detections before posting a batch. Default 5000ms. */ @@ -149,6 +229,38 @@ export interface CreateProtectionOptions { read(): unknown | Promise; write(envelope: unknown): unknown | Promise; }; + /** + * Declare which peers are this deployment's own reverse proxies, so a forwarded header can be believed. + * + * With no policy, the client address is whatever the transport observed — the socket peer on Node, and + * nothing at all in a runtime that exposes no peer, where the provenance reads `unavailable`. A + * forwarded header is never trusted implicitly: it is ordinary request input that any caller can send. + * + * A policy must say WHO is trusted, not just which header to read. Declare at least one of: + * + * - `peers` — CIDRs or bare addresses of your front end. An empty list means no peer is trusted, and + * one unparseable entry rejects the whole policy. + * - `hops` — the number of trusted proxies counting from the peer inward, as in the numeric form of + * Express's `trust proxy`. + * - `isTrusted` — a predicate over an address. + * + * `header` defaults to `x-forwarded-for`. The chain is read from the application side inward, stopping + * at the first address that is not trusted, because a proxy appends rather than replaces — so a value + * the caller prepended is ignored. + * + * Any unrecognised key, or any malformed value, rejects the policy rather than being ignored. There are + * no provider presets: a provider's name does not establish that the provider overwrote the header. + * + * Note that `req.ip` is never consulted on the Express path. Under `trust proxy` it is itself + * header-derived by a policy this guard has not verified, so an application behind a proxy sees the + * proxy's address until it declares a policy here. + */ + trustedProxy?: { + peers?: string[]; + hops?: number; + header?: string; + isTrusted?: (ip: string) => boolean; + }; /** Override the default response-phase (secret-leak) rule set. */ responseRules?: unknown[]; /** Override the default egress-phase (SSRF) rule set. */ @@ -175,7 +287,11 @@ export interface CreateProtectionOptions { message?: string; method?: string | null; path?: string | null; + /** The client address resolved for this request, or null when none could be established. */ ip?: string | null; + /** Where `ip` came from: the transport peer, a forwarded header a declared trusted proxy set, or + * nothing this guard can stand behind. `ip` is null when this is `unavailable`. */ + clientIpSource?: "runtime" | "trusted-proxy" | "unavailable"; userAgent?: string | null; }) => void; } diff --git a/src/protect/reporting-state.js b/src/protect/reporting-state.js new file mode 100644 index 00000000..0e7fc913 --- /dev/null +++ b/src/protect/reporting-state.js @@ -0,0 +1,136 @@ +/** + * Whether this guard reports security events, and if not, why not. + * + * Detection reporting is retained security-event evidence for sites the platform manages, not + * lightweight telemetry. So it turns on for an enrolled site with managed rules, and stays off + * everywhere else — a local install, a guard running its caller's own bundle, a site with no + * credential. + * + * The state is a single value with a reason built into it, because "no events arrived" has several + * causes that look identical from the platform: nothing matched, reporting was switched off, the site + * was never enrolled, or delivery is broken. + * + * Most of these travel: the state is declared on the rules fetch the guard already makes, so the platform + * learns it without an extra request and without waiting for a rule to fire. Two do not, and cannot: + * + * `not-enrolled` makes no site-addressed request at all, so there is nothing to carry it + * `unavailable-no-credential` cannot produce an authenticated request, and the declaration is withheld + * from an unauthenticated one because a claim about a site carries no + * weight without a verified token + * + * Both remain useful locally — `protect --check` reports them, and they are why a guard is silent. But the + * platform cannot infer them from this header, so server-side absence has to be modelled on its own terms + * (last seen, and how long ago) rather than treated as a state the guard reported. + */ + +/** + * Reporting states, and what each licenses. + * + * Only `on` reports. Every other value is a reason, and each is distinguishable at the platform so the + * dashboard can say which one it is. + */ +export const REPORTING_STATES = Object.freeze([ + /** Enrolled, managed rules, credential present, not opted out. Events are sent. */ + 'on', + /** `PATCHSTACK_REPORT_DETECTIONS=0`. An explicit, per-deployment opt-out. */ + 'disabled-by-config', + /** `PATCHSTACK_TELEMETRY=0`. The broader switch, which covers this along with everything else. */ + 'disabled-by-telemetry-opt-out', + /** No site identity: a local or unenrolled install. Nothing to report against. */ + 'not-enrolled', + /** + * A site identity, but the rules running are not the platform's — the caller's own bundle, or none. + * There is no managed rule document to attribute a detection to, so a report would name a rule id the + * platform never issued. + */ + 'no-managed-rules', + /** Enrolled with managed rules, but no credential resolved, so a report would be refused. */ + 'unavailable-no-credential', +]); + +/** + * Read an environment opt-out. + * + * Absent and empty both mean "not set" rather than "off": an unset variable is the default state, and a + * deployment that exports an empty value has not made a choice. + */ +function optedOut(value) { + if (value === undefined || value === null || value === '') return false; + + return /^(0|false|off|no)$/i.test(String(value)); +} + +/** + * Decide the reporting state. + * + * Pure, and separate from the runtime, so every combination can be enumerated in a test rather than + * reached by constructing a guard. The order of the checks is the meaning: an explicit opt-out outranks + * everything, because a deployment that switched reporting off should be told that is why — not that it + * lacks a credential it never needed. + * + * @param {{ + * siteUuid?: unknown, + * ruleOrigin?: 'api'|'cache'|'bundled'|'empty', + * hasCredential?: boolean, + * configOptOut?: boolean, + * env?: Record, + * }} input + * @returns {{ state: typeof REPORTING_STATES[number], reports: boolean }} + */ +export function reportingState(input) { + const env = input.env ?? (typeof process !== 'undefined' ? process.env : undefined) ?? {}; + + // Explicit opt-outs first, and reported distinctly. Collapsing them would tell an operator who set + // one variable to check the other. + // + // The programmatic flag is an opt-out ONLY. `reportDetections: false` switches reporting off; + // `true` cannot switch it on, because whether a site is managed is the platform's answer and not a + // caller's to assert. A guard that could self-declare managed status would report against rule ids + // the platform never issued. + if (input.configOptOut === true) return { state: 'disabled-by-config', reports: false }; + if (optedOut(env.PATCHSTACK_REPORT_DETECTIONS)) return { state: 'disabled-by-config', reports: false }; + if (optedOut(env.PATCHSTACK_TELEMETRY)) return { state: 'disabled-by-telemetry-opt-out', reports: false }; + + const siteUuid = input.siteUuid; + if (typeof siteUuid !== 'string' || siteUuid === '') return { state: 'not-enrolled', reports: false }; + + // The credential is checked BEFORE the managed-rules question, because a missing credential is what + // causes managed rules to be missing: the rules fetch is refused, resolution falls back to the + // caller's bundle or to nothing, and the origin is then `bundled` or `empty`. Asking about the origin + // first would report `no-managed-rules` for a site whose real and fixable problem is the credential, + // sending an operator to look for an enrolment that already exists. + if (input.hasCredential !== true) return { state: 'unavailable-no-credential', reports: false }; + + // `cache` counts as managed: the rules came from the platform, just not on this call. Excluding it + // would silence reporting for exactly the sites whose delivery is degraded — the ones whose evidence + // is most worth having. + const managed = input.ruleOrigin === 'api' || input.ruleOrigin === 'cache'; + if (!managed) return { state: 'no-managed-rules', reports: false }; + + return { state: 'on', reports: true }; +} + +/** + * A human-readable reason for `protect --check` and startup diagnostics. + * + * Every non-reporting state gets a sentence, because the state name alone is a label and an operator + * asking "why is nothing arriving" needs the answer, not the category. + */ +export function explainReportingState(state) { + switch (state) { + case 'on': + return 'Security events are reported to Patchstack for this site.'; + case 'disabled-by-config': + return 'Reporting is off because PATCHSTACK_REPORT_DETECTIONS is set to a false value.'; + case 'disabled-by-telemetry-opt-out': + return 'Reporting is off because PATCHSTACK_TELEMETRY is set to a false value, which covers all telemetry.'; + case 'not-enrolled': + return 'Reporting is off because this install has no site identity — it is not enrolled in Patchstack-managed mitigation.'; + case 'no-managed-rules': + return 'Reporting is off because the rules in force did not come from Patchstack, so a detection could not be attributed to a managed rule.'; + case 'unavailable-no-credential': + return 'Reporting is unavailable because no API credential resolved, so a report would be refused.'; + default: + return `Unrecognised reporting state: ${String(state)}.`; + } +} diff --git a/src/protect/rules/contract.js b/src/protect/rules/contract.js index 4098ae1a..cec9c830 100644 --- a/src/protect/rules/contract.js +++ b/src/protect/rules/contract.js @@ -12,7 +12,7 @@ // `rule-contract.json` is the published form. `tests/protect/rule-contract.test.ts` reads the engine's own // source and asserts these descriptions match what it implements. -export const CONTRACT_VERSION = '2.7'; +export const CONTRACT_VERSION = '2.8'; /** * Every parameter source, and what it accepts after the dot. @@ -354,8 +354,60 @@ export const RULE_PROPERTIES = Object.freeze([ 'message', 'enforcement', 'source_revision', 'prefilter', 'max_bytes', 'bypass_limit', 'set_headers', 'remove_headers', 'cookie_flags', 'ensure', + 'capture', ]); +/** + * The reviewed opt-in that lets one rule's evidence include raw request bytes. + * + * Versioned, because it authorises collection: a guard that met a `capture` it did not understand and + * guessed would be guessing about what may be gathered from an application. An unrecognised version + * grants nothing, which is the only safe direction — a newer server can then extend this without an + * older guard quietly capturing under rules it cannot read. + * + * `raw_chars` is a request for a bounded PREFIX of the body, never the whole of it, and the runtime caps + * it regardless of what is asked for. The property says a reviewer agreed raw bytes are needed for this + * rule; it does not let the rule set its own bounds. + */ +export const CAPTURE_VERSION = 1; +export const CAPTURE_KEYS = Object.freeze(['version', 'raw_chars']); +/** The most raw characters any opt-in can obtain, whatever it asks for. Published, so it is agreed. */ +export const CAPTURE_RAW_CHARS_MAX = 512; + +/** + * Properties whose null does NOT refuse the rule. + * + * A null is normally a value that was meant to be something and is not, and refusing it stops a rule + * running on a default nobody chose. These are the exception because they authorise collection rather + * than protection: `capture: null` says "collect nothing", which is already the default and costs no + * shielding. + */ +export const NULL_EXEMPT_PROPERTIES = Object.freeze(['capture']); + +/** + * @returns {string|null} why a `capture` opt-in cannot be honoured as written, or null + */ +export function captureProblem(capture) { + if (capture === undefined || capture === null) return null; + if (typeof capture !== 'object' || Array.isArray(capture)) return 'capture must be an object'; + + for (const key of Object.keys(capture)) { + if (!CAPTURE_KEYS.includes(key)) return `capture has no "${key}" — it takes ${CAPTURE_KEYS.join(', ')}`; + } + if (!Object.hasOwn(capture, 'version')) return 'capture must name the version it was written against'; + if (capture.version !== CAPTURE_VERSION) { + return `capture version ${String(capture.version)} is not one this guard understands`; + } + if (!Object.hasOwn(capture, 'raw_chars')) return 'capture must say what it is opting into'; + + const chars = capture.raw_chars; + if (typeof chars !== 'number' || !Number.isInteger(chars) || chars <= 0) { + return 'capture.raw_chars must be a positive whole number of characters'; + } + + return null; +} + /** * What each action needs, what it defaults, and which phases can carry it out. * @@ -447,6 +499,10 @@ export function parameterProblem(parameter) { if (Array.isArray(parameter)) { if (parameter.length === 0) return 'parameter list is empty'; for (const member of parameter) { + // One level, because the engine expands one level: it resolves each member of the list, and a + // member that is itself a list resolves to nothing. Accepting it would validate a rule that names + // parameters and matches on none of them. + if (Array.isArray(member)) return 'a parameter list holds parameters, not more lists'; const problem = parameterProblem(member); if (problem !== null) return problem; } @@ -598,6 +654,7 @@ export function nullPropertyProblem(rule) { if (!rule || typeof rule !== 'object') return null; for (const property of RULE_PROPERTIES) { + if (NULL_EXEMPT_PROPERTIES.includes(property)) continue; if (property in rule && rule[property] === null) { return `"${property}" is present but null; omit it to mean the default, since a null is a value that ` + 'was meant to be something and is not'; @@ -619,6 +676,12 @@ export function rulePropertyProblem(rule) { if (problem) return `"${property}" ${problem}`; } + // `capture` is deliberately NOT checked here. It authorises collection, and a rule is a mitigation: + // letting a capture value the guard cannot read decide whether the rule runs would turn a question + // about evidence into the loss of the protection itself — a newer server adding a capture version + // would switch off shielding on every older guard. An unreadable capture grants no capture; the rule + // still applies. `captureProblem` is where that is decided, and the plan is its only caller. + // // `enforcement` is deliberately NOT checked here, and this is the one place in the contract where a // value the runtime does not recognise is left alone on purpose. // @@ -697,9 +760,30 @@ export function ruleContract() { ), when_keys: [...WHEN_KEYS], null_valued_properties: NULL_VALUED_PROPERTIES, + // Except these. They authorise collection rather than protection, so a malformed one costs evidence + // and never the mitigation — a consumer deriving the general rule would otherwise refuse a document + // this guard runs. + null_exempt_properties: [...NULL_EXEMPT_PROPERTIES], rule_property_shapes: { ...RULE_PROPERTY_SHAPES }, enforcement_values: [...ENFORCEMENT_VALUES], rule_properties: [...RULE_PROPERTIES], + capture: { + version: CAPTURE_VERSION, + required: ['version', 'raw_chars'], + additional_properties: false, + properties: { + version: { type: 'integer', const: CAPTURE_VERSION }, + raw_chars: { type: 'integer', minimum: 1 }, + }, + // The runtime's ceiling, published apart from the schema on purpose. A schema maximum would have a + // consumer refuse a rule this guard accepts and runs: asking for more than the ceiling is not an + // authoring error, it is a request the runtime answers with the ceiling. + raw_chars_effective_maximum: CAPTURE_RAW_CHARS_MAX, + // What a guard does with a capture it cannot read, stated rather than left to be discovered. + unknown_version: 'grants no capture; the rule still applies', + unreadable: 'grants no capture; the rule still applies', + raw_chars_note: 'a request for a bounded PREFIX of the body; more than the effective maximum yields the maximum', + }, limits: { ...LIMITS }, }; } diff --git a/src/protect/rules/source.js b/src/protect/rules/source.js index 3314dfa8..3304ec9e 100644 --- a/src/protect/rules/source.js +++ b/src/protect/rules/source.js @@ -42,8 +42,31 @@ function reportRejections(rejected, options, label) { } /** A bundle plus the outcome of the attempt that produced it. `source` is never written to the store. */ -function fromSource(bundle, reason) { - return reason === undefined ? { ...bundle, source: { ok: true } } : { ...bundle, source: { ok: false, reason } }; +/** + * Wrap a resolved bundle with where it came from and whether the resolution was clean. + * + * `origin` is separate from `ok` because they answer different questions and a caller needs both. + * `ok: false` says the resolution hit a problem; `origin` says which leg actually supplied the rules + * that are now running: + * + * `api` delivered by the platform on this call + * `cache` last-known-good from the store — still platform-delivered, just not on this call + * `bundled` the caller's own `rules` option, which the platform never saw + * `empty` nothing at all + * + * Detection reporting depends on this distinction. Reporting is for sites the platform manages, so a + * guard running bundled or empty rules has nothing to report against: the platform has no rule document + * to attribute a hit to, and a detection naming a rule id it never issued is not evidence of anything. + * + * @param {object} bundle + * @param {'api'|'cache'|'bundled'|'empty'} origin + * @param {string} [reason] + */ +function fromSource(bundle, origin, reason) { + return { + ...bundle, + source: reason === undefined ? { ok: true, origin } : { ok: false, origin, reason }, + }; } export async function resolveRules(options, store, ctx = {}) { @@ -53,68 +76,68 @@ export async function resolveRules(options, store, ctx = {}) { const timeoutMs = ctx.timeoutMs; if (options.siteUuid) { const prior = await store.read(); // { bundle, etag } | null - const client = new PulseRuleClient({ siteUuid: options.siteUuid, baseUrl: options.pulseRulesUrl, etag: prior?.etag, timeoutMs, pulseAuth: ctx.pulseAuth, reportsDetections: options.reportDetections === true }); + const client = new PulseRuleClient({ siteUuid: options.siteUuid, baseUrl: options.pulseRulesUrl, etag: prior?.etag, timeoutMs, pulseAuth: ctx.pulseAuth, detectionState: ctx.detectionState }); const res = await client.getRules(); - if (res.success && res.notModified && prior?.bundle) return fromSource(normalizeBundle(prior.bundle, options)); + if (res.success && res.notModified && prior?.bundle) return fromSource(normalizeBundle(prior.bundle, options), 'cache'); if (res.success && !res.notModified) { const rejected = liveUpdateRejections(res, options); if (rejected.length > 0) { reportRejections(rejected, options, 'rule update rejected'); // Reached the source and refused what it sent. Not ok: the running rules are not the delivered // ones, and asking again at the normal interval re-downloads the same rejected bundle. - if (prior?.bundle) return fromSource(normalizeBundle(prior.bundle, options), 'update rejected'); - if (options.rules) return fromSource(normalizeBundle(options.rules, options), 'update rejected'); - return fromSource(emptyBundle(), 'update rejected'); + if (prior?.bundle) return fromSource(normalizeBundle(prior.bundle, options), 'cache', 'update rejected'); + if (options.rules) return fromSource(normalizeBundle(options.rules, options), 'bundled', 'update rejected'); + return fromSource(emptyBundle(), 'empty', 'update rejected'); } const bundle = normalizeBundle(res, options); await store.write({ bundle, etag: res.etag ?? null }); - return fromSource(bundle); + return fromSource(bundle, 'api'); } if (prior?.bundle) { notify(options.onError, new Error(`pulse rule fetch failed (${res.error ?? 'no usable response'}); using cached bundle`), 'onError'); - return fromSource(normalizeBundle(prior.bundle, options), res.error ?? 'no usable response'); + return fromSource(normalizeBundle(prior.bundle, options), 'cache', res.error ?? 'no usable response'); } if (options.rules) { notify(options.onError, new Error(`pulse rule fetch failed (${res.error ?? 'no usable response'}); using bundled fallback`), 'onError'); - return fromSource(normalizeBundle(options.rules, options), res.error ?? 'no usable response'); + return fromSource(normalizeBundle(options.rules, options), 'bundled', res.error ?? 'no usable response'); } notify(options.onError, new Error(`pulse rule fetch failed (${res.error ?? 'no usable response'}); no cache — running with no rules`), 'onError'); - return fromSource(emptyBundle(), res.error ?? 'no usable response'); + return fromSource(emptyBundle(), 'empty', res.error ?? 'no usable response'); } if (options.token) { const prior = await store.read(); const client = new PatchstackRuleClient({ token: options.token, baseUrl: options.baseUrl, etag: prior?.etag, timeoutMs }); const res = await client.getRules(); - if (res.success && res.notModified && prior?.bundle) return fromSource(normalizeBundle(prior.bundle, options)); + if (res.success && res.notModified && prior?.bundle) return fromSource(normalizeBundle(prior.bundle, options), 'cache'); if (res.success && !res.notModified) { const rejected = liveUpdateRejections(res, options); if (rejected.length > 0) { reportRejections(rejected, options, 'rule update rejected'); // Reached the source and refused what it sent. Not ok: the running rules are not the delivered // ones, and asking again at the normal interval re-downloads the same rejected bundle. - if (prior?.bundle) return fromSource(normalizeBundle(prior.bundle, options), 'update rejected'); - if (options.rules) return fromSource(normalizeBundle(options.rules, options), 'update rejected'); - return fromSource(emptyBundle(), 'update rejected'); + if (prior?.bundle) return fromSource(normalizeBundle(prior.bundle, options), 'cache', 'update rejected'); + if (options.rules) return fromSource(normalizeBundle(options.rules, options), 'bundled', 'update rejected'); + return fromSource(emptyBundle(), 'empty', 'update rejected'); } const bundle = normalizeBundle(res, options); await store.write({ bundle, etag: res.etag ?? null }); - return fromSource(bundle); + return fromSource(bundle, 'api'); } if (prior?.bundle) { notify(options.onError, new Error(`rule fetch failed (${res.error ?? 'no usable response'}); using cached bundle`), 'onError'); - return fromSource(normalizeBundle(prior.bundle, options), res.error ?? 'no usable response'); + return fromSource(normalizeBundle(prior.bundle, options), 'cache', res.error ?? 'no usable response'); } notify(options.onError, new Error(`rule fetch failed (${res.error ?? 'no usable response'}); no cache — running with no rules`), 'onError'); - return fromSource(emptyBundle(), res.error ?? 'no usable response'); + return fromSource(emptyBundle(), 'empty', res.error ?? 'no usable response'); } // No live source configured, so the bundle IS the source and cannot be behind one. if (options.rules) { - return fromSource(normalizeBundle(options.rules, options)); + return fromSource(normalizeBundle(options.rules, options), 'bundled'); } - return fromSource(emptyBundle()); + return fromSource(emptyBundle(), 'empty'); } // Every rule path (live fetch, cache, bundled fallback) funnels through here, so this is where the diff --git a/src/protect/rules/validate.js b/src/protect/rules/validate.js index 0aeaa68e..47d7509c 100644 --- a/src/protect/rules/validate.js +++ b/src/protect/rules/validate.js @@ -51,7 +51,7 @@ export function validateBundle(bundle, opts = {}) { rejected.push({ id: idOf(rule), reason: `bundle exceeds maxRules (${LIMITS.maxRules})` }); continue; } - const reason = ruleProblem(rule); + const reason = enforceableRuleProblem(rule); if (reason) rejected.push({ id: idOf(rule), reason }); else firewall.push(rule); } @@ -85,8 +85,16 @@ function idOf(rule) { return id === undefined || id === null ? '(unidentified)' : String(id); } -/** @returns {string|null} a reason the rule must be dropped, or null when it's acceptable. */ -function ruleProblem(rule) { +/** + * Why a rule would not be run, or null when it would. + * + * Exported because a capture permission depends on it: a permission exists to explain a detection, and a + * rule this refuses produces none. Capture validity is deliberately not part of the answer — it governs + * collection, never protection. + * + * @returns {string|null} a reason the rule must be dropped, or null when it's acceptable. + */ +export function enforceableRuleProblem(rule) { if (!rule || typeof rule !== 'object') return 'not an object'; // Before anything reads a property: a property that is PRESENT and null is not an omission. Every layer diff --git a/src/protect/runtime.js b/src/protect/runtime.js index 5a14a6ea..7cc9a36a 100644 --- a/src/protect/runtime.js +++ b/src/protect/runtime.js @@ -15,8 +15,11 @@ // Runtime guards: .express(), .node(), .fetch(handler) / .fetchGuard() — same policy, // every runtime an AI builder deploys to. // Vendored node-waf engine (this package is self-contained — no @patchstack/node-waf dep). +import { resolveClientIp } from './client-ip.js'; import { RuleEngine } from './engine/index.js'; import { matchValue, walkLeaves, safeRegExp, jwtClaimSpans } from './engine/engine.js'; +import { requestField } from './engine/normalizer.js'; +import { captureValues, createPlanCache, permitsAnything } from './capture-plan.js'; import { PulseRuleClient } from './engine/pulse-client.js'; import { fromFetchRequest } from './engine/fetch.js'; import { fromNodeRequest } from './engine/node.js'; @@ -29,6 +32,7 @@ import { makeStore } from './rules/store.js'; import { resolveRules } from './rules/source.js'; import { startRefresh, makeRefreshHandler } from './rules/refresh.js'; import { createDetectionReporter } from './detections.js'; +import { reportingState } from './reporting-state.js'; import { notify } from './notify.js'; import { createFirewallLogReporter, resolveApiBase, telemetryEnabled } from './firewall-log.js'; @@ -94,8 +98,29 @@ export async function createProtection(options = {}) { // Minimal payload by design; see `detections.js`. let detections = null; + /** + * The rule as a callback sees it: its identity, and nothing that carries policy. + * + * `onDetect` documents `rule` as `{ id, category }`, and that is what this returns. The rule the engine + * matches with IS the policy in force, and enforcement lives in its nested parts — `rule_v2`, `when`, + * and the match and action objects inside them — so handing out the object, or any copy that still + * shares them, lets a callback change what every later request through this guard is screened for. + * + * Projected rather than cloned because this runs on every detection, which an attacker can drive. A + * deep clone would allocate the whole rule per detection to hand back fields the contract does not + * promise. The internal reporters keep reading the real rule; only the callback's view is narrowed. + */ + const ruleIdentity = (rule) => ({ id: rule?.id, category: rule?.category }); + const onDetect = (detection) => { - notify(userOnDetect, detection, 'onDetect'); + // What the platform is told is the engine's account of the request, and a host callback cannot change + // it. A synchronous callback that replaced `ip`, `clientIpSource`, `rule` or `path` would otherwise + // decide what the platform is told a rule matched and who it matched — silently, and + // indistinguishably from a correct report. + // + // The callback gets its own object, so what it writes reaches nothing else; each internal reporter + // builds its own record from the detection itself. Reading before the callback runs is not what + // carries this — it is defence in depth for the case where the copy is later weakened. if (detections) detections.record(detection); if (firewallLog && detection?.mode === 'block') { firewallLog.record({ @@ -106,6 +131,16 @@ export async function createProtection(options = {}) { userAgent: detection.userAgent, }); } + + // Without `capture`: the documented callback carries the rule's identity and the request's own + // metadata, and a host already holds the request these values came from. Widening it to forward + // evidence is a decision about the callback contract, not a side effect of collecting any. + const { capture: _evidence, ...forCallback } = detection ?? {}; + notify( + userOnDetect, + { ...forCallback, ...(detection?.rule ? { rule: ruleIdentity(detection.rule) } : {}) }, + 'onDetect', + ); }; // One tiered store (memory → filesystem/pluggable) shared by the initial load and every refresh. @@ -141,42 +176,121 @@ export async function createProtection(options = {}) { notify(onError, new Error(message), 'onError'); console.warn(message); } - const bundle = await resolveRules(options, store, { timeoutMs: bootTimeoutMs, pulseAuth }); - // OPT-IN, deliberately. Two reasons, and the first is not about privacy: switching it on adds an - // outbound POST to every guard that has a site UUID, which is a change in what an installed app does - // on the network — the kind of thing that must be disclosed in the shipped docs before it is a default, - // not after. The second is that the default belongs to whoever owns that disclosure, so the capability - // lands here and the flip is a separate, deliberate change. + // The state sent on the fetch, computed from what is knowable before it: the store says whether the + // platform has ever delivered rules here, which is the honest origin at the moment of asking. The + // resolved state is recomputed from the actual origin immediately after, and every later fetch carries + // whatever the guard is in by then. + const cachedOrigin = async () => { + const prior = await store.read(); + if (prior?.bundle) return 'cache'; + + return options.rules ? 'bundled' : 'empty'; + }; + const stateFor = (origin) => + reportingState({ + siteUuid: options.siteUuid, + ruleOrigin: origin, + hasCredential: Boolean(pulseAuth), + configOptOut: options.reportDetections === false, + }); + + // Read once and reused below, so the state reported on the fetch and the state compared against the + // settled one are the same value rather than two reads of a store the fetch has since written to. + const preFetchState = stateFor(await cachedOrigin()).state; + const bundle = await resolveRules(options, store, { + timeoutMs: bootTimeoutMs, + pulseAuth, + detectionState: preFetchState, + }); + // ON by default for an enrolled site running Patchstack-delivered rules, and off otherwise — a local + // install and a guard running its own bundle send nothing. That default is a change in what an + // installed app does on the network, so it is disclosed in `AGENT-INSTALL.md` and in the option + // documentation rather than being inferred from behaviour. // - // And it needs a credential. The detections endpoint is site-addressed and site-bound-token-only, so a + // It needs a credential. The detections endpoint is site-addressed and site-bound-token-only, so a // reporter built without one queues events, posts them, and is refused — spending an outbound request // per batch to accomplish nothing, while `reportDetections: true` in the config says reporting is on. // Refusing to build it is the honest outcome; `protection.detectionReporting` says which it is. - let detectionReporting = 'off'; - if (options.reportDetections === true && options.siteUuid && telemetryEnabled()) { - if (!pulseAuth) { - detectionReporting = 'unavailable-no-credential'; - const message = - 'Patchstack: detection reporting is enabled for site ' + - options.siteUuid + - ' but no API credential resolved, so no report could be delivered. Reporting is off.'; - notify(onError, new Error(message), 'onError'); - console.warn(message); - } else { - detectionReporting = 'on'; + // + // Derived from enrolment rather than from a config flag: reporting is on for a site the platform + // manages, and off everywhere else. `reportingState` holds the whole decision so every combination is + // enumerable in a test instead of reachable only by constructing a guard. + let detectionReporting = 'not-enrolled'; + /** + * The origin of the rules in force. + * + * Held separately from the reported state so a request can carry a state derived FRESH from it. Sending + * the previously reported state would mean an opt-out that appeared under a running guard was not + * reported on the next request — only on the one after it. + */ + let currentOrigin; + + /** + * Bring reporting into line with the rules now in force. + * + * Called at boot and after every refresh, because the inputs are not all boot-time facts: a guard that + * started on a failed fetch and fell back to its cached or bundled rules can receive platform rules on + * a later refresh, and an opt-out can appear in the environment under a running process. A state fixed + * at startup leaves the first case silent for the life of the process. + * + * The credential is resolved once at boot, so losing it mid-process does not change the state. + * + * Starts and stops the reporter accordingly. Stopping flushes what it holds — the events already + * collected were collected while reporting was on, and dropping them would lose evidence rather than + * decline to gather it. + */ + const applyReportingState = async (origin) => { + currentOrigin = origin; + const next = stateFor(origin); + const changed = next.state !== detectionReporting; + detectionReporting = next.state; + + if (next.reports && !detections) { detections = createDetectionReporter({ siteUuid: options.siteUuid, baseUrl: options.pulseRulesUrl, pulseAuth, - // The bundle the guard is actually running, so a hit can be attributed to the rules that produced - // it rather than to whatever is current when the report is read. Kept current across refreshes — - // see the refresh tick below. + // The bundle the guard is actually running, so a hit can be attributed to the rules that + // produced it rather than to whatever is current when the report is read. rulesEtag: (await store.read())?.etag ?? null, fetchImpl: options.fetchImpl, flushMs: options.detectionFlushMs, }); + } else if (!next.reports && detections) { + detections.stop(); + detections = undefined; } - } + + if (changed && next.state === 'unavailable-no-credential') { + const message = + 'Patchstack: this site is enrolled and running managed rules, but no API credential resolved, ' + + 'so no security event could be delivered. Reporting is off.'; + notify(onError, new Error(message), 'onError'); + console.warn(message); + } + + return next; + }; + + /** + * Apply the settled state, and correct the platform if the request that just went out declared another. + * + * Both the boot fetch and every refresh declare a state BEFORE resolution decides where the rules came + * from, so either can settle somewhere else. This is the single place that reconciles the two, so the + * two paths cannot drift apart: a guard whose only refresh is a one-shot manual call would otherwise + * leave the platform holding the pre-resolution answer for the life of the process. + * + * @param {'api'|'cache'|'bundled'|'empty'|undefined} origin + * @param {string} declared the state carried by the request that produced `origin` + */ + const applyAndAcknowledge = async (origin, declared) => { + const settled = await applyReportingState(origin); + if (settled.state !== declared && detections) detections.announce(settled.state); + + return settled; + }; + + await applyAndAcknowledge(bundle.source?.origin, preFetchState); // Mode is mutable so a Pulse refresh can flip dry-run ↔ block when SaaS enables production. // Precedence: PATCHSTACK_MODE env (local override) > API enforcement > options.mode > dry-run. let mode = resolveMode(options, bundle); @@ -259,6 +373,39 @@ export async function createProtection(options = {}) { // before, so an older server that never sends the field behaves identically. const ruleMode = (rule) => (rule?.enforcement === 'dry-run' ? 'dry-run' : mode); + // Plans derived once per rule, and only ever consulted where there is somewhere for evidence to go. + const planCache = createPlanCache(); + + /** + * The evidence for one match, taken here and nowhere else. + * + * The resolver stays inside this function. What leaves is the bounded result — a fixed number of + * values, each of fixed length, and counts of what did not fit. Handing the resolver onward instead + * would put the whole request within reach of every consumer of a detection, which is the opposite of + * a plan that names what may be read. + * + * Nothing is derived when reporting is off. That is a cost decision rather than a safeguard — with no + * reporter there is no event for evidence to travel on, and the block log reads named fields only — so + * skipping the work changes what an app spends, not what leaves it. + */ + const evidenceFrom = (result) => { + if (!detections || !result?.rule) return undefined; + let entry; + try { + entry = planCache.for(result.rule); + } catch (err) { + notify(onError, err, 'onError'); + + return undefined; + } + + // The reference travels even when the plan permits nothing, because "this rule was allowed to show + // you nothing" and "this rule showed you nothing" are different facts, and only the first is policy. + if (!permitsAnything(entry.plan)) return { plan: entry.reference }; + + return { plan: entry.reference, ...captureValues(entry.plan, result.resolver) }; + }; + const decide = (phase, result, block, allow, ctx = {}) => { if (!result || !result.blocked) return allow(); const effectiveMode = ruleMode(result.rule); @@ -273,7 +420,12 @@ export async function createProtection(options = {}) { method: ctx.method, path: ctx.path, ip: ctx.ip, + // Provenance travels with the address. Without it a consumer cannot tell an observed peer from a + // value read out of a forwarded header, and `null` from "there was no address to establish". + clientIpSource: ctx.clientIpSource, userAgent: ctx.userAgent, + // Derived from the reading this decision was made on, before that reading goes out of scope. + capture: evidenceFrom(result), }); return effectiveMode === 'block' ? block() : allow(); }; @@ -309,7 +461,21 @@ export async function createProtection(options = {}) { // dry-run must not redact or withhold a body either: "detect until justified" is meaningless if the // rule still rewrites what the user sees. const responseMode = ruleMode(rule); - onDetect({ phase: 'response', mode: responseMode, category: rule.category, rule, message: result.message }); + // The same request metadata a request-phase detection carries, taken from the originating request's + // own resolution. Omitting it left a response detection with no client at all, so a reviewer could + // not tell which request produced it. + onDetect({ + phase: 'response', + mode: responseMode, + category: rule.category, + rule, + message: result.message, + ...requestMetaFromContext(reqCtx), + // A response detection names its capture policy like any other. Response sources are never + // capturable, so a response-only rule reports a plan that permitted nothing — which is the + // policy, and a different fact from a rule that found nothing. + capture: evidenceFrom(result), + }); if (responseMode !== 'block') continue; // dry-run: observe only if (redactors && redactors.length) { // Span redactors on a mutation-decoded rule can't map back to the raw body → fail closed. @@ -357,19 +523,122 @@ export async function createProtection(options = {}) { // Minimal request context for the response phase: what a response rule's `when` scope and any // request-header reference (Host/Origin) need — method, path, and request headers. No body. - const reqContextFromFetch = (request) => { + /** + * Present an Express request to the engine with the resolved address, without touching the original. + * + * Every field the engine reads is materialised as an OWN property. The engine normalises with + * `{ ...req, ...normalizeRequest(req) }`, and a spread copies own enumerable properties only — so a + * prototype-linked view would arrive carrying just the two properties added here, and a rule scoped to + * a method, or reading an uploaded file or a parsed cookie, would silently find nothing. + * + * The application's own request object is left alone: `req.ip` on Express is an accessor the framework + * defines from its own `trust proxy` setting, and overwriting it would change what the application sees. + */ + const shapeExpressRequest = (req) => { + const client = resolveClientIp({ + peer: req?.socket?.remoteAddress, + headers: req?.headers ?? {}, + trustedProxy: options.trustedProxy, + }); + + return { + client, + shaped: { + // Own copies of everything a rule can address. Listed rather than spread, so a field the engine + // gains has to be added here deliberately instead of appearing to work by accident. + // Through the same gate as the engine's own normalisation: this projection turns whatever it reads + // into an OWN property, so reading a polluted prototype here would launder it into evidence that + // no later own-property check could tell from the real thing. + method: requestField(req, 'method'), + url: requestField(req, 'url'), + originalUrl: requestField(req, 'originalUrl'), + headers: requestField(req, 'headers'), + query: requestField(req, 'query'), + body: requestField(req, 'body'), + files: requestField(req, 'files'), + cookies: requestField(req, 'cookies'), + socket: requestField(req, 'socket'), + // The verbatim body, when a caller kept one. A `raw` rule reads it directly, and the engine + // otherwise reconstructs raw by re-serialising the parsed body — which cannot carry what parsing + // did not keep: a body that failed to parse at all, a duplicate key where only the last value + // survives, or the exact bytes a signature was written against. + // + // Own property only. Evidence is what this request actually carried: a value reachable through a + // polluted prototype is not, and materialising it here would turn it into evidence indistinguish- + // able from real bytes — firing every raw rule that matches it. The adapters that capture real + // bytes define `_rawBody` on the request object itself. + ...(Object.hasOwn(req ?? {}, '_rawBody') ? { _rawBody: req._rawBody } : {}), + // The resolved address, and the resolution itself for the consumers downstream. + ip: client.ip ?? '', + _clientIp: client, + }, + }; + }; + + /** + * Screen a fetch request once, and hand back both the decision and the address it resolved. + * + * Shared by `fetchGuard()` and `fetch(handler)` so the response phase can reuse the request phase's + * resolution instead of making its own. + */ + const screenFetchRequest = async (request) => { + let result; + let shaped; + try { + shaped = await fromFetchRequest(request, { trustedProxy: options.trustedProxy }); + result = engine.evaluate(shaped); + } catch (err) { + notify(onError, err, 'onError'); + + return { blocked: null, client: undefined }; // fail open + } + const blocked = decide( + 'request', + result, + () => blockResponse(result, request), + () => null, + requestMeta(shaped, request), + ); + + return { blocked, client: shaped?._clientIp }; + }; + + const reqContextFromFetch = (request, client) => { try { const u = new URL(request.url); const headers = headerObject(request.headers); // A fetch Request doesn't expose the Host header (it's set at send time), so derive it from the // URL — response rules that compare origins (open-redirect / CORS) need the request Host. if (!headers.host) headers.host = u.host; - return { method: request.method, originalUrl: u.pathname + u.search, headers }; + // `client` is the resolution the request phase already made for this request, when there was one. + // Resolving again here could disagree with it, and a response detection naming a different address + // than the request detection for the same request describes two clients that do not exist. + const resolved = client ?? { ip: null, source: 'unavailable' }; + + return { + method: request.method, + originalUrl: u.pathname + u.search, + headers, + ip: resolved.ip ?? '', + _clientIp: resolved, + }; } catch { return undefined; } }; - const reqContextFromNode = (req) => (req ? { method: req.method, originalUrl: req.url, headers: req.headers || {} } : undefined); + // The response phase evaluates against the ORIGINATING request, so the resolution travels with it: a + // response rule reading `server.ip`, and a response detection's record, get the same address the + // request phase used. + const reqContextFromNode = (req, client) => + req + ? { + method: req.method, + originalUrl: req.url, + headers: req.headers || {}, + ip: client?.ip ?? '', + _clientIp: client ?? { ip: null, source: 'unavailable' }, + } + : undefined; // Screen a fetch Response (used by .fetch() and — via protection.screenResponse — the Supabase guard). const screenResp = async (response, reqCtx) => { @@ -490,7 +759,28 @@ export async function createProtection(options = {}) { // Same for egress: a dry-run rule records the outbound attempt without preventing it. Blocking a // request the app makes is at least as disruptive as blocking one it receives. const egressMode = ruleMode(result.rule); - onDetect({ phase: 'egress', mode: egressMode, category: result.rule?.category, rule: result.rule, message: result.message }); + // The outbound request's own method and path. An egress detection has no client address and no user + // agent by nature: the call is the application's, not a visitor's, so there is nobody to attribute it + // to. Its destination host reaches the report through capture, for a rule that names `egress.url` or + // `egress.host` — which the internal-host rules do. + let egressPath = null; + try { + const u = new URL(url); + egressPath = u.pathname + u.search; + } catch { + egressPath = typeof url === 'string' ? url : null; + } + + onDetect({ + phase: 'egress', + mode: egressMode, + category: result.rule?.category, + rule: result.rule, + message: result.message, + method: typeof method === 'string' ? method : null, + path: egressPath, + capture: evidenceFrom(result), + }); return egressMode === 'block'; }; @@ -514,30 +804,36 @@ export async function createProtection(options = {}) { // Screen a fetch Response through the response-phase rules (redact/block). Used by // .fetch(), and by the Supabase guard on its forwarded upstream response. - screenResponse: (response, request) => screenResp(response, request ? reqContextFromFetch(request) : undefined), + // A standalone response screen with no request phase of its own — the Supabase guard's forwarded + // upstream response. It resolves once here, which is the only resolution for this call. + screenResponse: (response, request) => + screenResp( + response, + request + ? reqContextFromFetch( + request, + resolveClientIp({ headers: headerObject(request.headers), trustedProxy: options.trustedProxy }), + ) + : undefined, + ), // (request) => Response | null (null = allow, caller proceeds). Request phase only. fetchGuard() { - return async (request) => { - let result; - try { - result = engine.evaluate(await fromFetchRequest(request)); - } catch (err) { - notify(onError, err, 'onError'); - return null; // fail open - } - return decide('request', result, () => blockResponse(result, request), () => null, fetchRequestMeta(request)); - }; + return async (request) => (await screenFetchRequest(request)).blocked; }, // Wrap a fetch handler: screens the request, then the response (redact/block). fetch(handler) { - const guard = protection.fetchGuard(); return async (request, ...rest) => { - const blocked = await guard(request); + // The request phase's own resolution is carried into the response phase rather than the response + // screening making a second one. Two resolutions for one request can disagree, and a response + // detection naming a different address than the request detection describes two clients that do + // not exist. + const { blocked, client } = await screenFetchRequest(request); if (blocked) return blocked; const response = await handler(request, ...rest); - return screenResp(response, reqContextFromFetch(request)); + + return screenResp(response, reqContextFromFetch(request, client)); }; }, @@ -546,11 +842,15 @@ export async function createProtection(options = {}) { express(exprOptions = {}) { return (req, res, next) => { let result; + // Resolved once, before evaluation, and reused by the engine, the response screening and the + // block record below. Three consumers deriving it separately could attribute one request to + // three different addresses. + const { shaped, client } = shapeExpressRequest(req); try { - result = engine.evaluate(req); + result = engine.evaluate(shaped); } catch (err) { notify(onError, err, 'onError'); - if (exprOptions.screenResponses) wrapNodeResponse(res, reqContextFromNode(req)); + if (exprOptions.screenResponses) wrapNodeResponse(res, reqContextFromNode(req, client)); return next(); } decide( @@ -564,10 +864,10 @@ export async function createProtection(options = {}) { } }, () => { - if (exprOptions.screenResponses) wrapNodeResponse(res, reqContextFromNode(req)); + if (exprOptions.screenResponses) wrapNodeResponse(res, reqContextFromNode(req, client)); next(); }, - nodeRequestMeta(req), + nodeRequestMeta(req, client), ); }; }, @@ -616,7 +916,7 @@ export async function createProtection(options = {}) { let shaped; let result; try { - shaped = fromNodeRequest(req, rawBody); + shaped = fromNodeRequest(req, rawBody, { trustedProxy: options.trustedProxy }); if (parsedBody !== undefined && parsedBody !== null) shaped.body = parsedBody; result = engine.evaluate(shaped); } catch (err) { @@ -640,10 +940,11 @@ export async function createProtection(options = {}) { // This guard consumed the request stream to screen it; re-expose the parsed // body so a downstream handler (without its own body-parser) can read it. if (req.body === undefined) req.body = shaped.body; - if (nodeOptions.screenResponses) wrapNodeResponse(res, reqContextFromNode(req)); + // The resolution the shaping already made, carried into the response phase and the record. + if (nodeOptions.screenResponses) wrapNodeResponse(res, reqContextFromNode(req, shaped?._clientIp)); next(); }, - nodeRequestMeta(req), + nodeRequestMeta(req, shaped?._clientIp), ); } }, @@ -694,9 +995,22 @@ export async function createProtection(options = {}) { notify(onError, err, 'onError'); // a failed report must not stop the rule refresh } } - const next = await resolveRules(options, store, { timeoutMs: options.refreshTimeoutMs, pulseAuth }); + // Derived from the origin in force, re-reading the environment, rather than resent from the last + // reported value: an opt-out that appeared since the previous request has to travel on THIS one. Held + // in a binding because the acknowledgement below compares against exactly what this request carried. + const declaredState = stateFor(currentOrigin).state; + const next = await resolveRules(options, store, { + timeoutMs: options.refreshTimeoutMs, + pulseAuth, + detectionState: declaredState, + }); mode = resolveMode(options, next); applyBundle(next); + // Recomputed from the origin this refresh resolved. Within one process the reachable changes are a + // rules source that STARTS being the platform's, and an opt-out appearing in the environment. It + // cannot stop being the platform's: once a bundle has been accepted the memory tier holds it, so a + // later failed fetch still resolves to `cache`. The credential is resolved once at boot. + await applyAndAcknowledge(next.source?.origin, declaredState); // After the swap, and only after it: later detections belong to the bundle now running. A refresh // that fell back to the cached or bundled ruleset kept the previous rules, and `store.read()` then // still holds the previous identity — which is exactly the answer that stays true. @@ -726,18 +1040,37 @@ export async function createProtection(options = {}) { // the block log, the detection reporter. Always present because a lifecycle method that exists only // for some configurations is one a caller cannot rely on — and each of these components can be the // only one installed, so any of them can be the one left running. + // + // Returns a promise that settles when the reporter has finished draining, so a host shutting down can + // await it rather than racing the last batch against process exit. Bounded and best-effort — a runtime + // that terminates regardless still wins — and ignoring the return behaves exactly as before. protection.stop = () => { loop?.stop(); - firewallLog?.stop(); - detections?.stop(); + // Both reporters, because the promise says every buffer this reaches is finished with. Waiting only + // for one would resolve while the other still had records outstanding — and resolve immediately in a + // configuration where the one being waited for was never built. + const outstanding = [firewallLog?.stop(), detections?.stop()].filter( + (wait) => wait && typeof wait.then === 'function', + ); + + return Promise.all(outstanding).then(() => undefined); }; // The name callers already have, kept as an alias for it. protection.stopRefresh = protection.stop; // Which of the three states reporting is in: requested and running, requested but undeliverable, or // not requested. A boolean would collapse the middle one into "off", which is the reassuring reading. - protection.detectionReporting = detectionReporting; - // Delivery health, when there is a reporter: what was attempted, acknowledged, refused, and dropped. - if (detections) protection.detectionHealth = () => detections.health(); + // A getter, because the state follows refreshes: a property assigned once would report the boot value + // for the life of the process, including after reporting started or stopped. + Object.defineProperty(protection, 'detectionReporting', { + get: () => detectionReporting, + enumerable: true, + }); + // Delivery health while there is a reporter: what was attempted, acknowledged, refused, and dropped. + // Undefined when there is none, so "no reporter" and "a reporter with nothing to show" stay apart. + Object.defineProperty(protection, 'detectionHealth', { + get: () => (detections ? () => detections.health() : undefined), + enumerable: true, + }); return protection; } @@ -1278,33 +1611,76 @@ function defaultOnDetect({ phase, mode, category, rule, message }) { console.warn(`[patchstack] ${tag} phase=${phase ?? 'request'} category=${category ?? '?'} rule=${rule?.id ?? '?'} ${message ?? ''}`.trim()); } -/** @param {Request} request */ -function fetchRequestMeta(request) { - if (!request) return {}; - let path = null; - try { - path = new URL(request.url).pathname; - } catch { - path = typeof request.url === 'string' ? request.url : null; - } +/** + * Request metadata for a detection or a block record, using the address already resolved for it. + * + * `shaped` is the object the engine evaluated, which carries the one resolution for this request. The + * original request is only consulted for the path, the method and the user agent — never for an address, + * because a second derivation could disagree with the first and attribute one request to two clients. + * + * @param {{ ip?: string, _clientIp?: { ip: string | null, source: string } } | undefined} shaped + * @param {Request | undefined} request + */ +/** + * Request metadata from a response-phase context. + * + * The context is the originating request, already carrying its own resolution — so a response detection + * names the same client as the request detection for that request. + */ +function requestMetaFromContext(reqCtx) { + if (!reqCtx) return {}; + const client = reqCtx._clientIp ?? { ip: null, source: 'unavailable' }; + return { - method: request.method ?? null, - path, - ip: request.headers?.get?.('x-forwarded-for') ?? null, - userAgent: request.headers?.get?.('user-agent') ?? null, + method: reqCtx.method ?? null, + // Path AND query. The reporter is what drops the query's VALUES, keeping its parameter names, so + // trimming it here would leave a Fetch or response detection unable to say what was requested. + path: typeof reqCtx.originalUrl === 'string' ? reqCtx.originalUrl : null, + ip: client.ip, + clientIpSource: client.source, + userAgent: reqCtx.headers?.['user-agent'] ?? null, }; } -/** @param {import('http').IncomingMessage & { ip?: string, originalUrl?: string }} req */ -function nodeRequestMeta(req) { +function requestMeta(shaped, request) { + const client = shaped?._clientIp ?? { ip: null, source: 'unavailable' }; + let path = null; + let method = null; + let userAgent = null; + + if (request) { + try { + const u = new URL(request.url); + path = u.pathname + u.search; + } catch { + path = typeof request.url === 'string' ? request.url : null; + } + method = request.method ?? null; + userAgent = request.headers?.get?.('user-agent') ?? null; + } + + return { method, path, ip: client.ip, clientIpSource: client.source, userAgent }; +} + +/** + * Request metadata on the Node and Express paths, using the address already resolved for the request. + * + * @param {import('http').IncomingMessage & { originalUrl?: string }} req + * @param {{ ip: string | null, source: string } | undefined} client + */ +function nodeRequestMeta(req, client) { if (!req) return {}; const headers = req.headers ?? {}; const ua = headers['user-agent'] ?? headers['User-Agent']; - const fwd = headers['x-forwarded-for'] ?? headers['X-Forwarded-For']; + const resolved = client ?? { ip: null, source: 'unavailable' }; + return { method: req.method ?? null, path: req.originalUrl || req.url || null, - ip: req.ip ?? (typeof fwd === 'string' ? fwd : null), + // Never `req.ip`: under Express's `trust proxy` that is header-derived by a policy this guard has not + // verified, and a second derivation could disagree with the one the engine evaluated. + ip: resolved.ip, + clientIpSource: resolved.source, userAgent: typeof ua === 'string' ? ua : Array.isArray(ua) ? ua[0] : null, }; } diff --git a/tests/endpoint-disclosure.test.ts b/tests/endpoint-disclosure.test.ts index 286f88c7..0095ed45 100644 --- a/tests/endpoint-disclosure.test.ts +++ b/tests/endpoint-disclosure.test.ts @@ -215,11 +215,135 @@ describe('shipped docs disclose every endpoint the package calls', () => { it('says what a detection report carries, and what it does not', () => { expect(agentInstall).toMatch(/reportDetections/); // Phrasing-tolerant, substance-strict: the claim has to be there, not any particular sentence. - for (const claim of [/query string|query-string/i, /matched value|value that matched/i, /request body/i]) { + for (const claim of [ + /query string|query-string/i, + /request body/i, + // A report can carry the values of the parameters a rule names, so the doc has to say which values + // and on what authority — an omission here reads as the older, narrower promise. + /values of the parameters a rule names/i, + /derived from the rule/i, + /capture\.plan/, + ]) { expect(agentInstall, `the payload description must address ${claim}`).toMatch(claim); } }); + it('states the limits on captured values, and that the limits report themselves', () => { + // A bound nobody documents is a bound a reader cannot rely on, and a truncated capture that does not + // say so invites a conclusion drawn from a sample. + for (const claim of [ + /at most 10 values/i, + /512 characters/i, + /shortened to fit is marked|marked/i, + /counted/i, + ]) { + expect(agentInstall, `the bounds must address ${claim}`).toMatch(claim); + } + }); + + it('says the same thing in the shipped type declaration and the module it documents', () => { + // `protect.d.ts` is copied into `dist/` and is what an editor shows a caller; the module docblock is + // what a reader of the source sees. A privacy statement that is current in one place and stale in + // another is worse than one stale everywhere, because the stale one still reads as authoritative. + const shipped = [ + readFileSync(new URL('../src/protect/protect.d.ts', import.meta.url), 'utf8'), + readFileSync(new URL('../src/protect/detections.js', import.meta.url), 'utf8'), + ]; + + for (const text of shipped) { + expect(text, 'must not still promise that no values are sent').not.toMatch( + /does NOT send the matched value|never carries: \*\*the matched value/i, + ); + expect(text, 'must say which values do travel').toMatch( + /values of the parameters the matched rule names|values of the parameters a rule names/i, + ); + expect(text, 'must scope the exclusion to unnamed parameters').toMatch( + /any (other )?parameter the matched rule does not name/i, + ); + // The User-Agent travels whether or not a rule names it, so an absolute exclusion would be false. + // Pinned in every place the claim is made, because a carve-out stated in one is not stated at all. + expect(text, 'must carve out the baseline user agent').toMatch( + /user.agent is the one exception|exception to the rule-scoped policy/i, + ); + // And the carve-out itself is phase-dependent: an egress detection has no client to attribute, so a + // surface that promised a user agent or an address on every detection would be wrong there. + expect(text, 'must scope the client fields to the phases that have a client').toMatch( + /request or response detection/i, + ); + expect(text, 'and must say an egress detection carries neither').toMatch( + /egress detection[^.]*(carries neither|no user agent)/i, + ); + } + }); + + it('qualifies the client fields in each place it lists them, not just once', () => { + // Two sections list what a report carries. A single "the phrase appears somewhere" check passes when + // one of them is qualified and the other still promises a user agent and an address on every + // detection — which is false for egress, and false in the direction that flatters us. + const sections = { + 'the per-detection summary': agentInstall.slice( + agentInstall.indexOf('What a detection report contains'), + agentInstall.indexOf('Every field is bounded'), + ), + 'the captured-values section': agentInstall.slice( + agentInstall.indexOf('### What values a report can contain'), + agentInstall.indexOf('One header value always travels'), + ), + }; + + for (const [where, text] of Object.entries(sections)) { + expect(text.length, `${where} should be findable`).toBeGreaterThan(200); + expect(text, `${where} must scope the client fields to the phases that have a client`).toMatch( + /request or response/i, + ); + expect(text, `${where} must not promise them on every detection`).not.toMatch( + /(every|any|each) detection[^.]*(user agent|client address)/i, + ); + } + }); + + it('states the egress baseline separately, since it is a different baseline', () => { + // An egress detection has no client and no user agent, and a single "every report carries" claim + // would be false for it in both directions. + expect(agentInstall, 'egress must have its own baseline').toMatch( + /\*\*An egress detection\*\*/i, + ); + expect(agentInstall, 'and must say it carries no client attribution').toMatch( + /no user agent and no\s+client address/i, + ); + }); + + it('qualifies the query-value exclusion as being about baseline metadata', () => { + // A rule naming `egress.url` captures that URL as it read it, query values included. An unqualified + // "query values are never sent" would be false, in the direction that flatters us. + expect(agentInstall, 'the qualification must be present').toMatch( + /One qualification on the query string/i, + ); + expect(agentInstall, 'and must name the case it covers').toMatch( + /names `egress\.url`[^.]*query values included|query values included/i, + ); + }); + + it('states which sources can never be captured, however a rule is written', () => { + // These are the refusals that hold regardless of what a rule names, so they are the ones a reader + // most needs stated rather than inferred. + expect(agentInstall, 'a whole-request read must grant nothing').toMatch( + /reading `raw` or `all`[^.]*permits \*\*nothing/i, + ); + expect(agentInstall, 'response values must never be captured').toMatch( + /\*\*response\*\* values are never captured/i, + ); + expect(agentInstall, 'raw bytes need a reviewed per-rule opt-in').toMatch( + /raw request bytes\*\* need an explicit, reviewed opt-in/i, + ); + expect(agentInstall, 'and the one header that always travels must be named').toMatch( + /User-Agent\*\*\. It is part of the baseline|One header value always travels/i, + ); + expect(agentInstall, 'and a parameter the rule does not name is never sent').toMatch( + /never contains:\*\* the value of any parameter the matched rule does not name/i, + ); + }); + it('separates parameter identifiers from values, and does not exclude what it sends', () => { // `ruleParameters` returns each condition's `parameter` verbatim, and those name a request region: // `cookie.session`, `server.HTTP_AUTHORIZATION`. So a rule inspecting a cookie or an Authorization @@ -229,7 +353,15 @@ describe('shipped docs disclose every endpoint the package calls', () => { expect(agentInstall, 'must say the identifiers carry their request region').toMatch(/request region/i); expect(agentInstall, 'must show a region-qualified example').toMatch(/cookie\.session/); expect(agentInstall, 'must show a header example, since that is the sensitive case').toMatch(/server\.HTTP_/); - expect(agentInstall, 'the exclusion must be about values').toMatch(/no values of any kind/i); + // The exclusion is now scoped to the parameters a rule does NOT name, since the ones it names may + // have their values captured. A blanket "no values of any kind" would be the opposite overclaim to + // the one this test was written for: understating what is sent rather than overstating it. + expect(agentInstall, 'the exclusion must be scoped to unnamed parameters').toMatch( + /the value of any parameter the matched rule does not name/i, + ); + expect(agentInstall, 'and must not still claim no values are sent at all').not.toMatch( + /no values of any kind/i, + ); // The regression itself: an exclusion clause that names headers or cookies without scoping to their // values. Asserted as an absence because the overclaim is a sentence someone would write again while diff --git a/tests/protect/capture-plan.test.ts b/tests/protect/capture-plan.test.ts new file mode 100644 index 00000000..7ae5065d --- /dev/null +++ b/tests/protect/capture-plan.test.ts @@ -0,0 +1,889 @@ +import { describe, it, expect } from 'vitest'; +import { RequestResolver } from '../../src/protect/engine/request.js'; +import { RuleEngine } from '../../src/protect/engine/engine.js'; +import { normalizeRequest } from '../../src/protect/engine/normalizer.js'; +import { + CAPTURE_LIMITS, + captureValues, + createPlanCache, + derivePlan, + permitsAnything, + planReference, +} from '../../src/protect/capture-plan.js'; +import { validateBundle } from '../../src/protect/rules/validate.js'; +import { captureProblem, ruleContract } from '../../src/protect/rules/contract.js'; + +/** + * What a rule permits to be captured, and — mostly — what it does not. + * + * The interesting direction is refusal. A rule earns each permission by naming what it reads, so the + * cases that matter are the ones where a rule reads broadly and must therefore permit nothing: those are + * where a capture policy quietly turns a security channel into a copy of an application's traffic. + */ +const leafFor = (parameter: string) => ({ parameter, match: { type: 'contains', value: 'x' } }); +const rule = (parameters: string[], extra: Record = {}) => ({ + id: 'r1', + rule_v2: parameters.map((parameter) => ({ parameter, match: { type: 'contains', value: 'x' } })), + ...extra, +}); + +describe('a rule permits what it names', () => { + it('reads every member of a parameter list', () => { + // A condition may name one parameter or a list of them, and the engine reads each. A plan blind to + // the list form would be empty for a rule that reads a dozen fields. + const list = { + id: 'r1', + rule_v2: [{ parameter: ['post.q', 'raw', 'cookie.session'], match: { type: 'contains', value: 'x' } }], + }; + + expect(derivePlan(list).named, 'the named members, and not the whole-request one').toEqual([ + 'cookie.session', + 'post.q', + ]); + }); + + it.each([ + 'get.redirect_to', + 'post.title', + 'cookie.session', + 'server.HTTP_AUTHORIZATION', + 'files.avatar', + 'egress.url', + 'request.q', + ])('permits the value of %s, because the rule was written to inspect it', (parameter) => { + expect(derivePlan(rule([parameter])).named).toEqual([parameter]); + }); + + it('takes the union across conditions and nested groups', async () => { + // The engine reports which RULE matched, not which condition, so a plan narrower than the rule would + // claim a precision the detection does not have. + const nested = { + id: 'r1', + rule_v2: [ + { parameter: 'post.title', match: { type: 'contains', value: 'x' } }, + { + parameter: 'rules', + rules: [ + { parameter: 'get.q', match: { type: 'contains', value: 'x' } }, + { parameter: 'cookie.session', match: { type: 'contains', value: 'x' } }, + ], + }, + ], + }; + + expect(derivePlan(nested).named).toEqual(['cookie.session', 'get.q', 'post.title']); + }); + + it('names each parameter once, however often the rule reads it', () => { + expect(derivePlan(rule(['post.title', 'post.title', 'post.title'])).named).toEqual(['post.title']); + }); +}); + +describe('a rule that reads everything permits nothing', () => { + it.each(['raw', 'all'])('derives no permission from %s', (parameter) => { + // These read the whole request. Deriving a permission from them would mean the broadest rules + // granting the broadest capture, which is exactly backwards. + const plan = derivePlan(rule([parameter])); + + expect(plan.named).toEqual([]); + expect(plan.prefixes).toEqual([]); + expect(permitsAnything(plan)).toBe(false); + }); + + it('derives nothing from a bare source wildcard', () => { + // `post.*` names nothing in particular: it is `all` wearing a different hat. + expect(permitsAnything(derivePlan(rule(['post.*'])))).toBe(false); + }); + + it('still permits the parameters a broad rule ALSO names', () => { + // `raw` adds nothing, but it does not poison what the rule names beside it. + expect(derivePlan(rule(['raw', 'post.title'])).named).toEqual(['post.title']); + }); +}); + +describe('a prefix permits matching keys and no others', () => { + it('records the prefix rather than the pattern', () => { + expect(derivePlan(rule(['post.field_*'])).prefixes).toEqual(['post.field_']); + }); + + it('bounds how many keys a prefix may match', () => { + // A prefix can match an unbounded number of keys, and "a rule that reads a prefix" must not become + // "a rule that reads the whole body". + expect(derivePlan(rule(['post.field_*'])).limits.prefixValues).toBe(CAPTURE_LIMITS.prefixValues); + expect(CAPTURE_LIMITS.prefixValues).toBeLessThan(CAPTURE_LIMITS.capturedValues + 1); + }); +}); + +describe('the contract decides what is a parameter at all', () => { + it.each([ + // `server` and `egress` enumerate their keys and do not fan out, so these are not parameters the + // engine can read — and a permission for something no rule can read is a permission with no rule + // behind it. + 'server.HTTP_*', + 'server.not-real', + 'egress.not-real', + 'egress.*', + 'response.header.*', + ])('derives nothing from %s, which no rule may read', (parameter) => { + expect(permitsAnything(derivePlan(rule([parameter])))).toBe(false); + }); + + it.each([ + 'server.HTTP_AUTHORIZATION', + 'server.REQUEST_URI', + 'egress.url', + 'files.avatar.filename', + // `files` keys are application field names, so the contract accepts any of them — including one + // whose suffix it does not enumerate. + 'files.avatar.invented', + ])( + 'still permits %s, which a rule may read', + (parameter) => { + expect(derivePlan(rule([parameter])).named).toEqual([parameter]); + }, + ); +}); + +describe('the response phase is never a capture source', () => { + it.each(['response.body', 'response.headers', 'response.status'])('refuses %s', (parameter) => { + // Those read what the application is about to SEND. The phase inspecting them exists to redact + // secrets, so capturing them would collect the values that redaction is there to stop leaving. + expect(permitsAnything(derivePlan(rule([parameter])))).toBe(false); + }); +}); + +describe('raw bytes need an explicit opt-in, and are bounded anyway', () => { + it('permits nothing without one', () => { + expect(derivePlan(rule(['post.title'])).raw).toBeNull(); + }); + + it('permits a bounded prefix when a rule carries one', () => { + expect(derivePlan(rule(['raw'], { capture: { version: 1, raw_chars: 128 } })).raw).toEqual({ chars: 128 }); + }); + + it('gives a rule the cap rather than what it asked for', () => { + // The opt-in says a reviewer agreed raw bytes are needed here, not that this rule sets its own + // bounds. + expect(derivePlan(rule(['raw'], { capture: { version: 1, raw_chars: 10_000 } })).raw).toEqual({ + chars: CAPTURE_LIMITS.valueChars, + }); + }); + + it.each([ + ['a negative request', -1], + ['zero', 0], + ['a fraction', 12.5], + // Coercion is not consent: none of these is a number a reviewer wrote. + ['a string of digits', '128'], + ['true', true], + ['a one-element array', [128]], + ['text', 'lots'], + ['nothing', undefined], + ])('refuses %s', (_what, raw_chars) => { + expect(derivePlan(rule(['raw'], { capture: { version: 1, raw_chars } })).raw).toBeNull(); + }); + + it.each([ + ['no version at all', { raw_chars: 128 }], + // An opt-in that authorises nothing is a property with no reason to exist, and a consumer cannot tell + // it from one that was meant to say something and does not. + ['nothing it is opting into', { version: 1 }], + ['a version this guard does not know', { version: 2, raw_chars: 128 }], + ['a version that is not a number', { version: '1', raw_chars: 128 }], + ['a key the contract does not define', { version: 1, raw_chars: 128, everything: true }], + ['a list instead of an object', [{ version: 1, raw_chars: 128 }]], + ['a string', 'raw'], + ])('refuses an opt-in with %s', (_what, capture) => { + // The opt-in authorises collection, so a guard meeting one it cannot read must grant nothing rather + // than guess. That is also what lets a newer server extend it without an older guard capturing under + // rules it does not understand. + expect(derivePlan(rule(['raw'], { capture })).raw).toBeNull(); + }); + + it('refuses an opt-in the rule does not own', () => { + // One write to a prototype would otherwise grant raw capture to every rule at once. + (Object.prototype as any).capture = { version: 1, raw_chars: 512 }; + try { + const r = rule(['post.title']); + + expect((r as any).capture, 'the chain does offer one').toBeTruthy(); + expect(derivePlan(r).raw, 'but it belongs to no rule').toBeNull(); + } finally { + delete (Object.prototype as any).capture; + } + }); +}); + +describe('a parameter list is one level deep', () => { + const nested = { + id: 'r1', + rule_v2: [{ parameter: [['post.q']], match: { type: 'contains', value: 'x' } }], + }; + + it('is refused by the validator rather than accepted and inert', () => { + // The engine expands one level: a member that is itself a list resolves to nothing. Accepting this + // would pass a rule that names a parameter and matches on none — protection that reads as present. + const { bundle, rejected } = validateBundle({ firewall: [nested], whitelists: [], whitelist_keys: {} }); + + expect(bundle.firewall).toEqual([]); + expect(rejected[0].reason).toMatch(/parameters, not more lists/); + }); + + it('grants no permission for a parameter buried inside one', () => { + expect(permitsAnything(derivePlan(nested as never))).toBe(false); + }); + + it('still reads a flat list, which is what the engine expands', () => { + const flat = { + id: 'r1', + rule_v2: [{ parameter: ['post.q', 'cookie.session'], match: { type: 'contains', value: 'x' } }], + }; + + expect(derivePlan(flat as never).named).toEqual(['cookie.session', 'post.q']); + }); +}); + +describe('a rule that cannot produce a detection permits nothing', () => { + it('grants no raw capture to a rule with no conditions', () => { + // A permission exists to explain a detection. The opt-in is a property ON a rule, not a licence of + // its own, so a rule the engine cannot read authorises nothing however the opt-in is written. + const plan = derivePlan({ id: 'r1', capture: { version: 1, raw_chars: 128 } } as never); + + expect(plan.raw).toBeNull(); + expect(permitsAnything(plan)).toBe(false); + }); + + it.each([ + ['conditions that are not a list', { id: 'r1', rule_v2: 'nonsense' }], + ['an empty condition list', { id: 'r1', rule_v2: [] }], + ['only parameters no rule may read', { id: 'r1', rule_v2: [{ parameter: 'server.HTTP_*' }] }], + // Spelled correctly and still not a rule: the guard refuses it, so it produces no detection to + // explain. A permission derived from spelling alone would outlive the rule that justified it. + ['a parameter and no match', { id: 'r1', rule_v2: [{ parameter: 'post.q' }] }], + [ + 'a match the engine does not have', + { id: 'r1', rule_v2: [{ parameter: 'post.q', match: { type: 'invented', value: 'x' } }] }, + ], + ['a phase that does not exist', { id: 'r1', rule_v2: [leafFor('post.q')], phase: 'sideways' }], + ])('grants nothing to a rule with %s', (_what, shape) => { + const plan = derivePlan({ ...shape, capture: { version: 1, raw_chars: 128 } } as never); + + expect(plan.raw, 'no raw permission').toBeNull(); + expect(permitsAnything(plan), 'and no named permission either').toBe(false); + }); + + it('grants capture to a rule that matches on the whole request and names no parameter', () => { + // The positive control the validator gate needs. `cross_origin` and its kin read the whole request + // and carry no parameter, so a gate asking for a parameter would deny a perfectly good rule the + // evidence it was opted into. + const wholeRequest = { + id: 'csrf', + rule_v2: [{ match: { type: 'cross_origin', value: '' } }], + capture: { version: 1, raw_chars: 128 }, + }; + + expect(derivePlan(wholeRequest as never).raw, 'it can fire, so it can explain itself').toEqual({ + chars: 128, + }); + }); + + it('still grants it to a rule that reads the raw body', () => { + // The positive control, and the opt-in's actual purpose: a rule matching on `raw` may capture a + // bounded prefix of it. + expect(derivePlan(rule(['raw'], { capture: { version: 1, raw_chars: 128 } })).raw).toEqual({ + chars: 128, + }); + }); +}); + +describe('an unreadable rule permits nothing', () => { + it.each([ + ['no rule at all', undefined], + ['null', null], + ['a rule with no conditions', { id: 'r1' }], + ['conditions that are not a list', { id: 'r1', rule_v2: 'nonsense' }], + ['conditions that are not objects', { id: 'r1', rule_v2: [null, 3, 'x'] }], + ['a parameter that is not a string', { id: 'r1', rule_v2: [{ parameter: 7 }] }], + ['a source nobody defines', { id: 'r1', rule_v2: [{ parameter: 'invented.thing' }] }], + // `all` and `raw` are whole-request reads, not sources with keys. Nothing resolves these, so a plan + // that granted them would be granting capture for a parameter the rule cannot even read. + ['a key hung off all', { id: 'r1', rule_v2: [{ parameter: 'all.x' }] }], + ['a key hung off raw', { id: 'r1', rule_v2: [{ parameter: 'raw.x' }] }], + ['a source with no key', { id: 'r1', rule_v2: [{ parameter: 'post.' }] }], + ])('permits nothing for %s', (_what, input) => { + // Failing to understand a rule must never be the reason something gets captured. + const plan = derivePlan(input as never); + + expect(permitsAnything(plan)).toBe(false); + expect(plan.named).toEqual([]); + }); + + it('does not recurse without bound on a self-referencing group', () => { + const loop: any = { id: 'r1', rule_v2: [{ parameter: 'rules', rules: [] }] }; + loop.rule_v2[0].rules.push(loop.rule_v2[0]); + + expect(() => derivePlan(loop)).not.toThrow(); + }); +}); + +describe('a plan reference names the policy, not the moment', () => { + it('is the same for the same permissions', () => { + // Two events carrying one reference were governed by the same permissions, across processes and + // releases — otherwise a reader cannot tell what a capture was allowed to include. + expect(planReference(derivePlan(rule(['post.title', 'get.q'])))).toBe( + planReference(derivePlan(rule(['get.q', 'post.title']))), + ); + }); + + it('does not depend on the order the permissions are listed in', () => { + // `derivePlan` sorts, so this cannot arise from it today — but the reference is what ties a captured + // value to the policy that allowed it, and that tie must not rest on an ordering somewhere else. + const limits = { capturedValues: 10, valueChars: 512, prefixValues: 5 }; + const one = { named: ['get.q', 'post.title'], prefixes: ['post.a.', 'post.b.'], raw: null, limits }; + const other = { named: ['post.title', 'get.q'], prefixes: ['post.b.', 'post.a.'], raw: null, limits }; + + expect(planReference(other)).toBe(planReference(one)); + }); + + it('covers the limits, not only the parameters', () => { + // Two plans naming the same parameters but allowing 512 and 4096 characters are different + // permissions. A reference that could not tell them apart would fail at exactly the claim it exists + // to support. + const base = derivePlan(rule(['post.title'])); + const looser = { ...base, limits: { ...base.limits, valueChars: 4096 } }; + const fewer = { ...base, limits: { ...base.limits, values: 1 } }; + + expect(planReference(looser)).not.toBe(planReference(base)); + expect(planReference(fewer)).not.toBe(planReference(base)); + }); + + it('is exactly this, for this literal plan', () => { + // A pinned vector for the ALGORITHM, over a plan written out here rather than derived. Every + // reference already emitted means whatever this algorithm and canonical form produced, so replacing + // either silently would change what all of them refer to while every relative assertion above still + // passed. Changing them deliberately takes a new prefix, not a new implementation under the old one. + // + // The limits are literal too. A future policy change to `CAPTURE_LIMITS` should give a different + // reference under the SAME algorithm, and a vector reading the current limits would call that a + // reason to change the prefix. + const plan = { + named: ['get.q', 'post.title'], + prefixes: [], + raw: null, + limits: { capturedValues: 10, valueChars: 512, prefixValues: 5 }, + }; + + expect(planReference(plan)).toBe('cp2-470617d87e6943b67e48ec6c4022705e'); + }); + + it('changes when a limit changes, without changing the algorithm', () => { + // The policy moving is not the algorithm moving: the reference follows the permissions, the prefix + // stays put. + const base = { named: ['post.title'], prefixes: [], raw: null, limits: { capturedValues: 10, valueChars: 512 } }; + const tighter = { ...base, limits: { capturedValues: 10, valueChars: 128 } }; + + expect(planReference(tighter)).not.toBe(planReference(base)); + expect(planReference(tighter).startsWith('cp2-')).toBe(true); + }); + + it('is what derivePlan produces under the limits in force', () => { + // Kept apart from the vector above: this one is allowed to change when the policy does. + const plan = derivePlan(rule(['post.title', 'get.q'])); + + expect(plan.named).toEqual(['get.q', 'post.title']); + expect(plan.limits).toEqual(CAPTURE_LIMITS); + }); + + it('is wide enough to be a durable identity', () => { + // It outlives the process and is compared across systems, so two different plans meeting on one + // reference must not be something a reader has to think about. + const reference = planReference(derivePlan(rule(['post.title']))); + + expect(reference).toMatch(/^cp2-[0-9a-f]{32}$/); + }); + + it('differs when the permissions differ', () => { + const reads = planReference(derivePlan(rule(['post.title']))); + + expect(planReference(derivePlan(rule(['post.body'])))).not.toBe(reads); + expect(planReference(derivePlan(rule(['post.title', 'get.q'])))).not.toBe(reads); + expect(planReference(derivePlan(rule(['post.title'], { capture: { version: 1, raw_chars: 64 } })))).not.toBe(reads); + }); +}); + +describe('a plan cannot change after its reference is computed', () => { + it('is frozen, along with everything it holds', () => { + // The reference identifies a set of permissions. A plan that could be edited afterwards would leave + // the reference naming permissions that no longer apply. + const plan = derivePlan(rule(['post.title'], { capture: { version: 1, raw_chars: 64 } })); + + expect(Object.isFrozen(plan)).toBe(true); + expect(Object.isFrozen(plan.named)).toBe(true); + expect(Object.isFrozen(plan.prefixes)).toBe(true); + expect(Object.isFrozen(plan.raw)).toBe(true); + expect(Object.isFrozen(plan.limits)).toBe(true); + expect(Object.isFrozen(CAPTURE_LIMITS)).toBe(true); + }); +}); + +describe('plans are derived once per rule', () => { + it('reuses the plan for a rule it has seen', () => { + const cache = createPlanCache(); + const r = { ...rule(['post.title']), source_revision: 'rev-1' }; + + expect(cache.for(r).plan).toBe(cache.for(r).plan); + expect(cache.derivations, 'derived once, answered twice').toBe(1); + }); + + it('does not answer for one rule with another rule\'s permissions', () => { + // A revision identifies a version of ONE rule, not a rule. Two rules can carry the same revision, and + // a cache keyed on that alone would capture a field the second rule never authorised. + const cache = createPlanCache(); + const first = { ...rule(['post.title']), id: 'rule-a', source_revision: 'shared-rev' }; + const second = { ...rule(['cookie.session']), id: 'rule-b', source_revision: 'shared-rev' }; + + expect(cache.for(first).plan.named).toEqual(['post.title']); + expect(cache.for(second).plan.named, 'its own permissions, not the first rule\'s').toEqual([ + 'cookie.session', + ]); + }); + + it('derives again for a rule that arrived as a new object', () => { + // A refreshed bundle brings new rule objects, so a changed rule is derived again rather than answered + // from an entry describing what it used to say. + const cache = createPlanCache(); + cache.for({ ...rule(['post.title']), source_revision: 'rev-1' }); + const updated = cache.for({ ...rule(['post.title', 'cookie.session']), source_revision: 'rev-2' }); + + expect(updated.plan.named).toEqual(['cookie.session', 'post.title']); + expect(cache.derivations).toBe(2); + }); + + it('answers for a rule that is not an object at all', () => { + const cache = createPlanCache(); + + expect(permitsAnything(cache.for(undefined as never).plan)).toBe(false); + expect(cache.for(null as never).reference).toMatch(/^cp2-/); + }); + + it('hands out an entry that cannot be edited', () => { + const cache = createPlanCache(); + const entry = cache.for({ ...rule(['post.title']), source_revision: 'rev-1' }); + + expect(Object.isFrozen(entry)).toBe(true); + expect(Object.isFrozen(entry.plan)).toBe(true); + }); +}); + +describe('capture metadata never costs the mitigation', () => { + const leaf = { parameter: 'get.q', match: { type: 'contains', value: 'x' } }; + const served = (capture: unknown) => ({ + firewall: [{ id: 'r1', title: 'a rule that still has to protect', rule_v2: [leaf], capture }], + whitelists: [], + whitelist_keys: {}, + }); + + it.each([ + ['a version this guard does not know', { version: 2, raw_chars: 128 }], + ['a version that is not a number', { version: '1', raw_chars: 128 }], + ['a key the contract does not define', { version: 1, raw_chars: 128, everything: true }], + ['a malformed size', { version: 1, raw_chars: -5 }], + ['nothing at all', null], + ['a string', 'raw'], + ])('keeps a rule carrying %s, and grants it no capture', (_what, capture) => { + // A rule is a mitigation. Letting a capture value the guard cannot read decide whether the rule runs + // would turn a question about evidence into the loss of the protection — a newer server adding a + // capture version would switch off shielding on every older guard. + const { bundle, rejected } = validateBundle(served(capture)); + + expect(rejected, 'the rule was not dropped').toEqual([]); + expect(bundle.firewall.length, 'the rule still protects').toBe(1); + expect(derivePlan(bundle.firewall[0]).raw, 'and collects nothing').toBeNull(); + }); + + it('grants capture for the opt-in it does understand', () => { + // The positive control: the separation is only meaningful if a valid opt-in still works. + const { bundle, rejected } = validateBundle(served({ version: 1, raw_chars: 128 })); + + expect(rejected).toEqual([]); + expect(bundle.firewall.length).toBe(1); + expect(derivePlan(bundle.firewall[0]).raw).toEqual({ chars: 128 }); + }); +}); + +describe('the published contract and the runtime give the same answer', () => { + const published = ruleContract().capture; + + it('does not publish a maximum a consumer would refuse a working rule over', () => { + // A schema maximum would have a consumer reject what this guard accepts and runs. Asking for more + // than the ceiling is a request, not an authoring error, so the ceiling is published beside the + // schema rather than inside it. + expect(published.properties.raw_chars.maximum, 'no schema maximum').toBeUndefined(); + expect(published.raw_chars_effective_maximum).toBe(CAPTURE_LIMITS.valueChars); + }); + + it('accepts a request above the ceiling and answers with the ceiling', () => { + const asked = published.raw_chars_effective_maximum * 20; + const served = { + firewall: [{ id: 'r1', rule_v2: [leafFor('raw')], capture: { version: 1, raw_chars: asked } }], + whitelists: [], + whitelist_keys: {}, + }; + const { bundle, rejected } = validateBundle(served); + + expect(rejected, 'the contract does not refuse it').toEqual([]); + expect(derivePlan(bundle.firewall[0]).raw, 'and the runtime answers with the ceiling').toEqual({ + chars: published.raw_chars_effective_maximum, + }); + }); + + it('refuses everything the published schema refuses', () => { + // The schema is what a consumer implements against, so the two must agree on rejection too. + expect(published.required).toEqual(['version', 'raw_chars']); + expect(published.additional_properties).toBe(false); + expect(published.properties.version.const).toBe(1); + expect(published.properties.raw_chars.minimum).toBe(1); + + expect(captureProblem({ raw_chars: 8 }), 'version is required').not.toBeNull(); + expect(captureProblem({ version: 1 }), 'raw_chars is required').not.toBeNull(); + expect(captureProblem({ version: 2, raw_chars: 8 }), 'version must be the one published').not.toBeNull(); + expect(captureProblem({ version: 1, raw_chars: 0 }), 'below the minimum').not.toBeNull(); + expect(captureProblem({ version: 1, raw_chars: 8, extra: 1 }), 'no other keys').not.toBeNull(); + expect(captureProblem({ version: 1, raw_chars: 8 }), 'and this one is valid').toBeNull(); + }); +}); + +describe('what a plan actually reads from a request', () => { + const resolverFor = (req: unknown) => new RequestResolver(normalizeRequestFor(req)); + // The engine reads a normalized request, so capture reads the same one — otherwise it would be holding + // something other than what the rule matched on. + function normalizeRequestFor(req: any) { + return { ...req, ...normalizeRequest(req) }; + } + const request = (over: Record = {}) => ({ + method: 'POST', + url: '/checkout', + originalUrl: '/checkout', + headers: { 'content-type': 'application/json', 'user-agent': 'scanner/1.0' }, + query: {}, + body: {}, + cookies: {}, + ...over, + }); + + it('reads the parameter the plan names, and nothing beside it', () => { + const plan = derivePlan(rule(['post.title'])); + const taken = captureValues(plan, resolverFor(request({ body: { title: 'payload', secret: 'not-permitted' } }))); + + expect(taken.values).toEqual([{ parameter: 'post.title', value: 'payload' }]); + expect(JSON.stringify(taken), 'a field the plan did not name is not read').not.toContain( + 'not-permitted', + ); + }); + + it('takes nothing when the plan permits nothing', () => { + // The common case, and the one that must cost nothing: a rule reading `raw` with no opt-in. + const taken = captureValues(derivePlan(rule(['raw'])), resolverFor(request({ body: { title: 'payload' } }))); + + expect(taken).toEqual({ values: [], omitted: 0, unsupported: 0, failed: 0, unavailable: false, raw: null }); + }); + + it('reads a header and a cookie the rule named', () => { + const plan = derivePlan(rule(['server.HTTP_USER_AGENT', 'cookie.session'])); + const taken = captureValues(plan, resolverFor(request({ cookies: { session: 'abc123' } }))); + + expect(taken.values.map((v: any) => v.parameter).sort()).toEqual([ + 'cookie.session', + 'server.HTTP_USER_AGENT', + ]); + }); + + it('shortens a long value and says which one', () => { + const plan = derivePlan(rule(['post.title'])); + const taken = captureValues(plan, resolverFor(request({ body: { title: 'x'.repeat(5000) } }))); + + expect(taken.values[0].value.length).toBe(CAPTURE_LIMITS.valueChars); + expect(taken.values[0].truncated).toBe(true); + }); + + it('says nothing about truncation when nothing was shortened', () => { + const taken = captureValues(derivePlan(rule(['post.title'])), resolverFor(request({ body: { title: 'short' } }))); + + expect(Object.hasOwn(taken.values[0], 'truncated')).toBe(false); + }); + + it('stops at the total it is allowed, and counts what it left', () => { + // A capture holding less than it appears to would have a reader drawing conclusions from a sample + // without knowing it was one. + const many = Array.from({ length: 30 }, (_, i) => `post.f${i}`); + const body = Object.fromEntries(many.map((_, i) => [`f${i}`, `value-${i}`])); + const taken = captureValues(derivePlan(rule(many)), resolverFor(request({ body }))); + + expect(taken.values.length).toBe(CAPTURE_LIMITS.capturedValues); + expect(taken.omitted).toBe(30 - CAPTURE_LIMITS.capturedValues); + }); + + it('bounds how many keys one prefix contributes', () => { + const body = Object.fromEntries(Array.from({ length: 20 }, (_, i) => [`field_${i}`, `v${i}`])); + const taken = captureValues(derivePlan(rule(['post.field_*'])), resolverFor(request({ body }))); + + expect(taken.values.length).toBe(CAPTURE_LIMITS.prefixValues); + expect(taken.values.every((v: any) => v.parameter === 'post.field_*')).toBe(true); + expect(taken.omitted).toBe(20 - CAPTURE_LIMITS.prefixValues); + }); + + it('does not serialise an object the plan named', () => { + // A permission for `post.profile` is not a permission for everything under it. + const plan = derivePlan(rule(['post.profile'])); + const taken = captureValues(plan, resolverFor(request({ body: { profile: { name: 'ada', password: 'hunter2' } } }))); + + expect(JSON.stringify(taken)).not.toContain('hunter2'); + expect(taken.unsupported, 'refused for its type, and said so').toBe(1); + expect(taken.omitted, 'which is not the same as a bound leaving it out').toBe(0); + }); + + it('reads a bounded prefix of the raw body only with the opt-in', () => { + const raw = 'not-json __proto__ ' + 'y'.repeat(1000); + const req = request({ headers: { 'content-type': 'application/json' }, body: {}, _rawBody: raw }); + + const without = captureValues(derivePlan(rule(['raw'])), resolverFor(req)); + expect(without.raw, 'no opt-in, no raw evidence').toBeNull(); + + const withOptIn = captureValues( + derivePlan(rule(['raw'], { capture: { version: 1, raw_chars: 64 } })), + resolverFor(req), + ); + expect(withOptIn.raw.value.length).toBe(64); + expect(withOptIn.raw.truncated).toBe(true); + expect(withOptIn.raw.value).toBe(raw.slice(0, 64)); + }); + + it('never fails a request over evidence', () => { + // Fail-open, like everything else on this path: a capture that cannot be taken is not taken. + const hostile = { + resolve() { + throw new Error('hostile parameter'); + }, + }; + + // No resolver to read with at all. + expect(captureValues(derivePlan(rule(['post.title'])), undefined as never)).toEqual({ + values: [], + omitted: 0, + unsupported: 0, + failed: 0, + // Distinguishable from "there was nothing to capture": a reviewer must not read incomplete + // evidence as complete. + unavailable: true, + raw: null, + }); + + const threw = captureValues(derivePlan(rule(['post.title'])), hostile as never); + + expect(threw.values).toEqual([]); + expect(threw.failed, 'the failure is recorded, not silently empty').toBe(1); + expect(threw.unavailable, 'and no read completed, so nothing was readable').toBe(true); + }); + + it('reports a partial read as partial, not as unreadable', () => { + // One parameter failing is not the request being unreadable, and the two lead to opposite + // conclusions from a short list of values. + const flaky = { + resolve(parameter: string) { + if (parameter === 'post.bad') throw new Error('nope'); + + return ['fine']; + }, + }; + const taken = captureValues(derivePlan(rule(['post.bad', 'post.good'])), flaky as never); + + expect(taken.values).toEqual([{ parameter: 'post.good', value: 'fine' }]); + expect(taken.failed).toBe(1); + expect(taken.unavailable).toBe(false); + }); + +}); + +describe('evidence records what it could not take, and why', () => { + const resolverFor = (req: unknown) => new RequestResolver({ ...(req as object), ...normalizeRequest(req as never) }); + const request = (over: Record = {}) => ({ + method: 'POST', + url: '/checkout', + originalUrl: '/checkout', + headers: { 'content-type': 'application/json' }, + query: {}, + body: {}, + cookies: {}, + ...over, + }); + + it('keeps a value that is present and empty', () => { + // A rule can be written so that its finding IS that a parameter is empty. Collapsing that into + // absence erases the evidence for exactly those rules. + const taken = captureValues(derivePlan(rule(['post.title'])), resolverFor(request({ body: { title: '' } }))); + + expect(taken.values).toEqual([{ parameter: 'post.title', value: '' }]); + }); + + it('tells a present-but-empty value apart from an absent one', () => { + const absent = captureValues(derivePlan(rule(['post.title'])), resolverFor(request({ body: {} }))); + + // An absent field resolves to no value at all, which is already a different answer — and neither is + // a failure, so nothing is counted against the bounds. + expect(absent.values).toEqual([]); + expect(absent).toMatchObject({ omitted: 0, unsupported: 0, failed: 0, unavailable: false }); + }); + + it('counts every value one NAMED parameter had excluded, not just the first', () => { + // A single named parameter can resolve to many values — several files uploaded under one field name + // fan out. Reporting one omission where there were several would have a reviewer take a truncated + // sample for a nearly complete one. + const upload = Array.from({ length: 15 }, (_, i) => ({ + filename: `f${i}.php`, + type: 'text/php', + content: `content-${i}`, + })); + const taken = captureValues(derivePlan(rule(['files.upload.content'])), resolverFor(request({ files: { upload } }))); + + expect(taken.values.length).toBe(CAPTURE_LIMITS.capturedValues); + expect(taken.omitted, 'every one that did not fit').toBe(15 - CAPTURE_LIMITS.capturedValues); + }); + + it('counts every value a prefix had excluded', () => { + const files = Object.fromEntries( + Array.from({ length: 15 }, (_, i) => [`f${i}`, { filename: `f${i}.php`, type: 'text/php', content: 'x' }]), + ); + const taken = captureValues(derivePlan(rule(['files.f*'])), resolverFor(request({ files }))); + + expect(taken.values.length).toBe(CAPTURE_LIMITS.prefixValues); + expect(taken.omitted).toBe(15 - CAPTURE_LIMITS.prefixValues); + }); + + it('judges an unsupported type wherever it appears, not by where the budget ran out', () => { + // Type is judged before capacity, so the same value is refused the same way at the front of a + // request and at the back of it. + // The object sorts LAST, so the budget is already full when it is reached. Judging capacity first + // would file it as omitted — a value a bound left out — rather than as one whose type is refused. + const body: Record = { z: { nested: true } }; + for (let i = 0; i < 12; i++) body[`f${i}`] = `v${i}`; + const plan = derivePlan(rule(['post.z', ...Array.from({ length: 12 }, (_, i) => `post.f${i}`)])); + + expect(plan.named[plan.named.length - 1], 'the object is reached last').toBe('post.z'); + + const taken = captureValues(plan, resolverFor(request({ body }))); + + expect(taken.unsupported, 'refused for its type, not for arriving late').toBe(1); + expect(taken.values.length).toBe(CAPTURE_LIMITS.capturedValues); + expect(taken.omitted).toBe(12 - CAPTURE_LIMITS.capturedValues); + }); + + it('classifies a prefix value before the prefix bound, not after', () => { + // Five strings, then an object. What a value IS does not depend on how many came before it: refused + // for its type reads as a value this channel will not carry, while left out by a bound reads as one + // that would have fitted in a larger event. Swapping them misreports why the evidence is short. + const body: Record = {}; + for (let i = 0; i < CAPTURE_LIMITS.prefixValues; i++) body[`field_${i}`] = `v${i}`; + body.field_last = { nested: true }; + + const taken = captureValues(derivePlan(rule(['post.field_*'])), resolverFor(request({ body }))); + + expect(taken.values.length).toBe(CAPTURE_LIMITS.prefixValues); + expect(taken.unsupported, 'the object was refused for its type').toBe(1); + expect(taken.omitted, 'and no bound left anything out').toBe(0); + }); + + it('gives raw its own allowance rather than a slot from the value total', () => { + // Raw is separately opted into and separately bounded. Making it consume a value slot would have an + // opt-in silently reduce the named evidence a reviewer needs. + const body = Object.fromEntries(Array.from({ length: 12 }, (_, i) => [`f${i}`, `v${i}`])); + const named = Array.from({ length: 12 }, (_, i) => `post.f${i}`); + const plan = derivePlan(rule([...named, 'raw'], { capture: { version: 1, raw_chars: 32 } })); + const taken = captureValues(plan, resolverFor(request({ body, _rawBody: 'r'.repeat(200) }))); + + expect(taken.values.length, 'the full value budget').toBe(CAPTURE_LIMITS.capturedValues); + expect(taken.raw, 'and raw besides').not.toBeNull(); + expect(taken.raw.value.length).toBe(32); + }); + + it('records the resolved value: normalised, but not mutated by the rule', () => { + // Three forms of a parameter exist. The engine normalises the request, then a condition applies its + // own mutations. This records the middle one — what the resolver answers with — because reading + // either of the others would be a second interpretation of the request. + const encoded = '%3Cscript%3E'; + const urlRule = { + id: 'r1', + rule_v2: [{ parameter: 'get.q', mutations: ['urldecode'], match: { type: 'contains', value: '', + // Parses, but names nobody — an event carrying it would attribute itself to no one, in a field + // whose whole job is attribution. + '0.0.0.0', + '::', + '::ffff:0.0.0.0', + ])('refuses %s as an address, whatever its provenance claims', (ip) => { + // This function states the payload invariant, so it checks rather than assumes. A resolver bypass — + // a future adapter, or a record rebuilt elsewhere — must not be able to put a hostname or a + // nobody-address into retained evidence. + for (const source of ['runtime', 'trusted-proxy'] as const) { + expect(clientIpFields({ ip, source } as never)).toEqual({ client_ip_source: 'unavailable' }); + } + }); + + it.each([ + ['::FFFF:198.51.100.9', '198.51.100.9'], + ['2001:0DB8:0000:0000:0000:0000:0000:0001', '2001:db8::1'], + ])('emits %s in its canonical form %s', (ip, canonical) => { + // One address must not appear as two records depending on how a hop happened to spell it. + expect(clientIpFields({ ip, source: 'trusted-proxy' } as never)).toEqual({ + client_ip: canonical, + client_ip_source: 'trusted-proxy', + }); + }); + + it('carry both when an address was established', () => { + expect(clientIpFields({ ip: PEER, source: 'runtime' })).toEqual({ + client_ip: PEER, + client_ip_source: 'runtime', + }); + }); +}); + +describe('a hop-count policy works on its own', () => { + it('trusts the peer as hop one with no CIDR list at all', () => { + // A policy declaring only `hops` states the trust numerically: the peer IS hop one. Requiring a CIDR + // match as well would make such a policy accepted by configuration and inert in practice. + const resolved = resolveClientIp({ + peer: PROXY, + headers: { 'x-forwarded-for': CLIENT }, + trustedProxy: { hops: 1 }, + }); + + expect(resolved).toEqual({ ip: CLIENT, source: 'trusted-proxy' }); + }); + + it('still lets an address list gate the peer when one is declared', () => { + // With `peers` present, that verdict governs: a connection from anywhere else did not arrive through + // the declared front end, whatever the hop count says. + const resolved = resolveClientIp({ + peer: PEER, + headers: { 'x-forwarded-for': CLIENT }, + trustedProxy: { peers: ['10.0.0.0/8'], hops: 1 }, + }); + + expect(resolved).toEqual({ ip: PEER, source: 'runtime' }); + }); + + it('applies the same rule to a predicate-only policy', () => { + const resolved = resolveClientIp({ + peer: PEER, + headers: { 'x-forwarded-for': CLIENT }, + trustedProxy: { isTrusted: () => false, hops: 1 }, + }); + + expect(resolved).toEqual({ ip: PEER, source: 'runtime' }); + }); +}); + +describe('trust configuration fails closed', () => { + it.each([ + ['an extra slash', ['10.0.0.0/8/typo']], + ['a prefix that is not a number', ['10.0.0.0/eight']], + ['a prefix wider than the family', ['10.0.0.0/33']], + ['an IPv6 prefix wider than the family', ['2001:db8::/129']], + ['an address that is not one', ['not-an-address']], + ['a bracketed address', ['[2001:db8::]/32']], + ['a non-string member', [42]], + ['peers that is not an array', '10.0.0.0/8'], + ])('rejects the whole policy for %s', (_label, peers) => { + // One unparseable entry invalidates everything. A policy that silently lost a member is a policy + // nobody wrote, and its behaviour would not reveal which half took effect. + expect(readTrustPolicy({ peers } as never)).toBeNull(); + }); + + it('rejects a mixed list rather than keeping the valid half', () => { + expect(readTrustPolicy({ peers: ['10.0.0.0/8', 'typo'] })).toBeNull(); + expect(readTrustPolicy({ peers: ['10.0.0.0/8'] })).not.toBeNull(); + }); +}); + +describe('addresses are parsed strictly and reported canonically', () => { + it.each(['[::1', '::1]', '[2001:db8::1]', '[2001:db8::1]:8080'])('rejects the bracketed form %s', (value) => { + // Brackets are rejected outright rather than stripped: stripping a delimiter whose partner may be + // missing accepts `[::1`. A proxy emitting the bracketed form is not understood, and the walk falls + // back to the observed peer. + expect(isIpAddress(value)).toBe(false); + }); + + it.each(['1.2.3.4%eth0', '203.0.113.1%0'])('rejects a zone on IPv4 (%s)', (value) => { + expect(isIpAddress(value)).toBe(false); + }); + + it.each(['fe80::1%eth0', 'fe80::1%2'])('accepts a zone on IPv6 (%s)', (value) => { + expect(isIpAddress(value)).toBe(true); + }); + + it.each(['fe80::1%', '::1%'])('rejects an empty zone (%s)', (value) => { + expect(isIpAddress(value)).toBe(false); + }); + + it.each([ + ['2001:0DB8:0000:0000:0000:0000:0000:0001', '2001:db8::1'], + ['::FFFF:203.0.113.1', '203.0.113.1'], + ['::1', '::1'], + ['203.0.113.1', '203.0.113.1'], + ['2001:db8:0:0:1:0:0:1', '2001:db8::1:0:0:1'], + ])('canonicalises %s to %s', (input, expected) => { + // One spelling everywhere: two records of one client that differ only in how the address was written + // are two records. + expect(canonicalIp(input)).toBe(expected); + }); + + it('reports the canonical spelling, not the one it received', () => { + const resolved = resolveClientIp({ + peer: '::ffff:10.0.0.7', + headers: { 'x-forwarded-for': '::FFFF:198.51.100.99' }, + trustedProxy: { peers: ['10.0.0.0/8'] }, + }); + + expect(resolved).toEqual({ ip: '198.51.100.99', source: 'trusted-proxy' }); + }); +}); + +describe('every present field is validated', () => { + it.each([ + ['a non-string header', { peers: ['10.0.0.0/8'], header: 42 }], + ['an empty header', { peers: ['10.0.0.0/8'], header: ' ' }], + ['a negative hop count', { peers: ['10.0.0.0/8'], hops: -1 }], + ['a zero hop count', { peers: ['10.0.0.0/8'], hops: 0 }], + ['a fractional hop count', { peers: ['10.0.0.0/8'], hops: 1.5 }], + ['a non-numeric hop count', { peers: ['10.0.0.0/8'], hops: 'two' }], + ['a non-function predicate', { peers: ['10.0.0.0/8'], isTrusted: 'nope' }], + ])('rejects the whole policy for %s, even with valid peers', (_label, config) => { + // Substituting a default for a malformed value, or dropping it, installs a policy the operator did + // not write — and its behaviour would not reveal which part took effect. + expect(readTrustPolicy(config as never)).toBeNull(); + }); + + it('rejects an explicitly empty peer list', () => { + // An empty list declares that no peer is trusted. Treating it as an absent field, so that a hop count + // supplies the trust instead, inverts what was written. + expect(readTrustPolicy({ peers: [] })).toBeNull(); + expect(readTrustPolicy({ peers: [], hops: 1 })).toBeNull(); + }); + + it('accepts a fully valid policy using every field', () => { + // The positive control: a rule rejecting everything would satisfy the cases above. + const policy = readTrustPolicy({ + peers: ['10.0.0.0/8'], + hops: 2, + header: 'X-Real-IP', + isTrusted: () => false, + }); + + expect(policy).not.toBeNull(); + expect(policy?.header).toBe('x-real-ip'); + expect(policy?.hops).toBe(2); + }); +}); + +describe('an IPv6 zone identifies an interface, so it is kept or refused', () => { + it('keeps the zone in the canonical spelling', () => { + // Dropping it would make two addresses on different interfaces compare equal. + expect(canonicalIp('fe80::1%eth0')).toBe('fe80::1%eth0'); + expect(canonicalIp('FE80::0001%eth0')).toBe('fe80::1%eth0'); + }); + + it('rejects a policy entry carrying a zone', () => { + // A zone plays no part in the numeric comparison, so such an entry would trust those bits on every + // interface — including the one it was written to exclude. + expect(readTrustPolicy({ peers: ['fe80::1%eth0'] })).toBeNull(); + expect(readTrustPolicy({ peers: ['fe80::/10%eth0'] })).toBeNull(); + }); + + it('does not let one zone stand in for another', () => { + const resolved = resolveClientIp({ + peer: 'fe80::1%eth1', + headers: { 'x-forwarded-for': '9.9.9.9' }, + trustedProxy: { peers: ['fe80::1%eth0'] }, + }); + + expect(resolved).toEqual({ ip: 'fe80::1%eth1', source: 'runtime' }); + }); + + it('still matches a zoneless prefix that covers the address', () => { + // A CIDR is about the address bits, so `fe80::/10` covering a scoped peer is correct — it is only an + // entry with an explicit zone that cannot be honoured. + const resolved = resolveClientIp({ + peer: 'fe80::1%eth0', + headers: { 'x-forwarded-for': CLIENT }, + trustedProxy: { peers: ['fe80::/10'] }, + }); + + expect(resolved).toEqual({ ip: CLIENT, source: 'trusted-proxy' }); + }); + + it.each(['fe80::1%eth0%oops', 'fe80::1%%', '::1%a%b'])('rejects more than one zone delimiter (%s)', (value) => { + expect(isIpAddress(value)).toBe(false); + }); +}); + +describe('a policy is read from its own properties only', () => { + it.each([ + ['a misspelled field', { peers: ['10.0.0.0/8'], heder: 'x-real-ip' }], + ['an unrelated field', { peers: ['10.0.0.0/8'], trustProxy: true }], + ['a plausible-looking extra', { peers: ['10.0.0.0/8'], trustedHops: 2 }], + ])('rejects %s rather than ignoring it', (_label, config) => { + // An unrecognised key is a typo, not an extension. Ignoring it uses the default header while the + // operator believes they configured another — the quiet substitution this function exists to avoid. + expect(readTrustPolicy(config as never)).toBeNull(); + }); + + it('does not read a field inherited through the prototype chain', () => { + // An inherited value was not written by whoever configured this object. It is ignored, so the policy + // falls back to the default header rather than adopting one from the prototype. + const config = Object.assign(Object.create({ header: 'x-real-ip' }), { peers: ['10.0.0.0/8'] }); + const policy = readTrustPolicy(config); + + expect(policy).not.toBeNull(); + expect(policy?.header, 'the inherited header must not be adopted').toBe('x-forwarded-for'); + }); + + it.each([ + ['hops', { hops: 1 }], + ['isTrusted', { isTrusted: () => true }], + ['peers', { peers: ['10.0.0.0/8'] }], + ])('does not let an inherited %s be the whole policy', (_field, proto) => { + // With nothing of its own, the object declares no policy at all. + expect(readTrustPolicy(Object.create(proto))).toBeNull(); + }); +}); + +describe('a forwarded header must be sent with the request', () => { + it('ignores a header inherited through the prototype chain', () => { + // Prototype pollution elsewhere in an application must not be able to supply an attributed client + // address. The peer is what the transport observed, so it stands. + const headers = Object.create({ 'x-forwarded-for': CLIENT }); + const resolved = resolveClientIp({ peer: PROXY, headers, trustedProxy: POLICY }); + + expect(resolved).toEqual({ ip: PROXY, source: 'runtime' }); + }); + + it('still reads a header the request actually carried', () => { + // The positive control: a check that rejected every header would satisfy the case above. + const headers = Object.assign(Object.create({ 'x-forwarded-for': '203.0.113.99' }), { + 'x-forwarded-for': CLIENT, + }); + const resolved = resolveClientIp({ peer: PROXY, headers, trustedProxy: POLICY }); + + expect(resolved).toEqual({ ip: CLIENT, source: 'trusted-proxy' }); + }); +}); + +describe('a zone identifier follows a conservative grammar', () => { + it.each(['eth0', 'en0', 'lo', '2', '15', 'eth0.100', 'br-abc123', 'wlan_0'])('accepts %s', (zone) => { + expect(isIpAddress(`fe80::1%${zone}`)).toBe(true); + }); + + it.each([ + ['a newline', 'bad\nzone'], + ['a space', 'a b'], + ['a path traversal', '../etc'], + ['a slash', 'eth0/1'], + ['brackets', 'zone[0]'], + ['non-ASCII', 'ünïcode'], + ['a quote', "zone'"], + ['a semicolon', 'zone;drop'], + ])('rejects %s in a zone', (_label, zone) => { + // A zone reaches logs and retained event payloads, so anything outside the forms real runtimes + // produce is a malformed address rather than an address with decoration. + expect(isIpAddress(`fe80::1%${zone}`)).toBe(false); + }); + + it('rejects a zone longer than the grammar allows', () => { + expect(isIpAddress(`fe80::1%${'a'.repeat(65)}`)).toBe(false); + expect(isIpAddress(`fe80::1%${'a'.repeat(64)}`)).toBe(true); + }); + + it.each([ + // Both spellings of the mapped form. Its canonical form is IPv4, which has no zone to carry, so + // keeping the address would mean discarding the scope — the one thing a zone must never do silently. + '::ffff:203.0.113.1%eth0', + '::FFFF:203.0.113.1%2', + '::ffff:c000:0280%eth0', + ])('refuses a mapped IPv4 address carrying a zone (%s)', (value) => { + expect(isIpAddress(value)).toBe(false); + expect(canonicalIp(value)).toBeNull(); + }); + + it.each([ + ['::ffff:203.0.113.1', '203.0.113.1'], + ['::ffff:c000:0280', '192.0.2.128'], + ])('still accepts the zoneless mapped form %s as %s', (input, expected) => { + // The positive control: refusing every mapped address would satisfy the case above. + expect(canonicalIp(input)).toBe(expected); + }); + + it('never reports an address carrying a rejected zone', () => { + const resolved = resolveClientIp({ peer: 'fe80::1%bad zone' }); + + expect(resolved).toEqual({ ip: null, source: 'unavailable' }); + }); +}); diff --git a/tests/protect/detection-delivery.test.ts b/tests/protect/detection-delivery.test.ts new file mode 100644 index 00000000..f0ae3e36 --- /dev/null +++ b/tests/protect/detection-delivery.test.ts @@ -0,0 +1,1237 @@ +import { describe, it, expect, vi, afterEach, beforeEach } from 'vitest'; +import { clearPulseToken } from '../../src/pulse-token.js'; +import { + byteLength, + createDetectionReporter, + retryDelayMs, + splitToFit, + worthRetrying, +} from '../../src/protect/detections.js'; + +/** + * Delivery, not payload. + * + * A report is evidence, so losing a batch to a restart or a rate limit is worth one more attempt — and a + * retry is only safe if a redelivery cannot be counted twice, and only sane if it is bounded. These + * assert those three together: retried when it is worth it, identified so a duplicate is recognisable, + * and given up on before an app spends itself on an endpoint that will not take it. + */ +const RULE = { id: 'r1', rule_v2: [{ parameter: 'post.title', match: { type: 'contains', value: 'x' } }] }; +const reporterFor = (fetchImpl: unknown, over: Record = {}) => + createDetectionReporter({ + siteUuid: 'site-1', + baseUrl: 'https://x.test/monitor/pulse', + fetchImpl: fetchImpl as typeof fetch, + ...over, + }); + +const one = (r: any, path = '/a') => r.record({ rule: RULE, phase: 'request', mode: 'block', path }); +const settle = async () => { await new Promise((r) => setTimeout(r, 0)); }; +/** Let every scheduled retry run, without waiting out the real backoff. */ +const runRetries = async () => { + for (let i = 0; i < MAX_ATTEMPTS + 1; i++) { + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS); + await Promise.resolve(); + } +}; +const MAX_ATTEMPTS = 4; +const RETRY_CAP_MS = 30_000; +const keysOf = (impl: any) => + impl.mock.calls.map((c: any[]) => (c[1]?.headers ?? {})['Idempotency-Key']).filter(Boolean); + +afterEach(() => { vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllGlobals(); }); + +describe('a batch worth retrying is retried, and only so far', () => { + it('retries a transient refusal and delivers the same batch', async () => { + vi.useFakeTimers(); + let attempts = 0; + const impl = vi.fn(async () => { + attempts++; + + return new Response('{}', { status: attempts < 3 ? 503 : 202 }); + }); + const r = reporterFor(impl); + + one(r); + r.flush(); + await runRetries(); + + expect(attempts, 'it kept trying until the endpoint took it').toBe(3); + // One event, delivered once — the retries are attempts at the same batch, not more events. + expect(r.health()).toMatchObject({ sent: 1, delivered: 1, failed: 0, retried: 2 }); + r.stop(); + }); + + it('sends every attempt of one batch under the same idempotency key', async () => { + vi.useFakeTimers(); + let attempts = 0; + const impl = vi.fn(async () => { + attempts++; + + return new Response('{}', { status: attempts < 3 ? 503 : 202 }); + }); + const r = reporterFor(impl); + + one(r); + r.flush(); + await runRetries(); + + const keys = keysOf(impl); + expect(keys.length, 'every attempt carried a key').toBe(3); + expect(new Set(keys).size, 'and it was the same key each time').toBe(1); + + // A different batch is a different key, or the server would discard it as a duplicate. + one(r, '/b'); + r.flush(); + await runRetries(); + + const all = keysOf(impl); + expect(new Set(all).size, 'the second batch is distinguishable').toBe(2); + r.stop(); + }); + + it('gives up after a bounded number of attempts and counts the loss', async () => { + vi.useFakeTimers(); + const impl = vi.fn(async () => new Response('{}', { status: 503 })); + const r = reporterFor(impl); + + one(r); + r.flush(); + await runRetries(); + + expect(impl.mock.calls.length, 'bounded, not a loop').toBe(MAX_ATTEMPTS); + expect(r.health()).toMatchObject({ sent: 1, delivered: 0, failed: 1 }); + + // And nothing is still scheduled: an exhausted batch leaves no timer behind. + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS * 4); + expect(impl.mock.calls.length, 'no attempt after the bound').toBe(MAX_ATTEMPTS); + r.stop(); + }); + + it('does not retry a refusal that will refuse again', async () => { + vi.useFakeTimers(); + const impl = vi.fn(async () => new Response('{}', { status: 400 })); + const r = reporterFor(impl); + + one(r); + r.flush(); + await runRetries(); + + // A rejected batch is rejected on its merits. Retrying it spends the app's time to be told the same + // thing, where a transient failure has some chance of a different answer. + expect(impl.mock.calls.length, 'one attempt only').toBe(1); + expect(r.health()).toMatchObject({ sent: 1, delivered: 0, failed: 1, retried: 0 }); + r.stop(); + }); + + it('retries when the endpoint could not be reached at all', async () => { + vi.useFakeTimers(); + let attempts = 0; + const impl = vi.fn(async () => { + attempts++; + if (attempts < 2) throw new Error('connection reset'); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl); + + one(r); + r.flush(); + await runRetries(); + + // Unreachable is not refused: nothing has said the endpoint is unwilling. + expect(r.health()).toMatchObject({ delivered: 1, failed: 0, retried: 1 }); + r.stop(); + }); + + it('schedules nothing new once stopped', async () => { + vi.useFakeTimers(); + const impl = vi.fn(async () => new Response('{}', { status: 503 })); + const r = reporterFor(impl); + + one(r); + r.stop(); + // Under fake timers a real `setTimeout` never fires, so the send is drained by advancing them. + await vi.advanceTimersByTimeAsync(1); + const afterStop = impl.mock.calls.length; + + expect(afterStop, 'stopping still sends what was buffered').toBe(1); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS * 4); + // A guard being torn down must not leave a timer holding a process open for a report nobody awaits. + expect(impl.mock.calls.length, 'and does not retry behind the guard').toBe(afterStop); + }); +}); + +describe('the backoff', () => { + it('grows, stays bounded, and is jittered', () => { + // Jitter is what keeps many guards from returning in step after one shared outage. + expect(retryDelayMs(1, null, () => 0.5)).toBe(1000); + expect(retryDelayMs(2, null, () => 0.5)).toBe(2000); + expect(retryDelayMs(3, null, () => 0.5)).toBe(4000); + expect(retryDelayMs(99, null, () => 0.5)).toBe(30_000); + + // Within ±25%, so jitter can neither collapse the delay to nothing nor exceed the cap. + expect(retryDelayMs(1, null, () => 0)).toBe(750); + expect(retryDelayMs(1, null, () => 0.999)).toBeLessThanOrEqual(1250); + expect(retryDelayMs(99, null, () => 0.999)).toBeLessThanOrEqual(30_000); + }); + + it('honours Retry-After, but not past the cap', () => { + // The endpoint saying what it can take beats a guess — capped, so a header cannot park a batch. + expect(retryDelayMs(1, '5')).toBe(5000); + expect(retryDelayMs(1, '99999')).toBe(30_000); + expect(retryDelayMs(1, 'not-a-date'), 'an unusable value falls back to the backoff').toBeGreaterThan(0); + expect(retryDelayMs(1, '-1'), 'and so does a negative one').toBeGreaterThan(0); + }); +}); + +describe('a reporting state supersedes rather than accumulates', () => { + it('declares only the newest state when several arrive before a send', async () => { + const bodies: any[] = []; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body ?? '{}'))); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl); + + r.announce('no-managed-rules'); + r.announce('on'); + await settle(); + + const declared = bodies.filter((b) => typeof b.reporting_state === 'string'); + expect(declared.length, 'one declaration, not two').toBe(1); + expect(declared[0].reporting_state, 'and it is the current state').toBe('on'); + expect(r.health().capability).toMatchObject({ announced: 1, acknowledged: 1 }); + // A declaration carries no events, so it must not move the event counters. + expect(r.health()).toMatchObject({ sent: 0, delivered: 0 }); + r.stop(); + }); + + it('carries a state and the queued events in one request', async () => { + const bodies: any[] = []; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body ?? '{}'))); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl); + + one(r); + r.announce('on'); + await settle(); + + expect(bodies.length, 'one request, not one each').toBe(1); + expect(bodies[0].reporting_state).toBe('on'); + expect(bodies[0].detections.length).toBe(1); + r.stop(); + }); + + it('declares a state that arrives while a send is in flight, once that send finishes', async () => { + const bodies: any[] = []; + let release: (() => void) | null = null; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body ?? '{}'))); + if (bodies.length === 1) await new Promise((r) => { release = r; }); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl); + + r.announce('no-managed-rules'); + await settle(); + expect(bodies.length, 'the first declaration is in flight').toBe(1); + + // Two more while it is held. Only the newest should follow. + r.announce('disabled-by-config'); + r.announce('on'); + release?.(); + await settle(); + await settle(); + + expect(bodies.length, 'one follow-up, not two').toBe(2); + expect(bodies[1].reporting_state).toBe('on'); + r.stop(); + }); +}); + +describe('an event is bounded in size, and says when it was', () => { + const capture = () => { + const bodies: any[] = []; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body ?? '{}'))); + + return new Response('{}', { status: 202 }); + }); + + return { bodies, impl }; + }; + + it('shortens a long route and says so', async () => { + const { bodies, impl } = capture(); + const r = reporterFor(impl); + + r.record({ rule: RULE, phase: 'request', mode: 'block', path: `/${'a'.repeat(500)}` }); + r.flush(); + await settle(); + + const event = bodies[0].detections[0]; + expect(event.route.length, 'capped').toBe(256); + // Without this a reader would take a shortened route for a different route. + expect(event.truncated).toContain('route'); + // And nothing else is claimed: no parameter was left out, so there is no total to report. + expect(event.truncated).not.toContain('parameters'); + expect(Object.hasOwn(event, 'parameters_total')).toBe(false); + }); + + it('keeps a missing route missing rather than turning it into an empty one', async () => { + const { bodies, impl } = capture(); + const r = reporterFor(impl); + + r.record({ rule: RULE, phase: 'request', mode: 'block' } as never); + r.flush(); + await settle(); + + // There being no route is not the same as the route being empty: an empty string reads as a known + // path that happens to be blank. + expect(bodies[0].detections[0].route).toBeNull(); + }); + + it('caps how many parameters an event names, and reports the real count', async () => { + const { bodies, impl } = capture(); + const r = reporterFor(impl); + const broad = { + id: 'broad', + rule_v2: Array.from({ length: 40 }, (_, i) => ({ + parameter: `post.field_${i}`, + match: { type: 'contains', value: 'x' }, + })), + }; + + r.record({ rule: broad, phase: 'request', mode: 'block', path: '/a' }); + r.flush(); + await settle(); + + const event = bodies[0].detections[0]; + expect(event.parameters.length).toBe(25); + expect(event.truncated).toContain('parameters'); + expect(event.parameters_total, 'so a reader knows what was left out').toBe(40); + }); + + it('says nothing about truncation when nothing was truncated', async () => { + const { bodies, impl } = capture(); + const r = reporterFor(impl); + + one(r); + r.flush(); + await settle(); + + const event = bodies[0].detections[0]; + // Absence is not a claim: the field appears only when something really was shortened. + expect(Object.hasOwn(event, 'truncated')).toBe(false); + expect(Object.hasOwn(event, 'parameters_total')).toBe(false); + }); + + it('marks a capped identifier instead of passing a shortened one off as whole', async () => { + const { bodies, impl } = capture(); + const r = reporterFor(impl, { rulesEtag: `"${'e'.repeat(400)}"` }); + + r.record({ + rule: { ...RULE, id: 'r'.repeat(400), rule_revision: 'v'.repeat(400) }, + phase: 'request', + mode: 'block', + path: '/a', + }); + r.flush(); + await settle(); + + const event = bodies[0].detections[0]; + expect(event.rule_id.length).toBe(256); + expect(event.rules_etag.length).toBe(256); + // A shortened identifier no longer names the rule it came from, so saying so is the whole point: a + // reader must not use it as a key believing it is complete. + expect(event.truncated).toContain('rule_id'); + expect(event.truncated).toContain('rules_etag'); + // The rule reads one parameter and the event names it, so a parameter total would be a false claim. + expect(Object.hasOwn(event, 'parameters_total'), 'no parameters were omitted').toBe(false); + r.stop(); + }); +}); + +describe('the byte bound on a batch', () => { + // Distinguishable, or `toEqual` on the remainder cannot tell a reordering from the right order. + const event = (bytes: number, id: number) => ({ rule_id: `r${id}`, route: `/${id}`.padEnd(bytes, 'a') }); + + it('keeps at least one event even when that one exceeds the bound', () => { + // A batch of none makes no progress and would retry forever against the same bound. + const [batch, rest] = splitToFit([event(5000, 1)], 1000); + + expect(batch.length).toBe(1); + expect(rest.length).toBe(0); + }); + + it('sends what fits and returns the rest in order', () => { + const events = [event(400, 1), event(400, 2), event(400, 3), event(400, 4)]; + const [batch, rest] = splitToFit(events, 1000); + + expect(batch.length, 'as many as fit').toBeLessThan(events.length); + expect(JSON.stringify(batch).length).toBeLessThanOrEqual(1000); + expect(batch.length + rest.length, 'nothing is lost in the split').toBe(events.length); + // Order matters: the remainder goes back to the front of the queue, so it must still be in sequence. + expect(rest).toEqual(events.slice(batch.length)); + }); + + it('delivers the remainder in a later batch rather than losing it', async () => { + // The split is only safe if what did not fit comes back. A remainder that is returned and then + // dropped looks identical to a batch that fit, and the events are simply gone. + const bodies: any[] = []; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body ?? '{}'))); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl, { rulesEtag: `"${'e'.repeat(254)}"` }); + // Worst-case events: a full batch of these exceeds the body bound, so the split is reached. + const wide = { + id: 'r'.repeat(256), + rule_revision: 'v'.repeat(256), + rule_v2: Array.from({ length: 25 }, (_, i) => ({ + parameter: `post.${'f'.repeat(58)}${i}`, + match: { type: 'contains', value: 'x' }, + })), + }; + for (let i = 0; i < 50; i++) { + r.record({ rule: wide, phase: 'request', mode: 'block', path: `/${i}`.padEnd(256, 'a') }); + } + + // Drain every batch the queue produces. + for (let i = 0; i < 6; i++) { + r.flush(); + await settle(); + } + + expect(bodies.length, 'it took more than one request').toBeGreaterThan(1); + for (const body of bodies) { + expect(JSON.stringify(body.detections).length, 'each body is under the bound').toBeLessThanOrEqual( + 64 * 1024, + ); + } + const total = bodies.reduce((n, b) => n + b.detections.length, 0); + expect(total, 'every event arrived').toBe(50); + expect(r.health()).toMatchObject({ delivered: 50, failed: 0, dropped: 0 }); + r.stop(); + }); + + it('sends everything when it all fits', () => { + const events = [event(10, 1), event(10, 2)]; + + expect(splitToFit(events, 1000)).toEqual([events, []]); + }); +}); + +describe('stopping leaves nothing outstanding and nothing scheduled', () => { + const refusing = () => vi.fn(async () => new Response('{}', { status: 503 })); + + it('makes a final attempt at a batch that was waiting to retry, and counts it', async () => { + vi.useFakeTimers(); + const impl = refusing(); + const r = reporterFor(impl); + + one(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + expect(impl.mock.calls.length, 'the first attempt failed and a retry is pending').toBe(1); + + r.stop(); + await vi.advanceTimersByTimeAsync(1); + + // Clearing the retry timer alone would leave this batch holding the only slot: never delivered, + // never abandoned, and absent from every counter. + expect(impl.mock.calls.length, 'the waiting batch got one last attempt').toBe(2); + expect(r.health()).toMatchObject({ sent: 1, delivered: 0, failed: 1 }); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS * 4); + expect(impl.mock.calls.length, 'and nothing after it').toBe(2); + }); + + it('accounts for a waiting batch that finally succeeds on the way out', async () => { + vi.useFakeTimers(); + let attempts = 0; + const impl = vi.fn(async () => { + attempts++; + + return new Response('{}', { status: attempts === 1 ? 503 : 202 }); + }); + const r = reporterFor(impl); + + one(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + r.stop(); + await vi.advanceTimersByTimeAsync(1); + + expect(r.health()).toMatchObject({ sent: 1, delivered: 1, failed: 0 }); + }); + + it('drains what is still queued, a batch at a time', async () => { + vi.useFakeTimers(); + const bodies: any[] = []; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body ?? '{}'))); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl); + + // More than one batch's worth, with nothing due, so it is all still queued. + for (let i = 0; i < 60; i++) one(r, `/p${i}`); + r.stop(); + await vi.advanceTimersByTimeAsync(1); + + const total = bodies.reduce((n, b) => n + b.detections.length, 0); + expect(bodies.length, 'more than one batch left').toBeGreaterThan(1); + expect(total, 'and all of it went').toBe(60); + expect(r.health()).toMatchObject({ delivered: 60, dropped: 0 }); + }); + + it('counts what it could not send rather than losing track of it', async () => { + vi.useFakeTimers(); + // A transport that is gone: nothing can be delivered, so the drain has to account for the queue. + const impl = vi.fn(async () => { throw new Error('gone'); }); + const r = reporterFor(impl); + + for (let i = 0; i < 60; i++) one(r, `/p${i}`); + r.stop(); + await vi.advanceTimersByTimeAsync(1); + + const h = r.health(); + // Every recorded event ends up somewhere: delivered, refused, or dropped. None simply disappears. + expect(h.delivered + h.failed + h.dropped, 'all 60 are accounted for').toBe(60); + expect(h.delivered).toBe(0); + }); + + it('does not start new work when a send already in flight completes after stop', async () => { + let release: (() => void) | null = null; + const impl = vi.fn(async () => { + if (impl.mock.calls.length === 1) await new Promise((r) => { release = r; }); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl); + + one(r); + r.flush(); + await settle(); + expect(impl.mock.calls.length, 'one send is in flight').toBe(1); + + // Queued behind it, then stopped while it is still running. + one(r, '/b'); + r.stop(); + release?.(); + await settle(); + await settle(); + + // The drain sends what was queued — deliberately, once each — rather than the completion quietly + // chaining fresh batches behind a stopped guard. + const total = impl.mock.calls.length; + await settle(); + expect(impl.mock.calls.length, 'and then it is finished').toBe(total); + expect(r.health().sent).toBe(2); + }); + + it('ignores a flush or a record that arrives after stopping', async () => { + const impl = vi.fn(async () => new Response('{}', { status: 202 })); + const r = reporterFor(impl); + + one(r); + r.stop(); + await settle(); + const afterStop = impl.mock.calls.length; + expect(afterStop, 'the drain sent what was buffered').toBe(1); + + // `flush` and `record` are public, so they can be called after a guard is torn down. Neither may + // start a new request or arm a new interval behind it. + one(r, '/late'); + r.flush(); + await settle(); + + expect(impl.mock.calls.length, 'nothing new was started').toBe(afterStop); + expect(r.health().sent, 'and the late event was never sent').toBe(1); + }); + + it('accounts for the queue when there is no transport to drain it through', async () => { + // A runtime with no usable `fetch` can still record. Those events go nowhere, so they have to be + // counted somewhere rather than sitting in a queue that appears in no number. + vi.stubGlobal('fetch', undefined); + const r = createDetectionReporter({ siteUuid: 'site-1', baseUrl: 'https://x.test/monitor/pulse' }); + + for (let i = 0; i < 7; i++) one(r, `/p${i}`); + r.stop(); + await settle(); + + const h = r.health(); + expect(h.dropped, 'every unsendable event is accounted for').toBe(7); + expect(h.sent).toBe(0); + expect(h.delivered + h.failed).toBe(0); + }); + + it('abandons an attempt that never settles instead of holding the only slot', async () => { + vi.useFakeTimers(); + const seen: Array = []; + const impl = vi.fn( + (_u: string, init?: RequestInit) => + new Promise((_resolve, reject) => { + seen.push(init?.signal ?? undefined); + init?.signal?.addEventListener('abort', () => reject(new Error('aborted'))); + }), + ); + const r = reporterFor(impl); + + one(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + expect(seen[0], 'the attempt carries a signal').toBeDefined(); + + // Ten seconds is the attempt bound; without it this request holds the single send slot for the life + // of the process and every later event is dropped for pressure. + await vi.advanceTimersByTimeAsync(10_000); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS); + expect(impl.mock.calls.length, 'it was abandoned and retried').toBeGreaterThan(1); + r.stop(); + }); + + it('aborts a request in flight when stopped', async () => { + let aborted = false; + const impl = vi.fn( + (_u: string, init?: RequestInit) => + new Promise((_resolve, reject) => { + init?.signal?.addEventListener('abort', () => { aborted = true; reject(new Error('aborted')); }); + }), + ); + const r = reporterFor(impl); + + one(r); + r.flush(); + await settle(); + r.stop(); + await settle(); + + expect(aborted, 'stopping does not wait on a request that may never answer').toBe(true); + }); +}); + +describe('what counts as worth retrying', () => { + it('retries any server error, not a chosen few', () => { + // The documented contract says a server error is retried. Picking a subset would abandon the rest on + // the first attempt while the documentation said otherwise. + for (const status of [500, 501, 502, 503, 504, 507, 508, 599]) { + expect(worthRetrying(status), `${status} is the endpoint's own fault`).toBe(true); + } + for (const status of [408, 425, 429]) expect(worthRetrying(status)).toBe(true); + expect(worthRetrying(null), 'unreachable says nothing about willingness').toBe(true); + }); + + it('does not retry a refusal on the merits', () => { + for (const status of [400, 401, 403, 404, 409, 413, 422, 200, 302]) { + expect(worthRetrying(status), `${status} would refuse again`).toBe(false); + } + }); +}); + +describe('the byte bound is measured in bytes, on the request that is sent', () => { + it('counts what goes on the wire, not UTF-16 code units', () => { + // A multi-byte character is one code unit and several bytes, so `length` understates the request. + const multi = '☂'.repeat(100); + expect(multi.length).toBe(100); + expect(byteLength(multi), 'three bytes each on the wire').toBe(300); + }); + + it('sizes the whole request, envelope included', () => { + const events = [{ rule_id: 'a' }, { rule_id: 'b' }]; + const wrap = (batch: unknown[]) => ({ detections: batch, dropped: 0, reporting_state: 'on' }); + const bare = byteLength(JSON.stringify({ detections: events })); + const full = byteLength(JSON.stringify(wrap(events))); + + // Measuring the events alone leaves the envelope out, so a body just under the bound goes over it. + expect(full).toBeGreaterThan(bare); + const [batch] = splitToFit(events, full - 1, wrap as never); + expect(batch.length, 'the envelope counted against the bound').toBe(1); + }); + + it('keeps a real request under the bound with multi-byte routes', async () => { + const bodies: string[] = []; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(String(init?.body ?? '')); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl, { rulesEtag: `"${'e'.repeat(254)}"` }); + const wide = { + id: 'r'.repeat(256), + rule_v2: Array.from({ length: 25 }, (_, i) => ({ + parameter: `post.${'f'.repeat(58)}${i}`, + match: { type: 'contains', value: 'x' }, + })), + }; + // Three bytes per character, so a batch sized by characters would be three times the bound. + for (let i = 0; i < 50; i++) { + r.record({ rule: wide, phase: 'request', mode: 'block', path: `/${'☂'.repeat(120)}${i}` }); + } + for (let i = 0; i < 8; i++) { + r.flush(); + await settle(); + } + + expect(bodies.length).toBeGreaterThan(1); + for (const body of bodies) { + // The actual bytes of the actual request. + expect(byteLength(body), 'the request that was sent is under the bound').toBeLessThanOrEqual(64 * 1024); + } + r.stop(); + }); +}); + +describe('a capability retry is not an event retry', () => { + it('counts a retried declaration against capability only', async () => { + vi.useFakeTimers(); + let attempts = 0; + const impl = vi.fn(async () => { + attempts++; + + return new Response('{}', { status: attempts === 1 ? 503 : 202 }); + }); + const r = reporterFor(impl); + + r.announce('on'); + await vi.advanceTimersByTimeAsync(1); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS); + + const h = r.health(); + // A declaration carries no events, so a retry of it describes no delivery of any event. + expect(h.retried, 'no event was retried, because none was sent').toBe(0); + expect(h.capability).toMatchObject({ announced: 1, acknowledged: 1, retried: 1 }); + expect(h.sent).toBe(0); + r.stop(); + }); + + it('counts both when one request carried both', async () => { + vi.useFakeTimers(); + let attempts = 0; + const impl = vi.fn(async () => { + attempts++; + + return new Response('{}', { status: attempts === 1 ? 503 : 202 }); + }); + const r = reporterFor(impl); + + one(r); + r.announce('on'); + await vi.advanceTimersByTimeAsync(1); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS); + + const h = r.health(); + expect(h.retried, 'the events were retried').toBe(1); + expect(h.capability.retried, 'and so was the declaration').toBe(1); + r.stop(); + }); +}); + +describe('a detection is posted authenticated, on a cold cache and after revocation', () => { + // `{secret}-{oauth id}`, the credential shape the exchange parses. + const AUTH = 'the-secret-40-chars-long-ish-value-here-987'; + + /** A transport that exchanges a credential for a token and records what each detection POST carried. */ + const stub = (opts: { tokens?: string[]; detectionStatus?: (n: number) => number } = {}) => { + const tokens = opts.tokens ?? ['jwt-first', 'jwt-second']; + let exchanges = 0; + const posts: Array<{ auth?: string; key?: string }> = []; + const impl = vi.fn(async (url: string, init?: RequestInit) => { + const target = String(url); + const headers = (init?.headers ?? {}) as Record; + if (target.endsWith('/token')) { + const token = tokens[Math.min(exchanges, tokens.length - 1)]; + exchanges++; + + return new Response(JSON.stringify({ access_token: token, expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + } + posts.push({ auth: headers.Authorization, key: headers['Idempotency-Key'] }); + + return new Response('{}', { status: opts.detectionStatus?.(posts.length) ?? 202 }); + }); + + return { impl, posts, exchanges: () => exchanges }; + }; + + beforeEach(() => { clearPulseToken(); }); + afterEach(() => { clearPulseToken(); }); + + it('exchanges a credential when nothing is cached', async () => { + // Boot happens to prime the shared token cache through the rules fetch, so a reporter that could not + // exchange one itself still looked authenticated — until the cache expired or it ran on its own. + const { impl, posts, exchanges } = stub(); + const r = reporterFor(impl, { pulseAuth: AUTH }); + + one(r); + r.flush(); + await settle(); + + expect(exchanges(), 'it obtained a token of its own').toBe(1); + expect(posts.length).toBe(1); + expect(posts[0].auth, 'and the detection went out authenticated').toBe('Bearer jwt-first'); + r.stop(); + }); + + it('exchanges again once the cached token has expired', async () => { + const { impl, posts } = stub({ tokens: ['jwt-short', 'jwt-fresh'] }); + const r = reporterFor(impl, { pulseAuth: AUTH }); + + one(r); + r.flush(); + await settle(); + expect(posts[0].auth).toBe('Bearer jwt-short'); + + // What a long-running guard reaches: the token it holds is past its life. + clearPulseToken(); + one(r, '/b'); + r.flush(); + await settle(); + + expect(posts[1].auth, 'a fresh token, not an unauthenticated request').toBe('Bearer jwt-fresh'); + r.stop(); + }); + + it('discards a revoked token, retries once, and keeps the same idempotency key', async () => { + // A credential can be rotated or revoked before the token's own expiry, so the server's 401 is + // authoritative over our clock. Without this a guard would present a dead token until local expiry + // and every event in between would be refused. + const { impl, posts, exchanges } = stub({ + tokens: ['jwt-revoked', 'jwt-reissued'], + detectionStatus: (n) => (n === 1 ? 401 : 202), + }); + const r = reporterFor(impl, { pulseAuth: AUTH }); + + one(r); + r.flush(); + await settle(); + + expect(posts.length, 'refused once, then sent again').toBe(2); + expect(posts[0].auth).toBe('Bearer jwt-revoked'); + expect(posts[1].auth, 'with a reissued token').toBe('Bearer jwt-reissued'); + expect(exchanges()).toBe(2); + // The redelivery must still be recognisable as the same batch. + expect(posts[1].key, 'the same key as the refused attempt').toBe(posts[0].key); + expect(r.health()).toMatchObject({ sent: 1, delivered: 1, failed: 0 }); + r.stop(); + }); + + it('counts a persistent refusal rather than retrying it forever', async () => { + vi.useFakeTimers(); + const { impl, posts } = stub({ detectionStatus: () => 401 }); + const r = reporterFor(impl, { pulseAuth: AUTH }); + + one(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS * 4); + + // The token path retries a 401 once with a fresh token; a still-refused batch is a refusal on the + // merits, so the outer retry does not repeat it. + expect(posts.length, 'two sends, not an endless series').toBe(2); + expect(r.health()).toMatchObject({ sent: 1, delivered: 0, failed: 1 }); + r.stop(); + }); + + it('bounds the credential exchange by the attempt, not by a longer app-wide setting', async () => { + // The exchange is a separate request, so an attempt's abort does not reach it. Bounded above the + // attempt it would hold the only send slot past the point the attempt was meant to end. + const seen: number[] = []; + const original = AbortSignal.timeout.bind(AbortSignal); + vi.spyOn(AbortSignal, 'timeout').mockImplementation((ms: number) => { + seen.push(ms); + + return original(ms); + }); + const { impl } = stub(); + const r = reporterFor(impl, { pulseAuth: AUTH, timeoutMs: 120_000 }); + + one(r); + r.flush(); + await settle(); + + expect(seen.length, 'the exchange bounded itself').toBeGreaterThan(0); + for (const ms of seen) { + expect(typeof ms, 'a number, or building the bound throws and the token is lost').toBe('number'); + expect(ms, 'never longer than one attempt').toBeLessThanOrEqual(10_000); + } + r.stop(); + }); +}); + +describe('stopping can be awaited', () => { + it('settles only once the outstanding batch has been delivered', async () => { + let release: ((r: Response) => void) | null = null; + const impl = vi.fn( + () => new Promise((resolve) => { release = resolve; }), + ); + const r = reporterFor(impl); + + one(r); + r.flush(); + await settle(); + + let settled = false; + const done = r.stop().then(() => { settled = true; }); + await settle(); + + // A shutdown handler that did not await this would race the last batch against process exit. + expect(settled, 'not while the request is still open').toBe(false); + release?.(new Response('{}', { status: 202 })); + await done; + + expect(settled).toBe(true); + expect(r.health()).toMatchObject({ delivered: 1 }); + }); + + it('settles when there was nothing outstanding', async () => { + const impl = vi.fn(async () => new Response('{}', { status: 202 })); + const r = reporterFor(impl); + + await expect(r.stop()).resolves.toBeUndefined(); + }); + + it('settles after the drain of a multi-batch queue', async () => { + const impl = vi.fn(async () => new Response('{}', { status: 202 })); + const r = reporterFor(impl); + + for (let i = 0; i < 60; i++) one(r, `/p${i}`); + await r.stop(); + + // Awaiting means the queue is finished with, not merely started on. + expect(r.health()).toMatchObject({ delivered: 60, dropped: 0 }); + }); + + it('ends the drain when the budget runs out, rather than only ending the wait', async () => { + vi.useFakeTimers(); + let landLate: ((r: Response) => void) | null = null; + let aborted = false; + const impl = vi.fn( + (_u: string, init?: RequestInit) => + new Promise((resolve) => { + landLate = resolve; + init?.signal?.addEventListener('abort', () => { aborted = true; }); + }), + ); + const r = reporterFor(impl); + + one(r); + one(r, '/b'); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + + let settled = false; + void r.stop().then(() => { settled = true; }); + // A host shutting down has its own deadline, so the wait is bounded. + await vi.advanceTimersByTimeAsync(5_000); + expect(settled, 'the wait ended').toBe(true); + + // And the promise means what it says. Resolving while the request was still open, the queue + // unaccounted and later batches free to follow would have been a claim of completion that had not + // happened. + const atBudget = r.health(); + expect(atBudget.delivered + atBudget.failed + atBudget.dropped, 'everything is accounted for').toBe(2); + // Not asserted here: `stop()` aborts whatever was open when it was called, so this request was + // already abandoned before the budget mattered. The test below covers the batch the budget is for. + expect(aborted, 'the request stop() found was abandoned').toBe(true); + const sendsAtBudget = impl.mock.calls.length; + + // A response landing after the drain ended must not move a number that has already been reported. + landLate?.(new Response('{}', { status: 202 })); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS * 4); + + expect(r.health(), 'the final numbers stayed final').toEqual(atBudget); + expect(impl.mock.calls.length, 'and nothing followed it').toBe(sendsAtBudget); + }); + + it('abandons a batch the drain itself started, when the budget runs out', async () => { + vi.useFakeTimers(); + // `stop()` aborts the request it finds. A LATER batch — one the drain starts on its own — is the one + // only the budget can end, so that is the request this hangs. + const aborts: boolean[] = []; + const impl = vi.fn((_u: string, init?: RequestInit) => { + if (impl.mock.calls.length === 1) return Promise.resolve(new Response('{}', { status: 202 })); + const index = aborts.length; + aborts.push(false); + + return new Promise(() => { + init?.signal?.addEventListener('abort', () => { aborts[index] = true; }); + }); + }); + const r = reporterFor(impl); + + // Two batches' worth: the first goes, the second hangs. + for (let i = 0; i < 60; i++) one(r, `/p${i}`); + const done = r.stop(); + await vi.advanceTimersByTimeAsync(1); + expect(impl.mock.calls.length, 'the drain moved on to a second batch').toBeGreaterThan(1); + + await vi.advanceTimersByTimeAsync(5_000); + await done; + + expect(aborts.some(Boolean), 'the hanging batch was let go of, not just left open').toBe(true); + const h = r.health(); + expect(h.delivered + h.failed + h.dropped, 'and all 60 are accounted for').toBe(60); + }); + + it.each([ + ['an acknowledgement', 202], + ['a refusal worth retrying', 503], + ['a refusal on the merits', 400], + ])('ignores %s that lands after the drain ended', async (_what, status) => { + vi.useFakeTimers(); + let landLate: ((r: Response) => void) | null = null; + const impl = vi.fn( + () => new Promise((resolve) => { landLate = resolve; }), + ); + const r = reporterFor(impl); + + one(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + // The budget only elapses once the timers move, so the wait is started and then advanced. + const done = r.stop(); + await vi.advanceTimersByTimeAsync(5_000); + await done; + + const atBudget = r.health(); + const sends = impl.mock.calls.length; + + // Each outcome takes a different path through the attempt, and none of them may reach a counter or + // schedule a retry once the numbers have been reported as final. + landLate?.(new Response('{}', { status })); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS * 4); + + expect(r.health()).toEqual(atBudget); + expect(impl.mock.calls.length).toBe(sends); + }); + + it('returns the same settled wait when stopped twice', async () => { + const impl = vi.fn(async () => new Response('{}', { status: 202 })); + const r = reporterFor(impl); + + one(r); + const first = r.stop(); + const second = r.stop(); + // The same shutdown, so the same wait — not a fresh promise that settles on its own schedule. + expect(second, 'the same wait, not a new one').toBe(first); + await Promise.all([first, second]); + + // A second stop must not restart a drain or hand back a promise nothing will settle. + expect(impl.mock.calls.length).toBe(1); + }); +}); + +describe('a redelivery after a refused token appears in the numbers', () => { + const AUTH = 'the-secret-40-chars-long-ish-value-here-987'; + + beforeEach(() => { clearPulseToken(); }); + afterEach(() => { clearPulseToken(); }); + + it('counts an authenticated redelivery, apart from a backoff retry', async () => { + let posts = 0; + const impl = vi.fn(async (url: string) => { + if (String(url).endsWith('/token')) { + return new Response(JSON.stringify({ access_token: `jwt-${posts}`, expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + } + posts++; + + return new Response('{}', { status: posts === 1 ? 401 : 202 }); + }); + const r = reporterFor(impl, { pulseAuth: AUTH }); + + one(r); + r.flush(); + await settle(); + + const h = r.health(); + expect(posts, 'the batch was sent twice').toBe(2); + // The credential path sends the second one, so it is not a backoff retry — but it IS a redelivery of + // this batch, and a number that ignored it would say the batch went out once. + expect(h.reauthorized, 'the redelivery is visible').toBe(1); + expect(h.retried, 'and is not confused with a backoff retry').toBe(0); + expect(h).toMatchObject({ sent: 1, delivered: 1 }); + r.stop(); + }); + + it('reports no redelivery when the token was accepted', async () => { + const impl = vi.fn(async (url: string) => + String(url).endsWith('/token') + ? new Response(JSON.stringify({ access_token: 'jwt', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }) + : new Response('{}', { status: 202 }), + ); + const r = reporterFor(impl, { pulseAuth: AUTH }); + + one(r); + r.flush(); + await settle(); + + expect(r.health().reauthorized).toBe(0); + r.stop(); + }); +}); + +describe('termination finalises everything the reporter was holding', () => { + const AUTH = 'the-secret-40-chars-long-ish-value-here-987'; + + beforeEach(() => { clearPulseToken(); }); + afterEach(() => { clearPulseToken(); }); + + it('does not let a credential refresh that lands late move the final numbers', async () => { + vi.useFakeTimers(); + let releaseSecond: ((r: Response) => void) | null = null; + let posts = 0; + const impl = vi.fn(async (url: string) => { + if (String(url).endsWith('/token')) { + return new Response(JSON.stringify({ access_token: `jwt-${posts}`, expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + } + posts++; + if (posts === 1) return new Response('{}', { status: 401 }); + + // The redelivery, held open past the shutdown budget. + return new Promise((resolve) => { releaseSecond = resolve; }); + }); + const r = reporterFor(impl, { pulseAuth: AUTH }); + + one(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + + const done = r.stop(); + await vi.advanceTimersByTimeAsync(5_000); + await done; + + const atBudget = r.health(); + releaseSecond?.(new Response('{}', { status: 202 })); + await vi.advanceTimersByTimeAsync(RETRY_CAP_MS); + + // The redelivery is counted as it is made, under this attempt's epoch — so one completing after the + // numbers were reported as final cannot change them. + expect(r.health(), 'the final numbers stayed final').toEqual(atBudget); + }); + + it('accounts for a state that was still waiting for a request', async () => { + vi.useFakeTimers(); + const impl = vi.fn(() => new Promise(() => {})); + const r = reporterFor(impl); + + one(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + // Queued behind the batch that is now stuck, so it never gets a request of its own. + r.announce('on'); + + const done = r.stop(); + await vi.advanceTimersByTimeAsync(5_000); + await done; + + // "The platform was never told my final state" is exactly what a reader of these numbers is after. + expect(r.health().capability).toMatchObject({ announced: 1, acknowledged: 0, failed: 1 }); + }); + + it('leaves no attempt timer scheduled behind a finished shutdown', async () => { + vi.useFakeTimers(); + const impl = vi.fn(() => new Promise(() => {})); + const r = reporterFor(impl); + + one(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + const whileRunning = vi.getTimerCount(); + + const done = r.stop(); + await vi.advanceTimersByTimeAsync(5_000); + await done; + + // The attempt's own ten-second bound outlives the five-second budget unless termination clears it. + expect(vi.getTimerCount(), 'nothing is still scheduled').toBeLessThan(whileRunning); + expect(vi.getTimerCount()).toBe(0); + }); +}); + +describe('captured evidence cannot carry an event past the body bound', () => { + it('bounds a parameter label a rule left unbounded', async () => { + // A parameter NAME comes from the rule, and rules carry no length limit — so a label alone can push + // an event past the body bound. A batch always sends at least one event, so an event that cannot fit + // could never be delivered at all: the bound has to hold at the wire, whatever produced the capture. + const bodies: string[] = []; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(String(init?.body ?? '')); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl); + const enormous = `post.${'p'.repeat(70_000)}`; + + r.record({ + rule: { id: 'r1', rule_v2: [{ parameter: enormous, match: { type: 'contains', value: 'x' } }] }, + phase: 'request', + mode: 'block', + path: '/a', + capture: { + plan: 'cp2-' + 'f'.repeat(32), + values: [{ parameter: enormous, value: 'v'.repeat(70_000) }], + }, + } as never); + r.flush(); + await settle(); + + expect(bodies.length).toBe(1); + expect(byteLength(bodies[0]), 'the body stayed inside the bound').toBeLessThanOrEqual(64 * 1024); + + const event = JSON.parse(bodies[0]).detections[0]; + expect(event.capture.values[0].parameter.length).toBeLessThanOrEqual(64); + expect(event.capture.values[0].value.length).toBeLessThanOrEqual(512); + // Marked, so a reader does not take a shortened label for the parameter's real name. + expect(event.capture.truncated).toContain('parameter'); + r.stop(); + }); + + it('bounds how many captured values one event can carry', async () => { + const bodies: string[] = []; + const impl = vi.fn(async (_u: string, init?: RequestInit) => { + bodies.push(String(init?.body ?? '')); + + return new Response('{}', { status: 202 }); + }); + const r = reporterFor(impl); + + r.record({ + rule: { id: 'r1', rule_v2: [{ parameter: 'post.a', match: { type: 'contains', value: 'x' } }] }, + phase: 'request', + mode: 'block', + path: '/a', + capture: { + plan: 'cp2-' + 'f'.repeat(32), + // More than any plan permits: the wire gate does not trust what produced the capture. + values: Array.from({ length: 200 }, (_, i) => ({ parameter: `post.f${i}`, value: `v${i}` })), + }, + } as never); + r.flush(); + await settle(); + + const event = JSON.parse(bodies[0]).detections[0]; + expect(event.capture.values.length).toBe(10); + expect(event.capture.truncated).toContain('values'); + r.stop(); + }); +}); diff --git a/tests/protect/detection-payload-contract.test.ts b/tests/protect/detection-payload-contract.test.ts index 3d1139eb..7ed03153 100644 --- a/tests/protect/detection-payload-contract.test.ts +++ b/tests/protect/detection-payload-contract.test.ts @@ -3,6 +3,9 @@ import { readFileSync } from 'node:fs'; import { join, dirname } from 'node:path'; import { fileURLToPath } from 'node:url'; import { createDetectionReporter } from '../../src/protect/detections.js'; +import { RequestResolver } from '../../src/protect/engine/request.js'; +import { normalizeRequest } from '../../src/protect/engine/normalizer.js'; +import { captureValues, derivePlan, planReference } from '../../src/protect/capture-plan.js'; /** * The disclosure, checked against a REAL serialized payload rather than against itself. @@ -31,14 +34,38 @@ const disclosure = readFileSync(join(root, 'AGENT-INSTALL.md'), 'utf8'); const FIELD_DISCLOSURE: Record = { rule_id: /rule id/i, route: /request path/i, + query_keys: /query string's parameter \*\*names\*\*|query string travels as names only/i, + method: /request's method|method/i, + user_agent: /user\s+agent/i, parameters: /parameter names/i, phase: /which phase matched/i, enforced: /whether it was enforced/i, rules_etag: /identifier of the rule bundle/i, rule_revision: /revision of the rule/i, detected_at: /timestamp/i, + client_ip: /client address/i, + client_ip_source: /where that address came from/i, + truncated: /which fields were shortened/i, + capture: /values of the parameters a rule names/i, + parameters_total: /how many parameters the rule reads/i, + query_keys_total: /how many query parameters the request carried/i, }; +/** + * Fields the payload omits rather than sends empty. + * + * `client_ip` is absent when no address could be established, which is the documented behaviour: a + * present-but-empty field reads as a failed lookup of a real address. So the completeness check below + * requires every OTHER documented field, and this one only when there was an address to report. + */ +const CONDITIONAL_FIELDS = new Set([ + 'client_ip', + 'truncated', + 'parameters_total', + 'query_keys_total', + 'capture', +]); + /** Envelope keys, described separately because they are per-batch rather than per-detection. */ const ENVELOPE_DISCLOSURE: Record = { detections: /per matched rule/i, @@ -51,11 +78,42 @@ const ENVELOPE_DISCLOSURE: Record = { * change and this fails. */ const VALUE_EXCLUSIONS = [ - { what: 'the value that matched', sentinel: 'SENTINEL-MATCHED-VALUE', disclosed: /value that matched|matched value/i }, - { what: 'the request body', sentinel: 'SENTINEL-REQUEST-BODY', disclosed: /request body/i }, - { what: 'a header value', sentinel: 'SENTINEL-HEADER-VALUE', disclosed: /header/i }, - { what: 'a cookie value', sentinel: 'SENTINEL-COOKIE-VALUE', disclosed: /cookie/i }, - { what: 'a query-string value', sentinel: 'SENTINEL-QUERY-VALUE', disclosed: /query.string|query string/i }, + { + what: 'the value of a parameter the rule does not name', + sentinel: 'SENTINEL-UNNAMED-FIELD', + disclosed: /the value of any parameter the matched rule does not name/i, + }, + { + what: 'a header value the rule does not name', + sentinel: 'SENTINEL-HEADER-VALUE', + disclosed: /the value of any parameter the matched rule does not name/i, + }, + { + what: 'the request body, where no reviewed opt-in permits it', + sentinel: 'SENTINEL-REQUEST-BODY', + disclosed: /request body, other than the reviewed raw prefix/i, + }, + { + what: 'a response body value', + sentinel: 'SENTINEL-RESPONSE-BODY', + disclosed: /response\*\* values are never captured|any response\s+body, header or status value/i, + }, + { + what: 'a query-string value in the route', + sentinel: 'SENTINEL-QUERY-VALUE', + disclosed: /query.string|query string/i, + }, +]; + +/** + * Values a rule's own plan DOES permit. + * + * Positive controls. Without them every exclusion above could pass because capture was not running at + * all, which is the failure mode a list of absences invites. + */ +const VALUE_INCLUSIONS = [ + { what: 'a named body field', sentinel: 'SENTINEL-NAMED-FIELD' }, + { what: 'a named cookie', sentinel: 'SENTINEL-NAMED-COOKIE' }, ]; /** @@ -107,10 +165,117 @@ async function capturePayload(): Promise<{ raw: string; body: Record }; } +/** + * The same reporter, on a detection large enough to be shortened. + * + * The fields that only appear when something was truncated would otherwise never be emitted here, and + * their disclosure entries would sit in the table above describing a payload this test never produces. + */ +async function captureTruncatedPayload(): Promise> { + let raw = ''; + const fetchImpl = vi.fn(async (_url: string, init: RequestInit) => { + raw = String(init.body); + + return new Response('{}', { status: 202 }); + }); + const reporter = createDetectionReporter({ + siteUuid: 'site-contract', + baseUrl: 'https://api.test/monitor/pulse', + fetchImpl: fetchImpl as unknown as typeof fetch, + }); + + reporter.record({ + rule: { + id: 'PS-CVE-2026-0002', + rule_v2: Array.from({ length: 40 }, (_, i) => ({ + parameter: `post.field_${i}`, + match: { type: 'contains', value: 'x' }, + })), + }, + phase: 'request', + mode: 'block', + // A long route AND more query parameters than one event carries, so every field that appears only + // when something was shortened is actually produced here. + path: `/${'a'.repeat(400)}?${Array.from({ length: 25 }, (_, i) => `k${i}=v${i}`).join('&')}`, + } as never); + reporter.flush(); + await vi.waitFor(() => expect(raw).not.toBe('')); + + return (JSON.parse(raw).detections as Array>)[0]; +} + +/** + * One detection carrying real evidence, taken the way the runtime takes it. + * + * The rule names two of the sentinels below and not the others, so the payload is the boundary itself: + * what a plan permits, against everything planted beside it. + */ +async function captureBearingPayload(): Promise { + const rule = { + id: 'PS-CVE-2026-0003', + rule_v2: [ + { parameter: 'post.title', match: { type: 'contains', value: 'SENTINEL' } }, + { parameter: 'cookie.session', match: { type: 'contains', value: 'SENTINEL' } }, + ], + }; + const req: any = { + method: 'POST', + url: '/checkout/confirm?token=SENTINEL-QUERY-VALUE', + originalUrl: '/checkout/confirm?token=SENTINEL-QUERY-VALUE', + headers: { 'content-type': 'application/json', authorization: 'SENTINEL-HEADER-VALUE' }, + query: {}, + body: { title: 'SENTINEL-NAMED-FIELD', secret: 'SENTINEL-UNNAMED-FIELD' }, + cookies: { session: 'SENTINEL-NAMED-COOKIE' }, + _rawBody: 'SENTINEL-REQUEST-BODY', + _response: { status: 200, body: 'SENTINEL-RESPONSE-BODY', headers: {} }, + }; + + const resolver = new RequestResolver({ ...req, ...normalizeRequest(req) }); + const plan = derivePlan(rule); + const capture = { plan: planReference(plan), ...captureValues(plan, resolver) }; + + let raw = ''; + const fetchImpl = vi.fn(async (_url: string, init: RequestInit) => { + raw = String(init.body); + + return new Response('{}', { status: 202 }); + }); + const reporter = createDetectionReporter({ + siteUuid: 'site-contract', + baseUrl: 'https://api.test/monitor/pulse', + fetchImpl: fetchImpl as unknown as typeof fetch, + }); + + reporter.record({ rule, phase: 'request', mode: 'block', path: req.originalUrl, capture } as never); + reporter.flush(); + await vi.waitFor(() => expect(raw).not.toBe('')); + + return raw; +} + describe('the detection payload matches what AGENT-INSTALL.md says about it', () => { it('describes every field it emits', async () => { const { body } = await capturePayload(); - const detection = (body.detections as Array>)[0]; + const truncatedDetection = await captureTruncatedPayload(); + // Every payload the reporter can emit, merged: a field that appears only when evidence is captured + // would otherwise sit outside this check, which is how an undisclosed field ships. + const withEvidence = (JSON.parse(await captureBearingPayload()).detections as any[])[0]; + const detection = { + ...(body.detections as Array>)[0], + ...truncatedDetection, + ...withEvidence, + }; + + expect(Object.keys(withEvidence), 'a capture-bearing payload names its evidence').toContain('capture'); + + // The conditional fields are only conditional; they still have to be produced somewhere. + expect(Object.keys(truncatedDetection), 'a shortened payload names what it shortened').toContain( + 'truncated', + ); + expect(Object.keys(truncatedDetection)).toContain('parameters_total'); + expect(Object.keys(truncatedDetection), 'and a shortened query list names its total').toContain( + 'query_keys_total', + ); for (const key of Object.keys(detection)) { const pattern = FIELD_DISCLOSURE[key]; @@ -136,7 +301,13 @@ describe('the detection payload matches what AGENT-INSTALL.md says about it', () const { body } = await capturePayload(); const detection = (body.detections as Array>)[0]; - expect(Object.keys(detection).sort()).toEqual(Object.keys(FIELD_DISCLOSURE).sort()); + const expected = Object.keys(FIELD_DISCLOSURE).filter( + (key) => !CONDITIONAL_FIELDS.has(key) || key in detection, + ); + + expect(Object.keys(detection).sort()).toEqual(expected.sort()); + // And the conditional field is absent for the right reason, not missing by accident. + if (!('client_ip' in detection)) expect(detection.client_ip_source).toBe('unavailable'); expect(Object.keys(body).sort()).toEqual(Object.keys(ENVELOPE_DISCLOSURE).sort()); }); @@ -151,8 +322,18 @@ describe('the detection payload matches what AGENT-INSTALL.md says about it', () expect(disclosure, 'the disclosure must say identifiers name their request region').toMatch(/request region/i); }); + it('sends the values its rule named, so the exclusions below mean something', async () => { + const raw = await captureBearingPayload(); + + for (const { what, sentinel } of VALUE_INCLUSIONS) { + expect(raw, `${what} is what the rule was written to inspect`).toContain(sentinel); + } + }); + it('excludes every value it promises to exclude', async () => { - const { raw } = await capturePayload(); + // Driven through the real plan and the real extractor, with a rule that names two fields and not the + // rest. A list of absences taken from a payload where capture never ran would prove nothing. + const raw = await captureBearingPayload(); for (const { what, sentinel, disclosed } of VALUE_EXCLUSIONS) { expect(raw, `${what} must not reach the wire`).not.toContain(sentinel); @@ -160,6 +341,12 @@ describe('the detection payload matches what AGENT-INSTALL.md says about it', () } }); + it('excludes them from a detection carrying no evidence at all, too', async () => { + const { raw } = await capturePayload(); + + for (const { sentinel } of VALUE_EXCLUSIONS) expect(raw).not.toContain(sentinel); + }); + it('keeps the route while dropping the query string, rather than dropping both', async () => { // The control for the query-string sentinel: a reporter that sent no route at all would pass the // exclusion check while losing the field the disclosure describes. diff --git a/tests/protect/detections.test.ts b/tests/protect/detections.test.ts index e9dc6e41..7158af4b 100644 --- a/tests/protect/detections.test.ts +++ b/tests/protect/detections.test.ts @@ -17,7 +17,23 @@ import { createProtection } from '../../src/protect/runtime.js'; const drain = () => new Promise((resolve) => setTimeout(resolve, 0)); /** Everything the payload is allowed to carry, and nothing else. */ -const ALLOWED_KEYS = ['rule_id', 'route', 'parameters', 'phase', 'enforced', 'rules_etag', 'rule_revision', 'detected_at']; +// `client_ip` is not here: it is omitted when no address could be established, which is the case for a +// reporter driven directly with no resolved address. `client_ip_source` is always present, because "this +// could not be established" is the part a reader needs. +const ALLOWED_KEYS = [ + 'rule_id', + 'route', + 'query_keys', + 'method', + 'user_agent', + 'parameters', + 'phase', + 'enforced', + 'rules_etag', + 'rule_revision', + 'client_ip_source', + 'detected_at', +]; const pinnedRule = { id: 'pulse-1', @@ -60,6 +76,7 @@ describe('the detection payload', () => { expect(event).toMatchObject({ rule_id: 'pulse-1', route: '/api/preview', + query_keys: ['url'], parameters: ['server.REQUEST_URI', 'get.url'], phase: 'request', // The point of the channel: this rule did not block, and that is the interesting case. @@ -69,7 +86,7 @@ describe('the detection payload', () => { expect(typeof event.detected_at).toBe('string'); }); - it('never puts a matched value, a query string, a body or a header on the wire', async () => { + it('never puts a matched value, a query-string value or a body on the wire', async () => { // The load-bearing test, and deliberately a scan of the serialized payload rather than of the object // we built: a field added later — `message`, `value`, `headers` — would pass every assertion above // and fail here, which is the direction this needs to fail in. @@ -91,11 +108,20 @@ describe('the detection payload', () => { await drain(); const wire = JSON.stringify(posts[0].body); - for (const forbidden of ['SUPER_SECRET', '169.254.169.254', 'meta-data', '203.0.113.9', 'curl/8.0', 'Blocked by']) { + for (const forbidden of ['SUPER_SECRET', '169.254.169.254', 'meta-data', '203.0.113.9', 'Blocked by']) { expect(wire, `${forbidden} must not reach the reporting endpoint`).not.toContain(forbidden); } - // And the route survived, so the scan above is not passing because nothing was sent. - expect(wire).toContain('/api/preview'); + + // The user agent DOES travel: attribution is what the channel is for, and a detection without it + // cannot be told from another client's. It is a header value, and the only one that is sent. + expect(wire).toContain('curl/8.0'); + + // The query's parameter NAMES travel, and none of its values do — which is what makes the scan above + // meaningful rather than a payload that simply dropped the URL. + const [event] = posts[0].body.detections; + expect(event.route).toBe('/api/preview'); + expect(event.query_keys).toEqual(['url', 'token']); + expect(wire).not.toContain('http://169'); }); it('reports the enforcement state, not the site mode', async () => { @@ -241,12 +267,21 @@ describe('declaring the capability', () => { reportDetections: true, }); - // Authenticated, so the claim carries weight and is made. - const claimed = seen.filter((h) => h['X-Patchstack-Detections'] === 'enabled'); + // Authenticated, so the claim carries weight and is made. The header carries the STATE, not a bit: + // "no events arrived" has several causes, and the platform can only tell them apart if the guard + // names which one it is in. + const claimed = seen.filter((h) => typeof h['X-Patchstack-Detections'] === 'string'); expect(claimed.length).toBeGreaterThan(0); for (const headers of claimed) { expect(headers.Authorization, 'the claim only travels on an authenticated request').toContain('Bearer'); + expect( + ['on', 'no-managed-rules', 'unavailable-no-credential'], + 'the header value is a reporting state', + ).toContain(headers['X-Patchstack-Detections']); } + // The first fetch of a site with no cached bundle honestly reports that it holds no managed rules + // yet; the state that follows the resolution is asserted separately below. + expect(p.detectionReporting).toBe('on'); p.stopRefresh?.(); }); @@ -484,7 +519,7 @@ describe('reporting that cannot be delivered', () => { expect(p.detectionReporting).toBe('unavailable-no-credential'); expect(p.detectionHealth, 'no reporter means no health to report').toBeUndefined(); expect(posted.some((url) => url.includes('/detections/'))).toBe(false); - expect(warnings.some((m) => m.includes('detection reporting is enabled'))).toBe(true); + expect(warnings.some((m) => m.includes('no API credential resolved'))).toBe(true); p.stop(); }); @@ -555,13 +590,18 @@ describe('the reporter can always be reached', () => { await p.fetchGuard()(new Request('https://app.test/api/x?q=boom')); await drain(); - expect(posts.length, 'still buffered — nothing has asked it to flush').toBe(0); + // Counted in EVENTS, not requests: a state announcement is a request carrying no events, and it is + // made once at boot. What this asserts is that no detection has left the buffer yet. + const events = () => posts.flatMap((body: any) => body.detections ?? []); + expect(events().length, 'still buffered — nothing has asked it to flush').toBe(0); p.stop(); await drain(); await drain(); - expect(posts.length).toBe(1); + // Exactly the one buffered event, delivered by the stop. `sent`/`delivered` count events, so the + // state announcement — which carries none — does not move them. + expect(events().length).toBe(1); expect(p.detectionHealth()).toMatchObject({ sent: 1, delivered: 1, failed: 0, dropped: 0 }); expect(p.detectionHealth().lastDeliveredAt).not.toBeNull(); }); @@ -572,7 +612,9 @@ describe('delivery health', () => { // The capability declaration says a guard intends to report. Only an acknowledgement says anything // arrived, and without counting the refusals a delivery path that rejects everything reads the same // as an app where no rule fired. - let status = 500; + // A terminal refusal, so the outcome is settled on the first attempt: a retryable status would be + // retried, and this test is about which counter moves, not about when. + let status = 400; const fetchImpl = vi.fn(async () => new Response('{}', { status })); const reporter = createDetectionReporter({ siteUuid: 'site-1', @@ -673,3 +715,264 @@ describe('the rule revision travels with the detection', () => { for (const event of posts[0].body.detections) expect(event.rule_revision).toBeNull(); }); }); + +describe('every shortened field says so, including the baseline ones', () => { + it('marks a capped method, user agent and query-key list, and gives the key total', async () => { + const { reporter, posts } = reporterWith(); + const manyKeys = Array.from({ length: 25 }, (_, i) => `k${i}=v${i}`).join('&'); + + reporter.record({ + rule: pinnedRule, + phase: 'request', + mode: 'block', + path: `/api/preview?${manyKeys}&${'n'.repeat(200)}=x`, + method: 'M'.repeat(40), + userAgent: 'u'.repeat(600), + } as never); + reporter.flush(); + await drain(); + + const [event] = posts[0].body.detections; + // A cap nobody reports is a cap a reader cannot allow for. + expect(event.truncated).toContain('method'); + expect(event.truncated).toContain('user_agent'); + expect(event.truncated).toContain('query_keys'); + expect(event.query_keys.length).toBe(10); + expect(event.query_keys_total, 'so a short list is not read as a complete one').toBe(26); + expect(event.method.length).toBeLessThanOrEqual(16); + expect(event.user_agent.length).toBeLessThanOrEqual(256); + }); + + it('says nothing about truncation when the baseline fitted', async () => { + const { reporter, posts } = reporterWith(); + + reporter.record({ + rule: pinnedRule, + phase: 'request', + mode: 'block', + path: '/api/preview?url=x', + method: 'GET', + userAgent: 'curl/8.0', + } as never); + reporter.flush(); + await drain(); + + const [event] = posts[0].body.detections; + expect(Object.hasOwn(event, 'truncated')).toBe(false); + expect(Object.hasOwn(event, 'query_keys_total')).toBe(false); + }); +}); + +describe('the wire gate validates what it is given', () => { + const withCapture = (capture: unknown) => ({ + rule: pinnedRule, + phase: 'request', + mode: 'block', + path: '/a', + capture, + }); + + it('counts what it dropped, so eleven values are not mistaken for two hundred', async () => { + const { reporter, posts } = reporterWith(); + + reporter.record( + withCapture({ + plan: 'cp2-abc', + omitted: 3, + values: Array.from({ length: 200 }, (_, i) => ({ parameter: `post.f${i}`, value: `v${i}` })), + }) as never, + ); + reporter.flush(); + await drain(); + + const { capture } = posts[0].body.detections[0]; + expect(capture.values.length).toBe(10); + // What the producer left out, plus what this gate did — in the matching counter. + expect(capture.omitted).toBe(3 + 190); + expect(Object.hasOwn(capture, 'unsupported'), 'nothing here was refused for its type').toBe(false); + expect(capture.truncated).toContain('values'); + }); + + it('refuses a value whose type this channel does not report, rather than coercing it', async () => { + const { reporter, posts } = reporterWith(); + // `String(x)` would run whatever `toString` an object carries, turning a refused value into content. + const hostile = { toString: () => 'SENTINEL-COERCED' }; + + reporter.record( + withCapture({ + plan: 'cp2-abc', + values: [ + { parameter: 'post.a', value: hostile }, + { parameter: 'post.b', value: 'kept' }, + ], + }) as never, + ); + reporter.flush(); + await drain(); + + const wire = JSON.stringify(posts[0].body); + expect(wire).not.toContain('SENTINEL-COERCED'); + const { capture } = posts[0].body.detections[0]; + expect(capture.values).toEqual([{ parameter: 'post.b', value: 'kept' }]); + // Refused for its type, which is a different fact from a bound leaving it out. + expect(capture.unsupported, 'counted as unsupported').toBe(1); + expect(Object.hasOwn(capture, 'omitted'), 'and not as omitted').toBe(false); + }); + + it('refuses a capture whose plan is not a plan', async () => { + const { reporter, posts } = reporterWith(); + + reporter.record(withCapture({ plan: { toString: () => 'SENTINEL-PLAN' }, values: [] }) as never); + reporter.flush(); + await drain(); + + const wire = JSON.stringify(posts[0].body); + expect(wire).not.toContain('SENTINEL-PLAN'); + expect(Object.hasOwn(posts[0].body.detections[0], 'capture')).toBe(false); + }); + + it('reads the capture once, so a getter cannot answer differently twice', async () => { + const { reporter, posts } = reporterWith(); + let reads = 0; + const detection: any = withCapture(undefined); + Object.defineProperty(detection, 'capture', { + get() { + reads += 1; + + return { plan: 'cp2-abc', values: [{ parameter: 'post.a', value: `read-${reads}` }] }; + }, + }); + + reporter.record(detection); + reporter.flush(); + await drain(); + + expect(reads, 'one read, so what was checked is what was sent').toBe(1); + expect(posts[0].body.detections[0].capture.values[0].value).toBe('read-1'); + }); +}); + +describe('the wire gate reads only what the capture itself carries', () => { + const record = (reporter: any, capture: unknown) => + reporter.record({ rule: pinnedRule, phase: 'request', mode: 'block', path: '/a', capture } as never); + + afterEach(() => { + for (const key of ['raw', 'values', 'plan', 'unavailable']) delete (Object.prototype as any)[key]; + }); + + it('does not transmit raw evidence that a prototype supplied', async () => { + // A plan that permitted nothing must transmit nothing. This guard shields applications against + // prototype pollution; its own reporting must not be the way one lands. + const { reporter, posts } = reporterWith(); + (Object.prototype as any).raw = { value: 'SENTINEL-INHERITED-RAW' }; + + record(reporter, { plan: 'cp2-abc' }); + reporter.flush(); + await drain(); + + const { capture } = posts[0].body.detections[0]; + expect(JSON.stringify(posts[0].body)).not.toContain('SENTINEL-INHERITED-RAW'); + expect(Object.hasOwn(capture, 'raw')).toBe(false); + }); + + it('does not transmit values that a prototype supplied', async () => { + const { reporter, posts } = reporterWith(); + (Object.prototype as any).values = [{ parameter: 'post.a', value: 'SENTINEL-INHERITED-VALUE' }]; + + record(reporter, { plan: 'cp2-abc' }); + reporter.flush(); + await drain(); + + expect(JSON.stringify(posts[0].body)).not.toContain('SENTINEL-INHERITED-VALUE'); + expect(Object.hasOwn(posts[0].body.detections[0].capture, 'values')).toBe(false); + }); + + it('does not accept a plan that only a prototype names', async () => { + const { reporter, posts } = reporterWith(); + (Object.prototype as any).plan = 'cp2-inherited'; + + record(reporter, { values: [{ parameter: 'post.a', value: 'v' }] }); + reporter.flush(); + await drain(); + + // No plan of its own means no capture at all: a report that named someone else's policy would be + // worse than one that named none. + expect(Object.hasOwn(posts[0].body.detections[0], 'capture')).toBe(false); + expect(JSON.stringify(posts[0].body)).not.toContain('cp2-inherited'); + }); + + it('does not let a prototype claim a capture was unavailable', async () => { + const { reporter, posts } = reporterWith(); + (Object.prototype as any).unavailable = true; + + record(reporter, { plan: 'cp2-abc', values: [{ parameter: 'post.a', value: 'v' }] }); + reporter.flush(); + await drain(); + + const { capture } = posts[0].body.detections[0]; + expect(capture.values.length, 'the values were read').toBe(1); + expect(Object.hasOwn(capture, 'unavailable'), 'and nothing claimed they were not').toBe(false); + }); + + it('does not let an entry inherit its parameter or value', async () => { + const { reporter, posts } = reporterWith(); + const bare: any = Object.create({ parameter: 'post.inherited', value: 'SENTINEL-INHERITED-ENTRY' }); + + record(reporter, { plan: 'cp2-abc', values: [bare, { parameter: 'post.a', value: 'kept' }] }); + reporter.flush(); + await drain(); + + const { capture } = posts[0].body.detections[0]; + expect(JSON.stringify(posts[0].body)).not.toContain('SENTINEL-INHERITED-ENTRY'); + expect(capture.values).toEqual([{ parameter: 'post.a', value: 'kept' }]); + expect(capture.unsupported).toBe(1); + }); +}); + +describe('reported query names are the names the guard addresses', () => { + const keysFor = async (query: string) => { + const { reporter, posts } = reporterWith(); + + reporter.record({ rule: pinnedRule, phase: 'request', mode: 'block', path: `/a?${query}` } as never); + reporter.flush(); + await drain(); + + return posts[0].body.detections[0]; + }; + + it('reads a plus as a space, as a rule addressing that parameter does', async () => { + // A rule addresses this parameter as `first name`. Reporting `first+name` would name something no + // reviewer could look up. + expect((await keysFor('first+name=x')).query_keys).toEqual(['first name']); + }); + + it('decodes a percent sequence', async () => { + expect((await keysFor('a%20b=1&%2Fslash=2')).query_keys).toEqual(['a b', '/slash']); + }); + + it('leaves an invalid percent sequence as written, rather than dropping the parameter', async () => { + expect((await keysFor('bad%ZZ=1')).query_keys).toEqual(['bad%ZZ']); + }); + + it('names a repeated parameter once, and counts distinct names', async () => { + const event = await keysFor('dup=1&dup=2&dup=3&other=4'); + + expect(event.query_keys).toEqual(['dup', 'other']); + // A parameter repeated three times is one name to look up, so the total describes the list. + expect(Object.hasOwn(event, 'query_keys_total'), 'nothing was left out').toBe(false); + }); + + it('reports a parameter with no value, and skips one with no name', async () => { + expect((await keysFor('flag&=novalue&real=1')).query_keys).toEqual(['flag', 'real']); + }); + + it('carries no names when there is no query at all', async () => { + const { reporter, posts } = reporterWith(); + + reporter.record({ rule: pinnedRule, phase: 'request', mode: 'block', path: '/a' } as never); + reporter.flush(); + await drain(); + + expect(posts[0].body.detections[0].query_keys).toEqual([]); + }); +}); diff --git a/tests/protect/fetch.test.ts b/tests/protect/fetch.test.ts index 42ff5968..970cb941 100644 --- a/tests/protect/fetch.test.ts +++ b/tests/protect/fetch.test.ts @@ -31,7 +31,11 @@ describe('fetch adapter', () => { ); assert.deepStrictEqual(req.query.q, ['1', '2']); assert.strictEqual(req.body.a, 1); - assert.strictEqual(req.ip, '1.2.3.4'); + // A WHATWG Request exposes no transport peer, so a forwarded header alone establishes nothing: it is + // indistinguishable from one the caller wrote. The address is absent and its provenance says why. + assert.strictEqual(req.ip, ''); + assert.strictEqual(req._clientIp.source, 'unavailable'); + assert.strictEqual(req._clientIp.ip, null); assert.strictEqual(req.headers['content-type'], 'application/json'); assert.strictEqual(req._rawBody, JSON.stringify({ a: 1 })); assert.strictEqual(req.originalUrl, '/api?q=1&q=2'); diff --git a/tests/protect/firewall-log.test.ts b/tests/protect/firewall-log.test.ts index 6c1a3fde..f14bdc58 100644 --- a/tests/protect/firewall-log.test.ts +++ b/tests/protect/firewall-log.test.ts @@ -192,3 +192,291 @@ describe('createProtection connector log reporting', () => { protection.stopRefresh?.(); }); }); + +describe('stopping the block log waits for what is outstanding, and is bounded', () => { + const KEY = 'abcdefghijabcdefghijabcdefghijabcdefghij-42'; + const reporter = (fetchImpl: unknown, over: Record = {}) => + createFirewallLogReporter({ + apiKey: KEY, + apiBase: 'https://api.test', + fetchImpl: fetchImpl as typeof fetch, + flushMs: 1, + ...over, + }); + let recorded = 0; + const record = (r: any, n = 1, tag = 'r') => { + for (let i = 0; i < n; i++) { + r.record({ rule: { id: `${tag}${i}` }, method: 'GET', path: `/a${recorded++}`, ip: '1.2.3.4' }); + } + }; + const tick = async () => { await new Promise((r) => setTimeout(r, 5)); }; + + afterEach(() => { vi.useRealTimers(); }); + + it('waits for a send that was already running when it was called', async () => { + // A flush that has already taken its batch leaves an empty queue behind it. A shutdown that looked + // only at the queue would see nothing to wait for while the post was still open. + let releasePost: ((r: Response) => void) | null = null; + const impl = vi.fn(async (url: string) => { + if (String(url).includes('/oauth/token')) { + return new Response(JSON.stringify({ access_token: 'jwt', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + } + + return new Promise((resolve) => { releasePost = resolve; }); + }); + const r: any = reporter(impl); + + record(r); + r.flush(); + await tick(); + expect(releasePost, 'the post is open and the queue is empty').not.toBeNull(); + + let settled = false; + const done = r.stop().then(() => { settled = true; }); + await tick(); + expect(settled, 'the wait found the send the queue no longer knew about').toBe(false); + + releasePost?.(new Response('{}', { status: 200 })); + await done; + expect(settled).toBe(true); + }); + + it('returns the same wait when stopped twice', async () => { + let releasePost: ((r: Response) => void) | null = null; + const impl = vi.fn(async (url: string) => { + if (String(url).includes('/oauth/token')) { + return new Response(JSON.stringify({ access_token: 'jwt', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + } + + return new Promise((resolve) => { releasePost = resolve; }); + }); + const r: any = reporter(impl); + + record(r); + r.flush(); + await tick(); + + // The second call must not hand back a resolved promise while the first drain is still running, nor + // start a second drain behind it: it is the same shutdown, so it is the same wait. + const firstCall = r.stop(); + const secondCall = r.stop(); + expect(secondCall, 'the same wait, not a new one').toBe(firstCall); + + let firstDone = false; + let secondDone = false; + const first = firstCall.then(() => { firstDone = true; }); + const second = secondCall.then(() => { secondDone = true; }); + await tick(); + + expect(firstDone || secondDone, 'neither has finished yet').toBe(false); + releasePost?.(new Response('{}', { status: 200 })); + await Promise.all([first, second]); + expect(firstDone && secondDone).toBe(true); + }); + + it.each([ + ['the token exchange', 'token'], + ['the log post', 'post'], + ])('aborts %s of a send that started before the shutdown did', async (_what, phase) => { + vi.useFakeTimers(); + // The sends worth ending have already started by the time a shutdown begins, so a signal created at + // that point reaches none of them. This starts the flush FIRST, which is the case the tracking set + // was added for in the first place. + const aborted: string[] = []; + const impl = vi.fn((url: string, init?: RequestInit) => { + const isToken = String(url).includes('/oauth/token'); + if (isToken && phase === 'post') { + return Promise.resolve( + new Response(JSON.stringify({ access_token: 'jwt', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }), + ); + } + + return new Promise(() => { + init?.signal?.addEventListener('abort', () => { aborted.push(isToken ? 'token' : 'post'); }); + }); + }); + const r: any = reporter(impl); + + record(r); + r.flush(); + await vi.advanceTimersByTimeAsync(1); + expect(impl.mock.calls.length, 'the send is already running').toBeGreaterThan(0); + + let settled = false; + void r.stop().then(() => { settled = true; }); + await vi.advanceTimersByTimeAsync(5_000); + await vi.advanceTimersByTimeAsync(1); + + expect(settled, 'the wait is bounded').toBe(true); + expect(aborted, `${phase} was let go of`).toContain(phase); + }); + + it('does not resolve while records buffered behind an armed interval are still going out', async () => { + vi.useFakeTimers(); + // Send A is open. Records B sit in the queue with the flush interval armed behind them. A drain that + // waited on the sends it happened to find at the start would resolve the moment A finished — while + // the interval had quietly taken B and started a send of its own. + const posts: Array<{ release: (r: Response) => void; body: string }> = []; + const impl = vi.fn((url: string, init?: RequestInit) => { + if (String(url).includes('/oauth/token')) { + return Promise.resolve( + new Response(JSON.stringify({ access_token: 'jwt', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }), + ); + } + + return new Promise((resolve) => { + posts.push({ release: resolve, body: String(init?.body ?? '') }); + }); + }); + // Short enough that the interval fires during the drain. + const r: any = reporter(impl, { flushMs: 10 }); + + record(r, 50, 'a'); // A: reaches the batch bound and starts a send + await vi.advanceTimersByTimeAsync(1); + record(r, 5, 'b'); // B: queued, with the interval now armed behind them + expect(posts.length, 'A is in flight').toBe(1); + + let settled = false; + const done = r.stop().then(() => { settled = true; }); + + // The interval's moment passes FIRST, while A is still open — that is the whole race. An interval + // left armed takes B here, starting a send the drain never learns about. + await vi.advanceTimersByTimeAsync(50); + + // Nothing took B: from the moment a shutdown begins, the drain owns the queue. Were the interval + // still armed it would have started B's send here, outside the drain's knowledge. + expect(posts.length, 'the interval did not take B').toBe(1); + + // A finishes. A drain waiting on the sends it found at the start is now satisfied, and would resolve + // with B's send open. + posts[0].release(new Response('{}', { status: 200 })); + await vi.advanceTimersByTimeAsync(5); + + expect(posts.length, "B's post has started").toBe(2); + expect(settled, 'and the shutdown is still waiting for it').toBe(false); + expect(posts[1].body, 'B is what is being sent').toContain('b0'); + + posts[1].release(new Response('{}', { status: 200 })); + await vi.advanceTimersByTimeAsync(1); + await done; + expect(settled).toBe(true); + }); + + it('runs nothing more once the shutdown has given up', async () => { + vi.useFakeTimers(); + let releaseToken: ((r: Response) => void) | null = null; + const impl = vi.fn((url: string) => { + if (String(url).includes('/oauth/token')) { + if (releaseToken) { + return Promise.resolve( + new Response(JSON.stringify({ access_token: 'jwt', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }), + ); + } + + return new Promise((resolve) => { releaseToken = resolve; }); + } + + return Promise.resolve(new Response('{}', { status: 200 })); + }); + // A long interval, so records stay in the queue instead of being swept into sends: the case here + // needs BOTH a send that is stuck and records still waiting behind it. + const r: any = reporter(impl, { flushMs: 60_000 }); + + record(r, 50); // reaches the batch bound and starts a send, which hangs on the token exchange + await vi.advanceTimersByTimeAsync(1); + record(r, 20); // and these stay queued + expect(impl.mock.calls.length, 'one send is stuck').toBe(1); + + const done = r.stop(); + await vi.advanceTimersByTimeAsync(5_000); + await done; + const callsAtBudget = impl.mock.calls.length; + + // The transport answers after the shutdown reported itself finished. Ending the WAIT but leaving the + // work alive would let the drain resume here and post the twenty records still queued behind it. + releaseToken?.( + new Response(JSON.stringify({ access_token: 'jwt', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }), + ); + await vi.advanceTimersByTimeAsync(1_000); + + expect(impl.mock.calls.length, 'nothing ran after the shutdown finished').toBe(callsAtBudget); + // And a later flush cannot restart it either. + r.flush(); + await vi.advanceTimersByTimeAsync(1_000); + expect(impl.mock.calls.length).toBe(callsAtBudget); + }); + + it('gives up on a transport that never answers, and lets go of both phases', async () => { + vi.useFakeTimers(); + const aborted: string[] = []; + const impl = vi.fn( + (url: string, init?: RequestInit) => + new Promise(() => { + init?.signal?.addEventListener('abort', () => { + aborted.push(String(url).includes('/oauth/token') ? 'token' : 'post'); + }); + }), + ); + const r: any = reporter(impl); + + record(r); + let settled = false; + void r.stop().then(() => { settled = true; }); + await vi.advanceTimersByTimeAsync(1); + expect(settled, 'still waiting on the token exchange').toBe(false); + + // A hung transport would otherwise keep a shutdown pending for as long as the process lived. + await vi.advanceTimersByTimeAsync(5_000); + await vi.advanceTimersByTimeAsync(1); + expect(settled, 'the wait is bounded').toBe(true); + expect(aborted, 'and the open phase was let go of').toContain('token'); + }); + + it('aborts a hanging post, not only a hanging token exchange', async () => { + vi.useFakeTimers(); + const aborted: string[] = []; + const impl = vi.fn((url: string, init?: RequestInit) => { + if (String(url).includes('/oauth/token')) { + return Promise.resolve( + new Response(JSON.stringify({ access_token: 'jwt', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }), + ); + } + + return new Promise(() => { + init?.signal?.addEventListener('abort', () => { aborted.push('post'); }); + }); + }); + const r: any = reporter(impl); + + record(r); + let settled = false; + void r.stop().then(() => { settled = true; }); + await vi.advanceTimersByTimeAsync(1); + await vi.advanceTimersByTimeAsync(5_000); + await vi.advanceTimersByTimeAsync(1); + + expect(settled, 'bounded on the post too').toBe(true); + expect(aborted).toContain('post'); + }); +}); diff --git a/tests/protect/refresh-backoff.test.ts b/tests/protect/refresh-backoff.test.ts index e43caf3a..2258a3ea 100644 --- a/tests/protect/refresh-backoff.test.ts +++ b/tests/protect/refresh-backoff.test.ts @@ -129,7 +129,9 @@ describe('the refresh loop', () => { reportManifest: false, }); - expect(await p.refresh()).toEqual({ ok: true }); + // Exact, including the origin: a successful refresh took the rules from the platform on this call, + // which is what makes the site's detections attributable to a managed rule. + expect(await p.refresh()).toEqual({ ok: true, origin: 'api' }); state.fail = true; const failed = await p.refresh(); diff --git a/tests/protect/reporting-runtime.test.ts b/tests/protect/reporting-runtime.test.ts new file mode 100644 index 00000000..a9dfe2b6 --- /dev/null +++ b/tests/protect/reporting-runtime.test.ts @@ -0,0 +1,374 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import { createProtection } from '../../src/protect/runtime.js'; + +/** + * Reporting at the runtime seam: is it on by default for a managed site, does each opt-out reach it, does + * the platform learn the state, and does it follow a refresh. + * + * The state calculator is covered exhaustively elsewhere. What these cover is the wiring — a correct + * calculator that the runtime never consults, or consults once at boot, produces exactly the failure the + * state exists to prevent: a managed site that silently never reports, or an unmanaged one that does. + */ +const AUTH = 'the-secret-40-chars-long-ish-value-here-987'; +const RULES = { + firewall: [{ id: 'r1', title: 't', rule_v2: [{ parameter: 'get.q', match: { type: 'contains', value: 'boom' } }] }], + whitelists: [], + enforcement: 'dry-run', +}; + +const drain = async () => { await new Promise((r) => setTimeout(r, 5)); }; + +/** A fetch stub that serves rules, and records the capability header of every rules request. */ +function stubFetch(opts: { rulesOk?: boolean; etag?: string; refuseDetections?: boolean } = {}) { + const capabilityHeaders: Array = []; + const posted: string[] = []; + const bodies: any[] = []; + let rulesOk = opts.rulesOk ?? true; + + const impl = vi.fn(async (url: string, init?: RequestInit) => { + const target = String(url); + // The credential is exchanged for a short-lived token before the rules fetch. Without answering this, + // the rules request carries no Authorization — and the capability header only travels on an + // authenticated request, so every capability assertion would fail for the wrong reason. + if (target.includes('token')) { + return new Response(JSON.stringify({ access_token: 'jwt-abc', expires_in: 3600 }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + } + if (target.includes('/detections/')) { + posted.push(target); + bodies.push(JSON.parse(String(init?.body ?? '{}'))); + + return new Response('{}', { + // Terminal, so a refusal is counted on the first attempt. Retry behaviour has its own tests. + status: opts.refuseDetections ? 400 : 200, + headers: { 'Content-Type': 'application/json' }, + }); + } + capabilityHeaders.push((init?.headers as Record)?.['X-Patchstack-Detections']); + if (!rulesOk) throw new Error('rules unreachable'); + + return new Response(JSON.stringify(RULES), { + status: 200, + headers: { 'Content-Type': 'application/json', ETag: opts.etag ?? '"v1"' }, + }); + }); + + vi.stubGlobal('fetch', impl); + + return { capabilityHeaders, posted, bodies, setRulesOk: (v: boolean) => { rulesOk = v; } }; +} + +afterEach(() => { + vi.unstubAllGlobals(); + // In afterEach, not at the end of a test body: an assertion that fails would otherwise leak a + // stubbed variable into every test after it. + vi.unstubAllEnvs(); +}); + +describe('reporting is on by default for a managed site', () => { + it('needs no config flag', async () => { + // The behaviour that changed. Nothing here asks for reporting: an enrolled site running rules the + // platform delivered, with a credential, reports. + const { posted } = stubFetch(); + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + detectionFlushMs: 1, + }); + + expect(p.detectionReporting).toBe('on'); + + await p.fetchGuard()(new Request('https://app.test/api/x?q=boom')); + p.stop(); + await drain(); + await drain(); + + expect(posted.length, 'an event reached the endpoint').toBeGreaterThan(0); + }); + + it('sends nothing for a local install running its own rules', async () => { + // A bare install: no site identity, so nothing to report against and no endpoint to report to. + const p: any = await createProtection({ rules: RULES, mode: 'dry-run', detectionFlushMs: 1 }); + + expect(p.detectionReporting).toBe('not-enrolled'); + expect(p.detectionHealth).toBeUndefined(); + p.stop(); + }); + + it('sends nothing for a site identity whose rules are not the platform’s', async () => { + // Enrolled-looking, but the rules in force are the caller's own, so a detection could not be + // attributed to a managed rule document. + const { posted, setRulesOk } = stubFetch(); + setRulesOk(false); + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + rules: RULES, + detectionFlushMs: 1, + onError: () => {}, + }); + + expect(p.detectionReporting).toBe('no-managed-rules'); + + await p.fetchGuard()(new Request('https://app.test/api/x?q=boom')); + p.stop(); + await drain(); + + expect(posted).toEqual([]); + }); +}); + +describe('each opt-out reaches the runtime', () => { + it.each([ + ['PATCHSTACK_REPORT_DETECTIONS', 'disabled-by-config'], + ['PATCHSTACK_TELEMETRY', 'disabled-by-telemetry-opt-out'], + ])('%s=0 switches reporting off, and says which switch did it', async (name, expected) => { + const { posted } = stubFetch(); + vi.stubEnv(name, '0'); + + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + detectionFlushMs: 1, + }); + + expect(p.detectionReporting).toBe(expected); + + await p.fetchGuard()(new Request('https://app.test/api/x?q=boom')); + p.stop(); + await drain(); + + expect(posted, 'an opt-out means no events leave the process').toEqual([]); + }); + + it('honours reportDetections: false as an opt-out', async () => { + const { posted } = stubFetch(); + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + reportDetections: false, + detectionFlushMs: 1, + }); + + expect(p.detectionReporting).toBe('disabled-by-config'); + await p.fetchGuard()(new Request('https://app.test/api/x?q=boom')); + p.stop(); + await drain(); + + expect(posted).toEqual([]); + }); +}); + +describe('the platform learns the state', () => { + it('carries the state on the rules request, not a bit', async () => { + const { capabilityHeaders } = stubFetch(); + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + }); + + const sent = capabilityHeaders.filter((v): v is string => typeof v === 'string'); + expect(sent.length).toBeGreaterThan(0); + // Never the legacy bit: a boolean cannot say which of the reasons applies. + expect(sent).not.toContain('enabled'); + p.stop(); + }); + + it('carries the opt-out state, rather than saying nothing', async () => { + // Silence would leave the platform unable to tell an opted-out site from one that never installed. + const { capabilityHeaders } = stubFetch(); + vi.stubEnv('PATCHSTACK_REPORT_DETECTIONS', '0'); + + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + }); + + expect(capabilityHeaders).toContain('disabled-by-config'); + p.stop(); + }); +}); + +describe('the settled state reaches the platform', () => { + it('is acknowledged when resolution settles somewhere other than the fetch declared', async () => { + // A site booting with an empty cache declares that it holds no managed rules, then receives them on + // that same request. Without an acknowledgement the platform keeps the pre-resolution answer — and a + // guard with refreshing switched off never sends another rules request. + const { capabilityHeaders, bodies } = stubFetch(); + + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + detectionFlushMs: 1, + }); + await drain(); + await drain(); + + // What the fetch carried, and what it settled on: different, which is why the acknowledgement exists. + expect(capabilityHeaders).toContain('no-managed-rules'); + expect(p.detectionReporting).toBe('on'); + + const announcements = bodies.filter((b) => typeof b.reporting_state === 'string'); + expect(announcements.map((b) => b.reporting_state)).toContain('on'); + // It carries no events — its only content is the state. + for (const body of announcements) expect(body.detections).toEqual([]); + + // And it is accounted for separately. The event counters are measured in events, so an announcement + // moving them would produce readings that describe no real delivery — `sent: 0` with `failed: 1` — and + // would make an acknowledgement look like a delivered detection. + const health = p.detectionHealth(); + expect(health.capability, 'the announcement is counted as a capability, not an event').toMatchObject({ + announced: 1, + acknowledged: 1, + failed: 0, + }); + expect(health.capability.lastAcknowledgedAt).not.toBeNull(); + expect( + { sent: health.sent, delivered: health.delivered, failed: health.failed, lastDeliveredAt: health.lastDeliveredAt }, + 'no event has been delivered, so the event counters have not moved', + ).toEqual({ sent: 0, delivered: 0, failed: 0, lastDeliveredAt: null }); + + p.stop(); + }); + + it('counts a refused announcement against capability, not against events', async () => { + // The failure direction of the same separation: a refused announcement must not appear as a refused + // detection, which is what `failed` counts. + const { bodies } = stubFetch({ refuseDetections: true }); + + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + detectionFlushMs: 1, + }); + await drain(); + await drain(); + + expect(bodies.filter((b) => typeof b.reporting_state === 'string').length).toBe(1); + + const health = p.detectionHealth(); + expect(health.capability).toMatchObject({ announced: 1, acknowledged: 0, failed: 1 }); + expect(health.failed, 'no event was refused, because none was sent').toBe(0); + expect(health.sent).toBe(0); + + p.stop(); + }); + + it('is not acknowledged when the fetch already declared the settled state', async () => { + // A site whose store already holds a platform bundle declares `on` before the fetch and settles on + // `on`, so there is nothing to correct and no extra request to make. + let cached: unknown = { bundle: { firewall: [], whitelists: [], whitelist_keys: {} }, etag: '"v0"' }; + const { bodies } = stubFetch(); + + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + detectionFlushMs: 1, + ruleCache: { read: () => cached, write: (e: unknown) => { cached = e; } }, + }); + await drain(); + await drain(); + + expect(p.detectionReporting).toBe('on'); + expect(bodies.filter((b) => typeof b.reporting_state === 'string')).toEqual([]); + + p.stop(); + }); +}); + +describe('reporting follows a refresh', () => { + it('stops when an opt-out appears under a running guard', async () => { + // The mirror of recovery, and the direction that matters more: a guard that keeps reporting after + // reporting is switched off is collecting retained evidence nobody asked it for. The state is read + // afresh on each refresh rather than fixed at boot, so an operator who sets the variable and waits + // for the next refresh gets what they asked for without a restart. + // + // Losing MANAGED status mid-process is not the case tested here: once a fetch has succeeded, the + // guard holds the platform's rules in its memory tier, so they remain managed and `cache` is the + // correct answer. The opt-out is the transition that is actually reachable. + const { posted, capabilityHeaders } = stubFetch(); + + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + detectionFlushMs: 1, + }); + + expect(p.detectionReporting).toBe('on'); + expect(p.detectionHealth).toBeTypeOf('function'); + + const beforeRefresh = capabilityHeaders.length; + vi.stubEnv('PATCHSTACK_REPORT_DETECTIONS', '0'); + await p.refresh(); + + expect(p.detectionReporting, 'the state follows the opt-out').toBe('disabled-by-config'); + expect(p.detectionHealth, 'and the health surface goes with it').toBeUndefined(); + // And the request made BY that refresh says so. Carrying the previously reported state would mean the + // platform learns of the opt-out only on the refresh after this one — or never, if there is none. + expect( + capabilityHeaders.slice(beforeRefresh), + 'the refresh request carries the state as of that request', + ).toContain('disabled-by-config'); + + const before = posted.length; + await p.fetchGuard()(new Request('https://app.test/api/x?q=boom')); + p.stop(); + await drain(); + await drain(); + + expect(posted.length, 'no event is sent after reporting stops').toBe(before); + }); + + it('starts once a refresh receives platform rules', async () => { + // The recovery path. A guard that started on its own bundle because the first fetch failed must begin + // reporting when the platform becomes reachable — not stay silent for the life of the process. + const { posted, bodies, setRulesOk } = stubFetch(); + setRulesOk(false); + + const p: any = await createProtection({ + siteUuid: 'site-1', + pulseRulesUrl: 'https://x.test/monitor/pulse', + pulseAuth: AUTH, + rules: RULES, + detectionFlushMs: 1, + onError: () => {}, + }); + + expect(p.detectionReporting).toBe('no-managed-rules'); + expect(p.detectionHealth).toBeUndefined(); + + setRulesOk(true); + const status = await p.refresh(); + await drain(); + + expect(status).toMatchObject({ ok: true, origin: 'api' }); + expect(p.detectionReporting, 'the state follows the refresh').toBe('on'); + expect(p.detectionHealth, 'and so does the health surface').toBeTypeOf('function'); + // And the platform is told. The refresh request declared `no-managed-rules` and resolution settled on + // `on`, which is the same mismatch the boot path corrects — a guard whose only refresh is this + // one-shot call would otherwise leave the platform on the pre-resolution answer for good. + expect( + bodies.filter((b) => typeof b.reporting_state === 'string').map((b) => b.reporting_state), + 'the settled state is acknowledged after a recovering refresh', + ).toContain('on'); + + await p.fetchGuard()(new Request('https://app.test/api/x?q=boom')); + p.stop(); + await drain(); + await drain(); + + expect(posted.length, 'events flow after recovery').toBeGreaterThan(0); + }); +}); diff --git a/tests/protect/reporting-state.test.ts b/tests/protect/reporting-state.test.ts new file mode 100644 index 00000000..87192c3b --- /dev/null +++ b/tests/protect/reporting-state.test.ts @@ -0,0 +1,159 @@ +import { describe, it, expect } from 'vitest'; +import { + REPORTING_STATES, + explainReportingState, + reportingState, +} from '../../src/protect/reporting-state.js'; + +/** + * Every combination that decides whether a guard reports security events. + * + * Enumerated rather than sampled. The states are the difference between retained evidence being + * collected and not, and between a dashboard saying "nothing matched" and "reporting is off" — so each + * input combination has one defined answer and there is no combination without one. + */ +type Input = Parameters[0]; + +const base: Input = { siteUuid: 'site-1', ruleOrigin: 'api', hasCredential: true, env: {} }; + +describe('reporting state', () => { + it('reports only for an enrolled site running managed rules with a credential', () => { + expect(reportingState(base)).toEqual({ state: 'on', reports: true }); + }); + + it('treats cached platform rules as managed', () => { + // Excluding `cache` would silence reporting for sites whose delivery is degraded — the ones whose + // evidence is most worth having. + expect(reportingState({ ...base, ruleOrigin: 'cache' })).toEqual({ state: 'on', reports: true }); + }); + + it.each([ + ['bundled', 'no-managed-rules'], + ['empty', 'no-managed-rules'], + ] as const)('does not report when the rules came from %s', (origin, expected) => { + // No managed rule document exists to attribute a detection to. + expect(reportingState({ ...base, ruleOrigin: origin })).toEqual({ state: expected, reports: false }); + }); + + it('blames the credential, not the rule origin, when both are missing', () => { + // A missing credential is what CAUSES managed rules to be missing: the fetch is refused and + // resolution falls back to the caller's bundle or to nothing. Reporting the origin first would send + // an operator looking for an enrolment they already have. + expect( + reportingState({ ...base, hasCredential: false, ruleOrigin: 'empty' }).state, + ).toBe('unavailable-no-credential'); + }); + + it.each(['', undefined, null, 42, {}])('does not report without a site identity (%s)', (siteUuid) => { + expect(reportingState({ ...base, siteUuid } as Input)).toEqual({ state: 'not-enrolled', reports: false }); + }); + + it('does not report without a credential, and says so distinctly', () => { + // Distinct from "off": the deployment intends to report and cannot, which is a delivery problem + // rather than a choice. + expect(reportingState({ ...base, hasCredential: false })).toEqual({ + state: 'unavailable-no-credential', + reports: false, + }); + }); + + it.each(['0', 'false', 'off', 'no', 'FALSE', 'Off'])( + 'honours PATCHSTACK_REPORT_DETECTIONS=%s', + (value) => { + expect(reportingState({ ...base, env: { PATCHSTACK_REPORT_DETECTIONS: value } })).toEqual({ + state: 'disabled-by-config', + reports: false, + }); + }, + ); + + it.each(['', undefined, '1', 'true', 'on', 'yes', 'anything-else'])( + 'does not read PATCHSTACK_REPORT_DETECTIONS=%s as an opt-out', + (value) => { + // An unset or empty variable is the default, not a choice; and only the false-ish words switch it + // off, so a deployment setting it to any other value is not silently disabling evidence. + expect(reportingState({ ...base, env: { PATCHSTACK_REPORT_DETECTIONS: value } }).reports).toBe(true); + }, + ); + + it('keeps the two opt-outs distinguishable', () => { + // An operator who set one variable must not be told to check the other. + expect(reportingState({ ...base, env: { PATCHSTACK_TELEMETRY: '0' } }).state).toBe( + 'disabled-by-telemetry-opt-out', + ); + expect(reportingState({ ...base, env: { PATCHSTACK_REPORT_DETECTIONS: '0' } }).state).toBe( + 'disabled-by-config', + ); + }); + + it('reports an explicit opt-out ahead of a missing credential', () => { + // The order is the meaning: a deployment that switched reporting off should be told that is why, + // not that it lacks a credential it never needed. + expect( + reportingState({ + ...base, + hasCredential: false, + siteUuid: undefined, + env: { PATCHSTACK_REPORT_DETECTIONS: '0' }, + }).state, + ).toBe('disabled-by-config'); + }); + + it.each([true, false, undefined])('honours configOptOut=%s as an opt-out only', (configOptOut) => { + // The programmatic flag can switch reporting off. It must never switch it on: whether a site is + // managed is the platform's answer, and a guard that could self-declare it would report against rule + // ids the platform never issued. + const offSite = { siteUuid: undefined, ruleOrigin: 'bundled', hasCredential: false, env: {}, configOptOut } as Input; + const onSite = { ...base, configOptOut } as Input; + + expect(reportingState(offSite).reports, 'an unmanaged site never reports').toBe(false); + expect(reportingState(onSite).reports).toBe(configOptOut !== true); + if (configOptOut === true) { + expect(reportingState(onSite).state).toBe('disabled-by-config'); + } + }); + + it('has exactly one answer for every combination of inputs', () => { + // Exhaustive over the axes. A combination with no defined state would surface as reporting silently + // on or silently off depending on which check happened to fall through. + const origins = ['api', 'cache', 'bundled', 'empty', undefined] as const; + const envs = [ + {}, + { PATCHSTACK_REPORT_DETECTIONS: '0' }, + { PATCHSTACK_TELEMETRY: '0' }, + { PATCHSTACK_REPORT_DETECTIONS: '0', PATCHSTACK_TELEMETRY: '0' }, + ]; + let count = 0; + + for (const siteUuid of ['site-1', '', undefined]) { + for (const ruleOrigin of origins) { + for (const hasCredential of [true, false]) { + for (const env of envs) { + for (const configOptOut of [true, false, undefined]) { + const result = reportingState({ siteUuid, ruleOrigin, hasCredential, env, configOptOut } as Input); + count++; + + expect(REPORTING_STATES).toContain(result.state); + // `reports` is true for exactly one state, so the two can never disagree. + expect(result.reports).toBe(result.state === 'on'); + // And an opt-out is absolute: no other input combination can override it. + if (configOptOut === true) expect(result.reports).toBe(false); + } + } + } + } + } + + expect(count).toBe(3 * 5 * 2 * 4 * 3); + }); + + it('explains every state it can produce', () => { + // A state name is a label; an operator asking why nothing arrived needs the sentence. + for (const state of REPORTING_STATES) { + const explanation = explainReportingState(state); + + expect(explanation.length).toBeGreaterThan(20); + expect(explanation).not.toContain('Unrecognised'); + } + }); +}); diff --git a/tests/protect/rule-contract.test.ts b/tests/protect/rule-contract.test.ts index b7d29760..f792af07 100644 --- a/tests/protect/rule-contract.test.ts +++ b/tests/protect/rule-contract.test.ts @@ -412,7 +412,12 @@ describe('what the delivered-bundle validator does with it', () => { // a quirk of those three fields rather than a rule about documents. It now covers all of them. const leaf = { parameter: 'get.q', match: { type: 'contains', value: 'x' } }; - for (const property of ruleContract().rule_properties) { + const { rule_properties, null_exempt_properties } = ruleContract(); + + for (const property of rule_properties) { + // Read from the contract, not written out here. A consumer has only the artifact, so an exception + // this test knew and the artifact did not would be an exception nobody else could honour. + if (null_exempt_properties.includes(property)) continue; // `rule_v2` last would be overwritten by the spread, so the null goes last and wins for every one. expect(ruleReasonFor({ rule_v2: [leaf], [property]: null }), property) .toMatch(/present but null/); @@ -423,6 +428,15 @@ describe('what the delivered-bundle validator does with it', () => { expect(ruleReasonFor({ rule_v2: [leaf] })).toBeNull(); expect(ruleReasonFor({ phase: 'request', action: 'block', rule_v2: [leaf] })).toBeNull(); + // The other side of the same policy: every exemption the artifact publishes really is exempt, and + // there is at least one — a published list nothing honours would be worse than no list. + expect(null_exempt_properties.length).toBeGreaterThan(0); + for (const property of null_exempt_properties) { + // These authorise collection rather than protection, so a malformed one costs evidence and never + // the mitigation. Dropping the rule would trade a working shield for a piece of metadata. + expect(ruleReasonFor({ rule_v2: [leaf], [property]: null }), property).toBeNull(); + } + // ...and it is a rule about the PROPERTY being present, not about nulls appearing anywhere in the // document. A null inside a condition is judged by the condition's own rules. expect(reasonFor({ parameter: 'get.q', match: { type: 'in_array', value: [1, null] } })) diff --git a/tests/protect/rule-origin.test.ts b/tests/protect/rule-origin.test.ts new file mode 100644 index 00000000..5fb358d9 --- /dev/null +++ b/tests/protect/rule-origin.test.ts @@ -0,0 +1,127 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import { resolveRules } from '../../src/protect/rules/source.js'; + +/** + * Where the rules in force came from. + * + * This is the fact detection reporting is gated on: security events are collected for sites the platform + * manages, so a guard must be able to say whether the rules that produced a hit were the platform's. + * Mislabelling the caller's own bundle as platform-delivered would start collecting retained evidence + * for a site that never enrolled — and mislabelling the other way would silently collect nothing for one + * that did. + * + * `ok` and `origin` answer different questions and both are asserted: `ok` is whether resolution was + * clean, `origin` is which leg supplied the rules that are now running. A degraded resolution that fell + * back to cache is `ok: false` with `origin: 'cache'` — still managed rules. + */ +const BUNDLE = { + firewall: [{ id: 'r1', title: 't', rule_v2: [{ parameter: 'get.q', match: { type: 'contains', value: 'x' } }] }], + whitelists: [], +}; + +/** A store that starts empty unless primed, and records what was written. */ +function memoryStore(initial: unknown = null) { + let held: any = initial; + + return { + read: async () => held, + write: async (next: any) => { held = next; }, + get held() { return held; }, + }; +} + +const ok = (body: unknown, etag = '"v1"') => + new Response(JSON.stringify(body), { status: 200, headers: { 'Content-Type': 'application/json', ETag: etag } }); + +afterEach(() => { vi.unstubAllGlobals(); }); + +describe('the origin of the rules in force', () => { + it('is `api` when the platform delivered them on this call', async () => { + vi.stubGlobal('fetch', vi.fn(async () => ok({ ...BUNDLE, enforcement: 'dry-run' }))); + + const res: any = await resolveRules({ siteUuid: 's1', pulseRulesUrl: 'https://x.test/p' }, memoryStore()); + + expect(res.source).toEqual({ ok: true, origin: 'api' }); + }); + + it('is `cache` when the platform revalidated with no change', async () => { + // 304: the running rules are the platform's, taken from the store rather than the wire. + vi.stubGlobal('fetch', vi.fn(async () => new Response(null, { status: 304 }))); + const store = memoryStore({ bundle: BUNDLE, etag: '"v1"' }); + + const res: any = await resolveRules({ siteUuid: 's1', pulseRulesUrl: 'https://x.test/p' }, store); + + expect(res.source).toEqual({ ok: true, origin: 'cache' }); + }); + + it('is `cache` when the fetch failed and last-known-good applied', async () => { + // Degraded, and still managed. Reporting stays on for exactly these sites: their evidence is the + // most worth having. + vi.stubGlobal('fetch', vi.fn(async () => { throw new Error('unreachable'); })); + const store = memoryStore({ bundle: BUNDLE, etag: '"v1"' }); + + const res: any = await resolveRules( + { siteUuid: 's1', pulseRulesUrl: 'https://x.test/p', onError: () => {} }, + store, + ); + + expect(res.source.origin).toBe('cache'); + expect(res.source.ok, 'a fallback is not a clean resolution').toBe(false); + }); + + it('is `bundled` when the fetch failed and the caller supplied its own rules', async () => { + // The platform never saw these rules, so it has no document to attribute a detection to. + vi.stubGlobal('fetch', vi.fn(async () => { throw new Error('unreachable'); })); + + const res: any = await resolveRules( + { siteUuid: 's1', pulseRulesUrl: 'https://x.test/p', rules: BUNDLE, onError: () => {} }, + memoryStore(), + ); + + expect(res.source.origin).toBe('bundled'); + }); + + it('is `bundled` when no live source is configured at all', async () => { + // A local install running its own rules: the common unenrolled shape. + const res: any = await resolveRules({ rules: BUNDLE }, memoryStore()); + + expect(res.source).toEqual({ ok: true, origin: 'bundled' }); + }); + + it('is `empty` when there is nothing anywhere', async () => { + const res: any = await resolveRules({}, memoryStore()); + + expect(res.source).toEqual({ ok: true, origin: 'empty' }); + expect(res.firewall).toEqual([]); + }); + + it('is `empty` when the fetch failed with no cache and no bundle', async () => { + vi.stubGlobal('fetch', vi.fn(async () => { throw new Error('unreachable'); })); + + const res: any = await resolveRules( + { siteUuid: 's1', pulseRulesUrl: 'https://x.test/p', onError: () => {} }, + memoryStore(), + ); + + expect(res.source.origin).toBe('empty'); + expect(res.source.ok).toBe(false); + }); + + it('never reports an origin the reporting gate does not know', async () => { + // The gate treats `api` and `cache` as managed and everything else as not. An origin outside the set + // would fall to "not managed" and silently disable reporting for a managed site. + const known = new Set(['api', 'cache', 'bundled', 'empty']); + vi.stubGlobal('fetch', vi.fn(async () => ok(BUNDLE))); + + for (const options of [ + {}, + { rules: BUNDLE }, + { siteUuid: 's1', pulseRulesUrl: 'https://x.test/p' }, + { siteUuid: 's1', pulseRulesUrl: 'https://x.test/p', rules: BUNDLE }, + ]) { + const res: any = await resolveRules(options, memoryStore()); + + expect(known, `origin was ${res.source?.origin}`).toContain(res.source?.origin); + } + }); +});