diff --git a/CHANGELOG.md b/CHANGELOG.md index 03f5e78..ea57f88 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## [1.32.0] - 2026-09-04 + +- Added an isolated consented Kingston Food Help profile on `/metrics/event`, with exact production origins, strict field/value rejection, GPC/DNT suppression, bounded input and scope-separated fail-closed minute abuse control. No raw event, identity, provider or journey history is stored. +- Added migration 0016 for constrained daily counts and separate page-view-only attribution, atomic writes and 400-day aggregate retention. Existing cron cadence is unchanged; pruning is independently fail-soft. +- Added protected `view=kfh` with strict shared contract 1.0, complete UTC comparison windows, nullable unavailable data and observed-only coverage. Raw-event fleet/source-health and existing site/CEO behavior remain separate and unchanged. +- Added producer fixtures and deterministic privacy, request, real SQLite/atomicity, retention, auth and reporting tests. Added pinned test-only `sql.js`; production runtime dependencies are unchanged. +- Review candidate only: no remote migration, Worker promotion, secret operation, live collection, report request or Discord interaction performed. The website producer remains disabled pending its separately reviewed change. Existing CEO consumer-first rollout restrictions remain in force. + + ## [1.31.0] - 2026-08-27 - Advanced only the strict CEO report contract from `1.1` to `1.2`; `metric_definition_version` remains `1.1`, so metric definitions, window boundaries, counts, and historical meaning do not reset. diff --git a/KFH_ANALYTICS_CONTRACT.md b/KFH_ANALYTICS_CONTRACT.md new file mode 100644 index 0000000..548670d --- /dev/null +++ b/KFH_ANALYTICS_CONTRACT.md @@ -0,0 +1,83 @@ +# Kingston Food Help analytics contract + +Version: ingestion `1`, report `1.0`; Lighthouse `1.32.0` review candidate, 2026-09-04. + +The owner authorized staging and review PRs. No migration, deployment, secret operation or website collection has been activated. Kingston's approved product policy remains the privacy ceiling; this implementation intentionally collects less context. Lighthouse owns measurements; Agent Smith owns the private `/kfh` presentation. + +## Ingestion + +`POST /metrics/event`, with `Origin` exactly `https://kingstonfoodhelp.ca` or `https://www.kingstonfoodhelp.ca`. These origins are registered with the dedicated `kfh_daily` reporting profile and `event_only` support class. No Pages preview, HTTP, localhost, wildcard or foreign origin is accepted. CORS does not allow credentials for Kingston. The browser must send with credentials omitted and no referrer. + +The Kingston-origin path reads at most 1,024 UTF-8 bytes and accepts only this exact common object: + +```json +{"site_key":"kingston_food_help","contract_version":1,"consent":true,"page":"directory","event_name":"page_view","source":"facebook","campaign":"launch_2026_09","content":"post_01"} +``` + +| Event | Allowed event value | Daily counter | +| --- | --- | --- | +| `page_view` | No `event_value` field | `page_views` | +| `contact_click` | `resource_call` | `resource_calls` | +| `contact_click` | `help_211` | `help_211` | +| `outbound_click` | `directions` | `directions` | +| `outbound_click` | `official_source` | `official_sources` | +| `pwa_install` | No `event_value` field | `pwa_installs` | + +Only page views may include `source`, `campaign` or `content`. The exact allowlists are: + +- Source: `direct_unknown`, `facebook`, `community`, `search`, `other`. Missing defaults to `direct_unknown`. +- Campaign: `none`, `launch_2026_09`. Missing defaults to `none`. +- Content: `none`, `post_01`, `poster_01`. Missing defaults to `none`. + +These are public campaign/creative labels, never group names, neighborhood tags or personalized links. Additional campaign labels require a governed change to both pinned contracts and the database constraint. Source, campaign and content are separate marginal counts: no join among these dimensions or with an action is retained. + +Unknown keys, unknown values, malformed JSON, wrong site/version/page, absent affirmative consent, or explicit HTTP `Sec-GPC: 1` / `DNT: 1` are dropped. Kingston-origin requests cannot impersonate an existing raw-event site. A Kingston site key from any other origin also cannot enter raw-event storage. No resource identifier/name, telephone, address, search, filter, eligibility, typed content, coordinates, URL, referrer, client timestamp, viewport, language, timezone, country, identity, test flag, device or request identifier is accepted. Server time supplies only the UTC day. + +The producer must still enforce default-off affirmative consent, GPC/DNT overriding older preferences, presence-based `dev_mode` suppression, `noAnalytics` suppression, production-origin-only emission, and a non-blocking site. There is no server-side proof that a consent assertion or event comes from a human. No offline queue, retry, delayed replay, service-worker caching or background tracking is permitted. The website producer and public consent wording are not part of this backend PR and remain disabled. + +## Storage and failure behavior + +Migration `0016_add_kfh_daily.sql` creates `kfh_daily(day, metric, value, count)`, a constrained aggregate-only table with no raw event rows. Event increments and the three page-view dimension increments use one atomic D1 batch. Action rows have no attribution. SQL constraints reject unsupported dimensions and values. + +Counting requires a client IP, the existing rate-limit secret, a non-ignored IP, and allowance under 50 events per IP per UTC minute. A keyed HMAC includes the Kingston scope and minute; raw IPs are never persisted. This is an approximate abuse control, not deduplication, identity or a unique-person measurement. Repeated clicks may count, and shared-network activity can be undercounted. The keyed bucket exists only in the existing rate table for about two days. An aggregation failure can consume rate allowance without recording activity. + +Kingston stores zero raw accepted or dropped events. Daily aggregates retain the current UTC day and previous 399 days. The existing daily cron gains an independently fail-soft prune task; cadence and other tasks are unchanged. Reports exclude older days even if cleanup has been delayed. Pruning is not performed by a report read. + +All public ingestion responses remain empty `204`, including rejected and failed submissions. `204` is transport completion, not a persistence acknowledgement. Missing rate storage or aggregate storage drops the event. Kingston failure logs are static and contain no payload, error object, IP, secret or raw context. + +## Protected report + +`GET /report?view=kfh` uses the existing report-read authorization. The optional GET-only `X-Report-Token` path and compatible admin path retain existing collision/fail-closed rules. The response is `Cache-Control: no-store`; there is no browser read-token CORS expansion. + +The exact shared TypeScript contract and strict runtime validator are `src/kfhContract.ts`, copied unchanged into Agent Smith `src/contracts/kfhContract.ts`. Representative producer fixtures are `contracts/kfh-v1/{sample,empty,unavailable}.json`. Contract copies and fixtures must be updated and reviewed together. The report never falls back to BUS Core, TGC or CEO data. + +| Window | UTC day span relative to generation day | +| --- | --- | +| `today` | Today, partial as of `generated_at` | +| `latest_complete_day` | Yesterday | +| `last_7_complete_days` | Days -7 through -1 | +| `previous_7_complete_days` | Days -14 through -8 | +| `last_30_complete_days` | Days -30 through -1 | + +Each window has exact dates, an explicit partial flag, and all six counters. A successful empty query yields zero **observed** counts and `no_observed_history`; failed, incompatible or missing storage yields null counts, null discovery and `query_failed`. These are not zero audience or outage assertions. First/last observed days describe retained activity only, not launch time, complete coverage, source health or reachability. + +`discovery_last_7_complete_days` reports only page-view source/campaign/content totals. Each dimension independently reconciles to the page-view total. Strict validation rejects unknown fields, unsafe numbers, wrong site/version, conflicting windows and invented coverage. Every report explicitly limits interpretation to observed consented activity; clicks are not completed calls, visits, installs, households or food received. There are no visitor counts, sessions, retention, engagement duration, conversion rates or population estimates. + +The dedicated report does not perform Cloudflare traffic refresh, probe, snapshot or outbound posting. It catches Kingston storage failure locally. Unexpected outer report-assembly failure retains Lighthouse's existing possible global error-counter side effect, so production reads still require explicit scope. + +Kingston is deliberately excluded from raw-event `view=fleet` and `view=source_health`; legacy and `view=site` selectors reject its key. This preserves those contracts without manufacturing raw-event zeroes or a heartbeat. Use `view=kfh` for Kingston. CEO contracts and existing clients are unchanged. + +## Review, rollout and rollback + +1. Review both PRs and the matching contract/fixtures. Run typecheck, the existing suites, new privacy/SQL/command tests, and Smith governance/bundle checks. +2. Separately approve and verify migration 0016 on the intended D1 before any Lighthouse promotion. Local SQL tests are SQLite WASM through a D1-shaped adapter; they do not prove remote D1 deployment or native Worker lifecycle. +3. Preserve the existing CEO consumer-first parity gate. Smith already contains the 0.26.1 validator patch; verify the required consumer deployment before promoting a Lighthouse version that emits CEO 1.2. +4. Coordinate owner-approved Smith merge/deployment/Discord registration and Lighthouse promotion. Smith's main merge can auto-deploy; Lighthouse publication may upload a preview/version. No PR merge, manual promotion, migration or live verification is performed by this review work. +5. If separately approved, provision Smith's optional GET-only read credential from the existing Lighthouse capability. Do not rotate/remove the broad credential still used for monthly snapshots. +6. Review and verify the separate default-off website producer/consent change before collecting anything. Cloudflare Web Analytics remains off. After authorized deployment, validate rejected payloads and controlled aggregate persistence without claiming a 204 proves storage. + +Rollback is an explicitly approved Worker/consumer operation, not a destructive database change. Retain the additive table; prior workers ignore it. Disable website emission first if it was activated. An older Lighthouse returns an unavailable `/kfh` product in Smith; existing BUS Core/CEO rollback compatibility remains intact. Daily retained aggregates cannot be attributed to individual visitors or selectively removed by an identity that is never stored. + +## Verification ownership + +`tests/kfh-analytics.test.mjs` covers strict ingestion, privacy rejection, real SQLite constraints/atomic rollback, route suppression/rate failure, report windows, null/zero honesty, retention, auth and no external refresh. Existing tests cover preserved clients and CEO contracts. `sql.js` is a pinned test-only dependency; production runtime dependencies are unchanged. diff --git a/OPERATIONS.md b/OPERATIONS.md index efe54fc..73a00aa 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -1,5 +1,11 @@ # Lighthouse Operations and Diagnostics +## Kingston review candidate — 2026-09-04 + +The new isolated Kingston path is governed by [KFH_ANALYTICS_CONTRACT.md](KFH_ANALYTICS_CONTRACT.md). Source version is `1.32.0`. No production state is inferred from this branch. `view=kfh` reads only stored Kingston aggregates and skips traffic refresh; no live read is performed. Migration 0016 requires separate approval/remote verification before Worker promotion. Existing CEO consumer-parity and external Workers Builds verification gates remain in force. + +The website emitter remains disabled. Cloudflare/Discord/D1 production access was unavailable and no endpoint, control-plane, secret or storage operation was attempted. Repository review publication is separately authorized and may trigger CI or non-promoting preview/version uploads. + - Status: current operational runbook - Scope: Lighthouse analytics access, evidence interpretation, incident diagnosis, and release-control boundaries - Repository baseline: Lighthouse `1.31.0` local governed bundle; CEO response contract `1.2`, metric-definition contract `1.1`; production promotion pending diff --git a/README.md b/README.md index 23d9957..4c2d4cc 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # buscore-lighthouse +Kingston Food Help review candidate: see [KFH_ANALYTICS_CONTRACT.md](KFH_ANALYTICS_CONTRACT.md) for the isolated aggregate report, privacy bounds and owner-controlled rollout. No live collection is enabled by this repository change. + ## 1.31.0 CEO activity truth and sparse probe health Version 1.31.0 advances the strict CEO response contract to `1.2` while keeping `metric_definition_version: "1.1"`. Direct aggregate sources now declare `freshness_basis: "activity"`: `data_through` is an activity watermark, freshness stays `unknown`, and `activity_only` prevents a quiet sparse source from being presented as a fresh, stale, healthy, or failed scheduled feed. A single-component watermark is its latest trusted observation; a composite watermark is the conservative earliest required-component watermark. A day-bucket watermark is normalized to a bounded reporting timestamp and is not an event timestamp. Only `service_probes` uses `freshness_basis: "scheduled_probe"` and the existing 36-hour scheduled fresh/stale rule. diff --git a/SOT.md b/SOT.md index b8de56e..e340ad1 100644 --- a/SOT.md +++ b/SOT.md @@ -1,5 +1,16 @@ # Lighthouse — Source of Truth +## Kingston Food Help — v1.32.0 review candidate (2026-09-04) + +The owner authorized a review branch and PR for Kingston analytics. This is an approved proposed behavior change, not a production receipt. `KFH_ANALYTICS_CONTRACT.md` governs this isolated aggregate profile and its Agent Smith `/kfh` consumer. + +- `POST /metrics/event` accepts the strict consented `kingston_food_help` profile only from the two production HTTPS origins. Unknown fields, resource context, identifiers, unapproved campaign tags, test traffic, GPC/DNT, and missing rate-control inputs are dropped. The profile uses six fixed action counters plus separate page-view-only source/campaign/content counts; it writes no raw events. +- Migration `0016_add_kfh_daily.sql` adds one aggregate table with 400 UTC-day-bucket retention. Existing two-day minute-HMAC abuse storage is reused with a Kingston-specific scope. Existing cron cadence is unchanged; Kingston pruning is an independent fail-soft task. +- Protected `GET /report?view=kfh` reads only Kingston aggregates and skips traffic refresh. It returns explicit UTC windows, nullable unavailable measurements, observed-only coverage and activity days, never a health/people/help-received claim. It has no CEO dependency or scheduled outbound delivery. +- Kingston is registered as `event_only` with a dedicated daily-report profile. It is excluded from raw-event fleet/source-health reports; legacy/site selectors reject its key rather than manufacture raw-event zeroes. Existing sites and CEO report contracts are preserved. +- Review publication does not authorize migration, Worker promotion, secret operations, website collection, or live report reads. Migration must be separately approved and verified before promotion. The previous CEO consumer-parity/promotion gate still applies. The website emitter remains disabled pending its separately reviewed producer change. + + ## CEO activity truth and sparse probe health — v1.31.0 Version 1.31.0 advances only the CEO response contract from `1.1` to `1.2`; `metric_definition_version` remains `1.1`. The change prevents sparse activity from being presented as a scheduled health signal, makes partial scheduled-probe evidence truthful, and clarifies that voluntary-inquiry totals count unique lead records. The authenticated route remains `GET /report?view=ceo`. No other report view, route, auth rule, header, CORS permission, Worker binding, environment variable, D1 table, retention rule, cron, probe target, canary, or producer contract changes. diff --git a/contracts/kfh-v1/empty.json b/contracts/kfh-v1/empty.json new file mode 100644 index 0000000..a3526f0 --- /dev/null +++ b/contracts/kfh-v1/empty.json @@ -0,0 +1,93 @@ +{ + "view": "kfh", + "report_contract_version": "1.0", + "site_key": "kingston_food_help", + "generated_at": "2026-09-04T12:00:00.000Z", + "source": { + "availability": "available", + "reason": "no_observed_history", + "first_observed_day": null, + "last_observed_day": null + }, + "windows": { + "today": { + "start_day": "2026-09-04", + "end_day": "2026-09-04", + "partial": true, + "counts": { + "page_views": 0, + "resource_calls": 0, + "help_211": 0, + "directions": 0, + "official_sources": 0, + "pwa_installs": 0 + } + }, + "latest_complete_day": { + "start_day": "2026-09-03", + "end_day": "2026-09-03", + "partial": false, + "counts": { + "page_views": 0, + "resource_calls": 0, + "help_211": 0, + "directions": 0, + "official_sources": 0, + "pwa_installs": 0 + } + }, + "last_7_complete_days": { + "start_day": "2026-08-28", + "end_day": "2026-09-03", + "partial": false, + "counts": { + "page_views": 0, + "resource_calls": 0, + "help_211": 0, + "directions": 0, + "official_sources": 0, + "pwa_installs": 0 + } + }, + "previous_7_complete_days": { + "start_day": "2026-08-21", + "end_day": "2026-08-27", + "partial": false, + "counts": { + "page_views": 0, + "resource_calls": 0, + "help_211": 0, + "directions": 0, + "official_sources": 0, + "pwa_installs": 0 + } + }, + "last_30_complete_days": { + "start_day": "2026-08-05", + "end_day": "2026-09-03", + "partial": false, + "counts": { + "page_views": 0, + "resource_calls": 0, + "help_211": 0, + "directions": 0, + "official_sources": 0, + "pwa_installs": 0 + } + } + }, + "discovery_last_7_complete_days": { + "sources": [], + "campaigns": [], + "contents": [] + }, + "limitations": { + "coverage": "observed_only", + "counts_are": "consented_activity_not_people_or_service_outcomes", + "attribution": "page_views_only_no_action_join", + "activity_is_health": false, + "raw_events_stored": false, + "identifiers_reported": false, + "aggregate_retention_days": 400 + } +} diff --git a/contracts/kfh-v1/sample.json b/contracts/kfh-v1/sample.json new file mode 100644 index 0000000..4586bb5 --- /dev/null +++ b/contracts/kfh-v1/sample.json @@ -0,0 +1,112 @@ +{ + "view": "kfh", + "report_contract_version": "1.0", + "site_key": "kingston_food_help", + "generated_at": "2026-09-04T12:00:00.000Z", + "source": { + "availability": "available", + "reason": "observed_activity", + "first_observed_day": "2026-08-22", + "last_observed_day": "2026-09-04" + }, + "windows": { + "today": { + "start_day": "2026-09-04", + "end_day": "2026-09-04", + "partial": true, + "counts": { + "page_views": 3, + "resource_calls": 2, + "help_211": 0, + "directions": 3, + "official_sources": 0, + "pwa_installs": 0 + } + }, + "latest_complete_day": { + "start_day": "2026-09-03", + "end_day": "2026-09-03", + "partial": false, + "counts": { + "page_views": 8, + "resource_calls": 2, + "help_211": 0, + "directions": 3, + "official_sources": 0, + "pwa_installs": 0 + } + }, + "last_7_complete_days": { + "start_day": "2026-08-28", + "end_day": "2026-09-03", + "partial": false, + "counts": { + "page_views": 17, + "resource_calls": 4, + "help_211": 0, + "directions": 6, + "official_sources": 0, + "pwa_installs": 0 + } + }, + "previous_7_complete_days": { + "start_day": "2026-08-21", + "end_day": "2026-08-27", + "partial": false, + "counts": { + "page_views": 4, + "resource_calls": 2, + "help_211": 0, + "directions": 3, + "official_sources": 0, + "pwa_installs": 0 + } + }, + "last_30_complete_days": { + "start_day": "2026-08-05", + "end_day": "2026-09-03", + "partial": false, + "counts": { + "page_views": 21, + "resource_calls": 6, + "help_211": 0, + "directions": 9, + "official_sources": 0, + "pwa_installs": 0 + } + } + }, + "discovery_last_7_complete_days": { + "sources": [ + { + "value": "facebook", + "count": 9 + }, + { + "value": "community", + "count": 8 + } + ], + "campaigns": [ + { + "value": "launch_2026_09", + "count": 17 + } + ], + "contents": [ + { + "value": "post_01", + "count": 17 + } + ] + }, + "limitations": { + "coverage": "observed_only", + "counts_are": "consented_activity_not_people_or_service_outcomes", + "attribution": "page_views_only_no_action_join", + "activity_is_health": false, + "raw_events_stored": false, + "identifiers_reported": false, + "aggregate_retention_days": 400 + } +} diff --git a/contracts/kfh-v1/unavailable.json b/contracts/kfh-v1/unavailable.json new file mode 100644 index 0000000..2f808d5 --- /dev/null +++ b/contracts/kfh-v1/unavailable.json @@ -0,0 +1,54 @@ +{ + "view": "kfh", + "report_contract_version": "1.0", + "site_key": "kingston_food_help", + "generated_at": "2026-09-04T12:00:00.000Z", + "source": { + "availability": "unavailable", + "reason": "query_failed", + "first_observed_day": null, + "last_observed_day": null + }, + "windows": { + "today": { + "start_day": "2026-09-04", + "end_day": "2026-09-04", + "partial": true, + "counts": null + }, + "latest_complete_day": { + "start_day": "2026-09-03", + "end_day": "2026-09-03", + "partial": false, + "counts": null + }, + "last_7_complete_days": { + "start_day": "2026-08-28", + "end_day": "2026-09-03", + "partial": false, + "counts": null + }, + "previous_7_complete_days": { + "start_day": "2026-08-21", + "end_day": "2026-08-27", + "partial": false, + "counts": null + }, + "last_30_complete_days": { + "start_day": "2026-08-05", + "end_day": "2026-09-03", + "partial": false, + "counts": null + } + }, + "discovery_last_7_complete_days": null, + "limitations": { + "coverage": "observed_only", + "counts_are": "consented_activity_not_people_or_service_outcomes", + "attribution": "page_views_only_no_action_join", + "activity_is_health": false, + "raw_events_stored": false, + "identifiers_reported": false, + "aggregate_retention_days": 400 + } +} diff --git a/migrations/0016_add_kfh_daily.sql b/migrations/0016_add_kfh_daily.sql new file mode 100644 index 0000000..d196499 --- /dev/null +++ b/migrations/0016_add_kfh_daily.sql @@ -0,0 +1,15 @@ +-- Aggregate counts only. No raw events, identity or source-to-action joins. +-- Apply only after owner approval; retain for 400 UTC day buckets. +CREATE TABLE IF NOT EXISTS kfh_daily ( + day TEXT NOT NULL CHECK (day GLOB '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'), + metric TEXT NOT NULL, + value TEXT NOT NULL, + count INTEGER NOT NULL CHECK (typeof(count) = 'integer' AND count > 0), + PRIMARY KEY (day, metric, value), + CHECK ( + (metric = 'event' AND value IN ('page_views', 'resource_calls', 'help_211', 'directions', 'official_sources', 'pwa_installs')) OR + (metric = 'source' AND value IN ('direct_unknown', 'facebook', 'community', 'search', 'other')) OR + (metric = 'campaign' AND value IN ('none', 'launch_2026_09')) OR + (metric = 'content' AND value IN ('none', 'post_01', 'poster_01')) + ) +) WITHOUT ROWID; diff --git a/package-lock.json b/package-lock.json index 837e253..1c38f79 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,17 +1,18 @@ { "name": "buscore-lighthouse", - "version": "1.31.0", + "version": "1.32.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "buscore-lighthouse", - "version": "1.31.0", + "version": "1.32.0", "license": "ISC", "devDependencies": { "@cloudflare/workers-types": "^4.20260305.0", "ajv": "^8.20.0", "ajv-formats": "^3.0.1", + "sql.js": "1.13.0", "typescript": "^5.9.3", "wrangler": "^4.69.0" } @@ -1444,6 +1445,13 @@ "@img/sharp-win32-x64": "0.34.5" } }, + "node_modules/sql.js": { + "version": "1.13.0", + "resolved": "https://registry.npmjs.org/sql.js/-/sql.js-1.13.0.tgz", + "integrity": "sha512-RJbVP1HRDlUUXahJ7VMTcu9Rm1Nzw+EBpoPr94vnbD4LwR715F3CcxE2G2k45PewcaZ57pjetYa+LoSJLAASgA==", + "dev": true, + "license": "MIT" + }, "node_modules/supports-color": { "version": "10.2.2", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-10.2.2.tgz", diff --git a/package.json b/package.json index 17b3eab..86e05c9 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "buscore-lighthouse", - "version": "1.31.0", + "version": "1.32.0", "description": "Standalone deterministic metrics worker: manifest proxy + fixed daily counters + protected on-demand reporting.", "scripts": { "dev": "wrangler dev", @@ -27,6 +27,7 @@ "@cloudflare/workers-types": "^4.20260305.0", "ajv": "^8.20.0", "ajv-formats": "^3.0.1", + "sql.js": "1.13.0", "typescript": "^5.9.3", "wrangler": "^4.69.0" } diff --git a/src/index.ts b/src/index.ts index 0b48a45..36ee65b 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,3 +1,5 @@ +import { KFH_SITE_KEY, KFH_ORIGINS } from "./kfhContract.js"; +import { ingestKfhEvent, readKfhBody, buildKfhReport, pruneKfhData } from "./kfhAnalytics.js"; import { BUSCORE_TELEMETRY_PATH, BUSCORE_TELEMETRY_PRODUCT_FAILURE_EVENTS, @@ -353,7 +355,7 @@ type SiteSectionAvailability = { identity: boolean; read: boolean; }; -type ReportView = "legacy" | "fleet" | "site" | "tgc" | "source_health" | "asset" | "monthly" | "ceo"; +type ReportView = "kfh" | "legacy" | "fleet" | "site" | "tgc" | "source_health" | "asset" | "monthly" | "ceo"; type ReportWindow = { start_day: string; end_day: string; @@ -442,6 +444,7 @@ type ReportRequestResolution = | { ok: true; view: "fleet" } | { ok: true; view: "site"; siteEventFilter: SiteEventFilter } | { ok: true; view: "tgc" } + | { ok: true; view: "kfh" } | { ok: true; view: "source_health" } | { ok: true; view: "asset" } | { ok: true; view: "monthly" } @@ -566,6 +569,7 @@ type CloudflareGraphQLResponse = { type SiteStatus = "active" | "staging" | "planned"; type TrackedSite = { + readonly report_profile?: "kfh_daily"; readonly site_key: string; readonly label: string; readonly status: SiteStatus; @@ -578,6 +582,13 @@ type TrackedSite = { }; const TRACKED_SITES: readonly TrackedSite[] = [ + { + site_key: KFH_SITE_KEY, label: "Kingston Food Help", status: "active", + report_profile: "kfh_daily", + production_hosts: ["kingstonfoodhelp.ca", "www.kingstonfoodhelp.ca"], + allowed_origins: KFH_ORIGINS, staging_hosts: [], + cloudflare_traffic_enabled: false, cloudflare_host: null, production_only_default: true, + }, { site_key: "buscore", label: "BUS Core", @@ -1306,6 +1317,7 @@ export function normalizeReportView(value: string | null): ReportView | null { } if ( + normalized === "kfh" || normalized === "fleet" || normalized === "site" || normalized === "tgc" || @@ -1582,6 +1594,7 @@ export function sanitizeAnalyticsLocation(value: string, allowEmpty: boolean = f export function parseCanonicalEventPayload(payload: unknown): SiteEventInput | null { const root = typeof payload === "object" && payload !== null ? (payload as Record) : {}; const siteKey = readRequiredString(root, "site_key"); + if (siteKey === KFH_SITE_KEY) return null; // Never fall back to raw-event storage. const eventName = readRequiredString(root, "event_name"); const clientTs = readRequiredString(root, "client_ts"); const path = readRequiredString(root, "path"); @@ -3065,6 +3078,7 @@ function parseBooleanQueryFlag(value: string | null, defaultValue: boolean): boo function normalizeSiteEventFilter(url: URL): SiteEventFilter | null { const siteKey = nullIfBlank(url.searchParams.get("site_key")); + if (siteKey === KFH_SITE_KEY) return null; // Use the dedicated aggregate view. if (!siteKey) { return null; } @@ -4850,7 +4864,7 @@ async function buildLegacyReport( async function buildFleetReport(db: D1Database, now: Date): Promise> { const { todayDay, last7StartDay } = reportDayBounds(now); const sites = await Promise.all( - TRACKED_SITES.map(async (site): Promise => { + TRACKED_SITES.filter(site => site.report_profile !== "kfh_daily").map(async (site): Promise => { const snapshot = await buildSiteSignalSnapshot(db, site, defaultSiteEventFilter(site), last7StartDay, todayDay); return { @@ -5126,7 +5140,7 @@ async function buildSourceHealthReport( ): Promise> { const { todayDay, last7StartDay } = reportDayBounds(now); const sites = await Promise.all( - TRACKED_SITES.map(async (site): Promise => { + TRACKED_SITES.filter(site => site.report_profile !== "kfh_daily").map(async (site): Promise => { const snapshot = await buildSiteSignalSnapshot(db, site, defaultSiteEventFilter(site), last7StartDay, todayDay); return { @@ -5293,7 +5307,7 @@ function withCors(request: Request, response: Response, allowMethods: string = " const activeOrigins = getAllActiveAllowedOrigins(); if (origin && activeOrigins.has(origin)) { headers.set("Access-Control-Allow-Origin", origin); - headers.set("Access-Control-Allow-Credentials", "true"); + if (!(KFH_ORIGINS as readonly string[]).includes(origin)) headers.set("Access-Control-Allow-Credentials", "true"); headers.set("Access-Control-Allow-Headers", "Content-Type"); headers.set("Vary", "Origin"); } else { @@ -6635,6 +6649,7 @@ export default { // Independent, fail-soft writers. One failing cannot break the others. await Promise.all([ + pruneKfhData(env.DB).catch(() => { console.warn("KFH retention cleanup unavailable."); }), prunePageviewData(env.DB).catch((error) => { console.warn("Pageview retention cleanup skipped after D1 failure.", error); }), @@ -6688,6 +6703,25 @@ export default { } if (url.pathname === SITE_EVENT_METRICS_PATH && request.method === "POST") { + if ((KFH_ORIGINS as readonly string[]).includes(request.headers.get("Origin") ?? "")) { + if (request.headers.get("Sec-GPC") === "1" || request.headers.get("DNT") === "1") { + return withCors(request, new Response(null, { status: 204 }), "POST, OPTIONS"); + } + const raw = await readKfhBody(request); + let payload: unknown; + try { payload = raw ? JSON.parse(raw) : null; } catch { payload = null; } + const now = new Date(); + const origin = request.headers.get("Origin"); + ctx.waitUntil(ingestKfhEvent(payload, env.DB, origin, async () => { + const clientIp = getClientIp(request); + const secret = env.TELEMETRY_RATE_LIMIT_SECRET?.trim(); + if (!clientIp || !secret || shouldSkipCounting(clientIp, env.IGNORED_IP)) return false; + const minute = utcMinuteBucket(now); + const key = await keyedRateIdentifier(secret, minute, `${KFH_SITE_KEY}:${clientIp}`); + return await incrementSiteEventRateLimitBucket(env.DB, minute, key) <= SITE_EVENT_RATE_LIMIT_PER_MINUTE; + }, now).catch(() => { console.warn("KFH ingest unavailable; submission dropped."); })); + return withCors(request, new Response(null, { status: 204 }), "POST, OPTIONS"); + } const requestContext = buildPageviewRequestContext(request); const capture = await readRawBodyCapture(request); ctx.waitUntil( @@ -7040,13 +7074,16 @@ export default { reportRequest.view !== "asset" && reportRequest.view !== "monthly" && reportRequest.view !== "ceo" && - reportRequest.view !== "tgc" + reportRequest.view !== "tgc" && + reportRequest.view !== "kfh" ) { await refreshPreviousCompletedTrafficBestEffort(env, now); } const payload = - reportRequest.view === "legacy" + reportRequest.view === "kfh" + ? await buildKfhReport(env.DB, now) + : reportRequest.view === "legacy" ? await buildLegacyReport(env.DB, env.BUSCORE_LEADS_DB, now, reportRequest.siteEventFilter) : reportRequest.view === "fleet" ? await buildFleetReport(env.DB, now) @@ -7064,7 +7101,7 @@ export default { return withCors( request, - Response.json(payload, { status: 200 }) + Response.json(payload, { status: 200, ...(reportRequest.view === "kfh" ? { headers: { "Cache-Control": "no-store" } } : {}) }) ); } catch { await incrementErrorCounterBestEffort(env.DB, day); diff --git a/src/kfhAnalytics.ts b/src/kfhAnalytics.ts new file mode 100644 index 0000000..2125d27 --- /dev/null +++ b/src/kfhAnalytics.ts @@ -0,0 +1,135 @@ +import { KFH_SITE_KEY, KFH_ORIGINS, KFH_SOURCES, KFH_CAMPAIGNS, KFH_CONTENTS, KFH_COUNT_KEYS, KFH_WINDOW_KEYS, KFH_LIMITATIONS, type CountKey, type Counts, type WindowKey, type KfhReport, isKfhReport } from "./kfhContract.js"; +type Row = { day: string; metric: string; value: string; count: number }; +type Dimension = { value: string; count: number }; + +function object(value: unknown): value is Record { + return !!value && typeof value === "object" && !Array.isArray(value); +} +function member(value: unknown, allowed: readonly string[]): value is string { + return typeof value === "string" && allowed.includes(value); +} + +export function parseKfhEvent(value: unknown): { counter: CountKey; source?: string; campaign?: string; content?: string } | null { + if (!object(value)) return null; + if (value.site_key !== KFH_SITE_KEY || value.contract_version !== 1 || value.consent !== true || value.page !== "directory") return null; + const keys = ["site_key", "contract_version", "consent", "page", "event_name"]; + if (value.event_name === "page_view") { + keys.push("source", "campaign", "content"); + if (Object.keys(value).some(key => !keys.includes(key))) return null; + const source = value.source === undefined ? "direct_unknown" : value.source; + const campaign = value.campaign === undefined ? "none" : value.campaign; + const content = value.content === undefined ? "none" : value.content; + if (!member(source, KFH_SOURCES) || !member(campaign, KFH_CAMPAIGNS) || !member(content, KFH_CONTENTS)) return null; + return { counter: "page_views", source, campaign, content }; + } + if (value.event_name !== "pwa_install") keys.push("event_value"); + if (Object.keys(value).some(key => !keys.includes(key))) return null; + if (value.event_name === "pwa_install") return { counter: "pwa_installs" }; + if (value.event_name === "contact_click" && value.event_value === "resource_call") return { counter: "resource_calls" }; + if (value.event_name === "contact_click" && value.event_value === "help_211") return { counter: "help_211" }; + if (value.event_name === "outbound_click" && value.event_value === "directions") return { counter: "directions" }; + if (value.event_name === "outbound_click" && value.event_value === "official_source") return { counter: "official_sources" }; + return null; +} + +export async function readKfhBody(request: Request): Promise { + if (!request.body) return null; + const reader = request.body.getReader(); + const chunks: Uint8Array[] = []; + let size = 0; + try { + while (true) { + const { done, value } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > 1024) { void reader.cancel().catch(() => {}); return null; } + chunks.push(value); + } + const bytes = new Uint8Array(size); + let offset = 0; + for (const chunk of chunks) { bytes.set(chunk, offset); offset += chunk.length; } + return new TextDecoder("utf-8", { fatal: true, ignoreBOM: false }).decode(bytes); + } catch { return null; } + finally { reader.releaseLock(); } +} + +const day = (date: Date) => date.toISOString().slice(0, 10); +const shiftDay = (now: Date, offset: number) => day(new Date(now.getTime() + offset * 86400000)); + +export async function ingestKfhEvent( + payload: unknown, db: D1Database, origin: string | null, + allowRate: () => Promise, now: Date = new Date(), +): Promise { + if (!origin || !(KFH_ORIGINS as readonly string[]).includes(origin)) return; + const event = parseKfhEvent(payload); + if (!event || !(await allowRate())) return; + const dimensions = [["event", event.counter]]; + if (event.counter === "page_views") { + dimensions.push(["source", event.source!], ["campaign", event.campaign!], ["content", event.content!]); + } + // D1 batch is atomic: a view cannot have a partially written attribution set. + await db.batch(dimensions.map(([metric, value]) => db.prepare( + "INSERT INTO kfh_daily(day, metric, value, count) VALUES (?, ?, ?, 1) ON CONFLICT(day, metric, value) DO UPDATE SET count = count + 1", + ).bind(day(now), metric, value))); +} + +export async function pruneKfhData(db: D1Database, now: Date = new Date()): Promise { + await db.prepare("DELETE FROM kfh_daily WHERE day < ?").bind(shiftDay(now, -399)).run(); +} + +export async function buildKfhReport(db: D1Database, now: Date = new Date()): Promise { + let rows: Row[] = []; + let available = true; + try { + const result = await db.prepare("SELECT day, metric, value, count FROM kfh_daily WHERE day >= ? AND day <= ? ORDER BY day, metric, value") + .bind(shiftDay(now, -399), day(now)).all(); + if (!result.success || !Array.isArray(result.results)) throw new Error("unavailable"); + rows = result.results; + // Fail closed on corrupt/incompatible aggregate rows; no raw values leave here. + for (const row of rows) { + const allowed = row.metric === "event" ? KFH_COUNT_KEYS : row.metric === "source" ? KFH_SOURCES + : row.metric === "campaign" ? KFH_CAMPAIGNS : row.metric === "content" ? KFH_CONTENTS : []; + if (!/^\d{4}-\d{2}-\d{2}$/.test(row.day) || !member(row.value, allowed) + || !Number.isSafeInteger(row.count) || row.count < 1) throw new Error("unavailable"); + } + } catch { available = false; rows = []; } + return kfhReportFromRows(available ? rows : null, now); +} + +function kfhReportFromRows(input: Row[] | null, now: Date): KfhReport { + const available = input !== null; + const rows = input ?? []; + const eventDays = rows.filter(row => row.metric === "event").map(row => row.day).sort(); + const ranges: Record = { + today: [0, 0], latest_complete_day: [-1, -1], last_7_complete_days: [-7, -1], + previous_7_complete_days: [-14, -8], last_30_complete_days: [-30, -1], + }; + const windows = {} as KfhReport["windows"]; + for (const key of KFH_WINDOW_KEYS) { + const [start, end] = ranges[key].map(offset => shiftDay(now, offset)); + const counts = Object.fromEntries(KFH_COUNT_KEYS.map(key => [key, 0])) as Counts; + for (const row of rows) if (row.metric === "event" && row.day >= start && row.day <= end) counts[row.value as CountKey] += row.count; + if (Object.values(counts).some(count => !Number.isSafeInteger(count))) return kfhReportFromRows(null, now); + windows[key] = { start_day: start, end_day: end, partial: key === "today", counts: available ? counts : null }; + } + const rank = (metric: string): Dimension[] => { + const totals = new Map(); + for (const row of rows) if (row.metric === metric && row.day >= shiftDay(now, -7) && row.day <= shiftDay(now, -1)) { + totals.set(row.value, (totals.get(row.value) ?? 0) + row.count); + } + return [...totals].map(([value, count]) => ({ value, count })).sort((a, b) => b.count - a.count || a.value.localeCompare(b.value)); + }; + const report: KfhReport = { + view: "kfh", report_contract_version: "1.0", site_key: KFH_SITE_KEY, generated_at: now.toISOString(), + source: { + availability: available ? "available" : "unavailable", + reason: !available ? "query_failed" : eventDays.length ? "observed_activity" : "no_observed_history", + first_observed_day: eventDays[0] ?? null, last_observed_day: eventDays[eventDays.length - 1] ?? null, + }, + windows, + discovery_last_7_complete_days: available ? { sources: rank("source"), campaigns: rank("campaign"), contents: rank("content") } : null, + limitations: KFH_LIMITATIONS, + }; + return isKfhReport(report) ? report : kfhReportFromRows(null, now); +} + diff --git a/src/kfhContract.ts b/src/kfhContract.ts new file mode 100644 index 0000000..5392a4c --- /dev/null +++ b/src/kfhContract.ts @@ -0,0 +1,112 @@ +// Kingston's sensitive-use directory keeps only separate daily totals. +// No raw event, provider context, identity or cross-dimension journey is stored. +export const KFH_SITE_KEY = "kingston_food_help"; +export const KFH_ORIGINS = ["https://kingstonfoodhelp.ca", "https://www.kingstonfoodhelp.ca"] as const; +export const KFH_SOURCES = ["direct_unknown", "facebook", "community", "search", "other"] as const; +export const KFH_CAMPAIGNS = ["none", "launch_2026_09"] as const; +export const KFH_CONTENTS = ["none", "post_01", "poster_01"] as const; +export const KFH_COUNT_KEYS = ["page_views", "resource_calls", "help_211", "directions", "official_sources", "pwa_installs"] as const; +export const KFH_WINDOW_KEYS = ["today", "latest_complete_day", "last_7_complete_days", "previous_7_complete_days", "last_30_complete_days"] as const; +export const KFH_LIMITATIONS = { + coverage: "observed_only", + counts_are: "consented_activity_not_people_or_service_outcomes", + attribution: "page_views_only_no_action_join", + activity_is_health: false, + raw_events_stored: false, + identifiers_reported: false, + aggregate_retention_days: 400, +} as const; + +export type CountKey = typeof KFH_COUNT_KEYS[number]; +export type WindowKey = typeof KFH_WINDOW_KEYS[number]; +export type Counts = Record; +type Dimension = { value: string; count: number }; +export type KfhReport = { + view: "kfh"; + report_contract_version: "1.0"; + site_key: typeof KFH_SITE_KEY; + generated_at: string; + source: { + availability: "available" | "unavailable"; + reason: "observed_activity" | "no_observed_history" | "query_failed"; + first_observed_day: string | null; + last_observed_day: string | null; + }; + windows: Record; + discovery_last_7_complete_days: { sources: Dimension[]; campaigns: Dimension[]; contents: Dimension[] } | null; + limitations: typeof KFH_LIMITATIONS; +}; + +const isObject = (value: unknown): value is Record => !!value && typeof value === "object" && !Array.isArray(value); +function exact(value: unknown, keys: readonly string[]): value is Record { + return isObject(value) && Object.keys(value).length === keys.length && keys.every(key => Object.prototype.hasOwnProperty.call(value, key)); +} +const isCount = (value: unknown): value is number => typeof value === "number" && Number.isSafeInteger(value) && value >= 0; +function validDay(value: unknown): value is string { + return typeof value === "string" && /^\d{4}-\d{2}-\d{2}$/.test(value) + && Number.isFinite(Date.parse(`${value}T00:00:00.000Z`)) && new Date(`${value}T00:00:00.000Z`).toISOString().slice(0, 10) === value; +} +export function kfhWindowDays(now: Date): Record { + const at = (offset: number) => new Date(now.getTime() + offset * 86400000).toISOString().slice(0, 10); + return { + today: [at(0), at(0)], latest_complete_day: [at(-1), at(-1)], last_7_complete_days: [at(-7), at(-1)], + previous_7_complete_days: [at(-14), at(-8)], last_30_complete_days: [at(-30), at(-1)], + }; +} + +// Strict shared producer/consumer contract. No runtime schema compiler is needed. +export function isKfhReport(value: unknown): value is KfhReport { + if (!exact(value, ["view", "report_contract_version", "site_key", "generated_at", "source", "windows", "discovery_last_7_complete_days", "limitations"])) return false; + if (value.view !== "kfh" || value.report_contract_version !== "1.0" || value.site_key !== KFH_SITE_KEY) return false; + if (typeof value.generated_at !== "string" || !/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d{3}Z$/.test(value.generated_at) + || !Number.isFinite(Date.parse(value.generated_at)) || new Date(value.generated_at).toISOString() !== value.generated_at) return false; + if (!exact(value.limitations, Object.keys(KFH_LIMITATIONS))) return false; + for (const [key, expected] of Object.entries(KFH_LIMITATIONS)) if (value.limitations[key] !== expected) return false; + const source = value.source; + if (!exact(source, ["availability", "reason", "first_observed_day", "last_observed_day"])) return false; + const unavailable = source.availability === "unavailable"; + if (!unavailable && source.availability !== "available") return false; + if (unavailable) { + if (source.reason !== "query_failed" || source.first_observed_day !== null || source.last_observed_day !== null) return false; + } else if (source.reason === "no_observed_history") { + if (source.first_observed_day !== null || source.last_observed_day !== null) return false; + } else { + if (source.reason !== "observed_activity" || !validDay(source.first_observed_day) || !validDay(source.last_observed_day) + || source.first_observed_day > source.last_observed_day || source.last_observed_day > value.generated_at.slice(0, 10)) return false; + } + if (!exact(value.windows, KFH_WINDOW_KEYS)) return false; + const ranges = kfhWindowDays(new Date(value.generated_at)); + for (const key of KFH_WINDOW_KEYS) { + const window = value.windows[key]; + if (!exact(window, ["start_day", "end_day", "partial", "counts"]) || window.start_day !== ranges[key][0] + || window.end_day !== ranges[key][1] || window.partial !== (key === "today")) return false; + if (unavailable) { if (window.counts !== null) return false; } + else { + if (!exact(window.counts, KFH_COUNT_KEYS) || !Object.values(window.counts).every(isCount)) return false; + if (source.reason === "no_observed_history" && Object.values(window.counts).some(count => count !== 0)) return false; + } + } + const discovery = value.discovery_last_7_complete_days; + if (unavailable) return discovery === null; + if (!exact(discovery, ["sources", "campaigns", "contents"])) return false; + const windows = value.windows as KfhReport["windows"]; + const views = windows.last_7_complete_days.counts!.page_views; + for (const [key, allowed] of [["sources", KFH_SOURCES], ["campaigns", KFH_CAMPAIGNS], ["contents", KFH_CONTENTS]] as const) { + const rows = discovery[key]; + if (!Array.isArray(rows) || rows.length > allowed.length) return false; + const seen = new Set(); + let total = 0; + for (const row of rows) { + if (!exact(row, ["value", "count"]) || typeof row.value !== "string" || !(allowed as readonly string[]).includes(row.value) + || seen.has(row.value) || !isCount(row.count) || row.count === 0) return false; + seen.add(row.value); total += row.count; + } + if (!Number.isSafeInteger(total) || total !== views) return false; + } + // These windows overlap by definition. A contradictory report is unavailable. + for (const key of KFH_COUNT_KEYS) { + if (windows.latest_complete_day.counts![key] > windows.last_7_complete_days.counts![key] + || windows.last_7_complete_days.counts![key] + windows.previous_7_complete_days.counts![key] > windows.last_30_complete_days.counts![key]) return false; + } + return true; +} diff --git a/tests/kfh-analytics.test.mjs b/tests/kfh-analytics.test.mjs new file mode 100644 index 0000000..6670576 --- /dev/null +++ b/tests/kfh-analytics.test.mjs @@ -0,0 +1,174 @@ +import test, { before, after, beforeEach } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import initSqlJs from "sql.js"; +import workerModule, { parseCanonicalEventPayload, resolveReportRequest } from "../dist/index.js"; +import { parseKfhEvent, ingestKfhEvent, buildKfhReport, pruneKfhData } from "../dist/kfhAnalytics.js"; +import { isKfhReport, KFH_SITE_KEY, KFH_ORIGINS, KFH_COUNT_KEYS } from "../dist/kfhContract.js"; + +const worker = workerModule.fetch ? workerModule : workerModule.default; +const now = new Date("2026-09-04T12:00:00.000Z"); +const payload = (overrides = {}) => ({ site_key: KFH_SITE_KEY, contract_version: 1, consent: true, page: "directory", event_name: "page_view", ...overrides }); +// Execute the checked-in SQL with SQLite WASM. This tests SQL/atomic batches, +// not Cloudflare's native Worker lifecycle or remote D1 deployment. +let sqlite, db; +before(async () => { + const SQL = await initSqlJs(); sqlite = new SQL.Database(); + const prepare = (sql, values = []) => ({ + bind(...next) { return prepare(sql, next); }, + async run() { sqlite.run(sql, values); return { success: true }; }, + async all() { + const statement = sqlite.prepare(sql); const results = []; + try { statement.bind(values); while (statement.step()) results.push(statement.getAsObject()); } + finally { statement.free(); } + return { success: true, results }; + }, + async first() { return (await this.all()).results[0] ?? null; }, + }); + db = { prepare, async exec(sql) { sqlite.exec(sql); }, async batch(statements) { + sqlite.run("BEGIN"); + try { const results = []; for (const statement of statements) results.push(await statement.run()); sqlite.run("COMMIT"); return results; } + catch (error) { sqlite.run("ROLLBACK"); throw error; } + } }; + for (const name of ["0008_add_site_event_rate_limit.sql", "0016_add_kfh_daily.sql"]) { + await db.exec(fs.readFileSync(new URL(`../migrations/${name}`, import.meta.url), "utf8")); + } +}); +after(() => { sqlite?.close(); }); +beforeEach(async () => { await db.exec("DELETE FROM kfh_daily; DELETE FROM site_event_rate_limit;"); }); + +test("strict Kingston payload rejects sensitive fields and action attribution", () => { + assert.equal(parseKfhEvent(payload()).counter, "page_views"); + for (const key of ["anon_user_id", "session_id", "provider", "resource_id", "phone", "search", "filter", "category", "lat", "url", "path", "referrer", "client_ts", "viewport", "country", "test_mode"]) { + assert.equal(parseKfhEvent(payload({ [key]: "sensitive fixture" })), null, key); + } + for (const patch of [{ consent: false }, { consent: "true" }, { site_key: "tgc_site" }, { event_name: "scroll" }, { campaign: "person@example.test" }, { content: "private-group" }, { page: "food-bank" }]) assert.equal(parseKfhEvent(payload(patch)), null); + for (const event_value of ["resource_call", "help_211"]) { + assert.ok(parseKfhEvent(payload({ event_name: "contact_click", event_value }))); + assert.equal(parseKfhEvent(payload({ event_name: "contact_click", event_value, source: "facebook" })), null); + } + assert.equal(parseKfhEvent(payload({ event_name: "pwa_install", event_value: "device" })), null); + assert.equal(parseCanonicalEventPayload(payload({ client_ts: now.toISOString(), path: "/", url: KFH_ORIGINS[0], referrer: "", device: "mobile", viewport: "320x600", lang: "en", tz: "UTC", utm: {} })), null); +}); + +test("D1 writes separate daily aggregates, with atomic page attribution and no raw rows", async () => { + await ingestKfhEvent(payload({ source: "facebook", campaign: "launch_2026_09", content: "post_01" }), db, KFH_ORIGINS[0], async () => true, now); + await ingestKfhEvent(payload({ event_name: "contact_click", event_value: "resource_call" }), db, KFH_ORIGINS[0], async () => true, now); + const rows = (await db.prepare("SELECT * FROM kfh_daily ORDER BY metric").all()).results; + assert.equal(rows.length, 5); + assert.ok(rows.every(row => Object.keys(row).sort().join(",") === "count,day,metric,value")); + assert.ok(rows.every(row => row.day === "2026-09-04")); + assert.equal(rows.filter(row => row.metric === "source")[0].count, 1); + const tables = (await db.prepare("SELECT name FROM sqlite_master WHERE type='table'").all()).results; + assert.ok(!tables.some(row => row.name === "site_events_raw")); + await assert.rejects(db.prepare("INSERT INTO kfh_daily VALUES ('2026-09-04', 'source', 'person@example.test', 1)").run()); +}); + +test("origin and rate failures write no Kingston measurements", async () => { + for (const origin of [null, "https://evil.example", "http://kingstonfoodhelp.ca", "https://kingstonfoodhelp.ca.evil.example", "https://preview.pages.dev"]) { + await ingestKfhEvent(payload(), db, origin, async () => { throw new Error("must not call rate gate"); }, now); + } + await ingestKfhEvent(payload(), db, KFH_ORIGINS[0], async () => false, now); + await assert.rejects(ingestKfhEvent(payload(), db, KFH_ORIGINS[0], async () => { throw new Error("fixture rate failure"); }, now)); + assert.equal((await db.prepare("SELECT COUNT(*) AS n FROM kfh_daily").first()).n, 0); +}); + +async function submit(body, options = {}) { + const pending = []; + const headers = { Origin: KFH_ORIGINS[0], "CF-Connecting-IP": "192.0.2.5", ...options.headers }; + const request = new Request("https://lighthouse.test/metrics/event", { method: "POST", headers, body: typeof body === "string" ? body : JSON.stringify(body) }); + const response = await worker.fetch(request, { DB: db, TELEMETRY_RATE_LIMIT_SECRET: "local-test-rate-secret", ...options.env }, { waitUntil(p) { pending.push(p); } }); + await Promise.all(pending); + return response; +} + +test("public route is fail-soft, production-only, privacy-suppressed and rate-bounded", async () => { + for (const options of [{ headers: { DNT: "1" } }, { headers: { "Sec-GPC": "1" } }, { headers: { "CF-Connecting-IP": "" } }, { env: { TELEMETRY_RATE_LIMIT_SECRET: "" } }, { env: { IGNORED_IP: "192.0.2.5" } }]) assert.equal((await submit(payload(), options)).status, 204); + for (const body of ["{invalid", "x".repeat(1025), payload({ site_key: "buscore", anon_user_id: "fixture" }), payload({ consent: false })]) assert.equal((await submit(body)).status, 204); + assert.equal((await db.prepare("SELECT COUNT(*) AS n FROM kfh_daily").first()).n, 0); + for (let i = 0; i < 51; i++) await submit(payload()); + const rate = (await db.prepare("SELECT * FROM site_event_rate_limit").all()).results; + assert.equal((await db.prepare("SELECT count FROM kfh_daily WHERE metric='event' AND value='page_views'").first()).count, rate.reduce((sum, row) => sum + Math.min(row.count, 50), 0)); + assert.ok(rate.length >= 1 && rate.length <= 2); // A UTC minute may turn during the test. + assert.match(rate[0].ip_hash, /^[a-f0-9]{64}$/); + assert.ok(!JSON.stringify(rate).includes("192.0.2.5")); + const response = await submit(payload()); + assert.equal(response.headers.get("Access-Control-Allow-Origin"), KFH_ORIGINS[0]); + assert.equal(response.headers.get("Access-Control-Allow-Credentials"), null); +}); + +test("failed attribution writes roll back the event total and migration reapplication preserves data", async () => { + await db.exec("CREATE TRIGGER fixture_reject_content BEFORE INSERT ON kfh_daily WHEN NEW.metric = 'content' BEGIN SELECT RAISE(ABORT, 'fixture'); END;"); + try { + await assert.rejects(ingestKfhEvent(payload(), db, KFH_ORIGINS[0], async () => true, now)); + assert.equal((await db.prepare("SELECT COUNT(*) AS n FROM kfh_daily").first()).n, 0); + } finally { await db.exec("DROP TRIGGER fixture_reject_content;"); } + await ingestKfhEvent(payload(), db, KFH_ORIGINS[0], async () => true, now); + await db.exec(fs.readFileSync(new URL("../migrations/0016_add_kfh_daily.sql", import.meta.url), "utf8")); + assert.equal((await db.prepare("SELECT count FROM kfh_daily WHERE metric='event'").first()).count, 1); +}); + +test("rate storage failure cannot fail delivery or leak the error/payload", async () => { + const warnings = []; const original = console.warn; + console.warn = (...args) => warnings.push(args); + try { + const response = await submit(payload(), { env: { DB: { prepare() { throw new Error("sensitive-fixture"); } } } }); + assert.equal(response.status, 204); + assert.equal(await response.text(), ""); + assert.deepEqual(warnings, [["KFH ingest unavailable; submission dropped."]]); + } finally { console.warn = original; } +}); + +test("report uses complete UTC windows and preserves empty versus unavailable data", async () => { + const empty = await buildKfhReport(db, now); + assert.ok(isKfhReport(empty)); + assert.equal(empty.source.reason, "no_observed_history"); + assert.equal(empty.windows.last_7_complete_days.counts.page_views, 0); + for (const date of ["2026-08-05", "2026-08-21", "2026-08-27", "2026-08-28", "2026-09-03", "2026-09-04"]) { + await ingestKfhEvent(payload(), db, KFH_ORIGINS[0], async () => true, new Date(`${date}T23:59:59Z`)); + } + const report = await buildKfhReport(db, now); + assert.ok(isKfhReport(report)); + assert.deepEqual(Object.values(report.windows).map(window => window.counts.page_views), [1, 1, 2, 2, 5]); + assert.equal(report.windows.last_7_complete_days.start_day, "2026-08-28"); + assert.equal(report.windows.previous_7_complete_days.end_day, "2026-08-27"); + assert.equal(report.discovery_last_7_complete_days.sources[0].count, 2); + const unavailable = await buildKfhReport({ prepare() { throw new Error("fixture failure"); } }, now); + assert.ok(isKfhReport(unavailable)); + assert.equal(unavailable.source.reason, "query_failed"); + assert.ok(Object.values(unavailable.windows).every(window => window.counts === null)); + assert.equal(unavailable.discovery_last_7_complete_days, null); +}); + +test("aggregate retention bounds both storage and reporting without raw event history", async () => { + const inside = new Date(now.getTime() - 399 * 86400000); + const outside = new Date(now.getTime() - 400 * 86400000); + for (const date of [inside, outside]) await ingestKfhEvent(payload(), db, KFH_ORIGINS[0], async () => true, date); + assert.equal((await buildKfhReport(db, now)).source.first_observed_day, inside.toISOString().slice(0, 10)); + await pruneKfhData(db, now); + assert.equal((await db.prepare("SELECT COUNT(DISTINCT day) AS n FROM kfh_daily").first()).n, 1); +}); + +test("dedicated report requires authentication, skips traffic refresh and cannot select another site", async () => { + assert.deepEqual(resolveReportRequest(new URL("https://lighthouse.test/report?view=kfh")), { ok: true, view: "kfh" }); + for (const query of [`?view=site&site_key=${KFH_SITE_KEY}`, `?site_key=${KFH_SITE_KEY}`]) assert.equal(resolveReportRequest(new URL(`https://lighthouse.test/report${query}`)).error, "invalid_site_key"); + const env = { DB: db, ADMIN_TOKEN: "local-admin", REPORT_READ_TOKEN: "r".repeat(32) }; + const ctx = { waitUntil() { throw new Error("no deferred writes"); } }; + assert.equal((await worker.fetch(new Request("https://lighthouse.test/report?view=kfh"), env, ctx)).status, 401); + const original = globalThis.fetch; + globalThis.fetch = () => { throw new Error("no external traffic refresh"); }; + try { + const response = await worker.fetch(new Request("https://lighthouse.test/report?view=kfh", { headers: { "X-Report-Token": env.REPORT_READ_TOKEN } }), env, ctx); + assert.equal(response.status, 200); + assert.equal(response.headers.get("Cache-Control"), "no-store"); + assert.ok(isKfhReport(await response.json())); + } finally { globalThis.fetch = original; } +}); + +test("strict response contract rejects identity, unsafe numbers and invented coverage", async () => { + const report = await buildKfhReport(db, now); + for (const change of [r => r.visitors = 1, r => r.windows.today.counts.resource_calls = null, r => r.windows.today.partial = false, r => r.source.last_observed_day = "2026-09-04", r => r.limitations.coverage = "full", r => r.site_key = "tgc_site", r => r.windows.today.counts.page_views = Number.MAX_SAFE_INTEGER + 1]) { + const changed = structuredClone(report); change(changed); assert.equal(isKfhReport(changed), false); + } + assert.equal(KFH_COUNT_KEYS.length, 6); +}); diff --git a/tests/release-control.test.mjs b/tests/release-control.test.mjs index 4fa1c96..d263f0f 100644 --- a/tests/release-control.test.mjs +++ b/tests/release-control.test.mjs @@ -54,7 +54,9 @@ test("operator scripts expose reads and upload without a direct production bypas }); test("the current governed bundle and historical release-control receipt stay synchronized", () => { - assert.equal(packageJson.version, "1.31.0"); + assert.equal(packageJson.version, "1.32.0"); + assert.match(sot, /^## Kingston Food Help — v1\.32\.0 review candidate \(2026-09-04\)$/m); + assert.match(changelog, /^## \[1\.32\.0\] - 2026-09-04$/m); assert.equal(packageLock.version, packageJson.version); assert.equal(packageLock.packages[""].version, packageJson.version); assert.match(sot, /^## CEO activity truth and sparse probe health — v1\.31\.0$/m);