diff --git a/CHANGELOG.md b/CHANGELOG.md index e9c2dcd..f6e6f58 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ The format is inspired by [Keep a Changelog](https://keepachangelog.com/en/1.1.0 ### Fixed +- Deployment smoke now proves the production Turnstile secret. Its bypass header skips Siteverify, so a wrong, rotated, or test secret used to pass smoke while every new session's first run failed. With `PBE_SMOKE_BYPASS_SECRET`, `scripts/smoke_deployment.py` calls the header-gated `POST /__smoke/turnstile`. The Worker sends its secret with Cloudflare's dummy token and reports `valid`, `invalid`, `testing_key`, `unverified`, `unexpected`, or `absent`. Smoke fails unless the secret is a working production secret with a site key, or Turnstile is off. + +- A Run whose Turnstile token the server rejects now ends with the server's message instead of re-challenging. The runner used to solve again every time the response carried the challenge marker, so a persistently failing Siteverify (wrong secret, hostname, or action) turned one click into an unbounded loop of solves, Worker POSTs, and Siteverify subrequests. Each Run now earns at most one challenge, and the failure message says to press Run again. +- Siteverify calls time out after 10 seconds, tokens longer than Cloudflare's documented 2048 characters are rejected without a subrequest, and a `success: true` response with a missing or non-string `hostname` fails closed instead of raising a 500. The gaps surfaced by comparing `_verify_turnstile` with the canonical handler in Cloudflare's Turnstile Spin skill. - Production Python packages are locked. Pywrangler 1.17.4 vendors a committed, hash-pinned `pylock.toml` instead of resolving an unpinned `fastapi` at deploy time, and `uv.lock` pins the same versions (FastAPI 0.141.1, Starlette 1.7.0, Pydantic 2.10.6) so the test suite runs against exactly what ships. `tests/test_dependency_locks.py` fails on drift, and CI and `make deploy` fail if a sync would change `pylock.toml`. Starlette 1.7.0 also clears the five advisories against the previously tested 1.0.0. - Example-page runner wiring (Run interception, Reset, the share button, and keyboard navigation) no longer waits for the CDN-backed highlighter and editor modules: `runner.js` loads `async`, so a slow or unreachable esm.sh cannot stall it — ordered module scripts otherwise execute strictly after every preceding module settles, including their top-level awaits. @@ -33,6 +37,7 @@ The format is inspired by [Keep a Changelog](https://keepachangelog.com/en/1.1.0 ### Changed +- The `turnstile` wide-event field distinguishes an issued challenge (`challenged`) from a rejected token (`fail`), and failures carry a closed-vocabulary `reason` plus allowlisted Siteverify `error_codes`. `scripts/learner_report.py` breaks failures down by reason and code and raises a configuration alert for `invalid-input-secret`, `missing-input-secret`, or a missing site key. Siteverify reports those secret errors with HTTP 400, so error codes are now read from non-2xx bodies instead of being logged as an outage. - Dependency refresh: Pillow 12.3.0 (from 11.3.0; 17 advisories), Pywrangler 1.17.4 (from 1.9.3; requires uv 0.12.3+), Shiki 4.4.3 (from 1.29.2; byte-identical output for every example block in both themes), and CodeMirror state 6.7.6, view 6.43.13, and language 6.12.4. - CI actions moved to their Node 24 releases (checkout, setup-node, and setup-python v7; setup-uv v10.2.0 pinned by commit), clearing the Node.js 20 deprecation warning; npm caching stays off. - Dependabot proposes weekly npm, GitHub Actions, and development-tool (Hypothesis, Pillow, Pywrangler) updates. Runtime packages refresh through `make upgrade-runtime-deps`. diff --git a/docs/learner-analytics.md b/docs/learner-analytics.md index 499d2f0..4e9b69e 100644 --- a/docs/learner-analytics.md +++ b/docs/learner-analytics.md @@ -21,7 +21,14 @@ time window: human-validated paths may suggest content candidates, but typos, scanners, and bots are confounders. - **Turnstile outcomes** — challenge and rejection counts used to monitor - runner availability and abuse controls, not learner quality. + runner availability and abuse controls, not learner quality. `challenged` + (a challenge was issued) is counted apart from `fail` (a token was + rejected), and failures break down by reason and Siteverify error code. + A **configuration alert** (`invalid-input-secret`, `missing-input-secret`, + or `site_key_missing`) means every challenged run is failing because of + the deployment, not the visitor. Deployment smoke probes the secret at + deploy time; this report catches a secret that breaks afterwards, such as + a rotation in the dashboard. ## Getting events diff --git a/docs/lessons-learned.md b/docs/lessons-learned.md index beaa9ec..6e9a944 100644 --- a/docs/lessons-learned.md +++ b/docs/lessons-learned.md @@ -131,6 +131,10 @@ git diff --check - **Deployment smoke belongs beside CI, and POST smoke must assert rendered output.** `scripts/smoke_deployment.py` checks rendered Worker pages, runtime-boundary pages, journey pages, prototype review pages, and representative Dynamic Worker POST runs for HTTP failures, exception markers, and stale edited-code output. Build success is not enough; the deployed Worker must render and execute edited examples. With Turnstile enabled, submitted code can appear in the editor textarea even when it did not run, so POST smoke must inspect the output panel rather than searching the whole HTML document. - **Observability smoke should assert the custom event, not the whole tail envelope.** Use unique `x-request-id` values, exercise cache miss/hit/bypass and client-error paths, and assert on the structured payload inside `logs[].message[]`. If Turnstile is enabled, Dynamic Worker error-path probes need the smoke bypass secret; otherwise they only verify the Turnstile-fail path. - **Turnstile should be secret-gated, session-scoped, and invisible until needed.** Protect edited-code POST runs only when `TURNSTILE_SECRET_KEY` and an explicit challenge mode are configured, lazy-load/render the Invisible-mode widget only after the server returns a challenge-required marker, and issue a signed clearance cookie so normal session runs skip Siteverify. The Cloudflare widget mode is `Invisible`; the client-side render option is `execution: "execute"`, not `size: "invisible"`. If production smoke must POST through a protected endpoint, use a separate `PBE_SMOKE_BYPASS_SECRET` header so smoke remains a deployment check rather than a CAPTCHA solver. See `docs/turnstile-runner-protection-spec.md` for the full runner-protection design. +- **A retry that can re-enter itself needs a bound.** The runner re-challenged whenever a response carried the challenge marker, including right after it had sent a token. A server that kept rejecting tokens (wrong secret, hostname, or action) turned one Run into an unbounded loop of Turnstile solves, Worker POSTs, and Siteverify subrequests. Cloudflare's Turnstile Spin skill treats tokens as single-use with at most one reset per retry; the runner now earns one challenge per Run and shows the server's message when the retried token is rejected. `scripts/check_browser_layout.mjs` drives a stub server that always rejects and fails if one Run costs more than one solve. +- **A bypassed check needs its own signal.** Deployment smoke uses `PBE_SMOKE_BYPASS_SECRET`, so it never proves the production `TURNSTILE_SECRET_KEY` works, and the wide event recorded both "challenge issued" and "Siteverify rejected" as `fail`. A wrong or rotated secret would have passed smoke while every new session's first run failed. Turnstile Spin's "Fix with Spin" banner is Cloudflare noticing widgets that serve traffic without working server-side validation; the self-hosted equivalent is `challenged` versus `fail` outcomes plus Siteverify `error_codes`, with `invalid-input-secret` reported by `scripts/learner_report.py` as a configuration alert rather than a bot failure. Verify the signal against the live service, not a stub: real Siteverify returns `invalid-input-secret` and `missing-input-secret` with HTTP 400, so the common `if (!response.ok) throw` pattern (Spin's reference handler included) discards the one code that names the misconfiguration. Unit stubs had assumed a 200. The deploy-time half of the fix is a smoke-gated probe (`POST /__smoke/turnstile`) that sends the production secret with Cloudflare's dummy token. Classify on more than the error code: Cloudflare's always-fail test secret answers the dummy token with the same `invalid-input-response` as a working production secret, and only `metadata.result_with_testing_key` tells them apart. +- **Check hand-rolled security code against the vendor's reference contract.** Reading the Turnstile Spin skill's canonical Siteverify handler beside `_verify_turnstile` found three gaps no test had asked about: no timeout, no 2048-character token bound, and a `null` hostname raising `AttributeError` (a 500) instead of failing closed. Use the reference as a checklist, and keep deliberate divergences written down: Invisible mode, session clearance instead of per-request verification, and the serving hostname instead of a configured allowlist (safe only while `workers.dev` and preview URLs stay disabled). +- **Credential-bearing commands run code from the dependency tree.** Turnstile Spin refuses to run `secret put` or other credential-bearing commands through `npx`, package scripts, or a project-local binary. `make deploy` runs the repository-pinned Wrangler with the account's credentials, and Dependabot now proposes npm updates weekly. Exact pins, the three-day cooldown, and `npm ci --ignore-scripts` limit the exposure; review Wrangler and Miniflare bumps as credential-bearing changes, not routine ones. ## Discoverability, theming, and learner analytics diff --git a/docs/observability-spec.md b/docs/observability-spec.md index 812966e..b8aba0a 100644 --- a/docs/observability-spec.md +++ b/docs/observability-spec.md @@ -404,7 +404,9 @@ Add: - `example.code_hash` — sha256 hex of submitted UTF-8 bytes, first 12 chars - `example.code_bytes` — `len(submitted.encode("utf-8"))` - `example.code_edited` — `submitted != example["code"]` -- `turnstile.outcome` — `pass` / `fail` / `bypass` / `disabled` +- `turnstile.outcome` — `challenged` / `pass` / `fail` / `bypass` / `disabled`. `challenged` means the request was sent a challenge; it is the normal first run of a session, not a failure. +- `turnstile.reason` — on `fail` only: `rejected` / `hostname_mismatch` / `action_mismatch` / `siteverify_unavailable` / `token_too_long` / `site_key_missing` / `missing_token` / `runtime_unavailable` +- `turnstile.error_codes` — on `rejected` only: Siteverify's documented `error-codes`, sorted, with anything undocumented recorded as `other` - `execution_ms` — duration around `_run_example`, recorded in a `finally` if execution started Example: @@ -419,7 +421,7 @@ request.state.wide_event["example"] = { } ``` -Change `_verify_turnstile(...)` to return `(ok, message, outcome)` so the caller can record the outcome without re-deriving it. Do not log the token or `CF-Connecting-IP`; the latter may still be sent to Turnstile verification but must not enter the event. +`_verify_turnstile(...)` returns `(ok, message, turnstile_fields)` so the caller records the outcome, failure reason, and error codes without re-deriving them. An `invalid-input-secret` code means the deployment is misconfigured, not that a visitor failed. Do not log the token or `CF-Connecting-IP`; the latter may still be sent to Turnstile verification but must not enter the event. #### `_run_example` @@ -567,7 +569,11 @@ Every event carries the context from `observability.py` plus a subset of the per | `example.code_hash` | string | handler | First 12 hex chars of sha256(submitted UTF-8 bytes). | | `example.code_bytes` | int | handler | Byte length of submitted UTF-8 code. | | `example.code_edited` | bool | handler | True when submitted differs from canonical. | -| `turnstile.outcome` | string | handler | `pass` / `fail` / `bypass` / `disabled`. | +| `turnstile.outcome` | string | handler | `challenged` / `pass` / `fail` / `bypass` / `disabled`. | +| `turnstile.reason` | string | handler | Closed-vocabulary failure reason; present only when `outcome` is `fail`. | +| `turnstile.error_codes` | array | handler | Allowlisted Siteverify `error-codes` (`other` for unknown); present only for `rejected`. | +| `turnstile_probe.secret` | string | probe handler | `valid` / `invalid` / `testing_key` / `unverified` / `unexpected` / `absent`; only on authenticated `POST /__smoke/turnstile`. | +| `turnstile_probe.ok` | bool | probe handler | Whether the deployed configuration can verify browsers. | | `execution_ms` | float | handler | Sandboxed run duration. | | `worker.outcome` | string | handler | Dynamic Worker outcome. | | `worker.status_code` | int | handler | Dynamic Worker HTTP status when fetch completes. | diff --git a/docs/pr-evidence/README.md b/docs/pr-evidence/README.md index 1d504f1..bbbee45 100644 --- a/docs/pr-evidence/README.md +++ b/docs/pr-evidence/README.md @@ -1,5 +1,24 @@ # PR visual evidence +## Turnstile rejection loop (2026-09-26) + +These captures show `/examples/values` six seconds after one Run click, with Siteverify rejecting every token. Both Workers used Cloudflare's invisible always-pass test site key (`1x00000000000000000000BB`), so the real Turnstile widget loaded and solved in the browser, and the always-fail test secret (`2x0000000000000000000000000000000AA`), so Siteverify rejected every solved token. + +| Evidence | Review point | Commit | SHA-256 | +| --- | --- | --- | --- | +| [Before](turnstile-rejection-before-runner.png) | The runner re-challenges after each rejection and stays busy on "Verification required…". Over 20 seconds, one click made 12 POSTs and 11 real widget solves, growing steadily. | `3f300af` (`origin/main`) | `5af0c28f339e6bc4d394186530c385e663046a7edf98671965d0a61ed1a3cdf3` | +| [After](turnstile-rejection-after-runner.png) | One solve, two POSTs, then the server's message and a free Run button. Still two POSTs after 20 seconds. | `eae0dc3` | `20ca27224fff2f0413004b8e2e2f35e7b1563b45f3941714a68f3e46e837f0af` | + +Reproduce each capture by serving the corresponding revision with the test keys: + +```bash +uv run --group workers pywrangler dev --port \ + --var TURNSTILE_SECRET_KEY:2x0000000000000000000000000000000AA \ + --var TURNSTILE_SITE_KEY:1x00000000000000000000BB +``` + +Open `http://127.0.0.1:/examples/values` at a 1200×900 viewport, click Run once, wait six seconds, and capture `.runner-grid`. Count `POST /examples/values` requests in DevTools to see the loop. `make browser-layout-test` covers the same contract with a stub server that always rejects. + ## Audit remediation (2026-07-10) These captures isolate the dark-mode Run-button contrast correction on the same `/examples/values` runner at a 1200×900 desktop viewport. diff --git a/docs/pr-evidence/turnstile-rejection-after-runner.png b/docs/pr-evidence/turnstile-rejection-after-runner.png new file mode 100644 index 0000000..d183ebb Binary files /dev/null and b/docs/pr-evidence/turnstile-rejection-after-runner.png differ diff --git a/docs/pr-evidence/turnstile-rejection-before-runner.png b/docs/pr-evidence/turnstile-rejection-before-runner.png new file mode 100644 index 0000000..3d00195 Binary files /dev/null and b/docs/pr-evidence/turnstile-rejection-before-runner.png differ diff --git a/docs/turnstile-runner-protection-spec.md b/docs/turnstile-runner-protection-spec.md index 581f7cf..cc68798 100644 --- a/docs/turnstile-runner-protection-spec.md +++ b/docs/turnstile-runner-protection-spec.md @@ -110,6 +110,7 @@ If `TURNSTILE_CHALLENGE_MODE=session` and both site/secret keys are configured: 7. The Worker validates the token through Siteverify. 8. On success, the Worker sets a signed `pbe_turnstile_clearance` cookie. 9. Later runs in that clearance window skip Turnstile and go straight to the Dynamic Worker. +10. If Siteverify rejects the token, the client shows the server's failure message and stops. Each Run earns at most one challenge; pressing Run again earns a fresh one. Siteverify endpoint from Cloudflare docs: @@ -160,6 +161,9 @@ x-pythonbyexample-smoke-secret: - The Turnstile widget should be configured in Cloudflare as **Invisible** mode. Client code uses explicit rendering with `execution: "execute"`; `size: "invisible"` is not a valid current Turnstile size option. - The widget is removed after callback success or failure. - Siteverify is called only when a challenge-required request retries with a token. +- A Run that sends a token and is challenged again does not solve again. Before this rule, a persistently failing Siteverify (wrong secret, hostname, or action) turned one click into an unbounded loop of solves, Worker POSTs, and Siteverify subrequests. +- Siteverify matches Cloudflare's reference contract (and the Turnstile Spin skill's canonical handler): `success is True`, the `run-example` action, the serving hostname, `remoteip` from `CF-Connecting-IP`, a 10-second `AbortSignal.timeout`, tokens over 2048 characters rejected before any subrequest, and every transport, status, payload, or type surprise failing closed. +- Each result lands in the wide event as `turnstile.outcome` (`challenged`, `pass`, `fail`, `bypass`, `disabled`) with a failure `reason` and allowlisted Siteverify `error_codes`, so `scripts/learner_report.py` can separate issued challenges from rejections and flag `invalid-input-secret` as a configuration alert. Siteverify reports a bad or missing secret with HTTP 400 and a JSON body, so the Worker reads error codes from non-2xx bodies too; a non-2xx response still never passes. - A valid challenge creates a signed, HttpOnly, Secure, SameSite=Lax clearance cookie scoped to `/examples`. - Failed or missing verification does not create a Dynamic Worker. - POST responses are still never cached. @@ -187,6 +191,26 @@ x-pythonbyexample-smoke-secret: This is only an app-level bypass. If Cloudflare Rate Limiting challenges smoke traffic before it reaches the Worker, either keep smoke below the threshold or add a Cloudflare-side skip/exception. +Because bypassed runs never touch Siteverify, the same header also unlocks a secret probe: + +```text +POST /__smoke/turnstile +x-pythonbyexample-smoke-secret: +``` + +The Worker sends its `TURNSTILE_SECRET_KEY` to Siteverify with Cloudflare's documented dummy token `XXXX.DUMMY.TOKEN.XXXX` and returns JSON (`Cache-Control: no-store`). It never includes the secret itself. Cloudflare documents that production secrets reject the dummy token; how they reject it tells the configurations apart: + +| Siteverify reply to the dummy token | `secret` | Smoke | +| --- | --- | --- | +| `invalid-input-response`, HTTP 200, no testing metadata | `valid` | passes when a site key is configured | +| `invalid-input-secret` or `missing-input-secret` (HTTP 400) | `invalid` | fails | +| `metadata.result_with_testing_key: true`, or `success: true` | `testing_key` | fails | +| unreachable, timeout, or unparseable | `unverified` | fails; re-run | +| anything else | `unexpected` | fails | +| no `TURNSTILE_SECRET_KEY` configured | `absent` | passes; Turnstile is off | + +The testing-metadata check matters: Cloudflare's always-fail test secret answers the dummy token with the same `invalid-input-response` as a working production secret, so without it a deployment where every learner fails would pass. Without the header, the route answers 404 and never calls Siteverify. The probe proves the secret is a working production secret. It cannot prove that the secret and site key belong to the same widget, or that the widget allows the production hostname; a browser run still covers those. + ## Recommended Cloudflare layer: Rate Limiting Rules The best first production protection for Dynamic Worker cost is Cloudflare Rate Limiting, because it runs before the Worker and limits repeated runner traffic at the edge. @@ -451,6 +475,8 @@ Deployment smoke OK The smoke script checks the rendered output panel for POST runs. This is important because a Turnstile-required response can echo submitted code in the editor textarea without executing it. +With `PBE_SMOKE_BYPASS_SECRET`, smoke also calls `POST /__smoke/turnstile` and fails unless the Worker's secret is a working production secret with a configured site key (see "Smoke-test behavior"). Without the secret, it prints `SKIP` for the probe. + ## Verification checklist ### Verify no visible widget on page load diff --git a/public/runner.96b105eef218.js b/public/runner.05ddb60aa9e7.js similarity index 97% rename from public/runner.96b105eef218.js rename to public/runner.05ddb60aa9e7.js index 52e2a6f..cdff252 100644 --- a/public/runner.96b105eef218.js +++ b/public/runner.05ddb60aa9e7.js @@ -210,6 +210,10 @@ function initializeRunner() { const document = new DOMParser().parseFromString(html, 'text/html'); const challengeRequired = document.querySelector('[data-turnstile-required]'); if (challengeRequired) { + // Each Run earns one challenge. If the server rejected the token just + // sent, report why instead of solving again: a wrong secret or hostname + // would otherwise loop through solves and Siteverify calls unbounded. + if (turnstileToken) throw new Error(responseErrorMessage(response, html)); outputPanel.querySelector('code').textContent = challengeRequired.textContent || 'Verification required before running edited code…'; const token = await requestTurnstileToken(signal); diff --git a/public/runner.js b/public/runner.js index 52e2a6f..cdff252 100644 --- a/public/runner.js +++ b/public/runner.js @@ -210,6 +210,10 @@ function initializeRunner() { const document = new DOMParser().parseFromString(html, 'text/html'); const challengeRequired = document.querySelector('[data-turnstile-required]'); if (challengeRequired) { + // Each Run earns one challenge. If the server rejected the token just + // sent, report why instead of solving again: a wrong secret or hostname + // would otherwise loop through solves and Siteverify calls unbounded. + if (turnstileToken) throw new Error(responseErrorMessage(response, html)); outputPanel.querySelector('code').textContent = challengeRequired.textContent || 'Verification required before running edited code…'; const token = await requestTurnstileToken(signal); diff --git a/scripts/check_browser_layout.mjs b/scripts/check_browser_layout.mjs index 38d476e..48bf30f 100755 --- a/scripts/check_browser_layout.mjs +++ b/scripts/check_browser_layout.mjs @@ -463,6 +463,57 @@ try { return { scriptAttempts, fetchCount, renderAction, submittedToken, firstFailure, finalOutput }; })()`); + // A server that keeps rejecting tokens (wrong secret, hostname, or action) + // must cost one challenge per Run. The capped stub stops a regressed + // challenge loop so this check reports it instead of hanging. + await client.send('Page.navigate', { url: `${new URL(target).origin}/?browser_turnstile_rejected=${Date.now()}` }); + await waitFor("document.readyState === 'complete'", 'Turnstile rejection fixture page'); + const turnstileRejected = await evaluateValue(`(async () => { + document.body.innerHTML = \` +
+
+ + +
+ + +
+
+

Output

Ready
+
\`; + const nativeFetch = window.fetch; + let solves = 0; + let widgetOptions = null; + window.turnstile = { + render: (_box, options) => { widgetOptions = options; return 9; }, + execute: () => queueMicrotask(() => widgetOptions.callback('rejected-token-' + (++solves))), + remove: () => {}, + reset: () => {}, + }; + let fetchCount = 0; + window.fetch = async () => { + fetchCount += 1; + const html = fetchCount < 20 + ? '
Verification required

Output

Turnstile verification failed. Press Run to try again.
' + : '

Output

loop cap reached
'; + return { ok: true, status: 200, text: async () => html }; + }; + await import(${JSON.stringify(runnerAsset)} + '?rejected=' + Date.now()); + const form = document.querySelector('form.runner-editor'); + const output = () => document.querySelector('.output-panel code')?.textContent || ''; + const run = async () => { + form.requestSubmit(); + for (let i = 0; i < 100 && form.hasAttribute('aria-busy'); i++) { + await new Promise(resolve => setTimeout(resolve, 10)); + } + return { fetchCount, solves, output: output(), busy: form.hasAttribute('aria-busy') }; + }; + const firstRun = await run(); + const secondRun = await run(); + window.fetch = nativeFetch; + return { firstRun, secondRun }; + })()`); + const failures = []; if (interaction.result.value?.ariaLabel !== 'Editable Python example code') { failures.push('CodeMirror editor is missing its accessible name'); @@ -504,6 +555,10 @@ try { if (!fragmentBound.unchanged || !fragmentBound.notice?.includes('invalid or too large')) failures.push('Oversized shared-code fragment was decoded or not announced'); if (turnstileRetry.scriptAttempts !== 2 || !turnstileRetry.firstFailure.includes('press Run to retry') || turnstileRetry.finalOutput !== 'retry succeeded') failures.push('Transient Turnstile CDN failure was cached instead of retried'); if (turnstileRetry.renderAction !== 'run-example' || turnstileRetry.submittedToken !== 'retry-token') failures.push('Turnstile browser action/token contract failed'); + const { firstRun: rejectedRun, secondRun: rejectedRetry } = turnstileRejected; + if (rejectedRun.fetchCount !== 2 || rejectedRun.solves !== 1) failures.push(`Rejected Turnstile token re-challenged in a loop (${rejectedRun.fetchCount} POSTs, ${rejectedRun.solves} solves for one Run)`); + if (!rejectedRun.output.includes('verification failed') || rejectedRun.busy) failures.push('Rejected Turnstile token did not end the run with the server message'); + if (rejectedRetry.fetchCount !== 4 || rejectedRetry.solves !== 2) failures.push('A new Run after a rejected token did not earn exactly one fresh challenge'); console.log(JSON.stringify({ runnerInteraction: interaction.result.value, @@ -519,6 +574,7 @@ try { offlineRunner, fragmentBound, turnstileRetry, + turnstileRejected, heldCdnRequests, }, null, 2)); client.close(); diff --git a/scripts/learner_report.py b/scripts/learner_report.py index 8738a5c..fcbf973 100755 --- a/scripts/learner_report.py +++ b/scripts/learner_report.py @@ -30,6 +30,14 @@ SERVICE = "pythonbyexample" +# Turnstile signals that mean the deployment is misconfigured rather than that +# a visitor failed: every challenged run fails until an operator fixes them. +TURNSTILE_CONFIG_ALERTS = { + "reason:site_key_missing": "TURNSTILE_SITE_KEY is not configured, so challenges cannot render", + "code:missing-input-secret": "Siteverify received no secret", + "code:invalid-input-secret": "TURNSTILE_SECRET_KEY is wrong, rotated, or from another widget", +} + def _looks_like_wide_event(value) -> bool: return isinstance(value, dict) and value.get("service") == SERVICE and "path" in value @@ -79,6 +87,8 @@ def aggregate_events(events: Iterable[dict]) -> dict: journey_views: Counter[str] = Counter() missing_example_paths: Counter[str] = Counter() turnstile_outcomes: Counter[str] = Counter() + turnstile_failure_reasons: Counter[str] = Counter() + turnstile_error_codes: Counter[str] = Counter() runs: dict[str, dict] = defaultdict( lambda: {"total": 0, "edited": 0, "errors": 0, "execution_ms": []} ) @@ -113,6 +123,11 @@ def aggregate_events(events: Iterable[dict]) -> dict: turnstile = event.get("turnstile") if isinstance(turnstile, dict) and turnstile.get("outcome"): turnstile_outcomes[str(turnstile["outcome"])] += 1 + if turnstile.get("reason"): + turnstile_failure_reasons[str(turnstile["reason"])] += 1 + error_codes = turnstile.get("error_codes") + if isinstance(error_codes, list): + turnstile_error_codes.update(str(code) for code in error_codes) run_summary = {} for slug, entry in runs.items(): @@ -130,10 +145,19 @@ def aggregate_events(events: Iterable[dict]) -> dict: "journey_views": dict(journey_views), "missing_example_paths": dict(missing_example_paths), "turnstile_outcomes": dict(turnstile_outcomes), + "turnstile_failure_reasons": dict(turnstile_failure_reasons), + "turnstile_error_codes": dict(turnstile_error_codes), + "turnstile_config_alerts": _turnstile_config_alerts(turnstile_failure_reasons, turnstile_error_codes), "runs": run_summary, } +def _turnstile_config_alerts(reasons: Counter[str], error_codes: Counter[str]) -> dict[str, int]: + signals = Counter({f"reason:{key}": count for key, count in reasons.items()}) + signals.update({f"code:{key}": count for key, count in error_codes.items()}) + return {signal: signals[signal] for signal in TURNSTILE_CONFIG_ALERTS if signals[signal]} + + def _top(counter: dict, limit: int) -> list[tuple[str, int]]: return sorted(counter.items(), key=lambda item: (-item[1], item[0]))[:limit] @@ -182,6 +206,24 @@ def render_report(report: dict, limit: int = 15) -> str: lines.append(f" {count:>6} {outcome}") lines.append("") + if report["turnstile_failure_reasons"]: + lines.append("Turnstile failure reasons") + for reason, count in _top(report["turnstile_failure_reasons"], limit): + lines.append(f" {count:>6} {reason}") + lines.append("") + + if report["turnstile_error_codes"]: + lines.append("Siteverify error codes") + for code, count in _top(report["turnstile_error_codes"], limit): + lines.append(f" {count:>6} {code}") + lines.append("") + + if report["turnstile_config_alerts"]: + lines.append("Turnstile configuration alerts (fix the deployment; these are not bot failures)") + for signal, count in _top(report["turnstile_config_alerts"], limit): + lines.append(f" {count:>6} {signal}: {TURNSTILE_CONFIG_ALERTS[signal]}") + lines.append("") + return "\n".join(lines) diff --git a/scripts/smoke_deployment.py b/scripts/smoke_deployment.py index cd4345b..eb43289 100755 --- a/scripts/smoke_deployment.py +++ b/scripts/smoke_deployment.py @@ -3,11 +3,16 @@ Usage: scripts/smoke_deployment.py https://www.pythonbyexample.dev + PBE_SMOKE_BYPASS_SECRET=... scripts/smoke_deployment.py https://www.pythonbyexample.dev + +With PBE_SMOKE_BYPASS_SECRET, POST runs skip Turnstile, so the script also +asks the Worker to prove its Turnstile secret against Siteverify. """ from __future__ import annotations import argparse import html +import json import os import re import sys @@ -35,6 +40,14 @@ ("subprocesses", "print('runtime-smoke-subprocess-boundary')\n", "runtime-smoke-subprocess-boundary"), ] ERROR_MARKERS = ["error code: 1101", "PythonError", "Traceback"] +SMOKE_BYPASS_HEADER = "x-pythonbyexample-smoke-secret" +TURNSTILE_PROBE_PATH = "/__smoke/turnstile" +TURNSTILE_PROBE_PROBLEMS = { + "invalid": "Siteverify rejects TURNSTILE_SECRET_KEY, so every challenged run fails", + "testing_key": "TURNSTILE_SECRET_KEY is a Cloudflare test secret, which ignores the visitor", + "unverified": "the Worker could not reach Siteverify; re-run smoke before trusting the secret", + "unexpected": "Siteverify answered the dummy token unexpectedly", +} def fetch(url: str) -> tuple[int, str]: @@ -51,7 +64,7 @@ def post_code(url: str, code: str, smoke_bypass_secret: str = "") -> tuple[int, "Content-Type": "application/x-www-form-urlencoded", } if smoke_bypass_secret: - headers["x-pythonbyexample-smoke-secret"] = smoke_bypass_secret + headers[SMOKE_BYPASS_HEADER] = smoke_bypass_secret request = urllib.request.Request( url, data=data, @@ -63,6 +76,30 @@ def post_code(url: str, code: str, smoke_bypass_secret: str = "") -> tuple[int, return response.status, body +def probe_turnstile(base_url: str, smoke_bypass_secret: str) -> dict: + request = urllib.request.Request( + urljoin(base_url, TURNSTILE_PROBE_PATH.lstrip("/")), + data=b"", + headers={"User-Agent": "pythonbyexample-smoke/1.0", SMOKE_BYPASS_HEADER: smoke_bypass_secret}, + method="POST", + ) + with urllib.request.urlopen(request, timeout=30) as response: + return json.loads(response.read().decode("utf-8")) + + +def turnstile_probe_problem(report: dict) -> str | None: + """Explain why the Worker's Turnstile configuration cannot verify browsers.""" + if report.get("ok") is True: + return None + secret = report.get("secret") + if secret == "valid" and not report.get("site_key_configured"): + return "TURNSTILE_SITE_KEY is not configured, so challenges cannot render" + problem = TURNSTILE_PROBE_PROBLEMS.get(secret, f"unrecognized probe report {report!r}") + if codes := report.get("error_codes"): + problem += f" ({', '.join(codes)})" + return problem + + def has_exception_marker(body: str) -> str | None: lowered = body.lower() for marker in ERROR_MARKERS: @@ -139,6 +176,21 @@ def main() -> int: failures.append(f"POST {url}: missing edited-code output {expected!r}") print(f"POST {status} {url} -> {expected}") + url = urljoin(base, TURNSTILE_PROBE_PATH.lstrip("/")) + if not smoke_bypass_value: + print(f"SKIP {url}: set PBE_SMOKE_BYPASS_SECRET to verify the deployed Turnstile secret") + else: + try: + report = probe_turnstile(base, smoke_bypass_value) + except urllib.error.HTTPError as exc: + failures.append(f"POST {url}: HTTP {exc.code} (deploy the probe route, or check PBE_SMOKE_BYPASS_SECRET)") + except Exception as exc: # noqa: BLE001 # pragma: no cover - report any transport failure + failures.append(f"POST {url}: {exc!r}") + else: + if problem := turnstile_probe_problem(report): + failures.append(f"POST {url}: {problem}") + print(f"POST {url} -> Turnstile secret {report.get('secret')}, mode {report.get('challenge_mode')}") + if failures: for failure in failures: print(failure, file=sys.stderr) diff --git a/src/asset_manifest.py b/src/asset_manifest.py index 6db62b1..d242400 100644 --- a/src/asset_manifest.py +++ b/src/asset_manifest.py @@ -1,3 +1,3 @@ # Generated by scripts/fingerprint_assets.py. Do not edit by hand. -ASSET_PATHS = {'SITE_CSS': '/site.07c59ae07f87.css', 'SYNTAX_JS': '/syntax-highlight.743e2535f133.js', 'EDITOR_JS': '/editor.3659b44c8480.js', 'RUNNER_JS': '/runner.96b105eef218.js', 'SEARCH_JS': '/search.e5a5822d9917.js', 'SEARCH_INDEX': '/search-index.332712b2e502.json'} -HTML_CACHE_VERSION = 'ce207aefedb8' +ASSET_PATHS = {'SITE_CSS': '/site.07c59ae07f87.css', 'SYNTAX_JS': '/syntax-highlight.743e2535f133.js', 'EDITOR_JS': '/editor.3659b44c8480.js', 'RUNNER_JS': '/runner.05ddb60aa9e7.js', 'SEARCH_JS': '/search.e5a5822d9917.js', 'SEARCH_INDEX': '/search-index.332712b2e502.json'} +HTML_CACHE_VERSION = 'deed3f3ebbb1' diff --git a/src/main.py b/src/main.py index c75dca1..d316c5b 100644 --- a/src/main.py +++ b/src/main.py @@ -39,7 +39,30 @@ TURNSTILE_VERIFY_URL = "https://challenges.cloudflare.com/turnstile/v0/siteverify" TURNSTILE_ACTION = "run-example" +TURNSTILE_FAILED_MESSAGE = "Turnstile verification failed. Press Run to try again." +# Cloudflare documents tokens as at most 2048 characters; longer values are +# rejected without spending a Siteverify subrequest. +MAX_TURNSTILE_TOKEN_CHARS = 2048 +SITEVERIFY_TIMEOUT_MS = 10_000 +# Siteverify's documented error codes. Only these names reach the wide event, +# so an unexpected payload cannot widen the logged vocabulary. +SITEVERIFY_ERROR_CODES = frozenset( + { + "missing-input-secret", + "invalid-input-secret", + "missing-input-response", + "invalid-input-response", + "bad-request", + "timeout-or-duplicate", + "internal-error", + } +) SMOKE_BYPASS_HEADER = "x-pythonbyexample-smoke-secret" +TURNSTILE_PROBE_PATH = "/__smoke/turnstile" +# Cloudflare's documented dummy token. Production secrets reject it with +# invalid-input-response and a bad secret gets invalid-input-secret, so it +# proves the deployed secret without solving a challenge. +TURNSTILE_PROBE_TOKEN = "XXXX.DUMMY.TOKEN.XXXX" TURNSTILE_CLEARANCE_COOKIE = "pbe_turnstile_clearance" DEFAULT_TURNSTILE_CLEARANCE_SECONDS = 60 * 60 * 8 # Generous ceiling for an edited example program (the largest curated @@ -48,11 +71,12 @@ MAX_SUBMITTED_BODY_BYTES = 100_000 try: - from js import Object, caches + from js import AbortSignal, Object, caches from js import Request as JsRequest from js import fetch as js_fetch from pyodide.ffi import create_once_callable, jsnull, to_js except ImportError: # Allows editor tooling outside Workers. + AbortSignal = None Object = None JsRequest = None jsnull = None @@ -320,12 +344,16 @@ async def run_example(slug: str, request: Request): turnstile_verified = False if needs_turnstile and not turnstile_token: - if event := _wide_event(request): - event["turnstile"] = {"outcome": "fail"} site_key = _turnstile_site_key(request) message = "Verification required before running edited code." + # Issuing a challenge is the normal first run of a session, not a + # failure; keeping it apart lets reports compare issued vs rejected. + turnstile_event = {"outcome": "challenged"} if not site_key: message = "Turnstile verification is required, but TURNSTILE_SITE_KEY is not configured." + turnstile_event = {"outcome": "fail", "reason": "site_key_missing"} + if event := _wide_event(request): + event["turnstile"] = turnstile_event return _html( render_example_page( example, @@ -337,9 +365,9 @@ async def run_example(slug: str, request: Request): ) if needs_turnstile: - ok, message, turnstile_outcome = await _verify_turnstile(request, turnstile_token) + ok, message, turnstile_event = await _verify_turnstile(request, turnstile_token) if event := _wide_event(request): - event["turnstile"] = {"outcome": turnstile_outcome} + event["turnstile"] = turnstile_event if not ok: return _html( render_example_page( @@ -377,6 +405,61 @@ async def run_example(slug: str, request: Request): return response +@app.post(TURNSTILE_PROBE_PATH) +async def turnstile_probe(request: Request): + # Deployment smoke skips challenges with the bypass header, so it needs its + # own proof that the production secret works. The same header gates this. + if not _smoke_bypass_ok(request): + return _html(render_not_found(), 404) + report = await _probe_turnstile(request) + if event := _wide_event(request): + event["turnstile_probe"] = {"secret": report["secret"], "ok": report["ok"]} + return Response(json.dumps(report, sort_keys=True), media_type="application/json") + + +async def _probe_turnstile(request: Request) -> dict: + """Report whether the deployed Turnstile configuration can verify browsers.""" + site_key_configured = bool(_turnstile_site_key(request)) + report = { + "challenge_mode": _turnstile_challenge_mode(request), + "site_key_configured": site_key_configured, + } + secret = _turnstile_secret(request) + if not secret: + report["secret"] = "absent" + elif js_fetch is None or JsRequest is None: + report["secret"] = "unverified" + else: + report["secret"], error_codes = _classify_probe_reply( + await _call_siteverify(secret, TURNSTILE_PROBE_TOKEN) + ) + if error_codes and report["secret"] != "valid": + report["error_codes"] = error_codes + # No secret leaves Turnstile off by configuration. A configured secret must + # be a working production secret with a site key, or challenges cannot pass. + report["ok"] = report["secret"] == "absent" or (report["secret"] == "valid" and site_key_configured) + return report + + +def _classify_probe_reply(reply: tuple[int, dict] | None) -> tuple[str, list[str]]: + if reply is None: + return "unverified", [] + status, result = reply + error_codes = _siteverify_error_codes(result) + metadata = result.get("metadata") + # Only test secrets accept the dummy token, and the always-fail test secret + # answers exactly like a production one; this flag tells them apart. + if result.get("success") is True or ( + isinstance(metadata, dict) and metadata.get("result_with_testing_key") is True + ): + return "testing_key", error_codes + if {"invalid-input-secret", "missing-input-secret"} & set(error_codes): + return "invalid", error_codes + if 200 <= status < 300 and error_codes == ["invalid-input-response"]: + return "valid", error_codes + return "unexpected", error_codes + + def _turnstile_not_required_outcome(request: Request) -> str: if not _turnstile_secret(request) or _turnstile_challenge_mode(request) == "off": return "disabled" @@ -387,50 +470,94 @@ def _turnstile_not_required_outcome(request: Request) -> str: return "disabled" -async def _verify_turnstile(request: Request, token: str) -> tuple[bool, str, str]: - secret = _turnstile_secret(request) - if not secret: - return True, "", "disabled" +def _turnstile_failure(reason: str, error_codes: list[str] | None = None) -> tuple[bool, str, dict]: + turnstile_event = {"outcome": "fail", "reason": reason} + if error_codes: + turnstile_event["error_codes"] = error_codes + return False, TURNSTILE_FAILED_MESSAGE, turnstile_event - if _smoke_bypass_ok(request): - return True, "", "bypass" - if not token: - return False, "Turnstile verification is required before running edited code. Please retry.", "fail" - if js_fetch is None or JsRequest is None: - return False, "Turnstile verification is unavailable outside the Cloudflare runtime.", "fail" +def _siteverify_error_codes(result: dict) -> list[str]: + codes = result.get("error-codes") + if not isinstance(codes, list): + return [] + return sorted( + {code if code in SITEVERIFY_ERROR_CODES else "other" for code in codes if isinstance(code, str)} + ) + +async def _call_siteverify(secret: str, token: str, remote_ip: str | None = None) -> tuple[int, dict] | None: + """POST one token to Siteverify; None when it is unreachable or unparseable.""" payload = {"secret": secret, "response": token} - remote_ip = request.headers.get("CF-Connecting-IP") if remote_ip: payload["remoteip"] = remote_ip - verify_request = JsRequest.new( - TURNSTILE_VERIFY_URL, - _to_js_object( - { - "method": "POST", - "body": urlencode(payload), - "headers": {"Content-Type": "application/x-www-form-urlencoded"}, - } - ), - ) + init = { + "method": "POST", + "body": urlencode(payload), + "headers": {"Content-Type": "application/x-www-form-urlencoded"}, + } + if AbortSignal is not None: + # A hung Siteverify must not hold a learner's run or a smoke probe open. + init["signal"] = AbortSignal.timeout(SITEVERIFY_TIMEOUT_MS) + verify_request = JsRequest.new(TURNSTILE_VERIFY_URL, _to_js_object(init)) try: response = await js_fetch(verify_request) - if int(getattr(response, "status", 0) or 0) < 200 or int(getattr(response, "status", 0) or 0) >= 300: - raise ValueError("Turnstile Siteverify returned a non-success status") + status = int(getattr(response, "status", 0) or 0) result = json.loads(await response.text()) - if not isinstance(result, dict): - raise TypeError("Turnstile Siteverify returned a non-object payload") - except Exception: # noqa: BLE001 - JS fetch failures must become a generic verification failure - return False, "Turnstile verification failed. Please refresh the challenge and try again.", "fail" - expected_hostname = urlparse(str(request.url)).hostname or "" - if ( - result.get("success") is True - and result.get("hostname", "").lower().rstrip(".") == expected_hostname.lower().rstrip(".") - and result.get("action") == TURNSTILE_ACTION - ): - return True, "", "pass" - return False, "Turnstile verification failed. Please refresh the challenge and try again.", "fail" + except Exception: # noqa: BLE001 - JS fetch failures, timeouts, and unparseable bodies fail closed + return None + return (status, result) if isinstance(result, dict) else None + + +async def _verify_turnstile(request: Request, token: str) -> tuple[bool, str, dict]: + """Return (ok, learner message, wide-event turnstile fields). + + Failures carry a closed-vocabulary ``reason`` and, for Siteverify + rejections, its documented ``error_codes``: ``invalid-input-secret`` + means the deployment is misconfigured, not that a learner is a bot. + """ + secret = _turnstile_secret(request) + if not secret: + return True, "", {"outcome": "disabled"} + + if _smoke_bypass_ok(request): + return True, "", {"outcome": "bypass"} + + if not token: + return ( + False, + "Turnstile verification is required before running edited code. Please retry.", + {"outcome": "fail", "reason": "missing_token"}, + ) + if len(token) > MAX_TURNSTILE_TOKEN_CHARS: + return _turnstile_failure("token_too_long") + if js_fetch is None or JsRequest is None: + return ( + False, + "Turnstile verification is unavailable outside the Cloudflare runtime.", + {"outcome": "fail", "reason": "runtime_unavailable"}, + ) + + reply = await _call_siteverify(secret, token, request.headers.get("CF-Connecting-IP")) + if reply is None: + return _turnstile_failure("siteverify_unavailable") + status, result = reply + # Siteverify answers a bad or missing secret with HTTP 400 and a JSON body + # naming it, so read error codes before judging the status: discarding + # non-2xx bodies would report a misconfigured deployment as an outage. + status_ok = 200 <= status < 300 + error_codes = _siteverify_error_codes(result) + if not status_ok and not error_codes: + return _turnstile_failure("siteverify_unavailable") + if not status_ok or result.get("success") is not True: + return _turnstile_failure("rejected", error_codes) + hostname = result.get("hostname") + expected_hostname = (urlparse(str(request.url)).hostname or "").lower().rstrip(".") + if not expected_hostname or not isinstance(hostname, str) or hostname.lower().rstrip(".") != expected_hostname: + return _turnstile_failure("hostname_mismatch") + if result.get("action") != TURNSTILE_ACTION: + return _turnstile_failure("action_mismatch") + return True, "", {"outcome": "pass"} async def _read_dynamic_response_text(response) -> tuple[str, bool]: diff --git a/tests/test_learner_report.py b/tests/test_learner_report.py index 389cb63..8cd1803 100644 --- a/tests/test_learner_report.py +++ b/tests/test_learner_report.py @@ -100,5 +100,58 @@ def test_render_report_is_readable(self): self.assertIn("/examples/not-a-real-example", text) +class TurnstileReportTests(unittest.TestCase): + def turnstile_run(self, turnstile): + event = example_run("closures") + event["turnstile"] = turnstile + return event + + def build_report(self): + return aggregate_events( + [ + self.turnstile_run({"outcome": "challenged"}), + self.turnstile_run({"outcome": "challenged"}), + self.turnstile_run({"outcome": "pass"}), + self.turnstile_run( + {"outcome": "fail", "reason": "rejected", "error_codes": ["invalid-input-secret"]} + ), + self.turnstile_run( + {"outcome": "fail", "reason": "rejected", "error_codes": ["timeout-or-duplicate"]} + ), + self.turnstile_run({"outcome": "fail", "reason": "hostname_mismatch"}), + self.turnstile_run({"outcome": "fail", "reason": "site_key_missing"}), + ] + ) + + def test_separates_issued_challenges_from_failures(self): + report = self.build_report() + self.assertEqual(report["turnstile_outcomes"], {"challenged": 2, "pass": 1, "fail": 4}) + self.assertEqual( + report["turnstile_failure_reasons"], + {"rejected": 2, "hostname_mismatch": 1, "site_key_missing": 1}, + ) + self.assertEqual( + report["turnstile_error_codes"], {"invalid-input-secret": 1, "timeout-or-duplicate": 1} + ) + + def test_flags_misconfiguration_apart_from_visitor_failures(self): + report = self.build_report() + self.assertEqual( + report["turnstile_config_alerts"], + {"code:invalid-input-secret": 1, "reason:site_key_missing": 1}, + ) + text = render_report(report) + self.assertIn("Turnstile configuration alerts", text) + self.assertIn("TURNSTILE_SECRET_KEY is wrong", text) + self.assertIn("Siteverify error codes", text) + + def test_visitor_failures_raise_no_configuration_alert(self): + report = aggregate_events( + [self.turnstile_run({"outcome": "fail", "reason": "rejected", "error_codes": ["invalid-input-response"]})] + ) + self.assertEqual(report["turnstile_config_alerts"], {}) + self.assertNotIn("configuration alerts", render_report(report)) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_main_turnstile.py b/tests/test_main_turnstile.py index 52744fb..8d0d385 100644 --- a/tests/test_main_turnstile.py +++ b/tests/test_main_turnstile.py @@ -8,6 +8,7 @@ from __future__ import annotations import asyncio +import json import sys import time import types @@ -309,82 +310,328 @@ def test_verified_token_sets_clearance_cookie(self): self._stub_run(output="verified-run") async def _fake_verify(request, token): - return True, "ok", "success" + return True, "", {"outcome": "pass"} main._verify_turnstile = _fake_verify env = session_env(TURNSTILE_SITE_KEY="site-key") - response = self._run(post_request(b"code=print(1)&cf-turnstile-response=tok", env=env)) + request = post_request(b"code=print(1)&cf-turnstile-response=tok", env=env) + request.state.wide_event = {"path": "/examples/values"} + response = self._run(request) self.assertEqual(self._ran_code, "print(1)") set_cookie = response.headers.get("set-cookie", "") self.assertIn(f"{main.TURNSTILE_CLEARANCE_COOKIE}=", set_cookie) + self.assertEqual(request.state.wide_event["turnstile"], {"outcome": "pass"}) + + def test_issued_challenge_is_recorded_apart_from_failures(self): + # A challenge is the normal first run of a session; recording it as + # "fail" hid real rejections among ordinary first runs. + self._stub_run() + request = post_request(b"code=print(1)", env=session_env(TURNSTILE_SITE_KEY="site-key")) + request.state.wide_event = {"path": "/examples/values"} + self._run(request) + self.assertEqual(request.state.wide_event["turnstile"], {"outcome": "challenged"}) + + misconfigured = post_request(b"code=print(1)", env=session_env()) + misconfigured.state.wide_event = {"path": "/examples/values"} + self._run(misconfigured) + self.assertEqual( + misconfigured.state.wide_event["turnstile"], {"outcome": "fail", "reason": "site_key_missing"} + ) + + def test_rejected_token_records_reason_and_does_not_run_code(self): + self._ran_code = None + self._stub_run() + + async def _rejecting_verify(request, token): + return main._turnstile_failure("rejected", ["invalid-input-secret"]) + + main._verify_turnstile = _rejecting_verify + request = post_request( + b"code=print(1)&cf-turnstile-response=tok", env=session_env(TURNSTILE_SITE_KEY="site-key") + ) + request.state.wide_event = {"path": "/examples/values"} + response = self._run(request) + body = response.body.decode() + self.assertIsNone(self._ran_code) + self.assertIn(main.TURNSTILE_FAILED_MESSAGE, body) + self.assertIn('data-turnstile-required="true"', body) + self.assertNotIn("set-cookie", response.headers) + self.assertEqual( + request.state.wide_event["turnstile"], + {"outcome": "fail", "reason": "rejected", "error_codes": ["invalid-input-secret"]}, + ) + + +class _SiteverifyResponse: + def __init__(self, status, body): + self.status, self.body = status, body + + async def text(self): + return self.body class TurnstileSiteverifyTests(unittest.TestCase): def setUp(self): - self.saved_fetch = main.js_fetch - self.saved_request = main.JsRequest + self.saved = {name: getattr(main, name) for name in ("js_fetch", "JsRequest", "AbortSignal")} main.JsRequest = type("Request", (), {"new": staticmethod(lambda url, options: (url, options))}) + self.sent = [] def tearDown(self): - main.js_fetch = self.saved_fetch - main.JsRequest = self.saved_request + for name, value in self.saved.items(): + setattr(main, name, value) - def test_transport_status_and_payload_failures_fail_closed(self): - class Response: - def __init__(self, status, body): - self.status, self.body = status, body - async def text(self): - return self.body + def _respond(self, body, status=200): + async def fetch(request): + self.sent.append(request) + return _SiteverifyResponse(status, body) + + main.js_fetch = fetch + def _verify(self, token="token"): + return asyncio.run(main._verify_turnstile(make_request(env=session_env()), token)) + + def test_transport_status_and_payload_failures_fail_closed(self): async def failing(_request): raise RuntimeError("network down") - request = make_request(env=session_env()) for fetch in ( failing, - lambda _request: _awaitable(Response(500, '{"success": true}')), - lambda _request: _awaitable(Response(200, "not json")), - lambda _request: _awaitable(Response(200, "[]")), - lambda _request: _awaitable(Response(200, '{"success": false}')), + lambda _request: _awaitable(_SiteverifyResponse(500, '{"success": true}')), + lambda _request: _awaitable(_SiteverifyResponse(503, "upstream down")), + lambda _request: _awaitable(_SiteverifyResponse(200, "not json")), + lambda _request: _awaitable(_SiteverifyResponse(200, "[]")), ): with self.subTest(fetch=fetch): main.js_fetch = fetch - ok, message, outcome = asyncio.run(main._verify_turnstile(request, "token")) + ok, message, details = self._verify() self.assertFalse(ok) self.assertIn("verification failed", message) - self.assertEqual(outcome, "fail") + self.assertEqual(details, {"outcome": "fail", "reason": "siteverify_unavailable"}) + + def test_siteverify_request_carries_a_timeout_signal(self): + main.AbortSignal = type("AbortSignal", (), {"timeout": staticmethod(lambda ms: ("timeout", ms))}) + self._respond('{"success": true, "hostname": "www.pythonbyexample.dev", "action": "run-example"}') + self._verify() + url, options = self.sent[0] + self.assertEqual(url, main.TURNSTILE_VERIFY_URL) + self.assertEqual(options["signal"], ("timeout", main.SITEVERIFY_TIMEOUT_MS)) + self.assertEqual(main.SITEVERIFY_TIMEOUT_MS, 10_000) + + def test_overlong_token_is_rejected_without_calling_siteverify(self): + self._respond('{"success": true, "hostname": "www.pythonbyexample.dev", "action": "run-example"}') + ok, _, details = self._verify("x" * (main.MAX_TURNSTILE_TOKEN_CHARS + 1)) + self.assertFalse(ok) + self.assertEqual(details, {"outcome": "fail", "reason": "token_too_long"}) + self.assertEqual(self.sent, []) + + ok, _, _ = self._verify("x" * main.MAX_TURNSTILE_TOKEN_CHARS) + self.assertTrue(ok) + self.assertEqual(len(self.sent), 1) def test_successful_siteverify_requires_matching_hostname_and_action(self): - class Response: - status = 200 - async def text(self): - return '{"success": true, "hostname": "www.pythonbyexample.dev", "action": "run-example"}' - - main.js_fetch = lambda _request: _awaitable(Response()) - ok, message, outcome = asyncio.run(main._verify_turnstile(make_request(env=session_env()), "token")) + self._respond('{"success": true, "hostname": "www.pythonbyexample.dev", "action": "run-example"}') + ok, message, details = self._verify() self.assertTrue(ok) self.assertEqual(message, "") - self.assertEqual(outcome, "pass") - - def test_siteverify_rejects_wrong_or_missing_hostname_and_action(self): - payloads = [ - '{"success": true}', - '{"success": true, "hostname": "other.example", "action": "run-example"}', - '{"success": true, "hostname": "www.pythonbyexample.dev", "action": "other-action"}', + self.assertEqual(details, {"outcome": "pass"}) + + def test_siteverify_rejects_wrong_missing_or_non_string_hostname_and_action(self): + cases = [ + ('{"success": true}', "hostname_mismatch"), + ('{"success": true, "hostname": null, "action": "run-example"}', "hostname_mismatch"), + ('{"success": true, "hostname": 7, "action": "run-example"}', "hostname_mismatch"), + ('{"success": true, "hostname": "other.example", "action": "run-example"}', "hostname_mismatch"), + ('{"success": true, "hostname": "www.pythonbyexample.dev", "action": "other-action"}', "action_mismatch"), + ('{"success": true, "hostname": "www.pythonbyexample.dev"}', "action_mismatch"), ] - for payload in payloads: + for payload, reason in cases: with self.subTest(payload=payload): - class Response: - status = 200 - - async def text(self, response_payload=payload): - return response_payload - - main.js_fetch = lambda _request: _awaitable(Response()) - ok, message, outcome = asyncio.run(main._verify_turnstile(make_request(env=session_env()), "token")) + self._respond(payload) + ok, message, details = self._verify() self.assertFalse(ok) self.assertIn("verification failed", message) - self.assertEqual(outcome, "fail") + self.assertEqual(details, {"outcome": "fail", "reason": reason}) + + def test_rejection_records_only_documented_error_codes(self): + cases = [ + ('{"success": false}', {"outcome": "fail", "reason": "rejected"}), + ( + '{"success": false, "error-codes": ["invalid-input-secret"]}', + {"outcome": "fail", "reason": "rejected", "error_codes": ["invalid-input-secret"]}, + ), + ( + '{"success": false, "error-codes": ["timeout-or-duplicate", "x-new-code", 5, {"a": 1}]}', + {"outcome": "fail", "reason": "rejected", "error_codes": ["other", "timeout-or-duplicate"]}, + ), + ('{"success": "true", "error-codes": "bad-request"}', {"outcome": "fail", "reason": "rejected"}), + ] + for payload, expected in cases: + with self.subTest(payload=payload): + self._respond(payload) + ok, _, details = self._verify() + self.assertFalse(ok) + self.assertEqual(details, expected) + + def test_http_400_secret_errors_are_configuration_codes_not_outages(self): + # Live Siteverify returns these exact bodies with HTTP 400. Treating + # every non-2xx as "unavailable" hid invalid-input-secret entirely. + cases = [ + ('{"error-codes":["invalid-input-secret"],"success":false,"messages":[]}', "invalid-input-secret"), + ('{"error-codes":["missing-input-secret"],"success":false,"messages":[]}', "missing-input-secret"), + ] + for payload, code in cases: + with self.subTest(code=code): + self._respond(payload, status=400) + ok, _, details = self._verify() + self.assertFalse(ok) + self.assertEqual(details, {"outcome": "fail", "reason": "rejected", "error_codes": [code]}) + + def test_non_2xx_never_passes_even_when_the_body_claims_success(self): + self._respond( + '{"success": true, "hostname": "www.pythonbyexample.dev", "action": "run-example", "error-codes": ["internal-error"]}', + status=500, + ) + ok, _, details = self._verify() + self.assertFalse(ok) + self.assertEqual(details, {"outcome": "fail", "reason": "rejected", "error_codes": ["internal-error"]}) + + +# Replies live Siteverify gave to the dummy token, per secret. +PRODUCTION_SECRET_REPLY = (200, '{"success": false, "error-codes": ["invalid-input-response"]}') +BAD_SECRET_REPLY = (400, '{"error-codes":["invalid-input-secret"],"success":false,"messages":[]}') +TEST_SECRET_REPLIES = { + "always-pass": ( + 200, + ( + '{"challenge_ts":"2026-09-25T22:08:06.790Z","error-codes":[],"hostname":"example.com",' + '"metadata":{"result_with_testing_key":true},"success":true}' + ), + ), + "always-fail": ( + 200, + ( + '{"error-codes":["invalid-input-response"],"success":false,"messages":[],' + '"metadata":{"result_with_testing_key":true}}' + ), + ), + "token-spent": ( + 200, + ( + '{"error-codes":["timeout-or-duplicate"],"success":false,"messages":[],' + '"metadata":{"result_with_testing_key":true}}' + ), + ), +} + + +class TurnstileProbeTests(unittest.TestCase): + """The smoke-gated probe proves the deployed secret with the dummy token. + + Response bodies are the ones live Siteverify returned for the dummy token. + """ + + def setUp(self): + self.saved = {name: getattr(main, name) for name in ("js_fetch", "JsRequest", "AbortSignal")} + main.JsRequest = type("Request", (), {"new": staticmethod(lambda url, options: (url, options))}) + main.AbortSignal = None + self.sent = [] + + def tearDown(self): + for name, value in self.saved.items(): + setattr(main, name, value) + + def _respond(self, status, body): + async def fetch(request): + self.sent.append(request) + return _SiteverifyResponse(status, body) + + main.js_fetch = fetch + + def _smoke_request(self, header="smoke-secret", **env): + values = {"PBE_SMOKE_BYPASS_SECRET": "smoke-secret", "TURNSTILE_SITE_KEY": "site-key"} + values.update(env) + headers = {main.SMOKE_BYPASS_HEADER: header} if header else {} + return make_request(headers=headers, env=session_env(**values)) + + def _probe(self, request): + return asyncio.run(main._probe_turnstile(request)) + + def test_working_production_secret_is_ok(self): + self._respond(*PRODUCTION_SECRET_REPLY) + report = self._probe(self._smoke_request()) + self.assertEqual( + report, + {"challenge_mode": "session", "site_key_configured": True, "secret": "valid", "ok": True}, + ) + _, options = self.sent[0] + self.assertIn(f"response={main.TURNSTILE_PROBE_TOKEN}", options["body"]) + self.assertIn("secret=server-secret", options["body"]) + self.assertNotIn("remoteip", options["body"]) + + def test_bad_secret_fails_with_its_error_code(self): + self._respond(*BAD_SECRET_REPLY) + report = self._probe(self._smoke_request()) + self.assertEqual(report["secret"], "invalid") + self.assertEqual(report["error_codes"], ["invalid-input-secret"]) + self.assertFalse(report["ok"]) + + def test_every_test_secret_fails_including_the_one_that_mimics_production(self): + for name, response in TEST_SECRET_REPLIES.items(): + with self.subTest(secret=name): + self._respond(*response) + report = self._probe(self._smoke_request()) + self.assertEqual(report["secret"], "testing_key") + self.assertFalse(report["ok"]) + + def test_valid_secret_without_site_key_is_not_ok(self): + self._respond(*PRODUCTION_SECRET_REPLY) + report = self._probe(self._smoke_request(TURNSTILE_SITE_KEY="")) + self.assertEqual(report["secret"], "valid") + self.assertFalse(report["site_key_configured"]) + self.assertFalse(report["ok"]) + + def test_absent_secret_reports_turnstile_off_without_calling_siteverify(self): + self._respond(*PRODUCTION_SECRET_REPLY) + report = self._probe(self._smoke_request(TURNSTILE_SECRET_KEY="")) + self.assertEqual(report["secret"], "absent") + self.assertTrue(report["ok"]) + self.assertEqual(self.sent, []) + + def test_unreachable_or_unexpected_siteverify_is_not_ok(self): + async def failing(_request): + raise RuntimeError("network down") + + main.js_fetch = failing + self.assertEqual(self._probe(self._smoke_request())["secret"], "unverified") + for status, body in ( + (200, '{"success": false, "error-codes": ["internal-error"]}'), + (200, '{"success": false}'), + (503, '{"success": false, "error-codes": ["invalid-input-response"]}'), + ): + with self.subTest(status=status, body=body): + self._respond(status, body) + report = self._probe(self._smoke_request()) + self.assertEqual(report["secret"], "unexpected") + self.assertFalse(report["ok"]) + + def test_route_requires_the_smoke_header_and_returns_json(self): + self._respond(*BAD_SECRET_REPLY) + for header in ("", "wrong-secret"): + with self.subTest(header=header): + response = asyncio.run(main.turnstile_probe(self._smoke_request(header=header))) + self.assertEqual(response.status_code, 404) + self.assertEqual(self.sent, [], "an unauthenticated probe must not reach Siteverify") + + request = self._smoke_request() + request.state.wide_event = {"path": main.TURNSTILE_PROBE_PATH} + response = asyncio.run(main.turnstile_probe(request)) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.media_type, "application/json") + report = json.loads(response.body) + self.assertEqual(report["secret"], "invalid") + self.assertFalse(report["ok"]) + self.assertEqual(request.state.wide_event["turnstile_probe"], {"secret": "invalid", "ok": False}) + self.assertNotIn("server-secret", response.body.decode()) async def _awaitable(value): diff --git a/tests/test_smoke_deployment.py b/tests/test_smoke_deployment.py index 9600d37..cb49a02 100644 --- a/tests/test_smoke_deployment.py +++ b/tests/test_smoke_deployment.py @@ -1,6 +1,12 @@ +import contextlib import importlib.util +import io +import os import pathlib +import sys import unittest +import urllib.error +from unittest import mock ROOT = pathlib.Path(__file__).resolve().parents[1] @@ -41,5 +47,81 @@ def test_smoke_bypass_secret_requires_https_origin(self): ) +OK_PROBE_REPORT = {"challenge_mode": "session", "ok": True, "secret": "valid", "site_key_configured": True} + + +class TurnstileProbeSmokeTests(unittest.TestCase): + """Bypass-authenticated smoke must still prove the deployed Turnstile secret.""" + + def setUp(self): + self.smoke = load_smoke_module() + self.probes = [] + + def run_smoke(self, report=None, *, secret="smoke-secret", probe_error=None): + smoke = self.smoke + + def fake_probe(base_url, smoke_bypass_secret): + self.probes.append((base_url, smoke_bypass_secret)) + if probe_error is not None: + raise probe_error + return report + + def fake_post(url, code, smoke_bypass_secret=""): + expected = next(marker for _, source, marker in smoke.POST_SMOKES if source == code) + return 200, f'
{expected}
' + + stdout, stderr = io.StringIO(), io.StringIO() + env = {"PBE_SMOKE_BYPASS_SECRET": secret} if secret else {} + with ( + mock.patch.object(smoke, "fetch", lambda url: (200, "ok")), + mock.patch.object(smoke, "post_code", fake_post), + mock.patch.object(smoke, "probe_turnstile", fake_probe), + mock.patch.object(sys, "argv", ["smoke_deployment.py", "https://www.pythonbyexample.dev"]), + mock.patch.dict(os.environ, env, clear=True), + contextlib.redirect_stdout(stdout), + contextlib.redirect_stderr(stderr), + ): + code = smoke.main() + return code, stdout.getvalue(), stderr.getvalue() + + def test_working_secret_passes_and_is_reported(self): + code, stdout, _ = self.run_smoke(OK_PROBE_REPORT) + self.assertEqual(code, 0) + self.assertEqual(self.probes, [("https://www.pythonbyexample.dev/", "smoke-secret")]) + self.assertIn("Turnstile secret valid, mode session", stdout) + self.assertIn("Deployment smoke OK", stdout) + + def test_bad_secret_fails_smoke_even_though_bypassed_runs_pass(self): + report = {**OK_PROBE_REPORT, "ok": False, "secret": "invalid", "error_codes": ["invalid-input-secret"]} + code, _, stderr = self.run_smoke(report) + self.assertEqual(code, 1) + self.assertIn("Siteverify rejects TURNSTILE_SECRET_KEY", stderr) + self.assertIn("invalid-input-secret", stderr) + + def test_missing_probe_route_or_wrong_bypass_secret_fails(self): + error = urllib.error.HTTPError("https://www.pythonbyexample.dev/__smoke/turnstile", 404, "Not Found", {}, None) + code, _, stderr = self.run_smoke(probe_error=error) + self.assertEqual(code, 1) + self.assertIn("HTTP 404", stderr) + + def test_without_bypass_secret_the_probe_is_skipped_not_failed(self): + code, stdout, _ = self.run_smoke(secret="") + self.assertEqual(self.probes, []) + self.assertIn("SKIP", stdout) + self.assertEqual(code, 0) + + def test_probe_problems_name_the_misconfiguration(self): + problem = self.smoke.turnstile_probe_problem + self.assertIsNone(problem(OK_PROBE_REPORT)) + self.assertIsNone(problem({"ok": True, "secret": "absent", "site_key_configured": False})) + self.assertIn("test secret", problem({"ok": False, "secret": "testing_key"})) + self.assertIn( + "TURNSTILE_SITE_KEY", problem({"ok": False, "secret": "valid", "site_key_configured": False}) + ) + self.assertIn("could not reach Siteverify", problem({"ok": False, "secret": "unverified"})) + self.assertIn("internal-error", problem({"ok": False, "secret": "unexpected", "error_codes": ["internal-error"]})) + self.assertIn("unrecognized", problem({"ok": False, "secret": "new-state"})) + + if __name__ == "__main__": unittest.main()