Bound the Turnstile retry loop and prove the production secret

CHANGELOG.md

@@ -8,6 +8,10 @@ The format is inspired by [Keep a Changelog](https://keepachangelog.com/en/1.1.0
 
 ### Fixed
 
+- Deployment smoke now proves the production Turnstile secret. Its bypass header skips Siteverify, so a wrong, rotated, or test secret used to pass smoke while every new session's first run failed. With `PBE_SMOKE_BYPASS_SECRET`, `scripts/smoke_deployment.py` calls the header-gated `POST /__smoke/turnstile`. The Worker sends its secret with Cloudflare's dummy token and reports `valid`, `invalid`, `testing_key`, `unverified`, `unexpected`, or `absent`. Smoke fails unless the secret is a working production secret with a site key, or Turnstile is off.
+
+- A Run whose Turnstile token the server rejects now ends with the server's message instead of re-challenging. The runner used to solve again every time the response carried the challenge marker, so a persistently failing Siteverify (wrong secret, hostname, or action) turned one click into an unbounded loop of solves, Worker POSTs, and Siteverify subrequests. Each Run now earns at most one challenge, and the failure message says to press Run again.
+- Siteverify calls time out after 10 seconds, tokens longer than Cloudflare's documented 2048 characters are rejected without a subrequest, and a `success: true` response with a missing or non-string `hostname` fails closed instead of raising a 500. The gaps surfaced by comparing `_verify_turnstile` with the canonical handler in Cloudflare's Turnstile Spin skill.
 - Production Python packages are locked. Pywrangler 1.17.4 vendors a committed, hash-pinned `pylock.toml` instead of resolving an unpinned `fastapi` at deploy time, and `uv.lock` pins the same versions (FastAPI 0.141.1, Starlette 1.7.0, Pydantic 2.10.6) so the test suite runs against exactly what ships. `tests/test_dependency_locks.py` fails on drift, and CI and `make deploy` fail if a sync would change `pylock.toml`. Starlette 1.7.0 also clears the five advisories against the previously tested 1.0.0.
 - Example-page runner wiring (Run interception, Reset, the share button, and keyboard navigation) no longer waits for the CDN-backed highlighter and editor modules: `runner.js` loads `async`, so a slow or unreachable esm.sh cannot stall it — ordered module scripts otherwise execute strictly after every preceding module settles, including their top-level awaits.
 
@@ -33,6 +37,7 @@ The format is inspired by [Keep a Changelog](https://keepachangelog.com/en/1.1.0
 
 ### Changed
 
+- The `turnstile` wide-event field distinguishes an issued challenge (`challenged`) from a rejected token (`fail`), and failures carry a closed-vocabulary `reason` plus allowlisted Siteverify `error_codes`. `scripts/learner_report.py` breaks failures down by reason and code and raises a configuration alert for `invalid-input-secret`, `missing-input-secret`, or a missing site key. Siteverify reports those secret errors with HTTP 400, so error codes are now read from non-2xx bodies instead of being logged as an outage.
 - Dependency refresh: Pillow 12.3.0 (from 11.3.0; 17 advisories), Pywrangler 1.17.4 (from 1.9.3; requires uv 0.12.3+), Shiki 4.4.3 (from 1.29.2; byte-identical output for every example block in both themes), and CodeMirror state 6.7.6, view 6.43.13, and language 6.12.4.
 - CI actions moved to their Node 24 releases (checkout, setup-node, and setup-python v7; setup-uv v10.2.0 pinned by commit), clearing the Node.js 20 deprecation warning; npm caching stays off.
 - Dependabot proposes weekly npm, GitHub Actions, and development-tool (Hypothesis, Pillow, Pywrangler) updates. Runtime packages refresh through `make upgrade-runtime-deps`.

docs/learner-analytics.md

@@ -21,7 +21,14 @@ time window:
   human-validated paths may suggest content candidates, but typos, scanners,
   and bots are confounders.
 - **Turnstile outcomes** — challenge and rejection counts used to monitor
-  runner availability and abuse controls, not learner quality.
+  runner availability and abuse controls, not learner quality. `challenged`
+  (a challenge was issued) is counted apart from `fail` (a token was
+  rejected), and failures break down by reason and Siteverify error code.
+  A **configuration alert** (`invalid-input-secret`, `missing-input-secret`,
+  or `site_key_missing`) means every challenged run is failing because of
+  the deployment, not the visitor. Deployment smoke probes the secret at
+  deploy time; this report catches a secret that breaks afterwards, such as
+  a rotation in the dashboard.
 
 ## Getting events
 

docs/lessons-learned.md

@@ -131,6 +131,10 @@ git diff --check
 - **Deployment smoke belongs beside CI, and POST smoke must assert rendered output.** `scripts/smoke_deployment.py` checks rendered Worker pages, runtime-boundary pages, journey pages, prototype review pages, and representative Dynamic Worker POST runs for HTTP failures, exception markers, and stale edited-code output. Build success is not enough; the deployed Worker must render and execute edited examples. With Turnstile enabled, submitted code can appear in the editor textarea even when it did not run, so POST smoke must inspect the output panel rather than searching the whole HTML document.
 - **Observability smoke should assert the custom event, not the whole tail envelope.** Use unique `x-request-id` values, exercise cache miss/hit/bypass and client-error paths, and assert on the structured payload inside `logs[].message[]`. If Turnstile is enabled, Dynamic Worker error-path probes need the smoke bypass secret; otherwise they only verify the Turnstile-fail path.
 - **Turnstile should be secret-gated, session-scoped, and invisible until needed.** Protect edited-code POST runs only when `TURNSTILE_SECRET_KEY` and an explicit challenge mode are configured, lazy-load/render the Invisible-mode widget only after the server returns a challenge-required marker, and issue a signed clearance cookie so normal session runs skip Siteverify. The Cloudflare widget mode is `Invisible`; the client-side render option is `execution: "execute"`, not `size: "invisible"`. If production smoke must POST through a protected endpoint, use a separate `PBE_SMOKE_BYPASS_SECRET` header so smoke remains a deployment check rather than a CAPTCHA solver. See `docs/turnstile-runner-protection-spec.md` for the full runner-protection design.
+- **A retry that can re-enter itself needs a bound.** The runner re-challenged whenever a response carried the challenge marker, including right after it had sent a token. A server that kept rejecting tokens (wrong secret, hostname, or action) turned one Run into an unbounded loop of Turnstile solves, Worker POSTs, and Siteverify subrequests. Cloudflare's Turnstile Spin skill treats tokens as single-use with at most one reset per retry; the runner now earns one challenge per Run and shows the server's message when the retried token is rejected. `scripts/check_browser_layout.mjs` drives a stub server that always rejects and fails if one Run costs more than one solve.
+- **A bypassed check needs its own signal.** Deployment smoke uses `PBE_SMOKE_BYPASS_SECRET`, so it never proves the production `TURNSTILE_SECRET_KEY` works, and the wide event recorded both "challenge issued" and "Siteverify rejected" as `fail`. A wrong or rotated secret would have passed smoke while every new session's first run failed. Turnstile Spin's "Fix with Spin" banner is Cloudflare noticing widgets that serve traffic without working server-side validation; the self-hosted equivalent is `challenged` versus `fail` outcomes plus Siteverify `error_codes`, with `invalid-input-secret` reported by `scripts/learner_report.py` as a configuration alert rather than a bot failure. Verify the signal against the live service, not a stub: real Siteverify returns `invalid-input-secret` and `missing-input-secret` with HTTP 400, so the common `if (!response.ok) throw` pattern (Spin's reference handler included) discards the one code that names the misconfiguration. Unit stubs had assumed a 200. The deploy-time half of the fix is a smoke-gated probe (`POST /__smoke/turnstile`) that sends the production secret with Cloudflare's dummy token. Classify on more than the error code: Cloudflare's always-fail test secret answers the dummy token with the same `invalid-input-response` as a working production secret, and only `metadata.result_with_testing_key` tells them apart.
+- **Check hand-rolled security code against the vendor's reference contract.** Reading the Turnstile Spin skill's canonical Siteverify handler beside `_verify_turnstile` found three gaps no test had asked about: no timeout, no 2048-character token bound, and a `null` hostname raising `AttributeError` (a 500) instead of failing closed. Use the reference as a checklist, and keep deliberate divergences written down: Invisible mode, session clearance instead of per-request verification, and the serving hostname instead of a configured allowlist (safe only while `workers.dev` and preview URLs stay disabled).
+- **Credential-bearing commands run code from the dependency tree.** Turnstile Spin refuses to run `secret put` or other credential-bearing commands through `npx`, package scripts, or a project-local binary. `make deploy` runs the repository-pinned Wrangler with the account's credentials, and Dependabot now proposes npm updates weekly. Exact pins, the three-day cooldown, and `npm ci --ignore-scripts` limit the exposure; review Wrangler and Miniflare bumps as credential-bearing changes, not routine ones.
 
 ## Discoverability, theming, and learner analytics
 

docs/observability-spec.md

@@ -404,7 +404,9 @@ Add:
 - `example.code_hash` — sha256 hex of submitted UTF-8 bytes, first 12 chars
 - `example.code_bytes` — `len(submitted.encode("utf-8"))`
 - `example.code_edited` — `submitted != example["code"]`
-- `turnstile.outcome` — `pass` / `fail` / `bypass` / `disabled`
+- `turnstile.outcome` — `challenged` / `pass` / `fail` / `bypass` / `disabled`. `challenged` means the request was sent a challenge; it is the normal first run of a session, not a failure.
+- `turnstile.reason` — on `fail` only: `rejected` / `hostname_mismatch` / `action_mismatch` / `siteverify_unavailable` / `token_too_long` / `site_key_missing` / `missing_token` / `runtime_unavailable`
+- `turnstile.error_codes` — on `rejected` only: Siteverify's documented `error-codes`, sorted, with anything undocumented recorded as `other`
 - `execution_ms` — duration around `_run_example`, recorded in a `finally` if execution started
 
 Example:
@@ -419,7 +421,7 @@ request.state.wide_event["example"] = {
 }
 ```
 
-Change `_verify_turnstile(...)` to return `(ok, message, outcome)` so the caller can record the outcome without re-deriving it. Do not log the token or `CF-Connecting-IP`; the latter may still be sent to Turnstile verification but must not enter the event.
+`_verify_turnstile(...)` returns `(ok, message, turnstile_fields)` so the caller records the outcome, failure reason, and error codes without re-deriving them. An `invalid-input-secret` code means the deployment is misconfigured, not that a visitor failed. Do not log the token or `CF-Connecting-IP`; the latter may still be sent to Turnstile verification but must not enter the event.
 
 #### `_run_example`
 
@@ -567,7 +569,11 @@ Every event carries the context from `observability.py` plus a subset of the per
 | `example.code_hash` | string | handler | First 12 hex chars of sha256(submitted UTF-8 bytes). |
 | `example.code_bytes` | int | handler | Byte length of submitted UTF-8 code. |
 | `example.code_edited` | bool | handler | True when submitted differs from canonical. |
-| `turnstile.outcome` | string | handler | `pass` / `fail` / `bypass` / `disabled`. |
+| `turnstile.outcome` | string | handler | `challenged` / `pass` / `fail` / `bypass` / `disabled`. |
+| `turnstile.reason` | string | handler | Closed-vocabulary failure reason; present only when `outcome` is `fail`. |
+| `turnstile.error_codes` | array | handler | Allowlisted Siteverify `error-codes` (`other` for unknown); present only for `rejected`. |
+| `turnstile_probe.secret` | string | probe handler | `valid` / `invalid` / `testing_key` / `unverified` / `unexpected` / `absent`; only on authenticated `POST /__smoke/turnstile`. |
+| `turnstile_probe.ok` | bool | probe handler | Whether the deployed configuration can verify browsers. |
 | `execution_ms` | float | handler | Sandboxed run duration. |
 | `worker.outcome` | string | handler | Dynamic Worker outcome. |
 | `worker.status_code` | int | handler | Dynamic Worker HTTP status when fetch completes. |

docs/pr-evidence/README.md

@@ -1,5 +1,24 @@
 # PR visual evidence
 
+## Turnstile rejection loop (2026-09-26)
+
+These captures show `/examples/values` six seconds after one Run click, with Siteverify rejecting every token. Both Workers used Cloudflare's invisible always-pass test site key (`1x00000000000000000000BB`), so the real Turnstile widget loaded and solved in the browser, and the always-fail test secret (`2x0000000000000000000000000000000AA`), so Siteverify rejected every solved token.
+
+| Evidence | Review point | Commit | SHA-256 |
+| --- | --- | --- | --- |
+| [Before](turnstile-rejection-before-runner.png) | The runner re-challenges after each rejection and stays busy on "Verification required…". Over 20 seconds, one click made 12 POSTs and 11 real widget solves, growing steadily. | `3f300af` (`origin/main`) | `5af0c28f339e6bc4d394186530c385e663046a7edf98671965d0a61ed1a3cdf3` |
+| [After](turnstile-rejection-after-runner.png) | One solve, two POSTs, then the server's message and a free Run button. Still two POSTs after 20 seconds. | `eae0dc3` | `20ca27224fff2f0413004b8e2e2f35e7b1563b45f3941714a68f3e46e837f0af` |
+
+Reproduce each capture by serving the corresponding revision with the test keys:
+
+```bash
+uv run --group workers pywrangler dev --port <port> \
+  --var TURNSTILE_SECRET_KEY:2x0000000000000000000000000000000AA \
+  --var TURNSTILE_SITE_KEY:1x00000000000000000000BB
+```
+
+Open `http://127.0.0.1:<port>/examples/values` at a 1200×900 viewport, click Run once, wait six seconds, and capture `.runner-grid`. Count `POST /examples/values` requests in DevTools to see the loop. `make browser-layout-test` covers the same contract with a stub server that always rejects.
+
 ## Audit remediation (2026-07-10)
 
 These captures isolate the dark-mode Run-button contrast correction on the same `/examples/values` runner at a 1200×900 desktop viewport.

docs/turnstile-runner-protection-spec.md

@@ -110,6 +110,7 @@ If `TURNSTILE_CHALLENGE_MODE=session` and both site/secret keys are configured:
 7. The Worker validates the token through Siteverify.
 8. On success, the Worker sets a signed `pbe_turnstile_clearance` cookie.
 9. Later runs in that clearance window skip Turnstile and go straight to the Dynamic Worker.
+10. If Siteverify rejects the token, the client shows the server's failure message and stops. Each Run earns at most one challenge; pressing Run again earns a fresh one.
 
 Siteverify endpoint from Cloudflare docs:
 
@@ -160,6 +161,9 @@ x-pythonbyexample-smoke-secret: <secret>
 - The Turnstile widget should be configured in Cloudflare as **Invisible** mode. Client code uses explicit rendering with `execution: "execute"`; `size: "invisible"` is not a valid current Turnstile size option.
 - The widget is removed after callback success or failure.
 - Siteverify is called only when a challenge-required request retries with a token.
+- A Run that sends a token and is challenged again does not solve again. Before this rule, a persistently failing Siteverify (wrong secret, hostname, or action) turned one click into an unbounded loop of solves, Worker POSTs, and Siteverify subrequests.
+- Siteverify matches Cloudflare's reference contract (and the Turnstile Spin skill's canonical handler): `success is True`, the `run-example` action, the serving hostname, `remoteip` from `CF-Connecting-IP`, a 10-second `AbortSignal.timeout`, tokens over 2048 characters rejected before any subrequest, and every transport, status, payload, or type surprise failing closed.
+- Each result lands in the wide event as `turnstile.outcome` (`challenged`, `pass`, `fail`, `bypass`, `disabled`) with a failure `reason` and allowlisted Siteverify `error_codes`, so `scripts/learner_report.py` can separate issued challenges from rejections and flag `invalid-input-secret` as a configuration alert. Siteverify reports a bad or missing secret with HTTP 400 and a JSON body, so the Worker reads error codes from non-2xx bodies too; a non-2xx response still never passes.
 - A valid challenge creates a signed, HttpOnly, Secure, SameSite=Lax clearance cookie scoped to `/examples`.
 - Failed or missing verification does not create a Dynamic Worker.
 - POST responses are still never cached.
@@ -187,6 +191,26 @@ x-pythonbyexample-smoke-secret: <secret>
 
 This is only an app-level bypass. If Cloudflare Rate Limiting challenges smoke traffic before it reaches the Worker, either keep smoke below the threshold or add a Cloudflare-side skip/exception.
 
+Because bypassed runs never touch Siteverify, the same header also unlocks a secret probe:
+
+```text
+POST /__smoke/turnstile
+x-pythonbyexample-smoke-secret: <secret>
+```
+
+The Worker sends its `TURNSTILE_SECRET_KEY` to Siteverify with Cloudflare's documented dummy token `XXXX.DUMMY.TOKEN.XXXX` and returns JSON (`Cache-Control: no-store`). It never includes the secret itself. Cloudflare documents that production secrets reject the dummy token; how they reject it tells the configurations apart:
+
+| Siteverify reply to the dummy token | `secret` | Smoke |
+| --- | --- | --- |
+| `invalid-input-response`, HTTP 200, no testing metadata | `valid` | passes when a site key is configured |
+| `invalid-input-secret` or `missing-input-secret` (HTTP 400) | `invalid` | fails |
+| `metadata.result_with_testing_key: true`, or `success: true` | `testing_key` | fails |
+| unreachable, timeout, or unparseable | `unverified` | fails; re-run |
+| anything else | `unexpected` | fails |
+| no `TURNSTILE_SECRET_KEY` configured | `absent` | passes; Turnstile is off |
+
+The testing-metadata check matters: Cloudflare's always-fail test secret answers the dummy token with the same `invalid-input-response` as a working production secret, so without it a deployment where every learner fails would pass. Without the header, the route answers 404 and never calls Siteverify. The probe proves the secret is a working production secret. It cannot prove that the secret and site key belong to the same widget, or that the widget allows the production hostname; a browser run still covers those.
+
 ## Recommended Cloudflare layer: Rate Limiting Rules
 
 The best first production protection for Dynamic Worker cost is Cloudflare Rate Limiting, because it runs before the Worker and limits repeated runner traffic at the edge.
@@ -451,6 +475,8 @@ Deployment smoke OK
 
 The smoke script checks the rendered output panel for POST runs. This is important because a Turnstile-required response can echo submitted code in the editor textarea without executing it.
 
+With `PBE_SMOKE_BYPASS_SECRET`, smoke also calls `POST /__smoke/turnstile` and fails unless the Worker's secret is a working production secret with a configured site key (see "Smoke-test behavior"). Without the secret, it prints `SKIP` for the probe.
+
 ## Verification checklist
 
 ### Verify no visible widget on page load

public/runner.96b105eef218.js → public/runner.05ddb60aa9e7.js

@@ -210,6 +210,10 @@ function initializeRunner() {
     const document = new DOMParser().parseFromString(html, 'text/html');
     const challengeRequired = document.querySelector('[data-turnstile-required]');
     if (challengeRequired) {
+      // Each Run earns one challenge. If the server rejected the token just
+      // sent, report why instead of solving again: a wrong secret or hostname
+      // would otherwise loop through solves and Siteverify calls unbounded.
+      if (turnstileToken) throw new Error(responseErrorMessage(response, html));
       outputPanel.querySelector('code').textContent = challengeRequired.textContent
         || 'Verification required before running edited code…';
       const token = await requestTurnstileToken(signal);