Cloudflare Browser Rendering fallback for JS/SPA pages

[harrymove-ctrl]

Context

The production extractor fetchPageContent() (apps/api/src/worker.ts ~2884) uses Firecrawl only when FIRECRAWL_API_KEY is set, otherwise falls back to raw fetchText + regex htmlToText() (line 5217). The Node core path (packages/core/src/web.ts fetchHtml) is also raw-fetch with no JS execution. As a result, SPA shells (Next.js/React <div id="root">) extract to near-empty content whenever Firecrawl is absent — a real risk during a live demo on a modern site. There is no Browser Rendering binding in apps/api/cloudflare/wrangler.jsonc (bindings today: D1, R2, AI, Queues).

Goal / user story

As a user extracting a JavaScript-rendered site, I want ContextMEM to render the page headlessly when a raw fetch returns an empty shell, so I get real content without Firecrawl being mandatory.

Acceptance criteria

• A browser binding is added to wrangler.jsonc and WorkerEnv, and @cloudflare/puppeteer is wired in the queue consumer.
• fetchPageContent order becomes firecrawl (if key) → Browser Rendering (if empty-shell detected) → raw fetch + htmlToText, with the engine recorded on FetchedText.engine.
• An empty-shell heuristic triggers rendering: extracted main-content text below a threshold while the HTML shows SPA markers (id="root", __next, app-root, or large inline script payload with little prose).
• Rendering honors a wait (CrawlOptions.waitForMs / networkidle) and a hard timeout; failures degrade gracefully to raw fetch rather than failing the run.
• Rendered HTML flows through the existing markdown/headings/brand path so downstream artifacts (chunks, facts, design-system) are unchanged in shape.

Implementation notes

• Touch apps/api/cloudflare/wrangler.jsonc (add "browser": { "binding": "BROWSER" }), apps/api/src/worker.ts (WorkerEnv, fetchPageContent, empty-shell detector). Run rendering only in the queue consumer, never the request path, due to CPU/time limits.
• Use puppeteer.launch(env.BROWSER) → page.goto(url, { waitUntil: 'networkidle0' }) → page.content() → feed into htmlToText/markdown. Reuse fetchText's 1.5MB size guard semantics.
• Gotchas: Browser Rendering has session/concurrency caps and per-invocation cost — gate it behind the empty-shell heuristic, cap concurrent sessions, and reuse a single browser per batch. Keep local Playwright (packages/core/src/screenshots.ts) for the CLI; this is the hosted parity path.
• Open question to resolve in PR: is Firecrawl permanent or does Browser Rendering become the default renderer (cost tradeoff)?

Sui Overflow angle

Many Sui/Walrus ecosystem sites and dapp docs are SPA-rendered; guaranteeing non-empty extraction without a paid Firecrawl key makes the demo robust on any URL a judge throws at it, including *.wal.app portals.

Dependencies

None (independent of chunking); complements the Walrus Sites on-chain enumeration issue for .wal.app targets.

Part of the ContextMEM roadmap (#4) • Sui Overflow build.

[harrymove-ctrl]

Implementation Plan

Approach
Add a Cloudflare Browser Rendering binding (BROWSER) and @cloudflare/puppeteer to the Worker, then insert a headless-render fallback into the production extractor fetchPageContent() (apps/api/src/worker.ts:2884). The render path is gated by an empty-shell heuristic: we still raw-fetch first (cheap, gives us the HTML to inspect + Walrus headers), and only spin up the browser when the fetched HTML looks like an un-hydrated SPA. Because fetchPageContent is reached only via extractTargetContext (:2452) → processExtractionJob (:1951) → async queue consumer (:545), the render path is already confined to the queue consumer and never touches the request path — no extra guard needed. Key design decision: render lazily and reuse one browser per extraction with a small concurrency cap, and degrade to the existing raw-fetch + htmlToText result on any render failure so a run never fails because of rendering.

Prerequisites & dependencies

• No blocking issues. Issue states "Dependencies: None"; independent of the chunking/Vectorize/Harbor tracks.
• Cloudflare account binding/quota: Browser Rendering must be enabled on the account for both contextmem-backend and contextmem-backend-staging. Browser Rendering has a free tier (limited minutes/day) but real usage typically needs a Workers Paid plan; confirm before relying on it in the demo. There is no --remote-free local browser — wrangler dev must use --remote to exercise it.
• New dep: @cloudflare/puppeteer in apps/api/package.json (Worker-native; do not confuse with the playwright stub already aliased in wrangler.jsonc).
• Roadmap open question to answer in the PR (cite issue): "is Firecrawl permanent or does Browser Rendering become the default renderer?" — this plan keeps Firecrawl first and Browser Rendering as the empty-shell fallback (recommended for cost); note the decision in the PR.

Step-by-step

1. Add the binding to apps/api/cloudflare/wrangler.jsonc. Add a top-level "browser": { "binding": "BROWSER" } and also add the same block under env.staging — bindings are NOT inherited in this config (see the comment block in the file; ai/queues/d1/r2 are redefined per-env).
2. Add the dependency: cd apps/api && bun add @cloudflare/puppeteer (adds to apps/api/package.json deps alongside @mysten/sui).
3. Extend WorkerEnv (apps/api/src/worker.ts:32): add BROWSER?: BrowserWorkerLike;. Define a minimal hand-rolled type next to the other *Like types (R2BucketLike:82, QueueLike:99) — the repo uses hand-rolled binding types, not @cloudflare/workers-types: type BrowserWorkerLike = { fetch: typeof fetch }; (puppeteer's launch() accepts a Fetcher-shaped binding; cast at the call site).
4. Widen the engine unions to add "browser": FetchedText at :298 (engine?: "fetch" | "firecrawl" | "browser") and the source-row type at :253 (engine?: "firecrawl" | "fetch" | "browser").
5. Add an empty-shell heuristic (new helper near crawlContentHash, :2904): function isEmptySpaShell(html: string, text: string): boolean. Return true when the stripped main-content length (reuse htmlToText(html).trim().length, :5217) is below a threshold (~300 chars) AND the HTML shows SPA markers: /id=["']root["']/, __next / id=["']__next["'], id=["']app(-root)?["'], ng-version, data-reactroot, OR a high script-to-prose ratio (sum of <script> payload length ≫ stripped-text length). Keep it conservative to avoid rendering normal pages.
6. Add the render helper (near firecrawlScrape, :2830): async function renderWithBrowser(url, env, opts): Promise<FetchedText>:
   • import puppeteer from "@cloudflare/puppeteer"; (top of file).
   • const browser = await puppeteer.launch(env.BROWSER as any); then const page = await browser.newPage();
   • await page.goto(url, { waitUntil: "networkidle0", timeout: opts.timeoutMs ?? 20000 });
   • Honor a wait: if opts.waitForMs (map from CrawlOptions.waitForMs, packages/core/src/types.ts:66) await new Promise(r => setTimeout(r, waitForMs)).
   • const html = await page.content(); — enforce the same 1.5 MB guard as fetchText (:2813/2816): if html.length > 1_500_000 truncate/throw.
   • Return { text: html, contentType: "text/html; charset=utf-8", headers: {}, engine: "browser", markdown: htmlToText(html) }.
   • Wrap in try/finally so the page/browser always close; never let a render error bubble — caller falls back.
7. Rewire fetchPageContent order (:2884) to firecrawl (if key) → Browser Rendering (if empty-shell) → raw fetch + htmlToText:
   • If env.FIRECRAWL_API_KEY: try firecrawlScrape (unchanged) — Firecrawl already executes JS, so on success return it.
   • const raw = await fetchText(url); (we need the HTML to run the heuristic and to keep Walrus headers).
   • if (env.BROWSER && isEmptySpaShell(raw.text, htmlToText(raw.text))) { try { return await renderWithBrowser(url, env, {...}); } catch (e) { console.warn("[browser-render] fallback to raw fetch", ...); } }
   • raw.markdown = htmlToText(raw.text); return raw; (unchanged final fallback).
8. Record the engine on the artifact (real fix, not the brittle string match). Today sourceFromSingleExtraction derives engine from sourceText.includes("firecrawl") (:2111) and manifest.source is "firecrawl-v2" | "cloudflare-fetch-extractor" (:2556). Add engine: home.engine to the manifest object in extractTargetContext (:2549), and change :2111 to engine: (manifest.engine as ... ) ?? (sourceText.includes("firecrawl") ? "firecrawl" : "fetch"). This makes FetchedText.engine (incl. "browser") flow into sources.json per AC.
9. Browser reuse + concurrency cap (cost control). The crawl loop fans out with Promise.allSettled over fetchPageContent (:2524) with no throttle. Add a tiny semaphore (cap ~2 concurrent renders) around the render branch, and lazily launch one browser per extractTargetContext invocation reused across that run's empty-shell pages, closed in a finally at the end of extractTargetContext (:2452). Simplest threading: a small RenderContext object (holds the lazily-launched browser + semaphore) created in extractTargetContext and passed into fetchPageContent/renderWithBrowser as an extra optional arg (default undefined → launch-per-call). Optionally use puppeteer.sessions(env.BROWSER) + puppeteer.connect() to reuse a warm session across batches (nice-to-have; document the cap).
10. Confirm downstream shape is unchanged (AC5). renderWithBrowser returns the same FetchedText shape with .text = rendered HTML, so the existing markdown/headings/brand path (home.markdown ?? htmlToText(home.text) at :2561/2781, brand/design enrichment that reads home.text) is untouched. No changes to chunk/fact/design-system code.

New / changed files

• apps/api/cloudflare/wrangler.jsonc — add browser binding (top-level + env.staging).
• apps/api/package.json — add @cloudflare/puppeteer.
• apps/api/src/worker.ts — import puppeteer; BrowserWorkerLike type; WorkerEnv.BROWSER (:32); widen engine unions (:253, :298); isEmptySpaShell() + renderWithBrowser() helpers; rewire fetchPageContent() (:2884); thread engine into manifest (:2549) + read it in sourceFromSingleExtraction (:2111); render context/semaphore in extractTargetContext (:2452).
• apps/api/src/worker.test.ts — new tests (below).

Test plan

• Unit (worker.test.ts): Mirror the existing test "uses Firecrawl when configured and falls back to raw fetch when it fails" (:454) and the fetch-stub helper at :1094. Add:
  • vi.mock("@cloudflare/puppeteer", () => ({ default: { launch: async () => fakeBrowser } })) where fakeBrowser.newPage() returns a page whose content() resolves to fully-rendered HTML.
  • Stub fetch so the source URL returns an SPA shell (<div id="root"></div> + a big inline <script>, near-empty prose). Provide an env with BROWSER: {} and no FIRECRAWL_API_KEY, run an extraction job, then assert sources.json contains "engine": "browser" and the rendered text landed in /site/index.md (read via CloudflareNamespaceStore.readArtifact, same as :478/500).
  • Negative test: a content-rich page (no SPA markers) → isEmptySpaShell false → engine: "fetch", browser never launched.
  • Graceful-degrade test: launch()/goto throws → result still engine: "fetch" with raw htmlToText content, run does not fail.
• Integration / manual: bunx wrangler dev --remote --config apps/api/cloudflare/wrangler.jsonc, POST an extraction for a known JS-rendered SPA (e.g. a *.wal.app portal or a Next.js docs site) with no Firecrawl key set; verify the run completes, sources.json shows "engine": "browser", and /site/index.md has real prose (not an empty shell). Capture the before/after content length as proof.
• CI: bun run check (typecheck + vitest + build) — confirm the worker still bundles under the Worker size limit with @cloudflare/puppeteer added (see Risks).

Risks & gotchas

• Worker bundle size / build: @cloudflare/puppeteer is sizeable; verify the deployed bundle stays under the gzip limit in bun-check.yml's build step. If it busts, lazy-import() puppeteer inside renderWithBrowser so it only loads when the binding exists.
• No local browser: Browser Rendering only works with wrangler dev --remote / deployed — unit tests must mock puppeteer; don't expect it under plain vitest.
• Cost / concurrency caps: Browser Rendering bills per session and has concurrent-session limits. The empty-shell gate + one-browser-per-run + semaphore (step 9) are the mitigations; do not render every page unconditionally.
• Hangs: networkidle0 can stall on pages with long-polling/analytics — the hard timeout in goto + try/finally close + graceful fallback (AC4) are mandatory.
• Heuristic tuning: too-aggressive thresholds will render normal pages (cost) or miss shells (empty extraction). Keep conservative defaults and log when the browser path triggers so it can be tuned from real runs.
• Network split: unrelated to mainnet/testnet storage — this is pure HTTP fetch; no Sui/Walrus tx involved.
• Engine-recording bug: the current sources.json engine field is derived from a string match on manifest.source (:2111), not FetchedText.engine — step 8 fixes this so "browser" is actually recorded; without it AC2 silently fails.

Acceptance-criteria mapping

• browser binding in wrangler.jsonc + WorkerEnv, @cloudflare/puppeteer wired in the queue consumer → Steps 1, 2, 3, 6 (render runs only via extractTargetContext→queue, already confined).
• Order firecrawl → Browser Rendering (if empty-shell) → raw fetch, engine recorded on FetchedText.engine → Steps 4, 7, 8.
• Empty-shell heuristic (root/__next/app-root/large script payload + low prose) → Step 5.
• Honors wait (waitForMs/networkidle) + hard timeout; failures degrade to raw fetch → Steps 6, 7 (try/catch fallback), 9.
• Rendered HTML flows through existing markdown/headings/brand path, downstream artifacts unchanged in shape → Steps 6 (returns standard FetchedText), 10; verified by the unit test reading /site/index.md + chunks.

Effort: M, ~2-3 person-days (1 day binding+wiring+heuristic, 0.5 day browser reuse/semaphore, 1 day tests + remote verification + threshold tuning).

---

Implementation plan generated for execution (planning phase). Part of the roadmap (#4).