Crawl Walrus Sites via on-chain resource enumeration (Site object + blobIds)

[harrymove-ctrl]

Context

The production Worker treats *.wal.app targets as plain HTTP through the portal — it sets mode: 'walrus' and scrapes x-walrus-* response headers (apps/api/src/worker.ts ~2462/2554) but never reads the on-chain Site object. The real, verifiable enumeration already exists in packages/walrus/src/resources.ts listWalrusResources(site), which reads each resource's blob_id from the Sui Site object (~line 103) using a WalrusSiteContext from resolve.ts — but it is Node-only (CLI materializeWalrusSite) and not used in the hosted crawl. So Walrus-native sites are crawled by HTML link-guessing instead of their authoritative on-chain manifest.

Goal / user story

As a user pointing ContextMEM at a Walrus Site (a .wal.app URL, SuiNS name, or 0x site object id), I want it to enumerate the site's resources and blobIds directly from the on-chain Site object, so extraction is complete and provenance is verifiable rather than guessed from anchors.

Acceptance criteria

• For a Walrus-Site target, the run resolves a WalrusSiteContext and enumerates resource paths + blobIds from the on-chain Site object (not HTML link discovery).
• Each resource is fetched via the Walrus aggregator (quilt.ts blobAggregatorEndpoint) and mapped into manifest pages/resources carrying source.blobId / resourcePath.
• buildSiteStructure (packages/core/src/site-structure.ts) populates its existing Walrus Provenance group from input.walrus.resources for these runs.
• Extracted chunks/facts carry blobId-backed FactSourceRef provenance so the "why" links resolve to on-chain blobs.
• If the on-chain read fails (RPC/grpc error), the crawl falls back to today's HTTP portal crawl and the run still completes.

Implementation notes

• Reuse packages/walrus/src/resolve.ts (resolveWalrusTarget) + resources.ts (listWalrusResources) from the Worker walrus branch, or port a Worker-safe variant.
• SPIKE first: confirm @mysten/sui/grpc SuiGrpcClient.getObject runs in the Worker under nodejs_compat (the roadmap notes proof.ts/history.ts/resolve.ts already use it for reads, but they run in Node today). resources.ts uses node:buffer — verify under nodejs_compat.
• Touch apps/api/src/worker.ts (walrus branch dispatch), wire WalrusResourceRecord[] into the existing SiteStructureInput.walrus.resources seam, and reuse aggregatorUrl from the resolved context.
• Gotchas: large sites need a resource cap/concurrency limit; aggregator fetch sizes must respect the Worker's existing 1.5MB guard; non-HTML resources (assets) should be enumerated but not chunked.

Sui Overflow angle

This is the uniquely Sui-native crawl: reading content straight from on-chain Walrus Site objects and Walrus blobs gives verifiable, tamper-evident provenance no traditional crawler can match. Paired with the chunkGraphDigest-keyed extract receipt, it tells a complete on-chain provenance story — the most differentiated thing to demo at a Sui hackathon.

Dependencies

Spike: @mysten/sui/grpc (SuiGrpcClient) in the Cloudflare Worker (shared with the on-chain receipt work). Coordinates with, but is not blocked by, the on-chain attribution-receipt issue.

Part of the ContextMEM roadmap (#4) • Sui Overflow build.

[harrymove-ctrl]

Implementation Plan

Approach
Wire the already-real on-chain enumeration (packages/walrus resolveWalrusTarget + listWalrusResources) into the production Worker's per-source extractor (apps/api/src/worker.ts extractTargetContext, ~L2452). For .wal.app / SuiNS / 0x targets we resolve a WalrusSiteContext, enumerate resource paths + blobIds from the Sui Site object, fetch HTML/markdown bodies through the Walrus aggregator (verifying blobHash), and emit manifest pages whose source carries blobId/resourcePath. The key design decision is to avoid the @contextmem/walrus barrel (it re-exports storage.ts's child_process/tar and resources.ts's node:http.createServer preview server, which break the Worker bundle) by adding a Worker-safe subpath @contextmem/walrus/enumerate, and to thread Walrus provenance all the way into chunks/facts so FactSourceRef.blobId resolves. On any RPC/grpc failure the existing HTTP-portal crawl runs unchanged.

Prerequisites & dependencies

• GATING SPIKE (do first): confirm @mysten/sui/grpc SuiGrpcClient (getObject, listDynamicFields, nameService.lookupName) runs in the Worker under nodejs_compat — bundle size, CPU, and whether the Connect/gRPC-over-fetch transport works on Workers. This is the same SuiGrpcClient-in-Worker spike the roadmap calls a shared dependency with the on-chain receipt work (roadmap §"Dependencies"). If it busts limits, the enumeration still runs inside processExtractionJob, which already executes in the queue consumer (worker.ts:553) with a larger CPU budget than a request handler — note this as the fallback host, not the Node CLI.
• No new creds/bindings. Enumeration is read-only against public mainnet Sui RPC (https://fullnode.mainnet.sui.io) + the public aggregator (https://aggregator.walrus-mainnet.walrus.space). No HARBOR_*, no Vectorize, no wallet/service key. (This zero-secret, verifiable read is the Sui-native demo angle.)
• @mysten/sui ^2.9.1 is already a dep of apps/api (apps/api/package.json:20) and @contextmem/walrus is already a workspace dep (line 19). p-limit is a pure-JS walrus dep and bundles fine.
• Network is fixed to mainnet for this issue: resolveWalrusTarget forces network:"mainnet" for .wal.app (resolve.ts:82-90) and defaults mainnet for 0x ids (resolve.ts:16). No mainnet/testnet split to reconcile here.

Step-by-step

1. SPIKE: SuiGrpcClient in the Worker. Add a temporary route or a tiny test Worker that calls resolveWalrusTarget("https://seal-docs.wal.app/") (the configured CONTEXTMEM_DEMO_SAMPLE_TARGET, wrangler.jsonc:8) then listWalrusResources(site) and logs resource count + first blobId. Verify it returns non-empty, the bundle builds with wrangler deploy --dry-run, and CPU stays in budget. Decide Worker vs queue-consumer host. Delete the spike route after.

2. Create the Worker-safe enumeration module. New packages/walrus/src/enumerate.ts exporting resolveWalrusTarget (re-export from ./resolve.js) plus Worker-safe listWalrusResources and fetchWalrusResource extracted from resources.ts — same Sui/BCS logic (SuiGrpcClient.listDynamicFields + getObject + DynamicFieldStruct(ResourcePathStruct, ResourceStruct), bcs.ts), blobAggregatorEndpoint/quiltAggregatorEndpoint (quilt.ts), sha256Base64 verify — but with no node:fs cache, no node:http, no materializeWalrusSite/startWalrusPreview. Refactor resources.ts to import the shared core from enumerate.ts (single source of truth; CLI keeps working via the barrel). Add to packages/walrus/package.json exports: "./enumerate": "./src/enumerate.ts". (Note: resolve.ts's top-level import fs from "node:fs/promises" is only used in the .json branch, never hit for .wal.app/0x, and resolves harmlessly under nodejs_compat.)

3. Add the Walrus enumeration branch in extractTargetContext (worker.ts:2452-2607). Detection already exists via target.hostname.endsWith(".wal.app") (used at 2462/2475/2554/2568). For Walrus targets, before the current header-scrape path:

   • import { resolveWalrusTarget, listWalrusResources, fetchWalrusResource } from "@contextmem/walrus/enumerate";
   • const site = await resolveWalrusTarget(target.toString());
   • const records = await listWalrusResources(site, { concurrency: 5 });
   • Cap to ~60-80 records; select HTML (/html/i.test(contentType) or .html?$) + markdown (.mdx?$) resources to fetch as pages; enumerate (but do not fetch bodies for) non-HTML assets — they still get blobId provenance in the structure.
   • For each page resource, await fetchWalrusResource(site, record) (omit cacheDir), respecting a per-resource byte cap aligned with the existing 1.5MB guard (worker.ts:2813/2816): skip/flag any resource > ~1.5MB instead of loading it. Convert HTML→markdown with the Worker's existing htmlToText/markdown path; verify sha256Base64(body) === record.blobHash (on mismatch, skip the page, don't fail the run).
   • Wrap the whole block in try/catch (Step 7).

4. Build manifest pages + walrus.resources with real provenance. Replace the fake walrusResources (worker.ts:2479-2485, blobId "") with the real WalrusResourceRecord[]. Each manifest page built from a Walrus resource must set source: { kind: "walrus", resourcePath: record.path, blobId: record.blobId, blobHash: record.blobHash, quiltPatchId: record.quiltPatchId } (matches PageArtifact.source, types.ts:335). Set manifest.walrus = { site: { network: site.network, siteObjectId: site.siteObjectId, aggregatorUrl: site.aggregatorUrl }, resources: records } (replacing the headers-derived block at 2568-2576). Keep mode: "walrus" (2554).

5. Populate the Walrus Provenance group via core buildSiteStructure. For Walrus runs, call buildSiteStructure({ target, pages, walrus: { resources: records } }) (site-structure.ts:16; already exported from core, index.ts:10) instead of the hand-rolled siteStructure object (worker.ts:2581-2607) — its walrus-provenance group (site-structure.ts:148) is built directly from input.walrus.resources. Because the final namespace manifest is assembled by buildCombinedNamespaceBundle (worker.ts:2125), also: (a) make sourceFromSingleExtraction (worker.ts:2069) carry the real WalrusResourceRecord[] for walrus sources so they flow through ExtractedNamespaceSource.resources into buildCombinedWalrusSummary (worker.ts:2244); and (b) in buildCombinedNamespaceBundle, when any source is walrus, set manifest.siteStructure = buildSiteStructure({ target: job.target, pages, walrus: { resources: combinedWalrusRecords } }) (replacing buildCombinedSiteStructure at 2166 for walrus runs; keep it for pure-web).

6. Thread blobId provenance into chunks → facts (AC4).

   • Extend ContextChunk (types.ts:726) with optional blobId?, resourcePath?, quiltPatchId?.
   • In buildChunks (chunks.ts:13-43) copy page.source?.blobId/resourcePath/quiltPatchId onto each emitted chunk. chunkGraphDigest (chunks.ts:59) only hashes chunkId:contentHash, so adding these fields does NOT change the digest — planMemoryWrite diffing stays stable (call this out in the PR).
   • In sourceFromChunk (facts.ts) add blobId: chunk.blobId, resourcePath: chunk.resourcePath to the returned FactSourceRef (the fields already exist on FactSourceRef, facts.ts:25-32 — just unpopulated today).
   • In the Worker, ensure factsPages (worker.ts:2660-2671) preserves each page's source (currently dropped) so buildSiteFacts/buildChunks emit blobId-backed sources. Verify source survives sourceFromSingleExtraction (...page spread, worker.ts:2080) and the combined buildChunks(pages…) (worker.ts:2186) so /context/chunks.ndjson and /context/facts.json both carry blobIds.

7. Fallback (AC5). Wrap Steps 3-4 in try/catch. On any error (SuiNS lookup failure, RPC/grpc error, empty enumeration), console.warn and fall through to the existing HTTP-portal crawl (the current extractWalrusHeaders + link-discovery path, worker.ts:2475-2606) so the run still completes with mode:"walrus".

New / changed files

• packages/walrus/src/enumerate.ts — NEW Worker-safe resolveWalrusTarget re-export + listWalrusResources/fetchWalrusResource (no fs/http/preview/cache).
• packages/walrus/package.json — add "./enumerate" to exports.
• packages/walrus/src/resources.ts — import shared enumeration core from enumerate.ts (de-dupe; keep Node caching/materialize wrappers here).
• packages/core/src/types.ts — ContextChunk += blobId?/resourcePath?/quiltPatchId? (L726).
• packages/core/src/chunks.ts — buildChunks copies page.source provenance (L13-43).
• packages/core/src/facts.ts — sourceFromChunk emits blobId/resourcePath.
• apps/api/src/worker.ts — Walrus enumeration branch + provenance in extractTargetContext (~L2452-2607); real walrus records through sourceFromSingleExtraction (L2069) + buildCombinedWalrusSummary (L2233); buildSiteStructure for walrus runs (replace L2166/L2581-2607); preserve source in factsPages (L2660).

Test plan

• Unit (bun test in packages/core): buildChunks on a PageArtifact with source.blobId emits chunks carrying that blobId; sourceFromChunk round-trips it into FactSourceRef.blobId; assert chunkGraphDigest is byte-identical before/after the field addition (proves diff stability).
• Integration (packages/walrus, network): against seal-docs.wal.app, resolveWalrusTarget returns a real siteObjectId; listWalrusResources returns >0 records each with a non-empty blobId; fetchWalrusResource body passes sha256Base64 === blobHash (the round-trip provenance assert). This is the verifiable, tamper-evident proof a normal crawler can't produce.
• Manual end-to-end: deploy to staging, enqueue a build for https://seal-docs.wal.app/, then inspect the run's /context/manifest.json (walrus.resources[].blobId populated, not ""), /context/site-structure.json (non-empty Walrus Provenance group), /context/facts.json + /context/chunks.ndjson (source.blobId/chunk blobId present). Record the resolved siteObjectId and one blobId in the issue as evidence.
• Fallback: point at a .wal.app host while forcing an RPC error (bad rpcUrl in the spike, or block the fullnode) and confirm the run still completes via the HTTP portal path.

Risks & gotchas

• Worker bundle/CPU from @mysten/sui/grpc — the gating spike. If over budget, keep enumeration in the queue consumer (already the host for processExtractionJob); do not regress to Node-only.
• Never import the @contextmem/walrus barrel in the Worker — it pulls storage.ts (node:child_process/spawn, node:os) and resources.ts (node:http.createServer), which fail under nodejs_compat. The ./enumerate subpath is mandatory.
• Aggregator latency / large sites: cap resource count, concurrency (~5), and per-resource size (1.5MB guard) so a big site can't blow the Worker time/memory limits; enumerate assets without fetching their bodies.
• Mainnet RPC rate limits / SuiNS lookup miss (.wal.app → nameService.lookupName, resolve.ts:109) — both are covered by the Step 7 try/catch fallback.
• blobHash mismatch: skip the offending page rather than fail the whole run.
• Combined-manifest plumbing: the final manifest is rebuilt by buildCombinedNamespaceBundle; if source/walrus records aren't threaded through sourceFromSingleExtraction, AC3/AC4 will pass per-source but vanish from the run output. Test the combined output, not just per-source.

Acceptance-criteria mapping

• Resolve WalrusSiteContext + enumerate paths/blobIds from the on-chain Site object (not HTML discovery) → Steps 2, 3.
• Each resource fetched via the aggregator (quilt.ts blobAggregatorEndpoint) and mapped into manifest pages/resources carrying source.blobId/resourcePath → Steps 3, 4.
• buildSiteStructure populates its existing Walrus Provenance group from input.walrus.resources → Step 5.
• Chunks/facts carry blobId-backed FactSourceRef provenance → Step 6.
• On-chain read failure falls back to the HTTP portal crawl and the run still completes → Step 7.

Effort
L — ~4-6 person-days. ~1 day gating spike; ~2-3 days Worker wiring + walrus subpath + combined-manifest plumbing; ~1-1.5 days core chunks/facts provenance + tests + staging verification.

---

Implementation plan generated for execution (planning phase). Part of the roadmap (#4).