StorageProvider seam + private SEAL-encrypted remember/recall (public→Tatum, private→Harbor)

[harrymove-ctrl]

Context

Today "private" is only a D1 read-token check (CloudflareNamespaceStore.authorizeNamespace, apps/api/src/worker.ts:502) — bytes are uploaded plaintext and readable by anyone with the blobId/aggregator URL — and WalrusStorageReceipt.provider is hard-wired to the literal 'tatum' (packages/core/src/types.ts:655) with packages/walrus/src/storage.ts as the only provider. The roadmap mandates coexistence + real encryption: a StorageProvider seam routes public→Tatum (mainnet, plaintext) and private→Harbor+SEAL (testnet, encrypted), where a private namespace gets an owned Seal policy and bytes are SEAL-encrypted before upload and only recallable by decrypt via SessionKey + seal_approve.

Goal / user story

As a user with a private namespace, when I remember an artifact it goes through a StorageProvider that SEAL-encrypts it and stores ciphertext on Harbor, and recall fetches + decrypts — while public namespaces keep going to Tatum plaintext, with the chosen provider recorded honestly in the receipt.

Acceptance criteria

• Define a StorageProvider interface { upload, status, fetch } in packages/walrus; refactor the existing Tatum functions behind a TatumStorageProvider implementing it, with no behavior change for current CLI/MCP callers.
• Widen WalrusStorageReceipt.provider (core/types.ts:655) from 'tatum' to 'tatum' | 'harbor' and add optional Harbor fields (spaceId, bucketId, fileId, sealPolicyId, sealIdentity).
• New packages/walrus/src/seal.ts wraps @mysten/seal: encrypt(bytes, { sealPolicyId, identity }) → ciphertext; decrypt(ciphertext, { sessionKey, sealPolicyId }) builds the seal_approve PTB.
• A selectStorageProvider({ visibility, network }) picks Harbor+SEAL for private namespaces and Tatum for public; private writes encrypt the proof bundle with a per-file identity against the namespace's sealPolicyId, then upload ciphertext via HarborClient; the chosen provider is written into the receipt, and rememberStorageIndex/renderStorageIndex (packages/memwal/src/index.ts:164/281) carry bucketId/fileId/sealPolicyId/sealIdentity (keep back-compat parsing of the [ctxm-storage] format).
• New GET /api/memwal/artifact (worker.ts) resolves provider + refs from D1, fetches ciphertext, decrypts via SessionKey + seal_approve, and streams plaintext — token-gated through authorizeNamespace (worker.ts:502); public namespaces fall back to the plaintext aggregator path.
• Fail-closed: non-private writes fall back to Tatum on Harbor failure, but private writes NEVER silently fall back to plaintext; creating/revoking a namespace read token maps to granting/revoking SEAL decrypt for that policy (wired near authorizeNamespace and token create/revoke worker.ts:992/:1013).
• Tests cover provider selection by visibility, public-only Tatum fallback, and a private round-trip (remember → recall artifact → bytes match; a request without a valid token cannot decrypt).

Implementation notes

• Land the interface + receipt widening early (no deps); uploadProofBundle (storage.ts:183) currently always returns provider: 'tatum' — route it through selectStorageProvider.
• SessionKey needs an Ed25519 signer (service key, or per-owner per the mapping issue's decision); execute the seal_approve PTB via SuiGrpcClient devInspect (same client as proof.ts). Per-file identity deterministic from namespace + run/chunk id so recall can reconstruct it.
• Public share pages CANNOT do SessionKey+seal_approve — confirm public namespaces stay plaintext (Tatum) and only private go through SEAL. Network reality: Tatum=mainnet, Harbor=testnet; document that public-mainnet and private-testnet blobs live on different networks.
• Encryption on write likely runs in the queue consumer / Node CLI+MCP path (packages/cli/src/index.ts:347, packages/mcp/src/index.ts:200) since packProofBundle is Node-only — gated by the seal Worker spike. Keep packProofBundle (tar) Node-only; the Worker private path needs an in-memory zip from R2 (separate roadmap item, out of scope here).

Sui Overflow angle

Turns "private" from a DB flag into real on-chain access control: decryption is gated by an on-chain seal_approve Move call. The headline Sui-native privacy demo — "revoke the token → recall can no longer decrypt" — lands here, shown side-by-side with a public Tatum/Walrus blob whose receipt names the provider.

Dependencies

The seal Worker spike, the HarborClient issue, and the namespace→Space/Bucket/Seal mapping + service-key issue. The interface + receipt widening have no deps and can land first.

Part of the ContextMEM roadmap (#4) • Sui Overflow build.

[harrymove-ctrl]

Implementation Plan

Approach
Introduce a thin StorageProvider seam in packages/walrus ({ upload, status, fetch }) so the existing Tatum code becomes one implementation (TatumStorageProvider) with zero behavior change, and a HarborStorageProvider (built on #6's HarborClient) becomes the second. SEAL crypto lives in a standalone seal.ts (decoupled from any provider) so encrypt runs on the write path and decrypt runs on the read path independently. selectStorageProvider({ visibility, network }) routes public→Tatum (mainnet, plaintext) and private→Harbor (testnet, SEAL ciphertext). Key design decision: private encryption happens on the Node CLI/MCP write path (where packProofBundle already runs, storage.ts:231), and the Worker only does fetch+decrypt at recall time via GET /api/memwal/artifact — keeping the heavy @mysten/seal decrypt the one thing we put in the Worker, which is exactly what spike #5 must clear first. The hosted (R2→Walrus) private write path stays out of scope (deferred to the in-memory-zip roadmap item).

Prerequisites & dependencies

• Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5 (seal Worker spike) — GATING. Decides whether seal.decrypt (SessionKey + seal_approve devInspect) fits the Worker bundle/CPU budget. If it busts limits, the recall route must move decrypt to a Node relayer off a Queue; do not start AC#5/AC#7 until Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5 lands green.
• Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6 (HarborClient). Provides packages/walrus/src/harbor.ts (createBucket reserve→sign→finalize, uploadFile + status poll, download ciphertext, the ~3s mirror_missing_grant 403 retry). HarborStorageProvider wraps it. AC#1 + AC#2 (interface + receipt widening) have no deps and land first.
• Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7 (namespace→Space/Bucket/Seal mapping + service-key). Persists spaceId/bucketId/sealPolicyId per namespace in D1 and decides service-key custody (shared vs per-owner). This issue reads those columns and the HARBOR_SERVICE_PRIVATE_KEY SessionKey signer; if Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7 hasn't landed, stub the mapping behind a single env default HARBOR_SPACE_ID + a deterministic bucket-per-namespace lookup.
• External creds/bindings (add to WorkerEnv worker.ts:32-55 + wrangler.jsonc + .env.example): HARBOR_API_KEY (hbr_), HARBOR_SERVICE_PRIVATE_KEY (suiprivkey1…), HARBOR_SPACE_ID, optional HARBOR_SEAL_PACKAGE_ID + HARBOR_TESTNET_RPC_URL. Mirror the header-or-secret resolution pattern of resolveMemwalCreds (worker.ts:3341).
• Open roadmap decisions that affect this issue: Decision docs: add product roadmap (Harbor storage + Talus on-chain direction) #4 (network for launch surface — confirm Tatum=mainnet / Harbor=testnet split is acceptable and documented), Decision Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6 (service-key custody — shared key decrypts everything vs per-owner). Both default to "shared service key, coexist" for this issue.
• Dep add: @mysten/seal in packages/walrus/package.json pinned compatibly with the existing @mysten/sui ^2.9.1 (root package.json:31); reuse @mysten/sui/grpc SuiGrpcClient already used in proof.ts:1.

Step-by-step

1. Define the seam (no deps). Create packages/walrus/src/provider.ts:

   • Types: StorageVisibility = "public"|"private", StorageNetwork = "mainnet"|"testnet", StorageRef = { jobId?; blobId?; fileId?; bucketId?; downloadUrls?: string[] }.
   • interface StorageProvider { readonly provider: "tatum"|"harbor"; upload(input): Promise<WalrusStorageReceipt>; status(ref): Promise<WalrusStorageJobState>; fetch(ref): Promise<Uint8Array>; }.
   • upload/status reuse the existing WalrusStorageUploadInput/WalrusStorageJobState/WaitOptions shapes from storage.ts:26-47,153.

2. Widen the receipt (no deps). Edit packages/core/src/types.ts:655 — provider: "tatum" → provider: "tatum" | "harbor"; add optional spaceId?, bucketId?, fileId?, sealPolicyId?, sealIdentity?: string. Keep all existing fields. (This is the early, dep-free landing per the issue's implementation note.)

3. Refactor Tatum behind TatumStorageProvider. In storage.ts, keep uploadToWalrusStorage/getWalrusStorageJob/waitForWalrusStorageCertified/uploadProofBundle/packProofBundle exported as-is (CLI/MCP import them directly — cli/src/index.ts:347, mcp/src/index.ts:200). Add class TatumStorageProvider implements StorageProvider: upload delegates to uploadProofBundle (already returns provider:"tatum", storage.ts:191); status → getWalrusStorageJob; fetch → GET ref.downloadUrls[0] (the aggregator URL) → Uint8Array. No behavior change for current callers (AC#1).

4. SEAL wrapper (gated by Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5). Create packages/walrus/src/seal.ts wrapping @mysten/seal:

   • encrypt(bytes, { sealPolicyId, identity }) → ciphertext Uint8Array (Node write path).
   • decrypt(ciphertext, { sessionKey, sealPolicyId, identity, suiClient }) → builds the seal_approve PTB (moveCall ${HARBOR_SEAL_PACKAGE_ID}::<module>::seal_approve(identity, policyObject)) and executes it via SuiGrpcClient devInspect (same client pattern as proof.ts:12), then SEAL SessionKey-based decrypt → plaintext.
   • Helper deriveSealIdentity(namespace, runId|chunkId) = deterministic sha256(${namespace}:${id}) hex so recall reconstructs the exact identity (per implementation note). Export all from packages/walrus/src/index.ts.

5. HarborStorageProvider (deps Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6/Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7). Create packages/walrus/src/harbor-provider.ts implementing StorageProvider over HarborClient (Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6): upload(ciphertextInput) → ensure bucket (from Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7 mapping) → uploadFile → poll status → return receipt { provider:"harbor", spaceId, bucketId, fileId, sealPolicyId, sealIdentity, blobId?, status, certified, … }; status(ref.fileId) → Harbor poll; fetch(ref.fileId) → HarborClient.download → ciphertext Uint8Array (decrypt is a separate seal.decrypt call, kept decoupled).

6. selectStorageProvider + fail-closed write path. In provider.ts: selectStorageProvider({ visibility, network }, deps) → returns HarborStorageProvider when visibility==="private", else TatumStorageProvider. Add an orchestrator storePrivateOrFallback(...):

   • private: seal.encrypt(bundle.data, { sealPolicyId, identity }) → HarborStorageProvider.upload(ciphertext); on Harbor/SEAL failure rethrow (NEVER plaintext fallback — AC#6).
   • public/non-private: TatumStorageProvider.upload; on Harbor failure for a non-private namespace that opted into Harbor, fall back to Tatum (AC#6). Route uploadProofBundle's callers through this selector (per implementation note "route it through selectStorageProvider").

7. Wire the write callers. Thread a visibility/network (+ sealPolicyId/bucketId from Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7) into the CLI storage push (cli/src/index.ts:341-378) and MCP upload_proof_to_walrus (mcp/src/index.ts:200) so private runs encrypt+Harbor and public stay Tatum. Default visibility=public preserves today's behavior.

8. Carry Harbor refs through the memory index (back-compat). Extend StorageIndexInput (memwal/src/index.ts:37) and renderStorageIndex (:281) to emit bucketId=, fileId=, sealPolicyId=, sealIdentity= lines inside the existing [ctxm-storage] block; mirror in sdk.ts:123. Add a parseStorageIndex(text) that reads the new keys and still parses legacy blocks without them (AC#4). These fields also get persisted to D1 (step 9) as the authoritative recall lookup; the MemWal text is the portable mirror.

9. D1 schema + namespace summary. Add migration apps/api/migrations/0007_harbor_seal_refs.sql: ALTER TABLE contextmem_namespaces ADD COLUMN space_id TEXT, bucket_id TEXT, seal_policy_id TEXT (and a per-version/artifact harbor_file_id + seal_identity if recall is per-artifact — store on contextmem_namespace_versions or a small contextmem_namespace_storage table keyed by namespace+version). Add the new columns to the getNamespace SELECT (worker.ts:415) and namespaceFromRow (worker.ts:4531) → HostedNamespaceSummary (sealPolicyId, bucketId, spaceId). (Mapping population is Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7's job; this issue just reads them.)

10. GET /api/memwal/artifact (recall fetch+decrypt; gated by Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5). Register in the route block (worker.ts:~630, beside /api/memwal/recall): GET /api/memwal/artifact?namespace=…&path=….

    • await store.authorizeNamespace(namespace, readAccessToken(request)) (worker.ts:502/616) — return its {status,message} on failure (token-gated, AC#5/AC#6).
    • If auth.summary.visibility === "public" → resolve downloadUrls/aggregator from the stored receipt/index and stream plaintext (Tatum fallback path, AC#5).
    • If private → resolve bucketId/fileId/sealPolicyId/sealIdentity from D1 (step 9) → HarborStorageProvider.fetch(fileId) → ciphertext → seal.decrypt(ciphertext, { sessionKey: serviceKeySessionKey, sealPolicyId, identity, suiClient }) → stream plaintext. Build the SessionKey from HARBOR_SERVICE_PRIVATE_KEY (resolved like resolveMemwalCreds, worker.ts:3341).

11. Token→SEAL decrypt mapping (AC#6). Wire near authorizeNamespace (:502) and token create/revoke (createNamespaceToken :992, revokeNamespaceToken :1013):

    • Primary deliverable: the Worker decrypts only after authorizeNamespace passes, so revoking the token → 403 → recall can no longer decrypt (the headline Sui-native demo) with the shared service-key SessionKey.
    • Add a grantSealDecrypt(namespace, token) / revokeSealDecrypt(namespace, tokenId) hook at those two call sites. Full on-chain per-grantee policy mutation depends on Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7's service-key/policy capability decision; land the token-gates-Worker-decrypt enforcement now and leave the on-chain grant as a hook (documented follow-up). Never silently downgrade a private namespace to plaintext.

12. Docs. Note the network split (public-mainnet Tatum vs private-testnet Harbor blobs live on different networks) in .env.example + a short comment near selectStorageProvider, per the issue's "document" requirement.

New / changed files

• packages/walrus/src/provider.ts (new — interface, TatumStorageProvider re-export glue, selectStorageProvider, storePrivateOrFallback)
• packages/walrus/src/seal.ts (new — @mysten/seal encrypt/decrypt + deriveSealIdentity)
• packages/walrus/src/harbor-provider.ts (new — HarborStorageProvider over Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6's HarborClient)
• packages/walrus/src/storage.ts (edit — add TatumStorageProvider; keep existing fns)
• packages/walrus/src/index.ts (edit — export provider/seal/harbor-provider)
• packages/walrus/package.json (edit — add @mysten/seal)
• packages/core/src/types.ts (edit :655 — widen provider, add Harbor fields)
• packages/memwal/src/index.ts (edit :37/:281 — Harbor fields + parseStorageIndex back-compat) and packages/memwal/src/sdk.ts (:123 mirror)
• packages/cli/src/index.ts (edit :341-378 — visibility/seal refs) and packages/mcp/src/index.ts (edit :200)
• apps/api/src/worker.ts (edit — WorkerEnv HARBOR_*, getNamespace/namespaceFromRow columns, GET /api/memwal/artifact, token→SEAL hooks)
• apps/api/migrations/0007_harbor_seal_refs.sql (new)
• apps/api/cloudflare/wrangler.jsonc + wrangler.example.jsonc + .env.example (edit — Harbor vars/secrets)

Test plan

• Unit (bun, packages/walrus): selectStorageProvider({visibility:"public"}) returns TatumStorageProvider and {visibility:"private"} returns HarborStorageProvider (AC#7). deriveSealIdentity is deterministic for the same (namespace, runId). parseStorageIndex reads new [ctxm-storage] blocks and a legacy block with no Harbor fields (AC#4 back-compat).
• Unit (fallback): mock Harbor failure → non-private upload falls back to Tatum and the receipt records provider:"tatum"; a private upload with Harbor failure throws and never produces a plaintext receipt (AC#6).
• Integration / round-trip (the proof it works): against Harbor testnet — seal.encrypt → HarborClient.uploadFile → poll certified → GET /api/memwal/artifact with a valid token → seal.decrypt → assert Buffer.equals(plaintextOut, bundle.data) (remember→recall→bytes match, AC#7). Capture the real Sui testnet tx digest from the seal_approve devInspect / bucket finalize as evidence.
• Negative auth: GET /api/memwal/artifact for the same private namespace without a valid read token → authorizeNamespace 401/403, no ciphertext fetched, no decrypt (AC#5/AC#6/AC#7). After revokeNamespaceToken, the previously-working token returns 403 and recall can no longer decrypt — the demo assertion.
• Manual: CLI contextmem storage push <run> --visibility private --remember produces a provider:"harbor" receipt with bucketId/fileId/sealPolicyId/sealIdentity; a public push is byte-identical to today (provider:"tatum"). Confirm CI bun-check.yml typecheck+test+build stays green.

Risks & gotchas

• @mysten/seal in the Worker (Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5). WebCrypto/BLS CPU + compressed-bundle cap; if decrypt busts limits, step 10's decrypt must move to a Node relayer off CONTEXTMEM_EXTRACT_QUEUE — design the recall handler so the decrypt step is swappable.
• Network split. Tatum=mainnet, Harbor=testnet — a public blob and a private blob for "the same" namespace live on different networks; recall must pick the path from the persisted provider, never guess. Document and never cross-reference.
• Sponsor-sig expiry / mirror_missing_grant. Bucket finalize sponsor sig expires fast and Harbor returns a ~3s post-finalize 403 — that retry belongs in Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6's HarborClient; this issue must surface (not swallow) a genuine private-write failure so it fails closed.
• Fail-closed correctness. The single most important invariant: a private namespace must NEVER fall back to Tatum plaintext on Harbor error (AC#6). Make the private branch rethrow with no catch-all.
• packProofBundle is Node-only (tar/child_process, storage.ts:231) — keep private encryption on the CLI/MCP/queue Node path; the Worker hosted→Walrus private write (in-memory R2 zip) is explicitly out of scope here.
• Migration 0007. ALTER TABLE ADD COLUMN on existing contextmem_namespaces is non-destructive; ensure new SELECTs tolerate NULLs for pre-migration rows (legacy public namespaces have no Harbor refs).
• Service-key custody (Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7/Decision Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6). Shared key = the Worker can decrypt everything (token gate is the only guard); per-owner = stronger isolation but needs the SessionKey signer plumbed per request. Ship shared now, leave a seam.

Acceptance-criteria mapping

• StorageProvider interface + TatumStorageProvider, no behavior change → Steps 1, 3.
• Widen provider union + Harbor fields (core/types.ts:655) → Step 2.
• packages/walrus/src/seal.ts encrypt/decrypt + seal_approve PTB → Step 4.
• selectStorageProvider, private encrypt+Harbor upload, receipt records provider, rememberStorageIndex/renderStorageIndex carry refs + back-compat → Steps 5, 6, 7, 8.
• GET /api/memwal/artifact fetch+decrypt, token-gated via authorizeNamespace, public→aggregator → Steps 9, 10.
• Fail-closed (non-private Tatum fallback, private never plaintext) + token create/revoke ↔ SEAL grant/revoke → Steps 6, 11.
• Tests: selection by visibility, public-only Tatum fallback, private round-trip + no-token can't decrypt → Test plan (unit + integration round-trip + negative auth).

Effort
L — ~5-8 person-days of focused work after #5/#6/#7 land. Dep-free slice (AC#1 + AC#2 + TatumStorageProvider: Steps 1-3) is ~1 day and can ship immediately; the SEAL/Harbor/recall slice (Steps 4-11) is the remaining ~4-7 days and is gated by the spike and the HarborClient/mapping issues.

---

Implementation plan generated for execution (planning phase). Part of the roadmap (#4).