Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets

[harrymove-ctrl]

Context

Namespaces are app-layer strings (namespaceForTarget, packages/core/src/utils.ts) tracked in the contextmem_namespaces D1 table (migrations 0002_hosted_namespaces.sql / 0003_namespace_platform.sql). To store private blobs in Harbor we must map each namespace to a Harbor Space/Bucket + Seal policy and persist those ids for recall, and the Worker needs the Harbor API key + SUI service key as secrets (mirroring resolveMemwalCreds, worker.ts:3341).

Goal / user story

As ContextMEM, I want a deterministic namespace → Space/Bucket/Seal-policy mapping persisted in D1, plus the Harbor/Sui credentials wired into Worker secrets — so any private remember/recall can resolve its on-chain storage targets.

Acceptance criteria

• New migration apps/api/migrations/0007_*.sql adds space_id, bucket_id, seal_policy_id, seal_identity_salt, and storage_provider columns to contextmem_namespaces.
• CloudflareNamespaceStore (worker.ts:401) reads/writes the new columns; a helper resolves-or-creates the Bucket + policy for a private namespace lazily on first private write and persists the ids.
• WorkerEnv (worker.ts:32) gains HARBOR_API_KEY, HARBOR_SERVICE_PRIVATE_KEY (suiprivkey1...), and HARBOR_SPACE_ID; .env.example and apps/api/cloudflare/wrangler.example.jsonc document them; a resolveHarborCreds(request, env) mirrors resolveMemwalCreds' header-or-secret precedence.
• One Space per environment (prod/staging); deterministic Bucket name per namespace string (documented mapping, e.g. sha256 of the namespace).
• DECISION recorded: shared service key (simple, decrypts everything) vs per-owner key (true isolation) — reflected in the credential helper.
• Secrets are never logged; handling is consistent with existing key redaction (sanitizeMemwalKey pattern).

Implementation notes

• Set secrets via wrangler secret put HARBOR_API_KEY --config apps/api/cloudflare/wrangler.jsonc (and the staging --env staging variant); use decodeSuiPrivateKey for the service key.
• Bucket creation goes through HarborClient.createBucket. Persisting the ids lets GET /api/memwal/artifact resolve refs without re-deriving.
• Keep the namespace→bucket mapping deterministic so the Node CLI/MCP path and the Worker agree on the same bucket for a given namespace.

Sui Overflow angle

Establishes the per-namespace on-chain object graph (Bucket + Seal-policy objects on Sui testnet) and the signing key that creates them — the reproducible backbone that makes the Sui-native private-memory demo work identically across prod and staging.

Dependencies

The HarborClient.createBucket issue. Feeds the StorageProvider + SEAL encrypted remember/recall issue.

Part of the ContextMEM roadmap (#4) • Sui Overflow build.

[harrymove-ctrl]

Implementation Plan

Approach
Add a 0007 D1 migration that bolts Harbor/SEAL target columns onto contextmem_namespaces, teach CloudflareNamespaceStore to read/write them, and add a lazy ensureHarborTargets(namespace) helper that — on the first private write — derives a deterministic bucket name, calls HarborClient.createBucket (reserve→sign→finalize with the service key), and persists space_id/bucket_id/seal_policy_id/seal_identity_salt/storage_provider. Wire HARBOR_API_KEY / HARBOR_SERVICE_PRIVATE_KEY / HARBOR_SPACE_ID into WorkerEnv with a resolveHarborCreds(request, env) that mirrors resolveMemwalCreds' header-or-secret precedence and redaction. Key design decision: one shared service key + one Space per environment, with a per-namespace seal_identity_salt providing in-policy isolation, and the bucket-name derivation (not the on-chain id) is the deterministic contract that lets the Node CLI/MCP and the Worker converge on the same bucket — so the derivation function lives in a shared package, computed from sha256(namespace) independently in each runtime.

Prerequisites & dependencies

• BLOCKING (code): the HarborClient issue (HarborClient.createBucket in packages/walrus/src/harbor.ts). ensureHarborTargets calls it. Build Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7's pure parts (migration, columns, resolveHarborCreds, deterministic naming, read paths) against an injected/typed HarborClient interface so they merge even before the client lands; gate only the create call on it.
• GATED BY the @mysten/seal Worker spike (Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5) only if SEAL policy creation happens at bucket-create time and pulls @mysten/seal into the Worker. Bucket create + Sui signing needs only @mysten/sui (already a dep — apps/api/package.json has @mysten/sui ^2.9.1, and decodeSuiPrivateKey/Ed25519Keypair.fromSecretKey("suiprivkey1…") is already used in packages/core/src/snapshot.ts:135), which is Worker-safe under nodejs_compat. If seal_policy_id provisioning forces @mysten/seal and that busts Worker bundle/CPU limits, ensureHarborTargets must run in the queue consumer or the Node CLI/MCP path instead of the request path.
• Roadmap DECISIONs to confirm (team): Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6 service-key custody (this plan recommends shared, AC5), Walrus storage + Memory via Tatum, plus CI deploy #1 coexist Tatum(public/mainnet)+Harbor(private/testnet), docs: add product roadmap (Harbor storage + Talus on-chain direction) #4 network for launch surface.
• External creds/bindings: a Harbor hbr_… API key, a suiprivkey1… service key, and a pre-created Harbor HARBOR_SPACE_ID per env (prod + staging). No new Cloudflare binding needed (reuses CONTEXTMEM_DB).

Step-by-step

1. Migration — create apps/api/migrations/0007_harbor_storage.sql. Mirror the ADD COLUMN … CHECK style proven in 0006_namespace_build_sources.sql:

   ALTER TABLE contextmem_namespaces ADD COLUMN storage_provider TEXT NOT NULL DEFAULT 'tatum' CHECK (storage_provider IN ('tatum','harbor'));
   ALTER TABLE contextmem_namespaces ADD COLUMN space_id TEXT;
   ALTER TABLE contextmem_namespaces ADD COLUMN bucket_id TEXT;
   ALTER TABLE contextmem_namespaces ADD COLUMN seal_policy_id TEXT;
   ALTER TABLE contextmem_namespaces ADD COLUMN seal_identity_salt TEXT;
   CREATE INDEX IF NOT EXISTS contextmem_namespaces_bucket_idx ON contextmem_namespaces(bucket_id);

   (SQLite ADD COLUMN: NOT NULL only with a constant default — only storage_provider is NOT NULL; the rest are nullable, filled lazily.)
2. Types + row mapping — in apps/api/src/worker.ts: extend NamespaceRow (worker.ts:103) with storage_provider, space_id, bucket_id, seal_policy_id, seal_identity_salt; add the same five SELECT columns to the four queries that hydrate NamespaceRow → getNamespace (worker.ts:415), listNamespaces owner + all (worker.ts:928, :937), listDirectoryNamespaces (worker.ts:1033); map them in namespaceFromRow (worker.ts:4531). Add storageProvider?, spaceId?, bucketId?, sealPolicyId?, sealIdentitySalt? to HostedNamespaceSummary in packages/mcp/src/hosted.ts:13.
3. Deterministic bucket naming (shared, runtime-agnostic) — add a pure formatter harborBucketName(namespaceSha256Hex: string): string (e.g. `cmem-${hex.slice(0, 40)}`, lowercase) to packages/walrus/src/harbor.ts (the HarborClient file) exported via a leaf path with no node: deps. It takes the already-computed sha256 hex so each runtime hashes with its own primitive (Worker = its async WebCrypto sha256Hex; Node CLI/MCP = sha256Hex in packages/core/src/utils.ts:14) yet produces an identical name — satisfying the Node↔Worker determinism requirement without bundling node:crypto into the Worker.
4. Credential helper + DECISION — in worker.ts near resolveMemwalCreds (worker.ts:3341), add sanitizeHarborKey() (strip Bearer /whitespace, mirror sanitizeMemwalKey worker.ts:3331) and resolveHarborCreds(request, env) returning { apiUrl, apiKey, serviceKey, spaceId } with header-first then env precedence (x-harbor-api-key / x-harbor-service-key / x-harbor-space-id, validate apiKey starts hbr_ and serviceKey starts suiprivkey1). Decode the service key with decodeSuiPrivateKey → Ed25519Keypair. Record the DECISION in a comment block: shared service key (single HARBOR_SERVICE_PRIVATE_KEY, can decrypt everything — simplest for the grant demo) over per-owner keys; the per-namespace seal_identity_salt is the lever that gives namespace-level separation under the shared key. The header override path is the BYO-Harbor power-user escape hatch (same pattern as the MemWal delegate headers), keeping the key out of any persisted/logged surface.
5. Resolve-or-create helper — add ensureHarborTargets(namespace) as a method on CloudflareNamespaceStore (worker.ts:401) plus a read-only getHarborTargets(namespace):
   • read the namespace; if storage_provider='harbor' and bucket_id + seal_policy_id present → return them (no tx);
   • else call resolveHarborCreds; if unconfigured → return null so callers fall back to Tatum for non-private (private write should error clearly);
   • compute harborBucketName(await sha256Hex(namespace)); generate a random seal_identity_salt (e.g. crypto.getRandomValues → hex);
   • call HarborClient.createBucket({ spaceId, name }) (reserve→sign→finalize as one tight sequence; tolerate the ~3s mirror_missing_grant 403 retry inside the client) → { bucketId, sealPolicyId };
   • persist with a conditional UPDATE contextmem_namespaces SET storage_provider='harbor', space_id=?, bucket_id=?, seal_policy_id=?, seal_identity_salt=?, updated_at=? WHERE namespace=? AND bucket_id IS NULL, then re-read (idempotent against concurrent first-writes; deterministic name means a duplicate create resolves to the same bucket).
     This is the lazy "on first private write" hook the StorageProvider/SEAL remember-recall issue will call.
6. Config + secrets docs —
   • WorkerEnv (worker.ts:32): add HARBOR_API_KEY?, HARBOR_SERVICE_PRIVATE_KEY?, HARBOR_SPACE_ID?, HARBOR_API_URL?.
   • apps/api/cloudflare/wrangler.example.jsonc: add HARBOR_API_KEY, HARBOR_SERVICE_PRIVATE_KEY to secrets.required; document HARBOR_SPACE_ID (an identifier, not a secret) as a vars entry.
   • apps/api/cloudflare/wrangler.jsonc: add HARBOR_SPACE_ID to top-level vars and to the env.staging.vars block (distinct space per env, AC4).
   • .env.example: add a "Harbor (private/encrypted storage)" section: HARBOR_API_KEY=, HARBOR_SERVICE_PRIVATE_KEY= (suiprivkey1…), HARBOR_SPACE_ID=, HARBOR_API_URL=, HARBOR_NETWORK=testnet.
   • Document the secret-set commands in a comment (as the OpenAI key is documented at wrangler.jsonc:12): echo "<hbr_…>" | npx wrangler secret put HARBOR_API_KEY --config apps/api/cloudflare/wrangler.jsonc and the --env staging variant.

New / changed files

• NEW apps/api/migrations/0007_harbor_storage.sql
• EDIT apps/api/src/worker.ts (WorkerEnv, NamespaceRow, 4 SELECTs, namespaceFromRow, sanitizeHarborKey, resolveHarborCreds, CloudflareNamespaceStore.ensureHarborTargets/getHarborTargets)
• EDIT packages/mcp/src/hosted.ts (HostedNamespaceSummary new optional fields)
• NEW (or co-located with HarborClient dep) packages/walrus/src/harbor.ts — pure harborBucketName(sha256Hex) formatter (leaf, no node: deps)
• EDIT apps/api/cloudflare/wrangler.example.jsonc, apps/api/cloudflare/wrangler.jsonc, .env.example
• NEW test(s): apps/api/src/worker.harbor.test.ts (or extend an existing harness alongside packages/mcp/src/hosted.test.ts)

Test plan

• Unit (bun test): harborBucketName is deterministic + stable (same namespace → same name across runs; verify Node sha256Hex and a WebCrypto sha256 produce the identical name for one fixture); resolveHarborCreds precedence (valid header overrides env, env fallback, malformed hbr_/suiprivkey1 rejected, missing → undefined); sanitizeHarborKey strips Bearer /0x/whitespace; assert no code path passes the raw service key to console.*.
• Integration (fake D1 + mocked HarborClient, mirror hosted.test.ts): first ensureHarborTargets call invokes createBucket once and writes all five columns; second call returns persisted ids with zero createBucket calls; concurrent first-writes create at most one bucket (conditional WHERE bucket_id IS NULL); namespaceFromRow round-trips the new columns.
• Migration test: wrangler d1 migrations apply contextmem --local then PRAGMA table_info(contextmem_namespaces) shows the five new columns + index (add to CI's migration step).
• Manual (proves it works): set the staging secrets, mark a namespace private and trigger a first private write → observe a real Sui testnet bucket-create tx digest in Worker logs (Sui explorer testnet) and a populated D1 row (bucket_id/seal_policy_id/storage_provider='harbor'); reload and confirm no second tx is signed.

Risks & gotchas

• Sponsor-sig expiry: reserve→sign→finalize must be one tight sequence; do all D1 persistence after finalize so slow writes never sit between reserve and finalize.
• mirror_missing_grant ~3s 403 after finalize — tolerate/retry (owned by HarborClient; ensureHarborTargets must not treat the first 403 as fatal).
• Worker limits / Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5 gate: Sui signing via @mysten/sui is fine in-Worker; if seal_policy_id provisioning needs @mysten/seal and busts bundle/CPU caps, move ensureHarborTargets to the queue consumer / Node path.
• Concurrency: two simultaneous first-writes — idempotent deterministic name + conditional UPDATE + re-read.
• Network split: Harbor=testnet vs Tatum=mainnet; storage_provider records the actual provider so recall never mixes networks.
• D1 migration is forward-only — NOT NULL columns need a constant default (only storage_provider); the rest stay nullable.
• Secrets never logged: route all Harbor key handling through sanitizeHarborKey/redaction consistent with the sanitizeMemwalKey pattern; never echo serviceKey in error envelopes (handleWorkerRequest at worker.ts:562 surfaces messages to clients).
• Determinism contract is the bucket NAME, not the on-chain id — keep the formatter in a shared, node:-free leaf so CLI/MCP and Worker agree.

Acceptance-criteria mapping

• Migration 0007_*.sql adds the 5 columns → Step 1.
• CloudflareNamespaceStore reads/writes new columns + lazy resolve-or-create helper → Steps 2 & 5.
• WorkerEnv gains the 3 vars; .env.example + wrangler.example.jsonc document them; resolveHarborCreds mirrors resolveMemwalCreds → Steps 4 & 6.
• One Space per env; deterministic Bucket name per namespace (documented sha256 mapping) → Steps 3 & 6 (staging/prod HARBOR_SPACE_ID vars + harborBucketName).
• DECISION recorded: shared vs per-owner key, reflected in the helper → Step 4 (shared key + per-namespace salt, documented in the helper comment).
• Secrets never logged; consistent with sanitizeMemwalKey redaction → Step 4 + Risks.

Effort
M — ~2-3 person-days for #7's own scope (migration, columns, helpers, config, tests). Excludes the blocking HarborClient.createBucket work; if that and the SEAL-policy path land in the Worker, add the #5 spike outcome as a dependency.

---

Implementation plan generated for execution (planning phase). Part of the roadmap (#4).