Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download

[harrymove-ctrl]

Context

There is no Harbor client today; storage is Tatum-only and plaintext (packages/walrus/src/storage.ts). Harbor's model is SPACES > BUCKETS > FILES, and bucket creation is a RESERVE → SIGN → FINALIZE handshake (an Enoki-sponsored Sui tx — no SUI balance needed). We need a pure-fetch, runtime-agnostic client that works in both the Worker and Node (the seal Worker spike decides which runtime actually signs/uploads).

Goal / user story

As ContextMEM, I want a HarborClient that can create a bucket (reserve→sign→finalize), upload a file (multipart ciphertext), poll its status to ready, and download it — so private namespaces can store encrypted blobs on Walrus via Sui.

Acceptance criteria

• New packages/walrus/src/harbor.ts exports a HarborClient (class or factory) with NO node:fs/node:child_process/node:os imports (runs in Worker + Node).
• createBucket(spaceId, name) performs RESERVE → sign the sponsored tx with Ed25519Keypair from decodeSuiPrivateKey(serviceKey) → FINALIZE as one tight back-to-back sequence; returns { bucketId, sealPolicyId }.
• uploadFile(bucketId, { name, data, contentType }) does the multipart upload and returns a fileId; getFileStatus(fileId) polls to a terminal/ready state mirroring waitForWalrusStorageCertified (storage.ts:160).
• A post-finalize 403 mirror_missing_grant is retried with backoff over the ~3s mirror window before surfacing an error.
• download(fileId) returns ciphertext bytes.
• Unit test with a mocked fetch covers reserve→sign→finalize ordering, the mirror_missing_grant retry, and an upload+poll happy path.

Implementation notes

• Auth: Authorization: Bearer hbr_.... Inject config (apiKey, serviceKey, spaceId, baseUrl, optional fetch) the same way TatumStorageConfig is injected in storage.ts:21.
• Sign with @mysten/sui/keypairs/ed25519 + decodeSuiPrivateKey from @mysten/sui/cryptography; build/execute the sponsored tx (Enoki sponsors gas). Sponsor sig expires fast — RESERVE and FINALIZE must be back-to-back with no awaiting of unrelated work between them.
• Add @mysten/seal + @mysten/sui to packages/walrus/package.json (currently only fast-xml-parser + p-limit) and re-export from packages/walrus/src/index.ts.
• This client takes ciphertext only — encryption lives in seal.ts (StorageProvider + SEAL issue). Mirror the polling/receipt shape already in storage.ts so the StorageProvider seam can wrap it.

Sui Overflow angle

The reserve→sign→finalize handshake is the FIRST time ContextMEM signs a real Sui tx. The resulting Bucket + Seal-policy objects are genuine on-chain artifacts (Enoki-sponsored, zero SUI), giving a concrete "where is the Sui contract" answer to show judges — with no custom Move authored yet.

Dependencies

The seal Worker spike decides whether sign/upload runs in Worker or Node. The namespace→Space/Bucket/Seal mapping issue supplies the service key + space id config this client consumes.

Part of the ContextMEM roadmap (#4) • Sui Overflow build.

[harrymove-ctrl]

Implementation Plan

Approach
Add a self-contained, runtime-agnostic packages/walrus/src/harbor.ts exporting a HarborClient built on pure fetch (Worker + Node) plus @mysten/sui for signing only (no @mysten/seal import here — encryption lives in the sibling seal.ts issue). The one hard design constraint is that the Enoki sponsor signature expires fast, so createBucket performs RESERVE → (synchronous CPU-only) SIGN → FINALIZE as one tight back-to-back sequence with zero unrelated awaits in between. Everything else (config injection, defensive JSON parsing, the status-poll loop) is mirrored from the proven Tatum code in storage.ts so the later StorageProvider seam can wrap it cleanly.

Prerequisites & dependencies

• Informs but does NOT block this build — Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5 (@mysten/seal Worker spike). It decides where sign/upload eventually runs (Worker vs queue vs Node), but HarborClient is runtime-agnostic by design and uses only @mysten/sui crypto (BLS-free Ed25519 signing), so it can be authored and unit-tested now. Do not import @mysten/seal from harbor.ts.
• Consumer issues (not blockers): the namespace→Space/Bucket/Seal-mapping issue supplies serviceKey + spaceId config this client consumes; the StorageProvider-abstraction/receipt-widening issue (which widens WalrusStorageReceipt.provider from the literal "tatum" at packages/core/src/types.ts:656) wraps this client. harbor.ts stays decoupled from both — it returns plain typed shapes.
• External creds/config to inject (constructor, mirroring TatumStorageConfig at storage.ts:21): apiKey (hbr_… Bearer), serviceKey (suiprivkey1…), spaceId, baseUrl, optional fetch. A Sui testnet service key is required, but it needs no SUI balance (Enoki sponsors bucket-creation gas). Worker-secret wiring (HARBOR_API_KEY/HARBOR_SERVICE_PRIVATE_KEY/HARBOR_SPACE_ID into WorkerEnv at apps/api/src/worker.ts:32) is the separate key-mgmt issue — out of scope here.
• OPEN DECISION (single biggest unknown) — exact Harbor alpha API surface. The precise endpoint paths and request/response field names for RESERVE / FINALIZE / upload / status / download are not in the repo. Before coding, confirm them against Harbor docs or a live curl (capture one real RESERVE response). Mitigation baked into the plan: parse defensively (port pickString/pickUrls from storage.ts:74-101) so minor field-name drift doesn't break the client.
• @mysten/sui is already a workspace dep at ^2.9.1 (root package.json:31, apps/api/package.json:20), so the version is known-good under nodejs_compat.

Step-by-step

1. Add deps + re-export. In packages/walrus/package.json (currently only fast-xml-parser + p-limit, lines 13-17) add "@mysten/sui": "^2.9.1" and "@mysten/seal": "^0.x" (seal added per issue note for the sibling seal.ts; harbor.ts itself must not import it). Add export * from "./harbor.js"; to packages/walrus/src/index.ts (after line 6). Run bun install.
2. Config + types in new packages/walrus/src/harbor.ts. Define HarborClientConfig = { apiKey, serviceKey, spaceId, baseUrl, fetch? } and a private resolveFetch/resolveConfig (mirror storage.ts:49, but read no process.env so it stays Worker-clean — caller injects everything). Define result types: HarborBucket = { bucketId, sealPolicyId, txDigest? }, HarborFileAccepted = { fileId, status? }, HarborFileState = { fileId, status, ready, failed, raw } (shape parallels WalrusStorageJobState at storage.ts:39). Port readBody/pickString/pickUrls (storage.ts:64-101) as module-private helpers. No node:fs/node:child_process/node:os/node:path imports (AC Walrus storage + Memory via Tatum, plus CI deploy #1).
3. createBucket(spaceId, name) — RESERVE → SIGN → FINALIZE, back-to-back:
   • Build the signer once: import { Ed25519Keypair } from "@mysten/sui/keypairs/ed25519" + import { decodeSuiPrivateKey } from "@mysten/sui/cryptography"; const keypair = Ed25519Keypair.fromSecretKey(decodeSuiPrivateKey(serviceKey).secretKey); (memoize on the instance).
   • RESERVE: POST {baseUrl}/…/buckets/reserve with Authorization: Bearer hbr_…, body { spaceId, name }. Response carries the Enoki-sponsored unsigned tx: { txBytes (base64 BCS TransactionData), sponsorSignature, reservationId } — parse via pickString with candidate keys (txBytes/transactionBytes/bytes, reservationId/id).
   • SIGN (synchronous, no network/await of unrelated work): const { signature } = await keypair.signTransaction(fromBase64(txBytes)); (the single unavoidable await; nothing else between reserve and finalize — this is the sponsor-sig-expiry guard, issue note line 17).
   • FINALIZE: immediately POST {baseUrl}/…/buckets/finalize body { reservationId, signature } (Harbor already holds txBytes+sponsorSignature from reserve). Parse { bucketId, sealPolicyId } from the response (candidate keys bucketId/bucket_id/id, sealPolicyId/seal_policy_id/policyId). Return HarborBucket.
4. uploadFile(bucketId, { name, data, contentType }) + download(fileId):
   • Upload via multipart exactly like storage.ts:109-112: new FormData() + new Blob([data], { type: contentType ?? "application/octet-stream" }), form.append("file", blob, name), POST {baseUrl}/…/buckets/{bucketId}/files with the Bearer header (no Content-Type — let fetch set the boundary). Wrap this POST in the mirror-grant retry (Step 5). Return { fileId }. The client takes ciphertext only (issue note line 19) — no encryption here.
   • download(fileId): GET {baseUrl}/…/files/{fileId}/download with Bearer → new Uint8Array(await res.arrayBuffer()) (ciphertext bytes) (AC Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5).
5. mirror_missing_grant retry. Add a private withMirrorGrantRetry(fn) helper: on a 403 whose body contains mirror_missing_grant, retry with backoff covering the ~3s window (e.g. delays 400, 800, 1200, 1600, 2000 ms via injectable sleep, ~6 attempts ≈ 6s of cushion); any other non-OK status throws immediately with the parsed body (mirror the error style of storage.ts:115). Wrap uploadFile's POST and the first getFileStatus call with it (AC docs: add product roadmap (Harbor storage + Talus on-chain direction) #4).
6. getFileStatus(fileId) + waitForFileReady. getFileStatus: GET {baseUrl}/…/files/{fileId}, normalize status to { ready, failed } (uppercase compare for ready states like READY/CERTIFIED/STORED and terminal FAILED/ERROR) — same normalization shape as getWalrusStorageJob (storage.ts:140-150). waitForFileReady(fileId, opts): deadline loop with pollIntervalMs/timeoutMs/onPoll, return on ready, throw on failed/timeout — a near-copy of waitForWalrusStorageCertified (storage.ts:160-177) so the receipt/poll shape matches Tatum (AC fix(ci): force bun + pin wrangler in Deploy workflow #3).
7. Unit test packages/walrus/src/harbor.test.ts (vitest — import { describe, expect, it, vi } from "vitest", matching quilt.test.ts:1). Inject a fake fetch via config and a use a deterministic test suiprivkey1… testnet key. Cover: (a) reserve→sign→finalize ordering — assert RESERVE called before FINALIZE, FINALIZE body carries a real signature produced by signTransaction over the reserve txBytes, and { bucketId, sealPolicyId } returned; (b) mirror-grant retry — fake fetch returns 403 {"error":"mirror_missing_grant"} twice then 200, assert upload eventually succeeds and call count == 3 (inject a no-op sleep); (c) upload+poll happy path — uploadFile → getFileStatus/waitForFileReady transitions PENDING→READY; (d) download returns the exact ciphertext bytes the fake fetch served (round-trip assert). Add a static guard test (or rely on AC#1 review) that harbor.ts source contains no node: import.

New / changed files

• packages/walrus/src/harbor.ts (new — HarborClient + types + helpers)
• packages/walrus/src/harbor.test.ts (new — vitest, mocked fetch)
• packages/walrus/src/index.ts (edit — add export * from "./harbor.js";)
• packages/walrus/package.json (edit — add @mysten/sui + @mysten/seal)

Test plan

• Unit (CI, no network): the four cases in Step 7, all on a mocked fetch + injected sleep so they run in bun run check/vitest run. The mirror-retry and reserve-before-finalize assertions are the load-bearing ones.
• Integration / manual (proves it really works): a bun-run scratch script using a real HARBOR_API_KEY + testnet suiprivkey1 against the live alpha: createBucket → log the returned bucketId/sealPolicyId and the Sui tx digest, then verify the Bucket + Seal-policy objects on a testnet explorer (this is ContextMEM's first real signed Sui tx — the concrete "where's the contract" artifact, issue "Sui Overflow angle"). Then uploadFile a small ciphertext blob → waitForFileReady to ready → download → assert bytes are byte-identical to what was uploaded (full round-trip). Exercise the upload immediately after finalize to confirm the mirror_missing_grant path fires on real infra.
• Acceptance: a captured testnet tx digest + a green round-trip assert = done.

Risks & gotchas

• Sponsor-sig expiry (highest): any await (DB write, logging round-trip, extra fetch) between RESERVE and FINALIZE can let the Enoki sponsor signature expire → finalize fails. Keep createBucket a single straight-line method; the only await between reserve and finalize is signTransaction. Surface a clear error if finalize returns an expiry/sponsor error so callers retry the whole reserve→finalize sequence (never reuse stale txBytes).
• API-shape uncertainty: Harbor is testnet alpha; endpoint paths/field names may differ from assumptions. Defensive pickString parsing + centralizing all paths/keys in one const block at the top of harbor.ts keeps adaptation to a one-spot edit once a real response is captured.
• Worker limits: harbor.ts deliberately avoids @mysten/seal (the BLS/CPU/bundle risk gated by Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5); @mysten/sui Ed25519 signing + fetch are Worker-safe under nodejs_compat. Multipart FormData/Blob upload already works in the Worker (proven by Tatum storage.ts).
• signTransaction input format: it expects raw BCS TransactionData bytes, not a serialized Transaction object. Decode the reserve txBytes with fromBase64 (or fromB64, version-dependent) before signing; pin against the installed @mysten/sui@2.9.1 API.
• Network split: these are testnet objects while Tatum stays mainnet — do not cross-reference Harbor fileId/bucketId with mainnet Tatum blobIds. Receipt/provider reconciliation is the StorageProvider issue's job.
• Mirror window tuning: ~3s is a guideline; size the retry budget to ~6s and make sleep injectable so tests stay instant.

Acceptance-criteria mapping

• New harbor.ts exports HarborClient, no node:fs/child_process/os imports → Steps 1, 2 (and the static no-node: guard in Step 7).
• createBucket(spaceId,name) RESERVE→sign(Ed25519Keypair+decodeSuiPrivateKey)→FINALIZE back-to-back, returns {bucketId, sealPolicyId} → Step 3.
• uploadFile multipart → fileId; getFileStatus polls mirroring waitForWalrusStorageCertified (storage.ts:160) → Steps 4, 6.
• Post-finalize 403 mirror_missing_grant retried with backoff over ~3s → Step 5.
• download(fileId) returns ciphertext bytes → Step 4.
• Unit test (mocked fetch) covers reserve→sign→finalize ordering, mirror retry, upload+poll happy path → Step 7.

Effort
L — ~3-4 person-days. ~1 day client skeleton (config/types/helpers/upload/status/download), ~1 day the reserve→sign→finalize signing path + mirror-grant retry, ~0.5 day tests, ~0.5-1 day reconciling the real Harbor alpha API shape from a live capture. Add a few hours of buffer if the alpha endpoint contract differs materially from the assumed shape.

---

Implementation plan generated for execution (planning phase). Part of the roadmap (#4).