Extract TS SDK, hoist pnpm workspace to repo root#160
Open
ilchu wants to merge 15 commits into
Open
Conversation
Part of #135 / #136 (PR A of two: examples + E2E migrate now, UIs + test-helpers follow in PR B). Workspace: pnpm-workspace.yaml, lockfile, and a private root package.json move to the repo root; members now include user-interfaces/*, shared/*, the shared descriptors package, packages/*, and examples/papi. This deletes the examples npm island (own lockfile, own generated .papi copy, polkadot-api manifest/lock drift) — the whole workspace resolves one polkadot-api and one descriptor instance. packages/sdk (@web3-storage/sdk): flat typed functions ported from examples/papi/{common,api,sc-api}.js — connect/readiness guards, signers (seedToKeypair, makeSigner, Alice..Ferdie), unified submitTx, provider HTTP helpers, and per-pallet extrinsic wrappers; pallet_revive helpers live on the ./revive subpath so viem stays out of non-contract graphs. Canonical tx semantics are encoded as API defaults: in-block ("best") submission, READ_OPTS best-block reads, finalization only for the challenge creators (ids embed the creation block), events extracted from the tx result. The wait helpers are watchValue-based (replays current value on subscribe -> no missed-event race, no poll interval), replacing the 1s-poll waitForAgreementAcceptance; waitForPrimaryProvider is ready to replace the three UI polling copies in PR B. PAPI v2 conventions adopted throughout (the old examples leaned on substrate-bindings 0.16): Binary/Enum come from "polkadot-api" itself, Vec<u8> fields take plain Uint8Array, fixed-size binary travels as 0x-hex (SizedHex) via a new asHex helper, and getWsProvider moved to "polkadot-api/ws". CLAUDE.md guidance updated accordingly. The tracked metadata snapshot in shared/papi had drifted from the runtime (187KB -> 225KB; it lacked pallet_revive entirely, and claim_expired_agreement had dropped its provider arg). Regenerated from a live chain, and the e2e CI job now re-fetches metadata and fails loudly on drift — the papi:check that PAPI_OVERHAUL designed but never shipped. CI: a setup-pnpm composite action (Node + pnpm 11 + store cache + frozen install at root) replaces the per-job setup-node/pnpm dances; the smoke, e2e, and sc jobs gain it because papi-setup is now a root pnpm install. c8 runs from the repo root so coverage spans packages/sdk alongside the suite. ui-checks/deploy-ui path filters extended to the hoisted workspace files. justfile invokes example scripts via node --import tsx (raw-TS sdk imported from plain node; the e2e runner forwards the loader to its child processes). Validated against a local dev chain + provider: e2e suite 10/10 workflows (95/95 tests), full-flow demo (2/2 ChallengeDefended), sc-demo, sc-coverage (15/15 selectors), sc-team-drive, sc-token-gated.
Inside the CI container, git refuses to discover the host-owned checkout (dubious-ownership guard surfaces as "Not a git repository"), so the drift check's git diff always failed regardless of metadata state. Compare a sha256 of the tracked snapshot before/after regeneration instead — no git involved, and the one-file content hash is the precise signal anyway. Also guard the always()-running coverage report against the case where an earlier step failed before the E2E suite ran (no tmp dir -> c8 ENOENT noise on top of the real failure).
…eb3Storage facade Delivers the #123 monorepo layout while keeping every existing import stable: @web3-storage/core (backend-free, browser-safe: byte/hex utils, retrying httpFetch, provider request signing per auth.rs, CID verification with CidMismatchError), @web3-storage/layer0 (the former packages/sdk content, now depending on core), @web3-storage/layer1 (FileSystemClient + S3Client over layer 0), and @web3-storage/sdk as the umbrella that re-exports all three and hosts the Web3Storage.connect facade. Dependency direction is strictly layer1 -> layer0 -> core; consumers keep importing @web3-storage/sdk (./revive, plus new ./fs and ./s3 subpaths). The layer-1 clients unify the three hand-rolled implementations: chain ops delegate to the pallet wrappers (which now accept a pass-through SubmitOpts param — closing PR B's noted gap — so clients run silent, in-block, no auto-retry), provider resolution shares one cached watchValue-based resolver, HTTP goes through core's retrying fetch and is signed (raw sr25519, ChainSigner now carries its keypair when locally derived) whenever possible. getObject verifies downloads against the on-chain CID: single-chunk mismatches throw CidMismatchError (data_root == chunk hash), multi-chunk payloads surface verified: false (DAG-walk parity with the Rust client is tracked separately), and path-based fs downloads are documented as unverified.
…age/sdk
user-interfaces/sdk/typescript/{file-system,s3} (2,212 LOC) had zero
consumers — not workspace members, never built, zero tests, never in CI
or the justfile — and both UIs had long since reimplemented everything
they covered. They were also browser-broken (Buffer in base64 helpers),
did no CID verification (audit finding B1; blake2AsU8a imported and
never called), and depended on the forbidden @polkadot/keyring +
@polkadot/util-crypto. The API shape worth keeping (type names,
bucket-name/object-key validation, x-amz-meta round-tripping) lives on
in @web3-storage/layer1.
docs/README.md, docs/filesystems/README.md (which falsely claimed
drive-ui used the stub), and console-ui's README now point at
@web3-storage/sdk; its README documents the #123 layout, tx semantics,
and the download-verification matrix.
bkontur
reviewed
Jun 11, 2026
|
|
||
| for (const providerAccount of providers) { | ||
| const provider = await api.query.StorageProvider.Providers.getValue(providerAccount); | ||
| const provider = await api.query.StorageProvider.Providers.getValue(providerAccount, { at: "best" }); |
Collaborator
There was a problem hiding this comment.
@ilchu I introduced for examples constant export const READ_OPTS = { at: "best" };, but for real usage, not sure if BestBlock is ok, maybe we should stick to the finalized (explicitly -> refactor to some constant, so we can change easily)
bkontur
reviewed
Jun 11, 2026
| const buckets: BucketInfo[] = []; | ||
| for (const bucketId of bucketIds) { | ||
| const bucket = await this.api!.query.S3Registry.S3Buckets.getValue(bucketId); | ||
| const bucket = await this.api!.query.S3Registry.S3Buckets.getValue(bucketId, READ_OPTS); |
Collaborator
There was a problem hiding this comment.
@ilchu for real UI, probably we should go with finalize, as I said before better refactor this and set the separate const as finalized for now
bkontur
approved these changes
Jun 11, 2026
bkontur
left a comment
bkontur
left a comment
Collaborator
There was a problem hiding this comment.
lgtm modulo I would keep the real UIs using "finalized":
- either remove READ_OPTS from UIs
- or introduce separate const for UIs READ_OPTS = "finalized-or-what-is-the-format"
we can keep READ_OPTS = "at: best" for examples and tests, that is ok
…09 through the layer-1 clients All 17 example/E2E scripts move from .js to .ts under a strict tsconfig (typecheck script + typescript devDep added; the stale .c8rc.json that still listed the deleted api.js/common.js goes away, as does e2e/format.js — its formatDispatchError has lived in the sdk since PR A). Typing is pragmatic: typed sdk imports drive inference, cross-closure ids are annotated, storage reads in tests use non-null assertions where an assert follows anyway, and provider HTTP JSON stays loose where the sdk itself returns any. The runner discovers *.ts workflows and forwards the tsx loader to its children; justfile file references updated. e2e/03 now drives the S3 surface through S3Client (chain ops + a new 3.12 object HTTP round-trip exercising put/get/list/delete WITH on-chain CID verification — the verified=true path), keeping one raw putChunk path for layer-0 coverage. e2e/09 drives drives through FileSystemClient and gains 9.8, the first e2e coverage of the provider's /fs HTTP routes from TS (mkdir/upload/ls/download/delete round-trip).
…ges + examples in CI 76 unit tests across the three packages: byte/hex/base64 roundtrips (incl. >64KiB chunked base64), CID verification incl. CidMismatchError payloads, httpFetch retry/backoff semantics (5xx retried, 4xx and aborts not), the provider auth-header format against the canonical message, signer derivation grammar parity, the submitTx engine against rxjs Subject fakes (best/finalized modes, TxDispatchError, stale-nonce retry on/off with fake timers), requireOneEvent, NonceManager (best-block read), firstMatch (skip/timeout/onTick), and the fs/s3 clients against injected fetch mocks (URL shapes, auth + x-amz-meta headers, wire-shape mapping, and the getObject verification matrix: verified single-chunk, hard-fail on corruption, unverified without on-chain metadata). ui-checks gains a typecheck step for the packages + examples and its change filter now covers examples/papi/** (example-only changes would previously skip CI entirely).
… + tampering spec The final unification step (#135's "UIs consume the shared code" end to end): drive-ui's DriveClient shrinks to a thin adapter over FileSystemClient — fs HTTP ops, retry/backoff, provider-URL cache, and acceptance waits all come from the sdk; what remains app-side is Blob conversion, status strings, and stateless signer swapping (public key recovered from the SS58 address; wallet flows carry no raw keypair, so provider requests stay unsigned exactly as before). Its stale-descriptor DriveCreated fallback is gone — CI fails loudly on metadata drift now. console-ui's StorageClient delegates its object-HTTP half (put/get/ list/delete + request signing) to S3Client; setSigner now derives one sdk ChainSigner (keypair + PolkadotSigner from a single makeSigner call), encryption stays app-side wrapping opaque bytes so the on-chain CID covers exactly what the provider stores, and on dev chains the client pins the local provider URL (the provider stores data regardless of agreements; the registered multiaddr points at the same endpoint). Downloads are now verified: getObject checks bytes against the on-chain S3Registry.Objects CID — the download toast says verified/unverified, and a single-chunk mismatch hard-fails with CidMismatchError surfaced in the failure toast. BucketRef.s3BucketId became optional for this (HTTP routes only need the layer-0 id; without the s3 id the result is flagged unverified). A new console-ui Playwright spec covers both outcomes, simulating a lying provider via route interception against metadata anchored from the test side. Also folds in the stub-era doc repointing for docs/README.md, docs/filesystems/README.md, and console-ui's README that a previous commit message claimed but did not actually stage. Validated locally: e2e suite 10/10 (TS scripts incl. the new 3.12 S3 round-trip and 9.8 fs round-trip), demo + sc-demo + sc-coverage, unit suites 76/76, Playwright drive-ui 19/19, console-ui 16/16 (incl. both verification tests), provider 11/11.
Per bkontur's review: READ_OPTS ({at: "best"}) and in-block submission
are the right trade for tests and examples, not for real UIs — those
keep the reorg-safe finalized view.
layer0 gains FINALIZED_READ_OPTS and READ_OPTS is documented as the
test/example default. The layer-1 clients now take readOpts + submitMode
options DEFAULTING to finalized reads and finalized submission
(UI-grade); the e2e workflows construct them with readOpts: READ_OPTS,
submitMode: "best" to keep test speed. resolveProviderEndpoint reads
finalized by default with the read view as a parameter — the resolver's
waitForProvider still resolves at best, because the acceptance it just
observed fired at the best head and a finalized read could lag the very
event that unblocked it.
The UI read sites that PR B had moved to best-block revert to the
finalized default (console-ui's ~15 reads, drive-ui's adapter reads),
and drive-ui's submission returns to finalized via the client default.
console-ui keeps its long-standing best-block submission (pre-dates this
series; Bulletin Chain pattern) with finalized reads, as it always had.
cleanProviderRegistry skipped dev providers with committed_bytes > 0 (can't wipe active agreements) — but a skipped provider that is still accepting_primary keeps getting picked by the runtime's auto-match, and with no node behind it every drive/bucket creation stalls until timeout. Seen live: the e2e suite leaves Charlie with a 2 MiB agreement and accepting_primary on, and drive-ui's whole Playwright suite then dies in createDriveViaApi. Now the cleanup flips accepting_primary off for anything it cannot remove, which is the same determinism trick ensureSoleAcceptingProvider uses. Validated against the exact chain state that reproduced the stall: drive-ui 19/19 after the fix.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…oss-platform reproducible The e2e job's drift check failed honestly: the committed parachain.scale (regenerated on a macOS toolchain) differs byte-wise — same 225,860 size — from the metadata the ci-unified Linux container produces when building the IDENTICAL runtime source. Verified by downloading the failing run's build artifact, booting a probe chain from that exact wasm, and extracting its metadata: the hash matches the "live chain" side of the check (9018bbf0…) precisely. The check compares bytes, so the committed snapshot must originate from the build environment CI uses. This commit adopts the CI-built bytes as canonical. All typechecks (packages, examples, three UIs) and unit suites pass against descriptors generated from it. DX note: refreshing this file from a macOS-built chain will fail the check again — regenerate from the CI build artifact (or a container build) instead. If that proves too annoying, the follow-up is a semantic (decoded) comparison instead of a byte hash.
…tion or class' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #135 / #136 (PR A of two: examples + E2E migrate now, UIs + test-helpers follow in PR B).
One workspace, one descriptor source
pnpm-workspace.yaml, the lockfile, and a private rootpackage.jsonmove to the repo root; members now coveruser-interfaces/*,shared/*, the shared descriptors package,packages/*, andexamples/papi. The examples npm island is deleted — its own lockfile (which had drifted: manifest saidpolkadot-api ^1.23.3, lock resolved 2.1.5), its own generated.papi/copy, and itsnpm installpath inpapi-setup. The whole repo now resolves exactly onepolkadot-apiinstance and one@polkadot-api/descriptorspackage.packages/sdk— seed of the dedicated TS SDK (#123)Flat typed functions ported from
examples/papi/{common,api,sc-api}.js: chain connect + readiness guards, signers (seedToKeypair,makeSigner,Alice..Ferdie), a unifiedsubmitTx, provider-HTTP helpers, and per-pallet extrinsic wrappers.pallet_revivehelpers live on the./revivesubpath so viem stays out of non-contract dependency graphs. Deliberately flat rather than a class facade: 1:1 with existing call sites, tree-shakeable from Vite bundles, and re-homeable into #123'score/layer0/layer1split without rewrites —Web3Storage.connect()later becomes a thin class delegating here. Test-only powers (sudo cleanup, expect-failure submission, Playwright fixtures) stay intest-helpers/examples by design; seepackages/sdk/README.md.The suite's finalization semantics (d3de2d9, 391f8bf, 2bd19cf, 18e1416) are encoded as API defaults, not conventions to remember:
submitTxresolves at in-block inclusion, reads target the best block viaREAD_OPTS, only the challenge creators finalize (challenge ids embed their creation block), and events come from the tx result — never from finalized-only event watches. The wait helpers arewatchValue-based (replays the current value on subscribe → no missed-event window, no poll interval to tune): the 1s-pollwaitForAgreementAcceptanceis gone, andwaitForPrimaryProvideris ready to replace the three identical ~150s polling copies in the UIs in PR B.Stale-metadata drift: found, fixed, and now gated
The tracked metadata snapshot in
shared/papihad silently drifted from the runtime — it lackedpallet_reviveentirely, andclaim_expired_agreementhad dropped itsproviderarg on-chain while the old wrapper kept passing it (silently discarded by the encoder). Regenerated from a live chain (187KB → 225KB), and the e2e CI job now re-fetches metadata from its running chain and fails on diff — thepapi:checkthat the descriptor unification designed but never wired.PAPI v2 conventions
The old examples only worked by pinning
substrate-bindings0.16. On v2:Binary/Enumcome frompolkadot-apiitself,Vec<u8>fields take plainUint8Array, fixed-size binary travels as 0x-hex (SizedHex) via a newasHexhelper, eventfilterreturns{payload}records, andgetWsProvidermoved topolkadot-api/ws. CLAUDE.md's guidance table said the opposite and is updated.CI
A
setup-pnpmcomposite action (Node + pnpm 11 + store cache + frozen root install) replaces the per-job setup dances; the smoke/e2e/sc jobs gain it becausepapi-setupis now a rootpnpm install. c8 runs from the repo root so the coverage gate spanspackages/sdkalongside the suite (92–100% per sdk module on the validation run).ui-checks/deploy-uipath filters extend to the hoisted workspace files. The justfile invokes example scripts vianode --import tsx(raw-TS sdk imported from plain node; the e2e runner forwards the loader to its child processes). pnpm 10 → 11 in CI to match the lockfile producer and the newallowBuildsworkspace key.Validation
Against a local omni-node dev chain + provider: e2e suite 10/10 workflows (95/95 tests),
just demo(2/2 ChallengeDefended through the finalized challenge path),sc-demo,sc-coverage(15/15 selectors),sc-team-drive,sc-token-gated; sdk typechecks undertsc --strict. The same suites gate this PR in CI, including the new metadata drift check.