test(provider-node): add HTTP and storage integration tests#147
Merged
Conversation
…den CI Extract chain interaction traits from 4 coordinators (agreement, challenge, checkpoint, replica-sync) so each can be tested with mock chain clients. Move Subxt implementations to dedicated `_subxt.rs` files. - Add unified `ProviderError` type in `error.rs` - Wire `Box<dyn Trait>` in `command.rs` startup - Make `sign()` fallible across API handlers - Add testability helpers to `fs_index.rs` and `s3_index.rs` - CI: combine 3 pallet coverage jobs, glob discovery, validation guards - CI: fix integration test flakiness - Fix `//Bob` nonce issue in S3 CI integration test
Add integration tests for the provider node's HTTP API layer: - API integration: all HTTP endpoints (upload, download, commit, checkpoint, etc.) - Auth integration: authentication middleware and signature verification - Disk integration: on-disk storage backend operations - FS integration: file-system API endpoint tests - S3 integration: S3-compatible API endpoint tests
…emove tempfile from PR 1 Replace manual `Pin<Box<dyn Future>>` return types with `#[async_trait]` across all 4 coordinator chain-client traits and their subxt impls: - AgreementChainClient - ChallengeChainClient - CheckpointChainClient - ReplicaSyncChainClient This preserves object safety (still `Box<dyn Trait>`) while making the code cleaner and easier to implement mock versions in tests. Also removes `tempfile` dev-dependency (unused in this PR; belongs in PR 3).
…ider-node-http-tests
…ider-node-http-tests
…struction Replace verbose Value::named_composite/unnamed_variant calls with the scale_value::value! macro across all _subxt.rs files. Also shorten fully-qualified subxt::dynamic::Value:: paths with a use import.
…paritytech/web3-storage into refactor/provider-node-trait-extraction
…ider-node-http-tests
ilchu
approved these changes
Jun 10, 2026
ilchu
left a comment
ilchu
left a comment
Collaborator
There was a problem hiding this comment.
Once again, just ergonomics here.
Each coordinator trait now has an `impl<T: Trait> Trait for Arc<T>` that delegates to the inner value, eliminating the need for SharedMock newtype wrappers in tests.
…paritytech/web3-storage into refactor/provider-node-trait-extraction
…ider-node-http-tests
- Remove unnecessary sleep after tokio::spawn in disk test server - Convert MembershipResolver trait to use #[async_trait] for compact return signature instead of manual Pin<Box<dyn Future>> - Update ChainMembershipResolver and MockResolver impls accordingly
ilchu
approved these changes
Jun 10, 2026
The merge from dev incorrectly removed the "Resolve chain-spec script" step and left a reference to matrix.runtime.chain_spec_script which doesn't exist in this job (no matrix strategy). Restore the env var approach that reads from runtimes-matrix.json.
…ider-node-http-tests
…ider-node-http-tests
* test: add coordinator mock tests and file-system pallet tests Add coordinator integration tests using mock chain clients (enabled by the trait extraction in PR 1) and file-system drive-registry success-path tests: - Agreement coordinator: mock chain client lifecycle tests - Challenge responder: mock chain client response tests - Checkpoint coordinator: mock chain client scheduling tests - Replica sync coordinator: mock chain client sync tests - Drive registry pallet: success-path tests for drive lifecycle * refactor(tests): update mock trait impls to use #[async_trait] Align test mocks with the async_trait conversion in the coordinator trait definitions from the base branch. * test(provider-node): consolidate coordinator tests into single binary Address PR #148 review feedback: - Merge 4 test binaries into tests/coordinators/ module for faster compile - Remove SharedMock wrappers (use Arc<T> directly via blanket impls) - Switch tokio::sync::Mutex to std::sync::Mutex in mocks - Add start_paused = true on tests that use tokio::time::sleep - Use full ALICE_SS58 constant instead of abridged addresses - Add shared wait_for() helper to replace inline timeout loops - Remove tautological variant-matching tests - Add test-util feature to tokio dev-dependency
franciscoaguirre
approved these changes
Jun 10, 2026
bkontur
reviewed
Jun 11, 2026
| use subxt::dynamic::At; | ||
| use subxt::{dynamic::Value, OnlineClient, PolkadotConfig}; | ||
|
|
||
| let api = OnlineClient::<PolkadotConfig>::from_url(&self.chain_rpc) |
Collaborator
There was a problem hiding this comment.
@franciscoaguirre is this the reason why you started #159?
From what I reading about subxt:
- new() / from_url() are the expensive parts (network handshake + metadata fetch). Call them once at startup, not per request.
- Don't reconnect on every operation. A common anti-pattern is calling from_url inside a loop or per request — that opens a fresh WS connection and re-downloads metadata each time.
A quick rule of thumb: treat OnlineClient<PolkadotConfig> like an Arc-backed handle (similar to a reqwest::Client or a DB pool) — construct once, clone everywhere.
@franciscoaguirre will you please check and refactor all this OnlineClient::<PolkadotConfig> also as a part of your #159 (or separate PR)?
Collaborator
|
Looks like some flaky test was introduced here, all the PRs on the top of this fails Basic tests. Potential fix introduced here: #161 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#144
Split from #139 (PR 3 of 4). Stacked on #145.
Add integration tests for the provider node's HTTP API layer:
PR stack
devdevTest plan
SKIP_WASM_BUILD=1 cargo clippy -p storage-provider-node --all-targetspassescargo test -p storage-provider-node— HTTP/storage tests pass