Open
Conversation
Add ably-common as a git submodule at submodules/ably-common, pinned to 6ff9a1a. This provides shared test fixtures and protocol definitions used by the UTS (Universal Test Suite) specs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add portable, language-independent test specifications covering the REST client: authentication (RSA), channels (RSL/RSP), encoding, batch publish, pagination, stats, time, and integration tests. Includes a mock HTTP helper and a skill for writing new test specs. These specs serve as the source of truth for expected SDK behaviour, independent of any specific programming language implementation.
Add test specs covering connection failures (RTN14/RTN15), open failures, error reason handling, fallback hosts (RSC15), heartbeats, update events, whenState helper, and a connection lifecycle integration test.
Ensure mock WebSocket connections are properly closed in connection failure and open-failure test specs to prevent resource leaks in tests.
…kill Separate the mock WebSocket specification into its own file for reuse across test specs, and add a skill document for writing test specs.
Replace TRY/CATCH error characterisation patterns with declarative EXPECT_THROW assertions for clearer, more portable test specifications.
…ence Add UNIQUE_CHANNEL_NAME() calls and randomised channel names throughout the test specs. Also adds new test specs for channel attach (RTL4), detach (RTL6), channel options, state events, and channels collection.
Substantially rework the heartbeat test specs for better coverage of RTN23 (heartbeat monitoring) and extend the mock WebSocket helper with additional transport simulation capabilities. Update the write-test-spec skill with improved patterns.
Correct the test approach for RTN15a immediate reconnection behaviour and update the write-test-spec skill with refined patterns.
Correct small errors in channel_attach and channel_state_events specs.
Add test specs for channel connection state handling, channel error reporting, server-initiated detach, channel properties (RTL15/RTL16), connection ID/key (RTN8/RTN9), and connection ping (RTN13). Also adds a completion-status tracker for spec point coverage.
Add comprehensive test specs covering channel message subscription, filtering, listener management, and unsubscribe behaviour.
Add test specs covering channel message publishing, including message encoding, connection state requirements, and error handling.
… specs Add realtime test spec stubs for stats (RSC6a) and time (RSC16) that reference the existing REST test specs, since behaviour is identical.
Add test specs covering the Realtime.request() method for making arbitrary REST requests through the realtime client.
Add test coverage for message queueing during connection state changes, publish behaviour across different connection states, and message delivery ordering guarantees.
…istory) Add specs covering connection state recovery options and realtime channel history retrieval.
Add specs covering log level configuration, log handler callbacks, and default logging behaviour for REST and Realtime clients.
Complete the authentication test spec coverage with specs for token reauth, auth error handling, and edge cases.
Add comprehensive test specs covering presence enter, leave, update, subscribe, presence map synchronisation, and presence history.
Update test specs to use encode_uri_component() for channel names in URL paths, ensuring correct handling of special characters. Add a README documenting the convention and update the write-test-spec skill.
Refine presence test specs based on implementation experience, add integration test specs for presence operations against a live server, and fix various issues in the presence specs.
Extend the skill documentation to note the importance of keeping UTS portable test specs synchronised with language-specific tests.
Add specs covering the batch presence API for retrieving presence state across multiple channels in a single request.
Add specs covering the revokeTokens API for invalidating issued authentication tokens.
Add specs covering the handling of channel UPDATE protocol messages, including resumed and non-resumed flag behaviour.
Add specs covering delta compression for channel messages using the VCDIFF format, including encoding, decoding, and error recovery.
…connect Add test specs for channel attributes (RTL15), channel whenState helper, realtime client timeout configuration, auto-connect behaviour (RTC1b), and REST channel attributes.
Add specs covering the mutable messages feature including message update and delete operations, action fields, and event handling.
Add specs covering push notification administration including device registration management and push channel subscription management.
Add a note to the RSC8a/b fallback host test specs clarifying their relationship with RSC8c (custom environment fallback).
Expand the RSC13 (HTTP request timeout) test spec with better guidance on how to structure timeout assertions in language-specific tests.
Remove an overly specific error message assertion from the RSC8e (fallback host failure) test spec that was ambiguous across SDKs.
Add a clarifying note to RSA4b (token auth with clientId) about when clientId detection occurs relative to the auth flow.
Strengthen the RSA4b4 (token renewal on 401) test spec to clarify that token renewal should be attempted at most once per failed request.
Update RSA5/RSA6 (token params defaults) assertions from SHOULD to MUST for null/absent default values of capability and clientId.
Update RSC10b test spec to explicitly assert that 401 errors unrelated to token expiry must not trigger the token renewal flow.
Add a note to RSL1b (publish) clarifying that a single message payload may be either a JSON object or array, per the spec.
Add a note to RSL6a (message decoding) documenting that intermediate states during chained encoding/decoding may be binary (Uint8Array).
Update TM3 (message type) assertions to explicitly use camelCase field names matching the JSON wire format (e.g. clientId, connectionId).
Correct the expected PresenceAction values in RSP4a (presence history) test assertions to match the wire protocol mapping.
Add notes to REC1a and REC2c1 (endpoint configuration) documenting the legacy host patterns alongside the newer domain patterns.
…rsing Add test cases covering msgpack binary protocol handling: message data deserialization with binary payloads, error response parsing with msgpack content type, and presence data round-tripping.
…P2h2b Fix several presence test spec issues: correct server echo expectations, handle wildcard clientId constraints, and fix RTL13b (presence SYNC) and RTP2h2b (presence re-entry) test logic.
Update REST integration test specs (auth, mutable messages, revoke tokens) to align assertions with actual Ably sandbox behaviour.
Add explicit fallbackHosts configuration to the RSC7c fallback retry test spec to ensure tests don't depend on default host lists.
paddybyers
force-pushed
the
uts-integration
branch
from
36c42b3 to
955fac7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Universal test suite
This PR proposes is a common test suite for Ably SDKs. This supersedes the approach in https://github.com/ably-labs/unified-test-suite.
This approach is based on the idea that:
This PR is an implementation of this approach. This is a set of test specs for the majority of the pubsub SDK feature spec.
The tests make certain assumptions about there being features in the target SDKs’ API in order for it to be effectively unit-testable. These are:
I'm not asking for this PR to be reviewed line-by-line. I have individually reviewed every test spec as it has been created, and these specs have been exercised by generating a Dart library and a Rust library. Feedback from each of those has been incorporated as bugfixes into this branch.
I propose that the next validation step on this set of test specs is to use them to generate tests for an existing library known to have good conformance with the spec. We will need to modify that target library so that the HTTP client etc are mockable in the way expected by these tests. Having done that, we would run the resulting tests against the library and, for any test failures, decide whether the problem is with the target library, or the test spec, or elsewhere.