From 6e229519542c6dd081a4c9091309211a58733f6c Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 14:58:44 +0100 Subject: [PATCH 01/10] test: provision a sandbox app for the integration specifications The integration tier talks to the real Ably sandbox, so it needs the app every specification's BEFORE ALL TESTS block provisions, and the pieces of that preamble that only mean something against a live server. helpers/sandbox.py provisions the app from the canonical app setup in ably-common, vendored under assets/ because the specifications index into its key ordering by position. Provisioning goes over plain httpx rather than through AblyRest: a client that cannot form a request should fail a test rather than look like a broken fixture. Teardown is best effort, since a sandbox app expires on its own. Alongside it are random_id(), the cipher the presence fixtures are encrypted with, and HS256 JWT signing, which the auth specification reaches for a third-party library to do. sandbox_rest_client and sandbox_realtime_client build clients that carry no test_options and reach the network, and wall_clock_poll_until waits on real time, which is what an integration specification's timeouts measure. Co-Authored-By: Claude Opus 5 (1M context) --- test/uts/README.md | 72 ++++++++- test/uts/assets/test-app-setup.json | 81 ++++++++++ test/uts/helpers/client.py | 95 +++++++++++ test/uts/helpers/sandbox.py | 223 ++++++++++++++++++++++++++ test/uts/rest/integration/__init__.py | 0 test/uts/rest/integration/conftest.py | 59 +++++++ 6 files changed, 529 insertions(+), 1 deletion(-) create mode 100644 test/uts/assets/test-app-setup.json create mode 100644 test/uts/helpers/sandbox.py create mode 100644 test/uts/rest/integration/__init__.py create mode 100644 test/uts/rest/integration/conftest.py diff --git a/test/uts/README.md b/test/uts/README.md index 2ba372e5..724ab9ce 100644 --- a/test/uts/README.md +++ b/test/uts/README.md @@ -17,11 +17,12 @@ upstream, and the choices behind how the specifications are adopted here. ``` helpers/ shared infrastructure the specifications assume, and its own tests +assets/ fixtures the specifications name, vendored from elsewhere rest/ specifications under uts/rest realtime/ specifications under uts/realtime ``` -Every directory needs an `__init__.py`, because `test` is a package. +Every directory holding tests needs an `__init__.py`, because `test` is a package. Unit tests serve every request from a mock and reach no network — neither the REST suite nor the realtime one. Integration tests run against a sandbox app. @@ -80,18 +81,87 @@ that closes its own leaves nothing to clean up if it fails first. | [helpers/client.py](helpers/client.py) | client constructors, and the `AWAIT_STATE` / `AWAIT UNTIL` equivalents | | [helpers/clock.py](helpers/clock.py) | `FakeClock`, `settle()` and `advance_to_connection_state()` — `enable_fake_timers()` and `ADVANCE_TIME(ms)` | | [helpers/presence.py](helpers/presence.py) | the presence-map stubs and wire-message builders the presence specifications share | +| [helpers/sandbox.py](helpers/sandbox.py) | the sandbox app the integration tier provisions, the presence-fixture cipher, `random_id()` and the JWT signing the auth specification asks a library for | | [helpers/deviations.py](helpers/deviations.py) | the `@deviation` and `@spec_error` gates | `SKILL.md` lists every name in each. The helpers have their own tests (`helpers/*_test.py`), which are not derived from a specification and are not counted in the derived-test totals. +## The integration tier + +`rest/integration/` runs against the real Ably sandbox, so it needs network access; +nothing there is served from a mock. Each specification's `BEFORE ALL TESTS` block +provisions an app from the canonical `test-resources/test-app-setup.json` in +[ably/ably-common](https://github.com/ably/ably-common), vendored at +[assets/test-app-setup.json](assets/test-app-setup.json), and deletes it afterwards. +One app serves the whole session, since an app per test would be slow and would invite +the sandbox's rate limiting. Provisioning goes over plain `httpx` rather than through +`AblyRest`: it is infrastructure, and a client that cannot form a request should fail a +test rather than look like a broken fixture. Teardown is best effort — a sandbox app +expires on its own, so a failed delete is logged and nothing more. + +The app arrives as the `sandbox` fixture, which is a specification's `app_config`: + +```python +async def test_rsl1n_publish_returns_serials(sandbox): + channel = sandbox_rest_client(sandbox.key(0).key_str).channels.get('test-serials-' + random_id()) +``` + +`sandbox.key(i)` is the specifications' `app_config.keys[i]`, carrying `key_str`, +`key_name`, `key_secret` and `capability`. The index means what it means in a +specification — `keys[0]` full access, `keys[1]` push admin, `keys[2]` the per-channel +capabilities, `keys[3]` subscribe-only, `keys[4]` revocable tokens — which is why the +asset is the `ably-common` file and not `test/assets/testAppSpec.json`, whose keys sit +at other indices. `sandbox.app_id` and `sandbox.key_str`, the full-access key, are +there too. + +`sandbox_rest_client(key, ...)` and `sandbox_realtime_client(key, ...)` in +[helpers/client.py](helpers/client.py) build the clients. They set the endpoint to the +sandbox, carry no `test_options`, and register the client for the same teardown the +mock-backed constructors use. A specification that authenticates some other way leaves +the key out and passes `token=`, `auth_callback=` or `auth_url=`. The protocol defaults +to JSON; the realtime client keeps `auto_connect` and the fallback hosts at the library +defaults, since connecting is the point. + +Waits are wall-clock here, which is the opposite of the unit tier: `poll_until` spins on +the event loop, so the integration tier uses `wall_clock_poll_until(condition, timeout, +description)`, which sleeps the specifications' interval between attempts and returns +whatever the condition answered with. Its condition may be sync or async. Nothing waits +on a fixed sleep for something that can be polled for. + +The five specifications carrying a `## Protocol Variants` section — `publish`, +`history`, `presence`, `batch_presence` and `mutable_messages` — run once per protocol. +A test asks for that by taking the `use_binary_protocol` fixture and passing it on; a +test that does not take it runs json only. + +```python +async def test_rsl1d_publish_failure(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(2).key_str, use_binary_protocol=use_binary_protocol) +``` + +Tests here are given 120 seconds each, per `uts/docs/integration-testing.md`, rather +than the 30 seconds `pyproject.toml` sets for the suite as a whole. The marker covers +the integration package alone. + ## Running ``` uv run --frozen --extra crypto --extra dev pytest test/uts -q ``` +The offline tiers alone, which need no network: + +``` +uv run --frozen --extra crypto --extra dev pytest test/uts/rest/unit test/uts/realtime/unit test/uts/helpers -q +``` + +The integration tier alone, which provisions a sandbox app and needs network access: + +``` +uv run --frozen --extra crypto --extra dev pytest test/uts/rest/integration -q +``` + `--frozen` is required: without it dependency resolution reaches past the environment's cutoff. `--extra dev` carries pytest. diff --git a/test/uts/assets/test-app-setup.json b/test/uts/assets/test-app-setup.json new file mode 100644 index 00000000..5e32a9d2 --- /dev/null +++ b/test/uts/assets/test-app-setup.json @@ -0,0 +1,81 @@ +{ + "post_apps": { + "limits": { + "presence": { + "maxMembers": 250 + } + }, + "keys": [ + {}, + { + "capability": "{ \"cansubscribe:*\":[\"subscribe\"], \"canpublish:*\":[\"publish\"], \"canpublish:andpresence\":[\"presence\",\"publish\"], \"pushenabled:*\":[\"publish\",\"subscribe\",\"push-subscribe\"], \"pushenabled:admin:*\":[\"publish\",\"subscribe\",\"push-admin\"] }" + }, + { + "capability": "{ \"channel0\":[\"publish\"], \"channel1\":[\"publish\"], \"channel2\":[\"publish\", \"subscribe\"], \"channel3\":[\"subscribe\"], \"channel4\":[\"presence\", \"publish\", \"subscribe\"], \"channel5\":[\"presence\"], \"channel6\":[\"*\"] }" + }, + { + "capability": "{ \"*\":[\"subscribe\"] }" + }, + { + "revocableTokens": true + }, + { + "capability": "{ \"[*]*\":[\"*\"] }" + } + ], + "namespaces": [ + { + "id": "persisted", + "persisted": true + }, + { + "id": "pushenabled", + "pushEnabled": true + }, + { + "id": "mutable", + "mutableMessages": true + } + ], + "channels": [ + { + "name": "persisted:presence_fixtures", + "presence": [ + { + "clientId": "client_bool", + "data": "true" + }, + { + "clientId": "client_int", + "data": "24" + }, + { + "clientId": "client_string", + "data": "This is a string clientData payload" + }, + { + "clientId": "client_json", + "data": "{ \"test\": \"This is a JSONObject clientData payload\"}" + }, + { + "clientId": "client_decoded", + "data": "{\"example\":{\"json\":\"Object\"}}", + "encoding": "json" + }, + { + "clientId": "client_encoded", + "data": "HO4cYSP8LybPYBPZPHQOtuD53yrD3YV3NBoTEYBh4U0N1QXHbtkfsDfTspKeLQFt", + "encoding": "json/utf-8/cipher+aes-128-cbc/base64" + } + ] + } + ] + }, + "cipher": { + "algorithm": "aes", + "mode": "cbc", + "keylength": 128, + "key": "WUP6u0K7MXI5Zeo0VppPwg==", + "iv": "HO4cYSP8LybPYBPZPHQOtg==" + } +} diff --git a/test/uts/helpers/client.py b/test/uts/helpers/client.py index 20f6e0bd..fe70c139 100644 --- a/test/uts/helpers/client.py +++ b/test/uts/helpers/client.py @@ -1,11 +1,13 @@ """Construction and teardown for the clients that derived tests drive.""" import asyncio +import inspect import logging from ably import AblyRealtime, AblyRest from ably.types.testoptions import TestOptions from test.uts.helpers.clock import settle +from test.uts.helpers.sandbox import SANDBOX_ENDPOINT log = logging.getLogger(__name__) @@ -20,6 +22,11 @@ # The wait the specifications quote for a connection state change. STATE_TIMEOUT = 5.0 +# What an integration specification's `poll_until(interval: 500ms, timeout: 10s)` +# is asking for, as wall-clock seconds. +POLL_TIMEOUT = 10.0 +POLL_INTERVAL = 0.5 + __open_clients = [] @@ -63,6 +70,51 @@ def realtime_client(mock_websocket=None, mock_http=None, clock=None, **kwargs): return client +def sandbox_rest_client(key=None, **kwargs): + """A REST client talking to the real sandbox, for the integration tier. + + Stands in for an integration specification's `Rest(options: ClientOptions( + key: api_key, endpoint: "nonprod:sandbox"))`. Where `rest_client` serves + every call from a mock, this one reaches the network: it carries no + `test_options`, because there is no transport to install in front of a + server the tests are there to talk to. + + `key` is the positional credential most specifications pass; one that + authenticates some other way passes `token=`, `auth_callback=` or + `auth_url=` instead and leaves it out. The protocol defaults to JSON, which + is what a specification without a `## Protocol Variants` section means; a + specification with one drives it from the `use_binary_protocol` fixture. + The client is closed when the test ends. + """ + if key is not None: + kwargs['key'] = key + kwargs.setdefault('endpoint', SANDBOX_ENDPOINT) + kwargs.setdefault('use_binary_protocol', False) + client = AblyRest(**kwargs) + __open_clients.append(client) + return client + + +def sandbox_realtime_client(key=None, **kwargs): + """A realtime client talking to the real sandbox, for the integration tier. + + Several REST specifications need presence members or presence history that + only a realtime connection can produce, and read them back over REST. + + Unlike `realtime_client`, this one leaves `auto_connect` and the fallback + hosts at their library defaults: connecting is the point, and a real + fallback host is a real host worth reaching for. The client is closed when + the test ends. + """ + if key is not None: + kwargs['key'] = key + kwargs.setdefault('endpoint', SANDBOX_ENDPOINT) + kwargs.setdefault('use_binary_protocol', False) + client = AblyRealtime(**kwargs) + __open_clients.append(client) + return client + + async def await_connection_state(client, state, timeout=STATE_TIMEOUT): """Waits for `client`'s connection to reach `state`. @@ -167,6 +219,49 @@ async def poll_until(condition, timeout=STATE_TIMEOUT, description='condition'): await asyncio.sleep(0) +async def wall_clock_poll_until(condition, timeout=POLL_TIMEOUT, description='condition', + interval=POLL_INTERVAL): + """Waits on real time until `condition` gives something truthy, and returns it. + + This is the integration tier's `poll_until`. `poll_until` above yields to + the event loop between attempts, which is the right thing against a mock, + where nothing but the loop can move the state being waited for. Against a + real server it would spin a core on a network wait, so this one sleeps the + specifications' interval instead. + + `condition` may be a plain callable or a coroutine function, since what a + specification usually polls for is the result of a request. The value the + condition gave is returned, so a condition that answers with the page it + fetched saves fetching it again: + + async def published_message(): + page = await channel.history() + return page if len(page.items) == 1 else None + + history = await wall_clock_poll_until( + published_message, description='the published message to reach history') + + A `PaginatedResult` is truthy whether or not it holds anything, so a + condition that returns one straight from `history()` is satisfied by the + first empty page. Answer with `None` until the page holds what the test is + waiting for, as above. + + A timeout raises, naming what was being waited for rather than leaving a + caller with the bare deadline. + """ + loop = asyncio.get_running_loop() + deadline = loop.time() + timeout + while True: + result = condition() + if inspect.isawaitable(result): + result = await result + if result: + return result + if loop.time() >= deadline: + raise AssertionError(f'Timed out after {timeout}s waiting for {description}') + await asyncio.sleep(interval) + + async def connected_client(mock_websocket, **kwargs): """A realtime client already CONNECTED through `mock_websocket`. diff --git a/test/uts/helpers/sandbox.py b/test/uts/helpers/sandbox.py new file mode 100644 index 00000000..0fb89fec --- /dev/null +++ b/test/uts/helpers/sandbox.py @@ -0,0 +1,223 @@ +"""The sandbox app the integration specifications run against, and its credentials. + +Every integration specification opens with the same `BEFORE ALL TESTS` block: +POST the canonical app setup body to the sandbox, take the keys out of the +response, and DELETE the app when the tests are done. + +Provisioning goes over plain `httpx` rather than through `AblyRest`. It is +infrastructure, and a client that cannot form a request would otherwise look +like a broken fixture rather than a failing test. + +Also here are the pieces of the specifications' preamble that only make sense +against a real server: `random_id()` for the channel and client names the +specifications build, and the JWT signing the auth specification uses in place +of a third-party library. +""" + +import base64 +import hashlib +import hmac +import json +import logging +import os +import secrets +import time + +import httpx + +log = logging.getLogger(__name__) + +# What a specification means by `endpoint: "nonprod:sandbox"`, and the URL the +# provisioning requests go to. The endpoint is for clients, which resolve it +# themselves; the URL is spelled out because provisioning does not go through +# the client. +SANDBOX_ENDPOINT = 'nonprod:sandbox' +SANDBOX_URL = 'https://sandbox.realtime.ably-nonprod.net' + +# The wait `integration-testing.md` gives a provisioning or teardown request. +PROVISION_TIMEOUT = 30.0 + +# The channel the app setup pre-populates with presence members, which the +# presence specifications read rather than write. +PRESENCE_FIXTURES_CHANNEL = 'persisted:presence_fixtures' + +# test/uts/assets/test-app-setup.json is a copy of +# test-resources/test-app-setup.json in ably/ably-common, carrying its +# `post_apps` and `cipher` objects and dropping the two comment keys beside +# them. The order of `post_apps.keys` is what the specifications index into — +# they name `keys[0]`, `keys[1]`, `keys[2]`, `keys[3]` and `keys[4]` and mean +# the capabilities that file gives at those positions, which are not the ones +# `test/assets/testAppSpec.json` gives. Refresh it from ably-common rather than +# editing it: +# +# gh api "repos/ably/ably-common/contents/test-resources/test-app-setup.json" \ +# --jq '.content' | base64 -d +APP_SETUP_PATH = os.path.join(os.path.dirname(__file__), '..', 'assets', 'test-app-setup.json') + +with open(APP_SETUP_PATH) as __asset: + __app_setup = json.load(__asset) + +APP_SETUP_BODY = __app_setup['post_apps'] +CIPHER_FIXTURE = __app_setup['cipher'] + + +def random_id(length=6): + """A short unique suffix, for the names the specifications build. + + This is the specifications' `random_id()`, as in + `"history-test-RSL2a-" + random_id()`. Channels and client ids have to be + unique across tests and across concurrent runs against the same sandbox + app, so the bytes come from `secrets` rather than from a seeded generator. + """ + return base64.urlsafe_b64encode(secrets.token_bytes(length)).decode('ascii').rstrip('=') + + +def fixture_cipher_params(): + """The cipher the app setup encrypted the `client_encoded` fixture with. + + A presence specification reading that member builds its channel with + `channels.get(name, cipher=fixture_cipher_params())`. The asset carries the + key and the IV base64-encoded, which is not what `CipherParams` wants: it + takes them as raw bytes and derives the key length from the key. + """ + from ably.util.crypto import CipherParams + + return CipherParams( + algorithm=CIPHER_FIXTURE['algorithm'], + mode=CIPHER_FIXTURE['mode'], + secret_key=base64.b64decode(CIPHER_FIXTURE['key']), + iv=base64.b64decode(CIPHER_FIXTURE['iv']), + ) + + +class SandboxKey: + """One of the app's API keys, as a specification's `app_config.keys[i]`.""" + + def __init__(self, app_id, key): + self.key_name = f"{app_id}.{key.get('id', '')}" + self.key_secret = key.get('value', '') + self.key_str = f'{self.key_name}:{self.key_secret}' + self.capability = json.loads(key.get('capability') or '{}') + + def __repr__(self): + return f'SandboxKey({self.key_name!r}, capability={self.capability!r})' + + +class SandboxApp: + """A provisioned sandbox app, as a specification's `app_config`.""" + + def __init__(self, response_body): + self.app_id = response_body.get('appId', '') + self.keys = [SandboxKey(self.app_id, key) for key in response_body.get('keys', [])] + + def key(self, index=0): + """The key at `index`, which is the position `APP_SETUP_BODY` gave it. + + The specifications pick their credentials by index — `keys[0]` for full + access, `keys[1]` for push admin, `keys[2]` for the per-channel + capabilities, `keys[3]` for subscribe-only, `keys[4]` for revocable + tokens — so an index here means the same thing it means in a spec. + """ + return self.keys[index] + + @property + def key_str(self): + """The full-access key, which is what most specifications authenticate with.""" + return self.key(0).key_str + + def __repr__(self): + return f'SandboxApp({self.app_id!r}, {len(self.keys)} keys)' + + +async def provision_app(): + """Creates a sandbox app from the canonical setup body. + + The app carries the keys the specifications index into and the presence + fixtures they read, and lives until `delete_app` removes it or the sandbox + expires it. + """ + async with httpx.AsyncClient(timeout=PROVISION_TIMEOUT) as http: + response = await http.post(f'{SANDBOX_URL}/apps', json=APP_SETUP_BODY) + if response.status_code < 200 or response.status_code >= 300: + raise AssertionError( + f'Provisioning a sandbox app failed: {response.status_code} {response.text}') + app = SandboxApp(response.json()) + log.info(f'provision_app(): created sandbox app {app.app_id}') + return app + + +async def delete_app(app): + """Deletes `app`, letting a failure to do so pass. + + Teardown is best effort. A sandbox app expires on its own, so an app left + behind costs nothing, whereas a teardown that raises would fail a suite + whose tests all passed. + """ + key = app.key(0) + try: + async with httpx.AsyncClient(timeout=PROVISION_TIMEOUT) as http: + response = await http.delete( + f'{SANDBOX_URL}/apps/{app.app_id}', + auth=(key.key_name, key.key_secret), + ) + if response.status_code < 200 or response.status_code >= 300: + log.warning( + f'delete_app(): sandbox app {app.app_id} was not deleted: ' + f'{response.status_code} {response.text}') + else: + log.info(f'delete_app(): deleted sandbox app {app.app_id}') + except Exception as error: + log.warning(f'delete_app(): sandbox app {app.app_id} was not deleted: {error!r}') + + +def extract_key_name(api_key): + """The key name half of `app_id.key_id:secret`.""" + return api_key.split(':', 1)[0] + + +def extract_key_secret(api_key): + """The secret half of `app_id.key_id:secret`.""" + return api_key.split(':', 1)[1] + + +def generate_jwt(key_name, key_secret, ttl=3600000, client_id=None, capability=None, expires_at=None): + """An Ably JWT signed with `key_secret`, as the auth specification's `generate_jwt`. + + The specification reaches for a third-party JWT library; an Ably JWT is + HS256 over the key secret, so signing it here keeps the integration tier + off a dependency the locked environment does not carry. + + `ttl` is milliseconds from now, as the specification passes it. `expires_at` + overrides it with a unix time in seconds, which is how the token-renewal + test asks for a JWT that has already expired. `capability` is a JSON string, + defaulting to the whole app. + + The server reads a JWT's lifetime as `exp - iat` and rejects a negative one + with 40003 before it ever considers whether the token has expired. So an + `expires_at` in the past backdates `iat` by `ttl` rather than leaving it at + now: the lifetime stays positive and the token is expired, which is the + condition a renewal test is after. + """ + seconds = int(ttl // 1000) + issued_at = int(expires_at) - seconds if expires_at is not None else int(time.time()) + header = {'typ': 'JWT', 'alg': 'HS256', 'kid': key_name} + claims = { + 'iat': issued_at, + 'exp': issued_at + seconds, + 'x-ably-capability': capability if capability is not None else '{"*":["*"]}', + } + if client_id is not None: + claims['x-ably-clientId'] = client_id + + signing_input = b'.'.join(__jwt_segment(part) for part in (header, claims)) + signature = hmac.new(key_secret.encode('utf-8'), signing_input, hashlib.sha256).digest() + return b'.'.join((signing_input, __base64url(signature))).decode('ascii') + + +def __jwt_segment(part): + return __base64url(json.dumps(part, separators=(',', ':')).encode('utf-8')) + + +def __base64url(raw): + # A JWT's segments are base64url with the padding stripped (RFC 7515 §2). + return base64.urlsafe_b64encode(raw).rstrip(b'=') diff --git a/test/uts/rest/integration/__init__.py b/test/uts/rest/integration/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/uts/rest/integration/conftest.py b/test/uts/rest/integration/conftest.py new file mode 100644 index 00000000..911bd8dd --- /dev/null +++ b/test/uts/rest/integration/conftest.py @@ -0,0 +1,59 @@ +"""Fixtures the REST integration specifications share. + +The sandbox app is provisioned once and read by every test that asks for it. +Each test creating its own app would make the tier several times slower and +would invite the sandbox's rate limiting. +""" + +import os + +import pytest +import pytest_asyncio + +from test.uts.helpers.sandbox import delete_app, provision_app + +# `integration-testing.md` puts a suite of this size at 120 seconds, against +# operations each bounded at 10 to 30. The repository default in +# `pyproject.toml` is 30 seconds, which suits a test served from a mock and not +# one that provisions an app, publishes over the network and polls for the +# result. The marker applies to this package alone, so the unit tiers keep the +# tighter default. +SUITE_TIMEOUT = 120 + +__package_dir = os.path.dirname(os.path.abspath(__file__)) + + +def pytest_collection_modifyitems(items): + for item in items: + if os.path.abspath(str(item.fspath)).startswith(__package_dir + os.sep): + item.add_marker(pytest.mark.timeout(SUITE_TIMEOUT)) + + +@pytest_asyncio.fixture(scope='session') +async def sandbox(): + """The provisioned sandbox app, as a specification's `app_config`. + + This is the specifications' `BEFORE ALL TESTS` / `AFTER ALL TESTS` pair. + `sandbox.key(0).key_str` is the full-access key they call `api_key`, and + the other indices are the capabilities named in each specification's app + provisioning section. + """ + app = await provision_app() + yield app + await delete_app(app) + + +@pytest.fixture(params=[False, True], ids=['json', 'msgpack']) +def use_binary_protocol(request): + """Runs the test once per protocol, which is the specifications' `PROTOCOL`. + + A specification carrying a `## Protocol Variants` section runs against both + json and msgpack, and passes this straight to its clients: + + client = sandbox_rest_client(api_key, use_binary_protocol=use_binary_protocol) + + Only a test that asks for it is parametrised. The six specifications + without that section are json only, and their clients take the JSON default + `sandbox_rest_client` already applies. + """ + return request.param From 9f65df06bea8665b01e8e55a8d0b567f5b319cc9 Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 15:16:04 +0100 Subject: [PATCH 02/10] test: derive the channel, pagination and stats integration specifications publish, history, pagination and time_stats, against the sandbox rather than a mock: 18 tests, the first two once per protocol. The specifications' timing assumptions need care against a real server. History is not consistent immediately after a publish, so every read polls until the page holds what it expects rather than reading once; a PaginatedResult is always truthy, so a poll returning the page directly would be satisfied by the first empty one. The stats specification guards its assertions on there being stats to read, which a freshly provisioned app has none of, so the tests inject an interval through the sandbox's own endpoint and assert unconditionally; real traffic is aggregated on the server's schedule, with no bounded wait after which it is certainly counted. RSL2b3 keeps the specification's four assertions and adds the converse they omit, that each window excludes the other batch. Without it the test passes with the time range dropped entirely. Co-Authored-By: Claude Opus 5 (1M context) --- test/uts/rest/integration/history_test.py | 174 +++++++++++++++++++ test/uts/rest/integration/pagination_test.py | 159 +++++++++++++++++ test/uts/rest/integration/publish_test.py | 127 ++++++++++++++ test/uts/rest/integration/time_stats_test.py | 118 +++++++++++++ 4 files changed, 578 insertions(+) create mode 100644 test/uts/rest/integration/history_test.py create mode 100644 test/uts/rest/integration/pagination_test.py create mode 100644 test/uts/rest/integration/publish_test.py create mode 100644 test/uts/rest/integration/time_stats_test.py diff --git a/test/uts/rest/integration/history_test.py b/test/uts/rest/integration/history_test.py new file mode 100644 index 00000000..975c9f37 --- /dev/null +++ b/test/uts/rest/integration/history_test.py @@ -0,0 +1,174 @@ +"""Derived from uts/rest/integration/history.md in ably/specification. + +Spec points: RSL2a, RSL2b1, RSL2b2, RSL2b3 +""" + +import asyncio + +from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until +from test.uts.helpers.sandbox import random_id + + +def history_page_of(channel, count): + """A poll condition answering with the history page once it holds `count` messages. + + A `PaginatedResult` is truthy whether or not it holds anything, so answering with the + page straight from `history()` would be satisfied by the first empty one. History is + not immediately consistent after a publish, and the first page usually is empty. + """ + async def condition(): + result = await channel.history() + return result if len(result.items) == count else None + + return condition + + +# UTS: rest/integration/RSL2a/history-returns-messages-0 +async def test_rsl2a_history_returns_messages(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + channel_name = 'history-test-RSL2a-' + random_id() + channel = client.channels.get(channel_name) + + await channel.publish(name='event1', data='data1') + await channel.publish(name='event2', data='data2') + await channel.publish(name='event3', data={'key': 'value'}) + + history = await wall_clock_poll_until( + history_page_of(channel, 3), description='three messages to reach history') + + assert len(history.items) == 3 + + # Default order is backwards (newest first) + assert history.items[0].name == 'event3' + assert history.items[0].data == {'key': 'value'} + + assert history.items[1].name == 'event2' + assert history.items[1].data == 'data2' + + assert history.items[2].name == 'event1' + assert history.items[2].data == 'data1' + + assert all(message.timestamp is not None for message in history.items) + + +# UTS: rest/integration/RSL2b1/history-direction-forwards-0 +async def test_rsl2b1_history_direction_forwards(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + channel_name = 'history-direction-' + random_id() + channel = client.channels.get(channel_name) + + # Publish messages - ordering is determined by server timestamp + await channel.publish(name='first', data='1') + await channel.publish(name='second', data='2') + await channel.publish(name='third', data='3') + + await wall_clock_poll_until( + history_page_of(channel, 3), description='three messages to reach history') + + # history() is history(direction=None, limit=None, start=None, end=None), so every + # argument goes by keyword. + history = await channel.history(direction='forwards') + + assert len(history.items) == 3 + assert history.items[0].name == 'first' + assert history.items[1].name == 'second' + assert history.items[2].name == 'third' + + +# UTS: rest/integration/RSL2b2/history-limit-parameter-0 +async def test_rsl2b2_history_limit_parameter(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + channel_name = 'history-limit-' + random_id() + channel = client.channels.get(channel_name) + + for i in range(1, 11): + await channel.publish(name=f'event-{i}', data=str(i)) + + await wall_clock_poll_until( + history_page_of(channel, 10), description='ten messages to reach history') + + history = await channel.history(limit=5) + + assert len(history.items) == 5 + + # Should get the 5 most recent (backwards direction by default) + assert history.items[0].name == 'event-10' + assert history.items[4].name == 'event-6' + + +# UTS: rest/integration/RSL2b3/history-time-range-0 +async def test_rsl2b3_history_time_range(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + channel_name = 'history-timerange-' + random_id() + channel = client.channels.get(channel_name) + + await channel.publish(name='early1', data='e1') + await channel.publish(name='early2', data='e2') + + # Small delay to help ensure server assigns distinct timestamps between batches + await asyncio.sleep(0.002) + + await channel.publish(name='late1', data='l1') + await channel.publish(name='late2', data='l2') + + async def four_messages(): + result = await channel.history() + return result.items if len(result.items) == 4 else None + + all_messages = await wall_clock_poll_until( + four_messages, description='four messages to reach history') + + # Use server-assigned timestamps to define the time boundary. Client-side now() must + # not be used here - client and server clocks may differ, and publishes may complete + # within the same client-clock millisecond. Message.timestamp is milliseconds since + # the epoch as a plain int, which is the form history's start and end take. + early_timestamps = [m.timestamp for m in all_messages if m.name.startswith('early')] + late_timestamps = [m.timestamp for m in all_messages if m.name.startswith('late')] + + max_early_ts = max(early_timestamps) + min_late_ts = min(late_timestamps) + time_boundary = (max_early_ts + min_late_ts) // 2 + + early_history = await channel.history( + start=max_early_ts - 1000, + end=time_boundary, + ) + + late_history = await channel.history( + start=time_boundary + 1, + end=min_late_ts + 1000, + ) + + assert len(early_history.items) >= 1 + assert len(late_history.items) >= 1 + + assert any(message.name.startswith('early') for message in early_history.items) + assert any(message.name.startswith('late') for message in late_history.items) + + # UTS SPEC ERROR: the four assertions above hold whether or not `start` and `end` are + # honoured. A server or client that dropped them entirely would answer both queries + # with all four messages, which is non-empty and does contain an "early" and a "late" + # name, so the test passes. What discriminates is that each window excludes the other + # batch, asserted below. The premise the specification's 2ms wait exists to establish + # is asserted first: with the two batches in the same millisecond the boundary + # arithmetic has no side to put them on, and the test should say so rather than fail + # on an exclusion that cannot hold. + assert min_late_ts > max_early_ts + assert not any(message.name.startswith('late') for message in early_history.items) + assert not any(message.name.startswith('early') for message in late_history.items) + + +# UTS: rest/integration/RSL2/history-empty-channel-0 +async def test_rsl2_history_empty_channel(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + # Use a fresh channel with no messages + channel_name = 'history-empty-' + random_id() + channel = client.channels.get(channel_name) + + history = await channel.history() + + assert isinstance(history.items, list) + assert len(history.items) == 0 + # has_next and is_last are methods here, not properties. + assert history.has_next() is False + assert history.is_last() is True diff --git a/test/uts/rest/integration/pagination_test.py b/test/uts/rest/integration/pagination_test.py new file mode 100644 index 00000000..d7cacd6d --- /dev/null +++ b/test/uts/rest/integration/pagination_test.py @@ -0,0 +1,159 @@ +"""Derived from uts/rest/integration/pagination.md in ably/specification. + +Spec points: TG1, TG2, TG3, TG4, TG5 +""" + +from ably.http.paginatedresult import PaginatedResult +from ably.types.message import Message +from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until +from test.uts.helpers.sandbox import random_id + +# The specifications publish their fixture messages one at a time. A round trip +# each turns the setup for the 25-message case into the slowest part of the +# tier, and the messages are the same set either way, so they go up in one +# request here. +PUBLISH_BATCH = 25 + +# `poll_until(interval: 500ms, timeout: 15s)`, as the specifications write it. +# History is not immediately consistent, so every test waits for the messages +# it published to be visible before it paginates over them. +HISTORY_TIMEOUT = 20.0 + + +async def publish_events(channel, count): + """The specifications' `FOR i IN 1..count: publish("event-" + i, str(i))`.""" + for start in range(1, count + 1, PUBLISH_BATCH): + batch = range(start, min(start + PUBLISH_BATCH, count + 1)) + await channel.publish([Message(f'event-{i}', str(i)) for i in batch]) + + +async def history_of_size(channel, count, timeout=HISTORY_TIMEOUT): + """Waits until `count` messages are visible in history, as the specifications' `poll_until`. + + A `PaginatedResult` is truthy whether or not it holds anything, so the + condition answers `None` until the page is the size the test needs. + """ + async def full_history(): + page = await channel.history() + return page if len(page.items) == count else None + + return await wall_clock_poll_until( + full_history, timeout=timeout, + description=f'the {count} published messages to reach history') + + +# UTS: rest/integration/TG1/items-and-navigation-0 +async def test_tg1_items_and_navigation(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + channel = client.channels.get(f'pagination-basic-{random_id()}') + + await publish_events(channel, 15) + await history_of_size(channel, 15) + + # Request with small limit to force pagination + page1 = await channel.history(limit=5) + + # TG1 - items contains array of results + assert isinstance(page1, PaginatedResult) + assert isinstance(page1.items, list) + assert len(page1.items) == 5 + + # TG2 - hasNext/isLast indicate more pages. Both are methods in this SDK, so + # the specification's `page1.hasNext() == true` is `page1.has_next()`; a bare + # `page1.has_next` would be a bound method and pass whatever the answer. + assert page1.has_next() is True + assert page1.is_last() is False + + +# UTS: rest/integration/TG3/next-retrieves-page-0 +async def test_tg3_next_retrieves_page(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + channel = client.channels.get(f'pagination-next-{random_id()}') + + await publish_events(channel, 12) + await history_of_size(channel, 12) + + page1 = await channel.history(limit=5) + page2 = await page1.next() + page3 = await page2.next() + + assert len(page1.items) == 5 + assert len(page2.items) == 5 + assert len(page3.items) == 2 # Remaining messages + + # Verify no duplicate messages across pages + all_ids = [] + for page in (page1, page2, page3): + for item in page.items: + assert item.id not in all_ids + all_ids.append(item.id) + + assert len(all_ids) == 12 + + +# UTS: rest/integration/TG4/first-retrieves-page-0 +async def test_tg4_first_retrieves_page(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + channel = client.channels.get(f'pagination-first-{random_id()}') + + await publish_events(channel, 10) + await history_of_size(channel, 10) + + page1 = await channel.history(limit=3) + page2 = await page1.next() + first_page = await page2.first() + + # first_page should have same items as page1 + assert first_page is not None + assert len(first_page.items) == len(page1.items) + + for expected, actual in zip(page1.items, first_page.items): + assert actual.id == expected.id + + +# UTS: rest/integration/TG5/iterate-all-pages-0 +async def test_tg5_iterate_all_pages(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + channel = client.channels.get(f'pagination-iterate-{random_id()}') + + message_count = 25 + await publish_events(channel, message_count) + await history_of_size(channel, message_count, timeout=30.0) + + all_messages = [] + page = await channel.history(limit=7) + + while True: + all_messages.extend(page.items) + + if not page.has_next(): + break + + page = await page.next() + + assert len(all_messages) == message_count + + # Verify all messages retrieved + event_names = [message.name for message in all_messages] + for i in range(1, message_count + 1): + assert f'event-{i}' in event_names + + +# UTS: rest/integration/TG3/next-last-page-null-1 +async def test_tg3_next_last_page_null(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + channel = client.channels.get(f'pagination-lastnext-{random_id()}') + + await publish_events(channel, 3) + await history_of_size(channel, 3, timeout=15.0) + + page = await channel.history(limit=10) # Larger than message count + + assert len(page.items) == 3 + assert page.has_next() is False + assert page.is_last() is True + + # `next()` answers None when the response carried no `next` link rel, which + # is the specification's "returns null on the last page". + next_page = await page.next() + assert next_page is None diff --git a/test/uts/rest/integration/publish_test.py b/test/uts/rest/integration/publish_test.py new file mode 100644 index 00000000..57f439f5 --- /dev/null +++ b/test/uts/rest/integration/publish_test.py @@ -0,0 +1,127 @@ +"""Derived from uts/rest/integration/publish.md in ably/specification. + +Spec points: RSL1d, RSL1k5, RSL1l1, RSL1m4, RSL1n +""" + +import pytest + +from ably.types.message import Message +from ably.util.exceptions import AblyException +from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until +from test.uts.helpers.sandbox import random_id + + +# UTS: rest/integration/RSL1d/publish-failure-error-0 +async def test_rsl1d_publish_failure_error(sandbox, use_binary_protocol): + restricted_key = sandbox.key(2) + + channel_name = 'forbidden-channel-' + random_id() + # keys[2] names channel0 to channel6 one by one, so a generated name is covered by + # none of them. Read off the key rather than assumed, so that a change to the app + # setup shows up here rather than turning the test into one that proves nothing. + assert channel_name not in restricted_key.capability + + restricted_client = sandbox_rest_client( + restricted_key.key_str, use_binary_protocol=use_binary_protocol) + restricted_channel = restricted_client.channels.get(channel_name) + + with pytest.raises(AblyException) as excinfo: + await restricted_channel.publish(name='event', data='data') + + assert excinfo.value.code == 40160 + assert excinfo.value.status_code == 401 + + +# UTS: rest/integration/RSL1n/publish-result-serials-0 +async def test_rsl1n_publish_result_serials(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + channel_name = 'test-serials-' + random_id() + channel = client.channels.get(channel_name) + + result1 = await channel.publish(name='event1', data='data1') + + assert isinstance(result1.serials, list) + assert len(result1.serials) == 1 + assert isinstance(result1.serials[0], str) + assert len(result1.serials[0]) > 0 + + result2 = await channel.publish(messages=[ + Message(name='event2', data='data2'), + Message(name='event3', data='data3'), + Message(name='event4', data='data4'), + ]) + + assert len(result2.serials) == 3 + assert all(isinstance(serial, str) and len(serial) > 0 for serial in result2.serials) + assert len(set(result2.serials)) == 3 + + +# UTS: rest/integration/RSL1k5/idempotent-client-ids-0 +async def test_rsl1k5_idempotent_client_ids(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + channel_name = 'idempotent-explicit-' + random_id() + channel = client.channels.get(channel_name) + + fixed_id = 'client-supplied-id-' + random_id() + + # publish(message=...) raises TypeError, so the Message goes positionally. The id is + # the client's own, so the library leaves it alone: it only generates ids when every + # message in the batch lacks one (RSL1k1). + for i in (1, 2, 3): + await channel.publish(Message(id=fixed_id, name='event', data=f'data-{i}')) + + async def any_message_in_history(): + result = await channel.history() + return result if len(result.items) > 0 else None + + history = await wall_clock_poll_until( + any_message_in_history, description='the published message to reach history') + + assert len(history.items) == 1 + assert history.items[0].id == fixed_id + # The data should be from the first publish (subsequent ones are no-ops) + assert history.items[0].data == 'data-1' + + +# UTS: rest/integration/RSL1l1/publish-params-force-nack-0 +async def test_rsl1l1_publish_params_force_nack(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + channel_name = 'force-nack-test-' + random_id() + channel = client.channels.get(channel_name) + + # publish(name=, data=, params=) raises; params reaches the query string only through + # the Message form, where it is the second positional argument. + with pytest.raises(AblyException) as excinfo: + await channel.publish(Message(name='event', data='data'), {'_forceNack': 'true'}) + + assert excinfo.value.code == 40099 + + +# UTS: rest/integration/RSL1m4/clientid-mismatch-rejected-0 +async def test_rsl1m4_clientid_mismatch_rejected(sandbox, use_binary_protocol): + key_client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + # request_token takes its token params as a positional dict; every keyword on it is an + # auth option. + token_details = await key_client.auth.request_token({'client_id': 'authenticated-client-id'}) + + # The spec passes the bare token string, not the TokenDetails, so the client does not + # know the clientId the token carries and cannot rule the publish out locally. The + # message goes to the server, which is what this test is here to exercise. Passing + # `token_details=` instead would have the library reject it in + # `Channel.__publish_request_body` with the same 400/40012 before any request left. + token_client = sandbox_rest_client( + token=token_details.token, use_binary_protocol=use_binary_protocol) + + channel_name = 'clientid-mismatch-' + random_id() + channel = token_client.channels.get(channel_name) + + with pytest.raises(AblyException) as excinfo: + await channel.publish(Message( + name='event', + data='data', + client_id='different-client-id', # Doesn't match authenticated clientId + )) + + assert excinfo.value.code == 40012 + assert excinfo.value.status_code == 400 diff --git a/test/uts/rest/integration/time_stats_test.py b/test/uts/rest/integration/time_stats_test.py new file mode 100644 index 00000000..39d6e465 --- /dev/null +++ b/test/uts/rest/integration/time_stats_test.py @@ -0,0 +1,118 @@ +"""Derived from uts/rest/integration/time_stats.md in ably/specification. + +Spec points: RSC16, RSC6 +""" + +import time as wall_clock +from datetime import datetime, timedelta, timezone + +import pytest_asyncio + +from ably import AblyRest +from ably.http.paginatedresult import PaginatedResult +from ably.types.stats import Stats +from test.uts.helpers.client import sandbox_rest_client +from test.uts.helpers.sandbox import SANDBOX_ENDPOINT + +# The tolerance the specification allows between the server's clock and this +# one, as milliseconds. +CLOCK_TOLERANCE_MS = 5000 + +# The units a stats interval may be aggregated over, which is the +# specification's `["minute", "hour", "day", "month"]`. +STATS_UNITS = ('minute', 'hour', 'day', 'month') + +# How far back the injected interval sits. A freshly provisioned app has no +# stats at all, so the interval has to be old enough to be complete and recent +# enough that a query bounded at `now` still reaches it. +STATS_INTERVAL_AGE = timedelta(minutes=3) + + +@pytest_asyncio.fixture(scope='module') +async def app_with_stats(sandbox): + """The sandbox app, with one minute of traffic recorded against it. + + The specification allows for `stats()` returning nothing — "stats may be + empty for a new sandbox app" — and guards its assertions on the interval's + shape behind `IF result.items.length > 0`. Against an app that has never + seen traffic that branch never runs, which would leave both stats tests + asserting only that the call returned. So the app is given a minute's worth + of traffic first, through the sandbox's own `POST /stats` injection + endpoint, and the assertions the specification guards are made + unconditionally below. + + Injecting rather than publishing is what the repository's own sandbox stats + suite does: real traffic is aggregated on the server's schedule, so there + is no bounded wait after which a published message is certain to be + counted, whereas an injected interval is queryable at once. + """ + interval = (datetime.now(timezone.utc).replace(tzinfo=None) - STATS_INTERVAL_AGE).replace( + second=0, microsecond=0) + client = AblyRest(key=sandbox.key(0).key_str, endpoint=SANDBOX_ENDPOINT, + use_binary_protocol=False) + try: + await client.http.post('/stats', body=[{ + 'intervalId': Stats.to_interval_id(interval, 'minute'), + 'inbound': {'realtime': {'messages': {'count': 50, 'data': 5000}}}, + 'outbound': {'realtime': {'messages': {'count': 20, 'data': 2000}}}, + }]) + finally: + await client.close() + return sandbox + + +# UTS: rest/integration/RSC16/time-returns-server-time-0 +async def test_rsc16_time_returns_server_time(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + + before_request = wall_clock.time() * 1000 + server_time = await client.time() + after_request = wall_clock.time() * 1000 + + # NOTE: the specification asserts the result IS DateTime. `time()` in this + # SDK returns milliseconds since the epoch, so the type assertion becomes a + # numeric one and the comparisons below are made in the same units. This is + # an idiomatic spelling of the same requirement, not a deviation. + assert isinstance(server_time, (int, float)) + assert not isinstance(server_time, bool) + + # Server time should be reasonably close to client time + # (allowing for network latency and minor clock differences) + assert server_time >= before_request - CLOCK_TOLERANCE_MS + assert server_time <= after_request + CLOCK_TOLERANCE_MS + + +# UTS: rest/integration/RSC6/stats-returns-result-0 +async def test_rsc6_stats_returns_result(app_with_stats): + client = sandbox_rest_client(app_with_stats.key(0).key_str) + + result = await client.stats() + + # Result should be a PaginatedResult + assert isinstance(result, PaginatedResult) + assert isinstance(result.items, list) + + # The specification guards these on `items.length > 0`; `app_with_stats` + # makes the app's stats non-empty so that they are reached. + assert len(result.items) > 0 + assert isinstance(result.items[0].interval_id, str) + assert result.items[0].unit in STATS_UNITS + + +# UTS: rest/integration/RSC6/stats-with-parameters-1 +async def test_rsc6_stats_with_parameters(app_with_stats): + client = sandbox_rest_client(app_with_stats.key(0).key_str) + + # Request stats with specific parameters + result = await client.stats(limit=5, direction='forwards', unit='hour') + + # Should succeed with parameters applied + assert isinstance(result, PaginatedResult) + assert len(result.items) <= 5 + + # The specification asserts only the limit, which an empty page satisfies + # whether or not the query reached the server. `app_with_stats` puts one + # interval in range, so `unit` shows that the parameter was applied rather + # than dropped. + assert len(result.items) > 0 + assert all(stat.unit == 'hour' for stat in result.items) From 4a115a055a6ce1ad63a8efc93a86c143bcfd476a Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 15:16:04 +0100 Subject: [PATCH 03/10] test: derive the presence integration specifications presence and batch_presence, 20 tests, both once per protocol. The presence specifications read the members the app setup pre-populates on persisted:presence_fixtures, one of which is encrypted; a channel carries the cipher from construction, so the test that decodes it holds its own client. Four tests reach for a clientId filter on RestPresence#get that ably-python does not have. Only the one that is about the filter is gated; the three decoding tests use it to pick a fixture, so they select in Python and adapt the count they assert. batch_presence needs Rest#batchPresence, which does not exist at all, so its three tests are gated against the spelling the unit tier already uses. Their setup runs, and the presence members reach the server, so what is gated is the read rather than the whole test. Co-Authored-By: Claude Opus 5 (1M context) --- .../rest/integration/batch_presence_test.py | 215 ++++++++++ test/uts/rest/integration/presence_test.py | 374 ++++++++++++++++++ 2 files changed, 589 insertions(+) create mode 100644 test/uts/rest/integration/batch_presence_test.py create mode 100644 test/uts/rest/integration/presence_test.py diff --git a/test/uts/rest/integration/batch_presence_test.py b/test/uts/rest/integration/batch_presence_test.py new file mode 100644 index 00000000..f41c58b7 --- /dev/null +++ b/test/uts/rest/integration/batch_presence_test.py @@ -0,0 +1,215 @@ +"""Derived from uts/rest/integration/batch_presence.md in ably/specification. + +Spec points: RSC24, BGR2, BGF2 + +DEVIATION: ably-python has no batch API. `AblyRest` exposes no `batch_presence`, and +the package defines neither `BatchResult` nor `BatchPresenceSuccessResult` / +`BatchPresenceFailureResult`; the word "batch" appears nowhere under `ably/`. Every +test here therefore departs from the specification and is gated behind +RUN_DEVIATIONS, against the same spelling +[test/uts/rest/unit/batch_presence_test.py](../unit/batch_presence_test.py) gates on — +`client.batch_presence([...])` giving a result with `success_count`, `failure_count` +and `results` — so that dropping the marker is the only change either tier needs when +RSC24 lands. + +The setup halves are real, and the responses they assert against were confirmed by +hand through `client.request('GET', '/presence', params={'channels': ...})` against +the sandbox: the server does return the `successCount` / `failureCount` / `results` +envelope the specification describes, with `code` 40160 and `statusCode` 401 for a +channel the key has no capability for. Two details of that confirmation are recorded +beside the assertions they bear on — the `presence` key the server omits for an empty +channel, and the presence members a closed connection takes with it. + +See [deviations-batch-push-channels-integration.md](../../deviations-batch-push-channels-integration.md). +""" + +from ably.realtime.connection import ConnectionState +from test.uts.helpers.client import ( + await_connection_state, + sandbox_realtime_client, + sandbox_rest_client, + wall_clock_poll_until, +) +from test.uts.helpers.deviations import deviation +from test.uts.helpers.sandbox import random_id + + +def result_for(result, channel_name): + """The specifications' `result.results.find(r => r.channel == channel_name)`.""" + for entry in result.results: + if entry.channel == channel_name: + return entry + raise AssertionError(f'No batch result for channel {channel_name!r}') + + +def is_success_result(entry): + """The specifications' `entry IS BatchPresenceSuccessResult`. + + Neither result class exists to name, so a success is told apart from a failure by + which attribute it carries, as the unit tier does. + """ + return getattr(entry, 'presence', None) is not None + + +def is_failure_result(entry): + """The specifications' `entry IS BatchPresenceFailureResult`.""" + return getattr(entry, 'error', None) is not None + + +def member_for(entry, client_id): + """The specifications' `presence.find(m => m.clientId == client_id)`.""" + for member in entry.presence: + if member.client_id == client_id: + return member + raise AssertionError(f'No presence member with clientId {client_id!r}') + + +async def entering_client(key, use_binary_protocol): + """A CONNECTED realtime client that may enter presence for any clientId. + + The specifications build this with the full-access key alone and call + `enterClient` straight away. ably-python needs `client_id='*'` on top: a basic-auth + connection is told `clientId: "*"` by the server, and `Auth._configure_client_id` + answers a wildcard from the server by marking the client id validated while leaving + it `None` (`ably/rest/auth.py:335`), after which `can_assume_client_id` refuses every + id and `enter_client` raises 40012. The repository's own presence suite carries the + same `client_id='*'` for the same reason + (`test/ably/realtime/realtimepresence_test.py:396`). + + The connection is awaited to CONNECTED before any enter, since a presence enter on a + connection that is still CONNECTING is queued rather than sent. + """ + client = sandbox_realtime_client(key, client_id='*', use_binary_protocol=use_binary_protocol) + await await_connection_state(client, ConnectionState.CONNECTED) + return client + + +async def enter_members(realtime, channel_name, members): + """Attaches `channel_name` and enters each `(client_id, data)` pair on it.""" + channel = realtime.channels.get(channel_name) + await channel.attach() + for client_id, data in members: + await channel.presence.enter_client(client_id, data) + return channel + + +@deviation +# UTS: rest/integration/RSC24/batch-presence-multiple-channels-0 +async def test_rsc24_batch_presence_multiple_channels(sandbox, use_binary_protocol): + channel_a_name = 'batch-presence-a-' + random_id() + channel_b_name = 'batch-presence-b-' + random_id() + + realtime = await entering_client(sandbox.key(0).key_str, use_binary_protocol) + await enter_members(realtime, channel_a_name, [('user-1', 'data-a1'), ('user-2', 'data-a2')]) + await enter_members(realtime, channel_b_name, [('user-3', 'data-b1')]) + + # The realtime client stays open: the members would leave with the connection, and + # closing it is the autouse fixture's job in any case. + rest = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + + result = await rest.batch_presence([channel_a_name, channel_b_name]) + + assert result.success_count == 2 + assert result.failure_count == 0 + assert len(result.results) == 2 + + result_a = result_for(result, channel_a_name) + result_b = result_for(result, channel_b_name) + + assert is_success_result(result_a) + assert len(result_a.presence) == 2 + client_ids_a = [member.client_id for member in result_a.presence] + assert 'user-1' in client_ids_a + assert 'user-2' in client_ids_a + + assert member_for(result_a, 'user-1').data == 'data-a1' + + assert is_success_result(result_b) + assert len(result_b.presence) == 1 + assert result_b.presence[0].client_id == 'user-3' + assert result_b.presence[0].data == 'data-b1' + + +@deviation +# UTS: rest/integration/RSC24/restricted-key-channel-failure-1 +async def test_rsc24_restricted_key_channel_failure(sandbox, use_binary_protocol): + # The specification hard-codes "channel6" as the channel `keys[2]` is allowed. The + # capability comes back with the provisioned app, so the channel is read off it + # rather than assumed; it is the only one of the seven given every operation, and + # presence is the operation this test needs. A wildcard pattern would not do: the + # specification records that the batch presence endpoint does not honour one. + capability = sandbox.key(2).capability + allowed = sorted(name for name, operations in capability.items() if '*' in operations) + assert allowed, f'keys[2] names no fully-permitted channel: {capability!r}' + allowed_channel = allowed[0] + denied_channel = 'denied-batch-' + random_id() + + realtime = await entering_client(sandbox.key(0).key_str, use_binary_protocol) + await enter_members(realtime, allowed_channel, [('member-1', 'hello')]) + await enter_members(realtime, denied_channel, [('member-2', 'world')]) + + # SPEC FAULT: the specification closes the realtime client here, then asserts that + # the allowed channel still holds `member-1`. Confirmed against the sandbox: closing + # the connection takes its presence members with it, and the allowed channel comes + # back with no `presence` at all. The connection is left open, as this file's other + # two tests say to do in so many words. + restricted_rest = sandbox_rest_client( + sandbox.key(2).key_str, use_binary_protocol=use_binary_protocol) + + # `allowed_channel` is a fixed name, so the json and msgpack runs enter `member-1` + # on the same channel one after the other. Polling lets the previous run's member + # finish leaving rather than counting it. + async def one_member_on_the_allowed_channel(): + result = await restricted_rest.batch_presence([allowed_channel, denied_channel]) + success = result_for(result, allowed_channel) + return result if is_success_result(success) and len(success.presence) == 1 else None + + result = await wall_clock_poll_until( + one_member_on_the_allowed_channel, + description=f'one presence member on {allowed_channel}') + + assert result.success_count == 1 + assert result.failure_count == 1 + assert len(result.results) == 2 + + success = result_for(result, allowed_channel) + failure = result_for(result, denied_channel) + + assert is_success_result(success) + assert len(success.presence) == 1 + assert success.presence[0].client_id == 'member-1' + + assert is_failure_result(failure) + assert failure.error.code == 40160 + assert failure.error.status_code == 401 + + +@deviation +# UTS: rest/integration/RSC24/empty-channel-presence-2 +async def test_rsc24_empty_channel_presence(sandbox, use_binary_protocol): + empty_channel = 'batch-empty-' + random_id() + populated_channel = 'batch-populated-' + random_id() + + realtime = await entering_client(sandbox.key(0).key_str, use_binary_protocol) + await enter_members(realtime, populated_channel, [('someone', 'here')]) + + rest = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + + result = await rest.batch_presence([empty_channel, populated_channel]) + + assert result.success_count == 2 + assert result.failure_count == 0 + assert len(result.results) == 2 + + empty_result = result_for(result, empty_channel) + populated_result = result_for(result, populated_channel) + + # The server counts the empty channel a success and leaves `presence` out of its + # result altogether rather than sending `[]`, so an implementation of BGR2 has to + # default the field for the specification's assertion to hold. + assert is_success_result(empty_result) + assert len(empty_result.presence) == 0 + + assert is_success_result(populated_result) + assert len(populated_result.presence) == 1 + assert populated_result.presence[0].client_id == 'someone' diff --git a/test/uts/rest/integration/presence_test.py b/test/uts/rest/integration/presence_test.py new file mode 100644 index 00000000..8250a8a7 --- /dev/null +++ b/test/uts/rest/integration/presence_test.py @@ -0,0 +1,374 @@ +"""Derived from uts/rest/integration/presence.md in ably/specification. + +Spec points: RSP1, RSP3, RSP3a, RSP4, RSP4b, RSP5 +""" + +import pytest + +from ably.http.paginatedresult import PaginatedResult +from ably.realtime.connection import ConnectionState +from ably.types.presence import Presence, PresenceAction, PresenceMessage +from ably.util.exceptions import AblyException +from test.uts.helpers.client import ( + await_connection_state, + sandbox_realtime_client, + sandbox_rest_client, + wall_clock_poll_until, +) +from test.uts.helpers.deviations import deviation +from test.uts.helpers.sandbox import PRESENCE_FIXTURES_CHANNEL, fixture_cipher_params, random_id + +# The clientIds the app setup pre-populates on `persisted:presence_fixtures`. +FIXTURE_CLIENT_IDS = ( + 'client_bool', 'client_int', 'client_string', 'client_json', 'client_decoded', 'client_encoded') + + +def member_for(page, client_id): + """The one fixture member with `client_id`, or None. + + The specifications reach for this with `presence.get(clientId: ...)`, which + `Presence.get` does not offer; see + [deviations-presence-integration.md](../../deviations-presence-integration.md). + """ + return next((item for item in page.items if item.client_id == client_id), None) + + +async def connected_realtime(key, client_id, use_binary_protocol): + """A realtime client already CONNECTED, ready to generate presence events. + + `presence.enter` resolves on the server's ACK, but a call made before the + connection is up is queued rather than sent, so the wait has to happen + first. + """ + client = sandbox_realtime_client( + key, client_id=client_id, use_binary_protocol=use_binary_protocol) + await await_connection_state(client, ConnectionState.CONNECTED) + return client + + +def presence_history_of_at_least(channel, count): + """A poll condition giving the presence history page once it holds `count` events.""" + async def condition(): + page = await channel.presence.history() + return page if len(page.items) >= count else None + + return condition + + +# UTS: rest/integration/RSP1/access-presence-from-channel-0 +async def test_rsp1_access_presence_from_channel(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL) + presence = channel.presence + + assert presence is not None + # NOTE: the spec asserts `presence IS RestPresence`. The class behind + # `channel.presence` in ably-python is `ably.types.presence.Presence`. + assert isinstance(presence, Presence) + + +# UTS: rest/integration/RSP3/get-presence-members-0 +async def test_rsp3_get_presence_members(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL) + result = await channel.presence.get() + + assert isinstance(result, PaginatedResult) + assert len(result.items) >= 5 + + client_ids = [message.client_id for message in result.items] + assert 'client_bool' in client_ids + assert 'client_string' in client_ids + assert 'client_json' in client_ids + + +# UTS: rest/integration/RSP3/presence-message-fields-1 +async def test_rsp3_presence_message_fields(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL) + result = await channel.presence.get() + + member = member_for(result, 'client_string') + + assert member is not None + assert isinstance(member, PresenceMessage) + assert member.action == PresenceAction.PRESENT + assert member.client_id == 'client_string' + assert member.data == 'This is a string clientData payload' + assert member.connection_id is not None + + +# UTS: rest/integration/RSP3a1/get-with-limit-0 +async def test_rsp3a1_get_with_limit(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL) + + result = await channel.presence.get(limit=2) + + assert len(result.items) <= 2 + if result.has_next(): + assert len(result.items) == 2 + + +@deviation +# UTS: rest/integration/RSP3a2/get-with-clientid-filter-0 +async def test_rsp3a2_get_with_clientid_filter(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL) + result = await channel.presence.get(client_id='client_json') + + assert len(result.items) == 1 + assert result.items[0].client_id == 'client_json' + # The fixture has no encoding field, so data is returned as a raw string. + assert isinstance(result.items[0].data, str) + assert result.items[0].data == '{ "test": "This is a JSONObject clientData payload"}' + + +# UTS: rest/integration/RSP3/get-empty-channel-2 +async def test_rsp3_get_empty_channel(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel_name = f'presence-empty-{random_id()}' + channel = client.channels.get(channel_name) + + result = await channel.presence.get() + + assert isinstance(result.items, list) + assert len(result.items) == 0 + assert result.has_next() is False + + +# UTS: rest/integration/RSP4/history-returns-events-0 +async def test_rsp4_history_returns_events(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel_name = f'presence-history-{random_id()}' + + realtime = await connected_realtime(sandbox.key_str, 'test-client', use_binary_protocol) + + realtime_channel = realtime.channels.get(channel_name) + await realtime_channel.presence.enter('entered') + await realtime_channel.presence.update('updated') + await realtime_channel.presence.leave('left') + # NOTE: the spec closes the realtime client here. The suite's autouse + # teardown closes every client it built, after the assertions have run. + + rest_channel = client.channels.get(channel_name) + + history = await wall_clock_poll_until( + presence_history_of_at_least(rest_channel, 3), + description='the three presence events to reach history') + + assert len(history.items) >= 3 + + actions = [message.action for message in history.items] + assert PresenceAction.ENTER in actions + assert PresenceAction.UPDATE in actions + assert PresenceAction.LEAVE in actions + + +# UTS: rest/integration/RSP4b1/history-time-range-0 +async def test_rsp4b1_history_time_range(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel_name = f'presence-history-time-{random_id()}' + + # NOTE: the spec reads `now_millis()` from the test's own clock. The bounds + # are read from the server instead, so that clock skew between the runner + # and the sandbox cannot put the events outside the window being asserted. + time_before = await client.time() + + realtime = await connected_realtime(sandbox.key_str, 'time-test-client', use_binary_protocol) + + realtime_channel = realtime.channels.get(channel_name) + await realtime_channel.presence.enter('test') + await realtime_channel.presence.leave() + + time_after = await client.time() + + rest_channel = client.channels.get(channel_name) + await wall_clock_poll_until( + presence_history_of_at_least(rest_channel, 2), + description='the enter and leave to reach history') + + history = await rest_channel.presence.history(start=time_before, end=time_after) + + assert len(history.items) >= 2 + + +# UTS: rest/integration/RSP4b2/history-direction-forwards-0 +async def test_rsp4b2_history_direction_forwards(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel_name = f'presence-direction-{random_id()}' + + realtime = await connected_realtime(sandbox.key_str, 'direction-client', use_binary_protocol) + + realtime_channel = realtime.channels.get(channel_name) + await realtime_channel.presence.enter('first') + await realtime_channel.presence.update('second') + await realtime_channel.presence.update('third') + + rest_channel = client.channels.get(channel_name) + await wall_clock_poll_until( + presence_history_of_at_least(rest_channel, 3), + description='the three ordered presence events to reach history') + + history_forwards = await rest_channel.presence.history(direction='forwards') + + assert len(history_forwards.items) >= 3 + assert history_forwards.items[0].data == 'first' + + history_backwards = await rest_channel.presence.history(direction='backwards') + + assert history_backwards.items[0].data == 'third' + + +# UTS: rest/integration/RSP4b3/history-limit-pagination-0 +async def test_rsp4b3_history_limit_pagination(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel_name = f'presence-limit-{random_id()}' + + realtime = await connected_realtime(sandbox.key_str, 'limit-client', use_binary_protocol) + + realtime_channel = realtime.channels.get(channel_name) + for index in range(1, 6): + await realtime_channel.presence.update(f'update-{index}') + + rest_channel = client.channels.get(channel_name) + await wall_clock_poll_until( + presence_history_of_at_least(rest_channel, 5), + description='the five presence updates to reach history') + + page1 = await rest_channel.presence.history(limit=2) + + assert len(page1.items) == 2 + assert page1.has_next() is True + + page2 = await page1.next() + + assert page2 is not None + assert len(page2.items) >= 1 + + +# UTS: rest/integration/RSP5/decode-string-data-0 +async def test_rsp5_decode_string_data(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL) + # NOTE: the spec calls `presence.get(clientId: "client_string")` and asserts + # `items.length == 1`. `Presence.get` takes no clientId filter, so the whole + # member set is fetched and filtered here instead. + result = await channel.presence.get() + member = member_for(result, 'client_string') + + assert member is not None + assert isinstance(member.data, str) + assert member.data == 'This is a string clientData payload' + + +# UTS: rest/integration/RSP5/decode-json-data-1 +async def test_rsp5_decode_json_data(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL) + # NOTE: the spec calls `presence.get(clientId: "client_decoded")` and asserts + # `items.length == 1`; filtered in Python for want of the clientId param. + result = await channel.presence.get() + member = member_for(result, 'client_decoded') + + assert member is not None + assert isinstance(member.data, dict) + assert member.data['example']['json'] == 'Object' + + +# UTS: rest/integration/RSP5/decode-encrypted-data-2 +async def test_rsp5_decode_encrypted_data(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + # `Presence` snapshots the channel's cipher when the channel is built, so + # the cipher goes in on the first `channels.get` for this client. + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL, cipher=fixture_cipher_params()) + + # NOTE: the spec calls `presence.get(clientId: "client_encoded")` and asserts + # `items.length == 1`; filtered in Python for want of the clientId param. + result = await channel.presence.get() + member = member_for(result, 'client_encoded') + + assert member is not None + assert member.data is not None + # The fixture encrypts the same payload `client_decoded` carries in the clear. + assert member.data == {'example': {'json': 'Object'}} + + +# UTS: rest/integration/RSP5/decode-history-messages-3 +async def test_rsp5_decode_history_messages(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel_name = f'presence-decode-history-{random_id()}' + + realtime = await connected_realtime(sandbox.key_str, 'decode-client', use_binary_protocol) + + json_data = {'key': 'value', 'number': 123} + realtime_channel = realtime.channels.get(channel_name) + await realtime_channel.presence.enter(json_data) + + rest_channel = client.channels.get(channel_name) + history = await wall_clock_poll_until( + presence_history_of_at_least(rest_channel, 1), + description='the entered presence member to reach history') + + assert isinstance(history.items[0].data, dict) + assert history.items[0].data['key'] == 'value' + assert history.items[0].data['number'] == 123 + + +# UTS: rest/integration/RSP3/full-pagination-3 +async def test_rsp3_full_pagination(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + + channel = client.channels.get(PRESENCE_FIXTURES_CHANNEL) + + page1 = await channel.presence.get(limit=2) + + all_members = list(page1.items) + + current_page = page1 + while current_page.has_next(): + current_page = await current_page.next() + all_members.extend(current_page.items) + + assert len(all_members) >= 5 + + client_ids = [member.client_id for member in all_members] + assert len(set(client_ids)) == len(client_ids) + assert set(client_ids) == set(FIXTURE_CLIENT_IDS) + + +# UTS: rest/integration/RSP3/invalid-credentials-rejected-4 +async def test_rsp3_invalid_credentials_rejected(sandbox, use_binary_protocol): + client = sandbox_rest_client('invalid.key:secret', use_binary_protocol=use_binary_protocol) + + with pytest.raises(AblyException) as excinfo: + await client.channels.get('test').presence.get() + + assert excinfo.value.status_code == 401 + assert 40100 <= excinfo.value.code < 40200 + + +# UTS: rest/integration/RSP3/subscribe-capability-sufficient-5 +async def test_rsp3_subscribe_capability_sufficient(sandbox, use_binary_protocol): + restricted_key = sandbox.key(3).key_str + + client = sandbox_rest_client(restricted_key, use_binary_protocol=use_binary_protocol) + + # Subscribe capability is sufficient for presence.get. + result = await client.channels.get(PRESENCE_FIXTURES_CHANNEL).presence.get() + assert result is not None + assert len(result.items) >= 5 From cb97916146a7f7b5aae4fdcbc80aa8ef6fce2e0c Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 15:16:04 +0100 Subject: [PATCH 04/10] test: derive the auth, revocation and mutable message integration specifications auth, revoke_tokens and mutable_messages, 20 tests, the last once per protocol. The auth specification signs its JWTs with the harness rather than a library. RSC10 is gated: Rest#request passes raise_on_error=False, so the HTTP layer never raises on the 401 and the reauthorise-and-retry branch never runs, while the pre-emptive check is separately inert without a time offset. The same expired token renews correctly through publish(). revoke_tokens needs Auth#revokeTokens, which does not exist, so all four tests are gated. Their assertions were checked against the endpoint directly, which moved two of them: a revoked token does not leave the connection DISCONNECTED with 40141 here, because the client the specification builds holds only a TokenDetails and cannot re-authorise, so RSA4a fails the connection with 40171. Co-Authored-By: Claude Opus 5 (1M context) --- test/uts/rest/integration/auth_test.py | 206 +++++++++++++ .../rest/integration/mutable_messages_test.py | 285 ++++++++++++++++++ .../rest/integration/revoke_tokens_test.py | 197 ++++++++++++ 3 files changed, 688 insertions(+) create mode 100644 test/uts/rest/integration/auth_test.py create mode 100644 test/uts/rest/integration/mutable_messages_test.py create mode 100644 test/uts/rest/integration/revoke_tokens_test.py diff --git a/test/uts/rest/integration/auth_test.py b/test/uts/rest/integration/auth_test.py new file mode 100644 index 00000000..eac29ee2 --- /dev/null +++ b/test/uts/rest/integration/auth_test.py @@ -0,0 +1,206 @@ +"""Derived from uts/rest/integration/auth.md in ably/specification. + +Spec points: RSA4, RSA8, RSC10 + +The specification's preamble asks for every test to run against both token +formats, JWT first and an Ably native token second. Its Test IDs already carry +that split — `token-auth-jwt-0` and `auth-callback-jwt-3` are the JWT pair, +`token-auth-native-1` and `auth-callback-token-request-2` the native pair — so +the two formats are separate tests here rather than a second axis of +parametrisation over all eight. + +There is no `## Protocol Variants` section, so these run against JSON only and +take no `use_binary_protocol`. +""" + +import time + +import pytest + +from ably.transport.defaults import Defaults +from ably.util.exceptions import AblyException +from test.uts.helpers.client import sandbox_rest_client +from test.uts.helpers.deviations import deviation +from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt, random_id + +# The specification's `ttl: 3600000`, an hour in milliseconds. +ONE_HOUR_MS = 3600000 + +# The specification's `expires_at: now() - 5_seconds`, in seconds. +EXPIRED_BY_S = 5 + + +async def channel_status(client, channel_name): + """The specification's `client.request("GET", "/channels/" + channel_name)`. + + `request` takes the protocol version as a required argument here, and + answers with an `HttpPaginatedResponse` that does not raise on an error + status, which is what lets the invalid-credentials test read a 401 off the + result rather than catching it. + """ + return await client.request('GET', f'/channels/{channel_name}', version=Defaults.protocol_version) + + +# UTS: rest/integration/RSA4/basic-auth-key-0 +async def test_rsa4_basic_auth_key(sandbox): + channel_name = f'test-RSA4-{random_id()}' + client = sandbox_rest_client(sandbox.key_str) + + result = await channel_status(client, channel_name) + + assert 200 <= result.status_code < 300 + + +# UTS: rest/integration/RSA8/token-auth-jwt-0 +async def test_rsa8_token_auth_jwt(sandbox): + api_key = sandbox.key_str + jwt = generate_jwt( + key_name=extract_key_name(api_key), + key_secret=extract_key_secret(api_key), + ttl=ONE_HOUR_MS, + ) + + channel_name = f'test-RSA8-jwt-{random_id()}' + client = sandbox_rest_client(token=jwt) + + result = await channel_status(client, channel_name) + + assert 200 <= result.status_code < 300 + + +# UTS: rest/integration/RSA8/token-auth-native-1 +async def test_rsa8_token_auth_native(sandbox): + key_client = sandbox_rest_client(sandbox.key_str) + + # `request_token` takes its token params as a positional dict; every + # keyword on it is an auth option. + token_details = await key_client.auth.request_token() + + channel_name = f'test-RSA8-native-{random_id()}' + token_client = sandbox_rest_client(token=token_details.token) + + result = await channel_status(token_client, channel_name) + + assert isinstance(token_details.token, str) + assert len(token_details.token) > 0 + assert token_details.expires > time.time() * 1000 + assert 200 <= result.status_code < 300 + + +# UTS: rest/integration/RSA8/auth-callback-token-request-2 +async def test_rsa8_auth_callback_token_request(sandbox): + token_request_client = sandbox_rest_client(sandbox.key_str) + + # The callback has to be a coroutine function: a synchronous one is + # reported as 401/40170 rather than being awaited. + async def auth_callback(params): + return await token_request_client.auth.create_token_request(params) + + channel_name = f'test-RSA8-callback-{random_id()}' + client = sandbox_rest_client(auth_callback=auth_callback) + + result = await channel_status(client, channel_name) + + assert 200 <= result.status_code < 300 + + +# UTS: rest/integration/RSA8/auth-callback-jwt-3 +async def test_rsa8_auth_callback_jwt(sandbox): + api_key = sandbox.key_str + + # The token params reach the callback as a dict with snake_case keys, and + # this client configures none, so both reads fall back the way the + # specification's `params.ttl OR 3600000` does. + async def auth_callback(params): + return generate_jwt( + key_name=extract_key_name(api_key), + key_secret=extract_key_secret(api_key), + client_id=params.get('client_id'), + ttl=params.get('ttl') or ONE_HOUR_MS, + ) + + channel_name = f'test-RSA8-jwt-callback-{random_id()}' + client = sandbox_rest_client(auth_callback=auth_callback) + + result = await channel_status(client, channel_name) + + assert 200 <= result.status_code < 300 + + +# UTS: rest/integration/RSA4/invalid-credentials-rejected-1 +async def test_rsa4_invalid_credentials_rejected(sandbox): + channel_name = f'test-RSA4-invalid-{random_id()}' + + # The real app id with a fabricated key name, so the server answers 401 + # with error code 40400 rather than rejecting the app. + invalid_key = f'{sandbox.app_id}.invalidKey:invalidSecret' + + client = sandbox_rest_client(invalid_key) + + result = await channel_status(client, channel_name) + + assert result.status_code == 401 + # `error_code` is the raw `X-Ably-Errorcode` header, so a string. + assert result.error_code == '40400' + + +# UTS: rest/integration/RSC10/token-renewal-expired-jwt-0 +@deviation +async def test_rsc10_token_renewal_expired_jwt(sandbox): + api_key = sandbox.key_str + issued_tokens = [] + + async def auth_callback(params): + if not issued_tokens: + # An already-expired JWT, so the first request is rejected with a + # token error: 401/40142, inside the 40140-40149 band RSC10 names. + token = generate_jwt( + key_name=extract_key_name(api_key), + key_secret=extract_key_secret(api_key), + expires_at=int(time.time()) - EXPIRED_BY_S, + ) + else: + token = generate_jwt( + key_name=extract_key_name(api_key), + key_secret=extract_key_secret(api_key), + ttl=ONE_HOUR_MS, + ) + issued_tokens.append(token) + return token + + channel_name = f'test-RSC10-renewal-{random_id()}' + client = sandbox_rest_client(auth_callback=auth_callback) + + result = await channel_status(client, channel_name) + + assert 200 <= result.status_code < 300 + assert len(issued_tokens) == 2 + # Counting the callbacks alone would not show that the renewal is what + # carried the request: the client has to end up holding the second JWT. + assert client.auth.token_details.token == issued_tokens[1] + + +# UTS: rest/integration/RSA8/capability-restriction-4 +async def test_rsa8_capability_restriction(sandbox): + api_key = sandbox.key_str + allowed_channel = f'test-RSA8-cap-allowed-{random_id()}' + denied_channel = f'test-RSA8-cap-denied-{random_id()}' + + jwt = generate_jwt( + key_name=extract_key_name(api_key), + key_secret=extract_key_secret(api_key), + capability=f'{{"{allowed_channel}":["publish","subscribe"]}}', + ttl=ONE_HOUR_MS, + ) + + client = sandbox_rest_client(token=jwt) + + # The allowed channel is publishable, which is the capability the JWT + # grants; a channel status request would need channel-metadata instead. + await client.channels.get(allowed_channel).publish(name='test', data='hello') + + with pytest.raises(AblyException) as excinfo: + await client.channels.get(denied_channel).publish(name='test', data='hello') + + assert excinfo.value.code == 40160 + assert excinfo.value.status_code == 401 diff --git a/test/uts/rest/integration/mutable_messages_test.py b/test/uts/rest/integration/mutable_messages_test.py new file mode 100644 index 00000000..b632b628 --- /dev/null +++ b/test/uts/rest/integration/mutable_messages_test.py @@ -0,0 +1,285 @@ +"""Derived from uts/rest/integration/mutable_messages.md in ably/specification. + +Spec points: RSL1n, RSL11, RSL14, RSL15, RSAN1, RSAN2, RSAN3 +""" + +from ably.http.paginatedresult import PaginatedResult +from ably.types.annotation import Annotation, AnnotationAction +from ably.types.message import Message, MessageAction +from ably.types.operations import MessageOperation, PublishResult, UpdateDeleteResult +from ably.util.exceptions import AblyException +from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until +from test.uts.helpers.sandbox import random_id + +# The specification's channel prefix. `test-app-setup.json` configures the +# `mutable` namespace with `mutableMessages: true`, and getMessage, +# updateMessage, deleteMessage, appendMessage and the annotation endpoints are +# all refused on a channel outside it. +MUTABLE_NAMESPACE = 'mutable:' + +# The annotation type the specification annotates with. +REACTION_TYPE = 'com.ably.reactions' + +# What the specification's `poll_until_success` gives a store read. The store is +# eventually consistent, so a serial-scoped read answers 404 until the write it +# is looking for has landed. +STORE_TIMEOUT = 20.0 + + +def mutable_channel(client, name): + """The channel a specification builds as `"mutable:test-..." + random_id()`.""" + return client.channels.get(f'{MUTABLE_NAMESPACE}{name}-{random_id()}') + + +def not_found(error): + """Whether `error` is the store saying "not yet", rather than a real failure. + + The specification's `poll_until_success` "treats any read error as keep + polling". Only the not-found is swallowed here: every other error means the + request was wrong rather than early, and swallowing it would leave the test + timing out on a message it could never explain. + """ + return isinstance(error, AblyException) and error.status_code == 404 + + +async def poll_get_message(channel, serial, action=None, description=None): + """The specification's `poll_until_success` around `getMessage`. + + `get_message` is not `@catch_all`-wrapped, so the store's 404 arrives as an + `AblyException` carrying that status rather than as a 50000. Returns the + message once it exists and, where `action` is given, once it carries that + action. + """ + async def visible(): + try: + message = await channel.get_message(serial) + except AblyException as error: + if not_found(error): + return None + raise + if action is not None and message.action != action: + return None + return message + + return await wall_clock_poll_until( + visible, + timeout=STORE_TIMEOUT, + description=description or f'message {serial} to be readable') + + +async def poll_message_versions(channel, serial, count): + """The specification's `poll_until_success` around `getMessageVersions`.""" + async def enough_versions(): + try: + result = await channel.get_message_versions(serial) + except AblyException as error: + if not_found(error): + return None + raise + return result if len(result.items) >= count else None + + return await wall_clock_poll_until( + enough_versions, + timeout=STORE_TIMEOUT, + description=f'{count} versions of message {serial}') + + +async def poll_annotations(channel, serial, count): + """The specification's `poll_until_success` around `annotations.get`.""" + async def enough_annotations(): + try: + result = await channel.annotations.get(serial) + except AblyException as error: + if not_found(error): + return None + raise + return result if len(result.items) >= count else None + + return await wall_clock_poll_until( + enough_annotations, + timeout=STORE_TIMEOUT, + description=f'{count} annotations on message {serial}') + + +# UTS: rest/integration/RSL1n/publish-returns-serials-0 +async def test_rsl1n_publish_returns_serials(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + channel = mutable_channel(client, 'test-RSL1n-serials') + + result1 = await channel.publish('event1', 'data1') + + assert isinstance(result1, PublishResult) + assert isinstance(result1.serials, list) + assert len(result1.serials) == 1 + assert isinstance(result1.serials[0], str) + assert len(result1.serials[0]) > 0 + + result2 = await channel.publish([ + Message('event2', 'data2'), + Message('event3', 'data3'), + Message('event4', 'data4'), + ]) + + assert len(result2.serials) == 3 + assert all(isinstance(serial, str) and len(serial) > 0 for serial in result2.serials) + + assert result2.serials[0] != result2.serials[1] + assert result2.serials[1] != result2.serials[2] + + +# UTS: rest/integration/RSL11/get-message-by-serial-0 +async def test_rsl11_get_message_by_serial(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + channel = mutable_channel(client, 'test-RSL11-getMessage') + + publish_result = await channel.publish('test-event', 'hello world') + serial = publish_result.serials[0] + + msg = await poll_get_message(channel, serial, description='the published message to be gettable') + + assert isinstance(msg, Message) + assert msg.name == 'test-event' + assert msg.data == 'hello world' + assert msg.serial == serial + assert msg.action == MessageAction.MESSAGE_CREATE + assert msg.timestamp is not None + + +# UTS: rest/integration/RSL15/update-message-0 +async def test_rsl15_update_message(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + channel = mutable_channel(client, 'test-RSL15-update') + + publish_result = await channel.publish('original', 'original-data') + serial = publish_result.serials[0] + + update_result = await channel.update_message( + Message(serial=serial, name='updated', data='updated-data'), + MessageOperation(description='edited content'), + ) + + assert isinstance(update_result, UpdateDeleteResult) + assert isinstance(update_result.version_serial, str) + assert len(update_result.version_serial) > 0 + + updated_msg = await poll_get_message( + channel, serial, action=MessageAction.MESSAGE_UPDATE, + description='the update to be visible') + + assert updated_msg.name == 'updated' + assert updated_msg.data == 'updated-data' + assert updated_msg.action == MessageAction.MESSAGE_UPDATE + assert updated_msg.version.description == 'edited content' + + +# UTS: rest/integration/RSL15/delete-message-1 +async def test_rsl15_delete_message(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + channel = mutable_channel(client, 'test-RSL15-delete') + + publish_result = await channel.publish('to-delete', 'delete-me') + serial = publish_result.serials[0] + + delete_result = await channel.delete_message(Message(serial=serial)) + + assert isinstance(delete_result, UpdateDeleteResult) + assert isinstance(delete_result.version_serial, str) + assert len(delete_result.version_serial) > 0 + + deleted_msg = await poll_get_message( + channel, serial, action=MessageAction.MESSAGE_DELETE, + description='the delete to be visible') + + assert deleted_msg.action == MessageAction.MESSAGE_DELETE + + +# UTS: rest/integration/RSL14/get-message-versions-0 +async def test_rsl14_get_message_versions(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + channel = mutable_channel(client, 'test-RSL14-versions') + + publish_result = await channel.publish('versioned', 'v1') + serial = publish_result.serials[0] + + await channel.update_message( + Message(serial=serial, data='v2'), + MessageOperation(description='first edit'), + ) + await channel.update_message( + Message(serial=serial, data='v3'), + MessageOperation(description='second edit'), + ) + + versions = await poll_message_versions(channel, serial, 3) + + assert isinstance(versions, PaginatedResult) + assert len(versions.items) >= 3 + + for item in versions.items: + assert isinstance(item, Message) + assert item.serial == serial + + +# UTS: rest/integration/RSL15/append-message-2 +async def test_rsl15_append_message(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + channel = mutable_channel(client, 'test-RSL15-append') + + publish_result = await channel.publish('appendable', 'original') + serial = publish_result.serials[0] + + append_result = await channel.append_message( + Message(serial=serial, data='appended-data'), + MessageOperation(description='appended content'), + ) + + assert isinstance(append_result, UpdateDeleteResult) + assert isinstance(append_result.version_serial, str) + assert len(append_result.version_serial) > 0 + + +# UTS: rest/integration/RSAN1/annotation-lifecycle-0 +async def test_rsan1_annotation_lifecycle(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + channel = mutable_channel(client, 'test-RSAN-lifecycle') + + publish_result = await channel.publish('annotatable', 'content') + serial = publish_result.serials[0] + + await channel.annotations.publish(serial, Annotation(type=REACTION_TYPE, name='like')) + + annotations = await poll_annotations(channel, serial, 1) + assert len(annotations.items) >= 1 + + found = False + for ann in annotations.items: + if ann.type == REACTION_TYPE and ann.name == 'like': + found = True + assert ann.action == AnnotationAction.ANNOTATION_CREATE + assert ann.message_serial == serial + assert found is True + + await channel.annotations.delete(serial, Annotation(type=REACTION_TYPE, name='like')) + + +# UTS: rest/integration/RSAN3/get-annotations-paginated-0 +async def test_rsan3_get_annotations_paginated(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key(0).key_str, use_binary_protocol=use_binary_protocol) + channel = mutable_channel(client, 'test-RSAN3-paginated') + + publish_result = await channel.publish('multi-annotated', 'content') + serial = publish_result.serials[0] + + await channel.annotations.publish(serial, Annotation(type=REACTION_TYPE, name='like')) + await channel.annotations.publish(serial, Annotation(type=REACTION_TYPE, name='heart')) + + result = await poll_annotations(channel, serial, 2) + + assert isinstance(result, PaginatedResult) + assert len(result.items) >= 2 + + for ann in result.items: + assert isinstance(ann, Annotation) + assert ann.message_serial == serial + assert ann.type == REACTION_TYPE + assert ann.timestamp is not None diff --git a/test/uts/rest/integration/revoke_tokens_test.py b/test/uts/rest/integration/revoke_tokens_test.py new file mode 100644 index 00000000..6fdf6309 --- /dev/null +++ b/test/uts/rest/integration/revoke_tokens_test.py @@ -0,0 +1,197 @@ +"""Derived from uts/rest/integration/revoke_tokens.md in ably/specification. + +Spec points: RSA17, RSA17b, RSA17c, RSA17d, RSA17e, RSA17f, RSA17g, TRS2, TRF2 + +ably-python has no token revocation: there is no `Auth#revokeTokens`, no +`TokenRevocationTargetSpecifier`, no `BatchResult` and no +`TokenRevocationSuccessResult`/`TokenRevocationFailureResult`. Every test here is +therefore an env-gated deviation, each failing with `AttributeError` when run with +`RUN_DEVIATIONS=1`. They are written against the API the specification describes, +spelled the way the unit tier spells it in +[rest/unit/auth/revoke_tokens_test.py](../unit/auth/revoke_tokens_test.py) +(`revoke_tokens`, `issued_before`, `allow_reauth_margin`, `success_count`, +`failure_count`, `applies_at`), with the target specifiers as plain dicts since +the specifier type does not exist either. The two tiers gate on the same names, so +they go green together when the API lands. + +Everything either side of the revocation call is real: the app, the revocable key, +the issued token and the realtime connection the revocation drops. The two +connection tests assert FAILED with 40171 where the specification asserts +DISCONNECTED with 40141, which is an adaptation recorded in +[deviations-revoke-tokens-integration.md](../../deviations-revoke-tokens-integration.md) +and explained where it is made. +""" + +import asyncio + +import pytest + +from ably.realtime.connection import ConnectionState +from ably.util.exceptions import AblyException +from test.uts.helpers.client import ( + await_connection_state, + next_connection_state, + sandbox_realtime_client, + sandbox_rest_client, +) +from test.uts.helpers.deviations import deviation +from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt, random_id + +# `keys[4]` is the one the canonical app setup marks `revocableTokens: true`. The +# revocation endpoint is only served for such a key, so the index is part of what +# these tests assert rather than an arbitrary choice of credential. +REVOCABLE_KEY_INDEX = 4 + +# How long a revoked token's connection is given to be dropped. The revocation +# applies about a quarter of a second after the request and the server pushes the +# disconnect at once, so this is generous; it is bounded well inside the package's +# 120 seconds so a stalled wait names itself. +DROP_TIMEOUT = 20.0 + + +async def start_drop_watch(client): + """Begins waiting for `client`'s connection to be dropped, before the caller revokes. + + This is the specification's `disconnected_promise = connection.once("disconnected")`, + set up before the revocation so the state change cannot be missed. The client + auto-connects, so a listener registered afterwards can wait for a change that has + already gone past. The wait runs as a task, and the event loop is yielded to once, + so the listener is in place by the time this returns. + + FAILED stands in for the specification's DISCONNECTED; see the note in each caller. + """ + watch = asyncio.ensure_future( + next_connection_state(client, ConnectionState.FAILED, timeout=DROP_TIMEOUT)) + await asyncio.sleep(0) + return watch + + +async def connected_token_client(key_client, client_id): + """A realtime client connected with a native token issued for `client_id`. + + The specification's setup: `requestToken(clientId: client_id)` followed by a + `Realtime` carrying that token, awaited to CONNECTED. Token params go to + `request_token` as a positional dict, since its keyword arguments are all + auth options; the `TokenDetails` goes to the client as `token_details`, since + `token` wants a string. + """ + token_details = await key_client.auth.request_token({'client_id': client_id}) + realtime_client = sandbox_realtime_client(token_details=token_details) + await await_connection_state(realtime_client, ConnectionState.CONNECTED) + return realtime_client + + +# UTS: rest/integration/RSA17g/revoke-token-prevents-use-0 +@deviation +async def test_rsa17g_revoke_token_prevents_use(sandbox): + client_id = f'revoke-client-{random_id()}' + key_client = sandbox_rest_client(sandbox.key(REVOCABLE_KEY_INDEX).key_str) + realtime_client = await connected_token_client(key_client, client_id) + + dropped = await start_drop_watch(realtime_client) + try: + revoke_result = await key_client.auth.revoke_tokens([{'type': 'clientId', 'value': client_id}]) + + assert revoke_result.success_count == 1 + assert revoke_result.failure_count == 0 + assert len(revoke_result.results) == 1 + + # NOTE: the spec asserts `success IS TokenRevocationSuccessResult`. No such type + # exists in ably-python, so the attributes it defines stand in for the type check. + success = revoke_result.results[0] + assert success.target == f'clientId:{client_id}' + assert isinstance(success.issued_before, (int, float)) + assert isinstance(success.applies_at, (int, float)) + + # The spec asserts a DISCONNECTED state change whose `reason.code` is 40141. The + # server does push exactly that — action 6 carrying `{"code": 40141, "message": + # "token revoked"}` — but a connection holding only a `TokenDetails` has no way to + # renew, so ably-python fails it under RSA4a with 40171 and the 40141 never reaches + # the connection's state. See deviations-revoke-tokens-integration.md. + state_change = await dropped + assert state_change.reason.code == 40171 + assert state_change.reason.status_code == 403 + finally: + dropped.cancel() + + +# UTS: rest/integration/RSA17d/token-auth-revoke-rejected-0 +@deviation +async def test_rsa17d_token_auth_revoke_rejected(sandbox): + revocable_key = sandbox.key(REVOCABLE_KEY_INDEX).key_str + jwt = generate_jwt(extract_key_name(revocable_key), extract_key_secret(revocable_key), ttl=3600000) + token_rest = sandbox_rest_client(token=jwt) + + with pytest.raises(AblyException) as excinfo: + await token_rest.auth.revoke_tokens([{'type': 'clientId', 'value': 'anyone'}]) + + assert excinfo.value.code == 40162 + assert excinfo.value.status_code == 401 + + +# UTS: rest/integration/RSA17e/issued-before-reauth-margin-0 +@deviation +async def test_rsa17e_issued_before_reauth_margin(sandbox): + client_id = f'revoke-margin-client-{random_id()}' + key_client = sandbox_rest_client(sandbox.key(REVOCABLE_KEY_INDEX).key_str) + + # `time()` answers with a number of milliseconds since the epoch, not a date. + server_time = await key_client.time() + + # An `issuedBefore` in the past, so no token in use anywhere is revoked. + issued_before = int(server_time) - 20 * 60 * 1000 + + revoke_result = await key_client.auth.revoke_tokens( + [{'type': 'clientId', 'value': client_id}], + issued_before=issued_before, + allow_reauth_margin=True, + ) + + assert revoke_result.success_count == 1 + assert len(revoke_result.results) == 1 + + # RSA17e: issuedBefore should reflect what was sent. + assert revoke_result.results[0].issued_before == issued_before + + # RSA17f: allowReauthMargin delays appliesAt by ~30 seconds. + assert revoke_result.results[0].applies_at > server_time + 30 * 1000 + + +# UTS: rest/integration/RSA17c/mixed-success-failure-0 +@deviation +async def test_rsa17c_mixed_success_failure(sandbox): + client_id = f'revoke-mixed-client-{random_id()}' + key_client = sandbox_rest_client(sandbox.key(REVOCABLE_KEY_INDEX).key_str) + realtime_client = await connected_token_client(key_client, client_id) + + dropped = await start_drop_watch(realtime_client) + try: + revoke_result = await key_client.auth.revoke_tokens([ + {'type': 'clientId', 'value': client_id}, + {'type': 'invalidType', 'value': 'abc'}, + ]) + + assert revoke_result.success_count == 1 + assert revoke_result.failure_count == 1 + assert len(revoke_result.results) == 2 + + # NOTE: the spec asserts `success IS TokenRevocationSuccessResult` and + # `failure IS TokenRevocationFailureResult`; neither type exists in ably-python, + # so the attributes each defines stand in for the type checks. + success = revoke_result.results[0] + assert success.target == f'clientId:{client_id}' + assert isinstance(success.issued_before, (int, float)) + assert isinstance(success.applies_at, (int, float)) + + # The sandbox answers an invalid target type with 40001 rather than the 40000 the + # spec's example shows, so only the status code the spec asserts is asserted here. + failure = revoke_result.results[1] + assert failure.target == 'invalidType:abc' + assert failure.error.status_code == 400 + + # FAILED with 40171 for the spec's DISCONNECTED with 40141, as in RSA17g above. + state_change = await dropped + assert state_change.reason.code == 40171 + assert state_change.reason.status_code == 403 + finally: + dropped.cancel() From 6b86dd3370a90662dd0751a279116493cf17bbbd Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 15:16:04 +0100 Subject: [PATCH 05/10] test: derive the push integration specifications MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit push_admin and push_channels, 18 tests, json only. A filtered list needs proving. The push admin filter parameters are camelCase on the wire, and the server drops one it does not recognise rather than rejecting it, so a snake_case filter returns the whole unfiltered page. A test asserting that the row it just created is present then passes with the filter doing nothing. Every filtered list here carries a control — a decoy row, or a count taken before the call — so that narrowing is what the assertion rests on. Deletion is asynchronous, so the counts that follow a removal poll. push_channels needs channel.push, client.device and LocalDevice, none of which exist, so both tests are gated. The specification's hard-coded device identity token is also rejected by the server, so the test takes the one the registration issues. Co-Authored-By: Claude Opus 5 (1M context) --- test/uts/rest/integration/push_admin_test.py | 400 ++++++++++++++++++ .../rest/integration/push_channels_test.py | 160 +++++++ 2 files changed, 560 insertions(+) create mode 100644 test/uts/rest/integration/push_admin_test.py create mode 100644 test/uts/rest/integration/push_channels_test.py diff --git a/test/uts/rest/integration/push_admin_test.py b/test/uts/rest/integration/push_admin_test.py new file mode 100644 index 00000000..cb4f768a --- /dev/null +++ b/test/uts/rest/integration/push_admin_test.py @@ -0,0 +1,400 @@ +"""Derived from uts/rest/integration/push_admin.md in ably/specification. + +Spec points: RSH1, RSH1a, RSH1b1, RSH1b2, RSH1b3, RSH1b4, RSH1b5, RSH1c1, RSH1c2, +RSH1c3, RSH1c4, RSH1c5 +""" + +import pytest + +from ably import AblyException, DeviceDetails, PushChannelSubscription +from ably.http.paginatedresult import PaginatedResult +from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until +from test.uts.helpers.sandbox import random_id + +# The push-enabled namespace the canonical app setup provisions. A channel +# subscription may only be saved against a channel whose namespace carries +# `pushEnabled: true`. +PUSH_NAMESPACE = 'pushenabled' + +# An APNs device token as the sandbox expects to receive one: 32 bytes, hex. +# `test/ably/rest/restpush_test.py` registers the same value. +DEVICE_TOKEN = '740f4707bebcf74f9b7c25d48e3358945f6aa01da5ddb387462c7eaf61bb78ad' + + +def apns_device(device_id, client_id=None, device_token=DEVICE_TOKEN, platform='ios', + form_factor='phone'): + """The specifications' `DeviceDetails(... push: DevicePushDetails(recipient: ...))`. + + There is no `DevicePushDetails` type in ably-python; `DeviceDetails.push` is + the plain dict the wire carries, so the recipient goes in directly. + """ + return DeviceDetails( + id=device_id, + client_id=client_id, + platform=platform, + form_factor=form_factor, + push={'recipient': {'transportType': 'apns', 'deviceToken': device_token}}, + ) + + +async def remove_device_quietly(client, device_id): + try: + await client.push.admin.device_registrations.remove(device_id) + except AblyException: + pass + + +async def remove_subscription_quietly(client, subscription): + try: + await client.push.admin.channel_subscriptions.remove(subscription) + except AblyException: + pass + + +# UTS: rest/integration/RSH1a/push-publish-clientid-0 +async def test_rsh1a_push_publish_clientid(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + + result = await client.push.admin.publish( + {'clientId': 'test-client-push'}, + {'notification': {'title': 'Integration Test', 'body': 'Hello from push admin'}}, + ) + + # The spec only requires the call not to throw. `publish` returns nothing. + assert result is None + + +# UTS: rest/integration/RSH1a/push-publish-invalid-recipient-1 +async def test_rsh1a_push_publish_invalid_recipient(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + + # NOTE: the spec expects the server to reject the empty recipient and the + # error to carry a `code`. `PushAdmin.publish` validates the recipient + # itself (`ably/rest/push.py:57`) and raises `ValueError` before any request + # is made, so there is no server error and no code to read. See + # test/uts/deviations-push-admin-integration.md. + with pytest.raises(ValueError): + await client.push.admin.publish({}, {'notification': {'title': 'Test'}}) + + +# UTS: rest/integration/RSH1b3/save-and-get-device-0 +async def test_rsh1b3_save_and_get_device(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + registrations = client.push.admin.device_registrations + device_id = f'test-device-{random_id()}' + device_token = f'{random_id(16)}{random_id(16)}' + + saved = await registrations.save(apns_device(device_id, device_token=device_token)) + try: + assert isinstance(saved, DeviceDetails) + assert saved.id == device_id + assert saved.platform == 'ios' + assert saved.form_factor == 'phone' + assert saved.push['recipient']['transportType'] == 'apns' + + retrieved = await registrations.get(device_id) + assert isinstance(retrieved, DeviceDetails) + assert retrieved.id == device_id + assert retrieved.platform == 'ios' + finally: + await remove_device_quietly(client, device_id) + + +# UTS: rest/integration/RSH1b3/update-device-registration-1 +async def test_rsh1b3_update_device_registration(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + registrations = client.push.admin.device_registrations + device_id = f'test-device-update-{random_id()}' + token_v1 = f'{random_id(16)}{random_id(16)}' + token_v2 = f'{random_id(16)}{random_id(16)}' + + await registrations.save(apns_device(device_id, device_token=token_v1)) + try: + updated = await registrations.save(apns_device(device_id, device_token=token_v2)) + + assert updated.id == device_id + assert updated.push['recipient']['deviceToken'] == token_v2 + + retrieved = await registrations.get(device_id) + assert retrieved.push['recipient']['deviceToken'] == token_v2 + finally: + await remove_device_quietly(client, device_id) + + +# UTS: rest/integration/RSH1b1/get-unknown-device-error-0 +async def test_rsh1b1_get_unknown_device_error(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + + with pytest.raises(AblyException) as excinfo: + await client.push.admin.device_registrations.get(f'nonexistent-device-{random_id()}') + + assert excinfo.value.status_code == 404 + + +# UTS: rest/integration/RSH1b2/list-devices-filtered-0 +async def test_rsh1b2_list_devices_filtered(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + registrations = client.push.admin.device_registrations + device_id = f'test-device-list-{random_id()}' + # A second registration the filter has to exclude. Without it a `deviceId` + # filter that was dropped altogether would still leave one item behind. + decoy_id = f'test-device-decoy-{random_id()}' + + await registrations.save(DeviceDetails( + id=device_id, + platform='android', + form_factor='tablet', + push={'recipient': {'transportType': 'gcm', 'registrationToken': f'token-{random_id()}'}}, + )) + await registrations.save(apns_device(decoy_id)) + try: + result = await registrations.list(deviceId=device_id) + + assert isinstance(result, PaginatedResult) + assert len(result.items) == 1 + assert result.items[0].id == device_id + assert result.items[0].platform == 'android' + finally: + await remove_device_quietly(client, device_id) + await remove_device_quietly(client, decoy_id) + + +# UTS: rest/integration/RSH1b2/list-devices-pagination-1 +async def test_rsh1b2_list_devices_pagination(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + registrations = client.push.admin.device_registrations + client_id = f'test-client-list-{random_id()}' + device_ids = [] + + for i in (1, 2, 3): + device_id = f'test-device-limit-{i}-{random_id()}' + device_ids.append(device_id) + await registrations.save(apns_device(device_id, client_id=client_id)) + try: + # The filter has to hold before the limit means anything: three + # registrations share this clientId and nothing else does. + unlimited = await registrations.list(clientId=client_id) + assert len(unlimited.items) == 3 + + result = await registrations.list(clientId=client_id, limit=2) + + assert len(result.items) <= 2 + # NOTE: the spec writes `result.hasNext == true`. `has_next` is a method + # here, and a bound method is truthy whatever the page holds. + assert result.has_next() is True + finally: + for device_id in device_ids: + await remove_device_quietly(client, device_id) + + +# UTS: rest/integration/RSH1b4/remove-device-0 +async def test_rsh1b4_remove_device(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + registrations = client.push.admin.device_registrations + device_id = f'test-device-remove-{random_id()}' + + await registrations.save(apns_device(device_id)) + + response = await registrations.remove(device_id) + # `remove` answers with the raw response rather than nothing. + assert response.status_code == 204 + + with pytest.raises(AblyException) as excinfo: + await registrations.get(device_id) + assert excinfo.value.status_code == 404 + + +# UTS: rest/integration/RSH1b4/remove-nonexistent-device-1 +async def test_rsh1b4_remove_nonexistent_device(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + + response = await client.push.admin.device_registrations.remove( + f'nonexistent-device-{random_id()}') + + assert response.status_code == 204 + + +# UTS: rest/integration/RSH1b5/remove-where-clientid-0 +async def test_rsh1b5_remove_where_clientid(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + registrations = client.push.admin.device_registrations + client_id = f'test-client-removeWhere-{random_id()}' + survivor_client_id = f'test-client-survivor-{random_id()}' + survivor_id = f'test-device-survivor-{random_id()}' + device_ids = [] + + for i in (1, 2): + device_id = f'test-device-rw-{i}-{random_id()}' + device_ids.append(device_id) + await registrations.save(apns_device(device_id, client_id=client_id)) + # A registration under a different clientId, which removeWhere must leave alone. + await registrations.save(apns_device(survivor_id, client_id=survivor_client_id)) + try: + response = await registrations.remove_where(clientId=client_id) + assert response.status_code == 204 + + # Deletion is asynchronous on the server side. + async def removed(): + result = await registrations.list(clientId=client_id) + return result if len(result.items) == 0 else None + + result = await wall_clock_poll_until( + removed, timeout=20.0, + description='the devices registered under the clientId to be removed') + assert len(result.items) == 0 + + assert (await registrations.get(survivor_id)).id == survivor_id + finally: + for device_id in device_ids + [survivor_id]: + await remove_device_quietly(client, device_id) + + +# UTS: rest/integration/RSH1c3/save-and-list-subscriptions-0 +async def test_rsh1c3_save_and_list_subscriptions(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + subscriptions = client.push.admin.channel_subscriptions + device_id = f'test-device-sub-{random_id()}' + channel_name = f'{PUSH_NAMESPACE}:test-sub-{random_id()}' + decoy_channel = f'{PUSH_NAMESPACE}:test-sub-decoy-{random_id()}' + + await client.push.admin.device_registrations.save(apns_device(device_id)) + decoy = PushChannelSubscription(decoy_channel, device_id=device_id) + await subscriptions.save(decoy) + try: + saved = await subscriptions.save( + PushChannelSubscription(channel_name, device_id=device_id)) + + assert isinstance(saved, PushChannelSubscription) + assert saved.channel == channel_name + assert saved.device_id == device_id + + result = await subscriptions.list(channel=channel_name) + assert isinstance(result, PaginatedResult) + assert len(result.items) >= 1 + # The same device is subscribed to a second channel, so a `channel` + # filter that was dropped would show that subscription too. + assert {sub.channel for sub in result.items} == {channel_name} + + found = False + for sub in result.items: + if sub.device_id == device_id: + found = True + assert sub.channel == channel_name + assert found is True + finally: + await remove_subscription_quietly( + client, PushChannelSubscription(channel_name, device_id=device_id)) + await remove_subscription_quietly(client, decoy) + await remove_device_quietly(client, device_id) + + +# UTS: rest/integration/RSH1c3/save-subscription-clientid-1 +async def test_rsh1c3_save_subscription_clientid(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + subscriptions = client.push.admin.channel_subscriptions + client_id = f'test-client-sub-{random_id()}' + channel_name = f'{PUSH_NAMESPACE}:test-clientsub-{random_id()}' + + subscription = PushChannelSubscription(channel_name, client_id=client_id) + saved = await subscriptions.save(subscription) + try: + assert saved.channel == channel_name + assert saved.client_id == client_id + finally: + await remove_subscription_quietly(client, subscription) + + +# UTS: rest/integration/RSH1c2/list-channels-with-subscriptions-0 +async def test_rsh1c2_list_channels_with_subscriptions(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + subscriptions = client.push.admin.channel_subscriptions + client_id = f'test-client-lc-{random_id()}' + channel_name = f'{PUSH_NAMESPACE}:test-listchannels-{random_id()}' + + subscription = PushChannelSubscription(channel_name, client_id=client_id) + await subscriptions.save(subscription) + try: + # The channel appears once the subscription has propagated. + async def listed(): + result = await subscriptions.list_channels() + return result if channel_name in result.items else None + + result = await wall_clock_poll_until( + listed, timeout=20.0, + description='the subscribed channel to appear in listChannels') + + assert isinstance(result, PaginatedResult) + assert channel_name in result.items + finally: + await remove_subscription_quietly(client, subscription) + + +# UTS: rest/integration/RSH1c4/remove-channel-subscription-0 +async def test_rsh1c4_remove_channel_subscription(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + subscriptions = client.push.admin.channel_subscriptions + client_id = f'test-client-rm-{random_id()}' + channel_name = f'{PUSH_NAMESPACE}:test-remove-{random_id()}' + + subscription = PushChannelSubscription(channel_name, client_id=client_id) + await subscriptions.save(subscription) + # The filter has to narrow for the assertion below to mean anything. + assert len((await subscriptions.list(channel=channel_name, clientId=client_id)).items) == 1 + + response = await subscriptions.remove(subscription) + assert response.status_code == 204 + + async def removed(): + result = await subscriptions.list(channel=channel_name, clientId=client_id) + return result if len(result.items) == 0 else None + + result = await wall_clock_poll_until( + removed, timeout=20.0, description='the removed subscription to disappear') + assert len(result.items) == 0 + + +# UTS: rest/integration/RSH1c4/remove-nonexistent-subscription-1 +async def test_rsh1c4_remove_nonexistent_subscription(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + + response = await client.push.admin.channel_subscriptions.remove(PushChannelSubscription( + f'{PUSH_NAMESPACE}:nonexistent-{random_id()}', client_id='nonexistent-client')) + + assert response.status_code == 204 + + +# UTS: rest/integration/RSH1c5/remove-where-subscriptions-0 +async def test_rsh1c5_remove_where_subscriptions(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + subscriptions = client.push.admin.channel_subscriptions + client_id = f'test-client-rwsub-{random_id()}' + survivor_client_id = f'test-client-rwsub-keep-{random_id()}' + survivor_channel = f'{PUSH_NAMESPACE}:test-rwsub-keep-{random_id()}' + created = [] + + for i in (1, 2): + channel_name = f'{PUSH_NAMESPACE}:test-rwsub-{i}-{random_id()}' + subscription = PushChannelSubscription(channel_name, client_id=client_id) + created.append(subscription) + await subscriptions.save(subscription) + # A subscription under a different clientId, which removeWhere must leave alone. + survivor = PushChannelSubscription(survivor_channel, client_id=survivor_client_id) + await subscriptions.save(survivor) + try: + response = await subscriptions.remove_where(clientId=client_id) + assert response.status_code == 204 + + async def removed(): + result = await subscriptions.list(clientId=client_id) + return result if len(result.items) == 0 else None + + result = await wall_clock_poll_until( + removed, timeout=20.0, + description='the subscriptions under the clientId to be removed') + assert len(result.items) == 0 + + survived = await subscriptions.list(clientId=survivor_client_id) + assert len(survived.items) == 1 + finally: + for subscription in created + [survivor]: + await remove_subscription_quietly(client, subscription) diff --git a/test/uts/rest/integration/push_channels_test.py b/test/uts/rest/integration/push_channels_test.py new file mode 100644 index 00000000..d509e1f6 --- /dev/null +++ b/test/uts/rest/integration/push_channels_test.py @@ -0,0 +1,160 @@ +"""Derived from uts/rest/integration/push_channels.md in ably/specification. + +Spec points: RSH7a, RSH7b, RSH7c, RSH7d + +DEVIATION: ably-python implements neither the PushChannel interface (RSH7, the `push` +field on a channel) nor LocalDevice (RSH8). `ably/rest/channel.py` gives a channel no +`push`, `AblyRest` no `device`, and `ably/types/device.py` defines only `DeviceDetails`. +Both tests here therefore depart from the specification and are gated behind +RUN_DEVIATIONS, against the same spelling +[test/uts/rest/unit/push/push_channels_test.py](../unit/push/push_channels_test.py) +gates on — `ably.types.device.LocalDevice`, `client.device` and +`channel.push.subscribe_device()` and friends — so that dropping the marker is the only +change either tier needs when RSH7 lands. + +The halves that do not depend on the missing API are real, and were confirmed by hand +against the sandbox: a device registered under the id and APNs token shape the +specification builds, an admin-created channel subscription in the `pushenabled` +namespace, and `channel_subscriptions.list(...)` finding it and then not finding it +once removed. Those filter keys must be camelCase — `list(**params)` hands the dict to +`format_params` positionally, which converts only its own `**kw` +(`ably/rest/push.py:147`, `ably/http/paginatedresult.py:18`), so `device_id=` reaches +the server as an unknown `device_id` query parameter that it ignores, leaving the +channel filter to match on its own. + +That confirmation also turned up a fault in the specification, recorded against the +RSH7a setup below: the `deviceIdentityToken` it hard-codes is rejected, and the real +one from the registration response is used instead. + +See [deviations-batch-push-channels-integration.md](../../deviations-batch-push-channels-integration.md). +""" + +from ably import AblyException, DeviceDetails +from test.uts.helpers.client import sandbox_rest_client +from test.uts.helpers.deviations import deviation +from test.uts.helpers.sandbox import random_id + +# A channel subscription may only be saved against a channel whose namespace carries +# `pushEnabled: true`, which in the canonical app setup is this one. +PUSH_NAMESPACE = 'pushenabled' + +# The specifications' `deviceIdentityToken: "test-device-identity-token"`, which stands +# in wherever no registration issued a real one. `subscribeClient` and +# `unsubscribeClient` do not authenticate as the device — RSH7b2 and RSH7d2 send the +# clientId, and the unit tier pins that neither sends `X-Ably-DeviceToken` — so a +# placeholder is all the RSH7b test needs. +PLACEHOLDER_DEVICE_IDENTITY_TOKEN = 'test-device-identity-token' + + +def set_local_device(client, device_id, device_identity_token=PLACEHOLDER_DEVICE_IDENTITY_TOKEN, + client_id=None): + """The specifications' `client.device = LocalDevice(...)`. + + The import is deliberately inside the call, so that a file-level import of a name + that does not exist does not take the collection of the whole package down with it. + """ + from ably.types.device import LocalDevice + + client.device = LocalDevice( + id=device_id, + device_identity_token=device_identity_token, + client_id=client_id, + ) + + +def issued_device_identity_token(registration): + """The device identity token `deviceRegistrations.save` came back with. + + `DeviceDetails.device_identity_token` holds the whole object the server sends — + `{token, keyName, issued, expires, capability}` — where a `LocalDevice` carries the + token itself, which is what push device authentication puts in `X-Ably-DeviceToken`. + """ + issued = registration.device_identity_token + assert issued, 'the registration response carried no deviceIdentityToken' + return issued['token'] if isinstance(issued, dict) else issued + + +async def remove_device_quietly(client, device_id): + try: + await client.push.admin.device_registrations.remove(device_id) + except AblyException: + pass + + +@deviation +# UTS: rest/integration/RSH7a/subscribe-unsubscribe-device-0 +async def test_rsh7a_subscribe_unsubscribe_device(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + + device_id = 'test-device-pushchan-' + random_id() + channel_name = f'{PUSH_NAMESPACE}:test-rsh7a-' + random_id() + device_token = 'test-apns-token-' + random_id() + + # A device has to be registered before a device channel subscription will take. + # There is no `DevicePushDetails` type in ably-python; `DeviceDetails.push` is the + # plain dict the wire carries, so the recipient goes in directly. + registration = await client.push.admin.device_registrations.save(DeviceDetails( + id=device_id, + platform='ios', + form_factor='phone', + push={'recipient': {'transportType': 'apns', 'deviceToken': device_token}}, + )) + try: + # SPEC FAULT: the specification's own comment says the deviceIdentityToken is + # obtained from the registration response, and then its pseudocode hard-codes + # "test-device-identity-token". The sandbox will not have it: RSH7a2 and RSH7c2 + # authenticate as the device, and a token the server did not issue comes back + # 40005 / 400, `Invalid accessToken in request`. The comment is right and the + # code beside it is not, so the issued token is what goes in. + set_local_device(client, device_id, + device_identity_token=issued_device_identity_token(registration)) + + channel = client.channels.get(channel_name) + + await channel.push.subscribe_device() + + result = await client.push.admin.channel_subscriptions.list( + channel=channel_name, deviceId=device_id) + assert len(result.items) >= 1 + assert any( + subscription.device_id == device_id and subscription.channel == channel_name + for subscription in result.items) + + await channel.push.unsubscribe_device() + + result_after = await client.push.admin.channel_subscriptions.list( + channel=channel_name, deviceId=device_id) + assert len(result_after.items) == 0 + finally: + # Removing the registration removes the device's channel subscriptions with it. + await remove_device_quietly(client, device_id) + + +@deviation +# UTS: rest/integration/RSH7b/subscribe-unsubscribe-client-0 +async def test_rsh7b_subscribe_unsubscribe_client(sandbox): + client = sandbox_rest_client(sandbox.key(0).key_str) + + client_id = 'test-client-pushchan-' + random_id() + channel_name = f'{PUSH_NAMESPACE}:test-rsh7b-' + random_id() + + # `subscribeClient` subscribes by clientId rather than by deviceId, so no device + # registration is needed here. + set_local_device(client, 'test-device-' + random_id(), client_id=client_id) + + channel = client.channels.get(channel_name) + + await channel.push.subscribe_client() + + result = await client.push.admin.channel_subscriptions.list( + channel=channel_name, clientId=client_id) + assert len(result.items) >= 1 + assert any( + subscription.client_id == client_id and subscription.channel == channel_name + for subscription in result.items) + + await channel.push.unsubscribe_client() + + result_after = await client.push.admin.channel_subscriptions.list( + channel=channel_name, clientId=client_id) + assert len(result_after.items) == 0 From 096d4fd309bde812baf7d6b7288d190b3351c227 Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 15:23:11 +0100 Subject: [PATCH 06/10] docs: cover the integration tier in the translation skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The skill described a suite where every request came from a mock. It now covers the sandbox-backed tier too: the harness names, the rule that only a specification carrying `## Protocol Variants` takes the protocol fixture, and the traps that came out of deriving the eleven REST integration specifications. Most of those traps produce a test that passes while proving nothing rather than one that fails — a filter the server ignores rather than rejects, a paginated result that is truthy when empty, a guarded assertion that never runs against a fresh app. Each is recorded with the measurement behind it. The timers section now distinguishes three regimes rather than one, since the realtime tier has a clock seam and the integration tier uses real time on purpose. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/uts-to-python/SKILL.md | 169 +++++++++++++++++++++++--- 1 file changed, 152 insertions(+), 17 deletions(-) diff --git a/.claude/skills/uts-to-python/SKILL.md b/.claude/skills/uts-to-python/SKILL.md index b81ed692..cba62713 100644 --- a/.claude/skills/uts-to-python/SKILL.md +++ b/.claude/skills/uts-to-python/SKILL.md @@ -7,24 +7,34 @@ allowed-tools: Bash, Read, Edit, Write, WebFetch ## Sources -Fetch both fresh at the start of every run; do not work from memory. +Fetch the governing doc and the spec fresh at the start of every run; do not work from +memory. ```bash gh api repos/ably/specification/contents/uts/docs/writing-derived-tests.md --jq '.content' | base64 -d +gh api repos/ably/specification/contents/uts/docs/integration-testing.md --jq '.content' | base64 -d gh api repos/ably/specification/contents/uts/rest/unit/.md --jq '.content' | base64 -d gh api repos/ably/specification/contents/uts/realtime/unit/.md --jq '.content' | base64 -d +gh api repos/ably/specification/contents/uts/rest/integration/.md --jq '.content' | base64 -d ``` -`writing-derived-tests.md` governs. This file covers only what is particular to ably-python. +`writing-derived-tests.md` governs, and `integration-testing.md` alongside it for +`uts/rest/integration`. This file covers only what is particular to ably-python. ## Layout A spec at `uts//.md` becomes `test/uts//_test.py`, so -`uts/rest/unit/auth/token_renewal.md` becomes `test/uts/rest/unit/auth/token_renewal_test.py`. +`uts/rest/unit/auth/token_renewal.md` becomes `test/uts/rest/unit/auth/token_renewal_test.py` +and `uts/rest/integration/history.md` becomes `test/uts/rest/integration/history_test.py`. Every directory needs an `__init__.py`, as `test` is a package. -`test/uts/rest/unit/time_test.py` is the reference example for REST, and -`test/uts/realtime/unit/connection/auto_connect_test.py` for realtime. Follow their shape. +There are two kinds of tier. `rest/unit` and `realtime/unit` serve every request from a +mock and reach no network; `rest/integration` runs against the real Ably sandbox and has +no mock at all. See **The integration tier** below. + +`test/uts/rest/unit/time_test.py` is the reference example for REST unit, +`test/uts/realtime/unit/connection/auto_connect_test.py` for realtime, and +`test/uts/rest/integration/history_test.py` for integration. Follow their shape. ## Anatomy of a derived test @@ -81,6 +91,9 @@ async def test_rsc16_time_returns_server_time(): | `install_mock(m)` + `Realtime(options: ...)` | `realtime_client(m, ...)` from `test.uts.helpers.client` | | `AWAIT_STATE client.connection.state == X` | `await await_connection_state(client, ConnectionState.X)` | | `mock_ws.active_connection` | the same, on `MockWebSocket` | +| `Rest(ClientOptions(key: api_key, endpoint: "nonprod:sandbox"))` | `sandbox_rest_client(key)`; `sandbox_realtime_client(key)` for realtime | +| `app_config.keys[i]` / `BEFORE ALL TESTS` app setup | the `sandbox` fixture and `sandbox.key(i)` | +| `poll_until(interval: 500ms, timeout: 10s)` in an integration spec | `await wall_clock_poll_until(condition, description='...')`, **not** `poll_until` | Client options are snake_case throughout. Check the actual signature in `ably/types/options.py` before assuming an option exists. @@ -269,6 +282,57 @@ All in `test.uts.helpers.clock`. | `settle(passes=20)` | `process_pending_events()`: twenty yields, because the realtime paths chain `create_task` several levels deep | | `advance_to_connection_state(client, clock, state, step, limit=60)` | the specifications' `LOOP up to N: ADVANCE_TIME(x)`, for driving the connection to SUSPENDED | +## The integration tier + +`uts/rest/integration/.md` becomes `test/uts/rest/integration/_test.py` and +runs against the real Ably sandbox — eleven specifications, 76 tests. There is no mock +and no `test_options`: the client reaches the network. `test/uts/README.md` covers the +same ground for someone reading the suite; this is what someone writing a test needs. + +What differs from the mock-backed tiers: + +- **Nothing sits in front of the client.** No `install_mock`, no captured request to + assert on, no way to make the server answer a chosen way. A spec point that can only + be shown through a stubbed response belongs in the unit tier. +- **One sandbox app serves the whole session**, standing in for `BEFORE ALL TESTS`. + So every channel name, client id and device id takes a `random_id()` suffix, and + anything a test registers it removes in a `finally`. +- **Waits are wall-clock**, the inverse of the unit tier's rule. See Timers below. +- **The per-test timeout is 120 seconds**, set by the package's own `conftest.py`, not + the 30 `pyproject.toml` gives the rest of the suite. + +| Name | Is | +|---|---| +| `sandbox` fixture, in `rest/integration/conftest.py` | a specification's `app_config`. Session-scoped: provisioned once from the vendored `assets/test-app-setup.json` and deleted afterwards | +| `sandbox.key(i)` | `app_config.keys[i]`, carrying `key_str`, `key_name`, `key_secret` and `capability`. The index means what it means in a spec — 0 full access, 1 push admin, 2 per-channel capabilities, 3 subscribe-only, 4 revocable tokens. `sandbox.key_str` (the full-access key) and `sandbox.app_id` are shorthands | +| `use_binary_protocol` fixture | runs the test once per protocol. **Only a spec carrying a `## Protocol Variants` section takes it** — `publish`, `history`, `presence`, `batch_presence`, `mutable_messages`. A test that does not take it runs json only, which is the clients' default here | +| `sandbox_rest_client(key=None, **kwargs)` | `Rest(ClientOptions(key: api_key, endpoint: "nonprod:sandbox"))`. Registered for the same teardown as `rest_client`. Leave `key` out and pass `token=`, `auth_callback=` or `auth_url=` where the spec authenticates some other way | +| `sandbox_realtime_client(key=None, **kwargs)` | the same for realtime, for the REST specs that need presence members or presence history a connection has to produce. Unlike `realtime_client` it keeps `auto_connect` and the fallback hosts at the **library** defaults | +| `wall_clock_poll_until(condition, timeout=10.0, description='condition', interval=0.5)` | this tier's `poll_until`. Sleeps `interval` between attempts, takes a sync or async condition, and **returns whatever the condition answered with**, so a condition that fetches a page saves fetching it again | +| `random_id(length=6)` | the specifications' `random_id()`, url-safe base64 over `secrets` bytes | +| `fixture_cipher_params()` | the `CipherParams` the app setup encrypted the `client_encoded` presence fixture with. The asset holds key and IV base64; this decodes them | +| `PRESENCE_FIXTURES_CHANNEL` | `'persisted:presence_fixtures'`, the channel the app setup pre-populates with members the presence specs read rather than write | +| `generate_jwt(key_name, key_secret, ttl=3600000, client_id=None, capability=None, expires_at=None)` | the auth specification's `generate_jwt`. Signed HS256 here rather than pulling in a JWT library the locked environment does not carry | +| `extract_key_name(api_key)` / `extract_key_secret(api_key)` | the two halves of `app_id.key_id:secret` | + +`sandbox_rest_client` and `sandbox_realtime_client` are in `test.uts.helpers.client` +alongside the mock-backed constructors; everything else is in `test.uts.helpers.sandbox`. + +```python +# UTS: rest/integration/RSL2a/history-returns-messages-0 +async def test_rsl2a_history_returns_messages(sandbox, use_binary_protocol): + client = sandbox_rest_client(sandbox.key_str, use_binary_protocol=use_binary_protocol) + channel = client.channels.get('history-test-RSL2a-' + random_id()) + + await channel.publish(name='event1', data='data1') + + async def one_message(): + page = await channel.history() + return page if len(page.items) == 1 else None + + history = await wall_clock_poll_until(one_message, description='the message to reach history') +``` + ## Traps that cost the most time Ordered by how much they cost, not by subject. Every one was hit for real while @@ -477,20 +541,85 @@ Infrastructure Limitation. RTN23a via `send_to_client(HEARTBEAT_MESSAGE)` works. `log_handler` among them — and raise `TypeError` rather than being ignored. Check `ably/types/options.py` first. +## Traps found while deriving the REST integration specs + +Each was established by measurement against the sandbox. Most of them produce a test +that **passes while proving nothing**, rather than one that fails. + +- **Push admin filter parameters must be camelCase, and the server drops an + unrecognised query parameter rather than rejecting it.** Measured against one app: + `device_registrations.list(clientId=x)` returned 2, `list(client_id=x)` returned 3, + and the unfiltered list returned 3 — the snake_case filter was silently the whole + page. A test that asserts only that the row it just created is present therefore + passes with the filter doing nothing, so **a filtered list needs a control proving it + narrowed**: a decoy row under another id, or an unfiltered count to compare against. + The cause is that `list`, `list_channels` and `DeviceRegistrations.remove_where` hand + their dict to `format_params` positionally, and `format_params` camel-cases only its + own `**kw` (`ably/http/paginatedresult.py:18`). + `PushChannelSubscriptions.remove_where` is the one call that spreads + (`format_params(**params)`), so it does accept snake_case — do not generalise from it. +- **A `PaginatedResult` is always truthy and defines no `__len__`.** So a poll + condition that answers with the page straight from `history()` or `list()` is + satisfied by the first empty one and the assertions then run against nothing. Return + `None` until the page holds what is wanted, and read `len(page.items)`. `has_next()` + is a method too, and a bound method is truthy whatever the page holds. +- **Nothing is consistent immediately after a write.** History and presence lag a + publish or an enter, and device deletion is asynchronous — `remove_where` answers 204 + while the rows are still listed. Any count that follows a write goes through + `wall_clock_poll_until`; a fixed sleep either flakes or spends the budget. +- **`Message.timestamp` is a raw int of milliseconds, `PresenceMessage.timestamp` is a + `datetime`.** `PresenceMessage.from_dict` converts and `Message` does not, so a + history time boundary is integer arithmetic, a presence one is not, and comparing the + two raises. Derive a boundary from server-assigned timestamps rather than from a + client-side `now()`, which can land inside the same millisecond as the messages. +- **A channel captures its cipher and its protocol when it is constructed**, and + `Channels.get` caches by name. A cipher passed on a later `get` reaches + `channel.cipher` but never `channel.presence`, which snapshotted it in + `Presence.__init__` (`ably/types/presence.py:207`). Pass the cipher on the **first** + `get` for that client. +- **A fresh sandbox app has no stats.** A spec guarding its assertions on there being + stats to read is vacuous against a new app — the guarded branch never runs and the + test asserts nothing. Inject an interval first through the sandbox's own + `POST /stats`, as `time_stats_test.py`'s `app_with_stats` fixture does. +- **An Ably JWT's lifetime is read as `exp - iat`, and a negative one is rejected 40003 + before expiry is ever considered.** An already-expired JWT cannot be made by leaving + `iat` at now and putting `exp` in the past; that is a malformed token, not an expired + one, and a renewal test would be exercising the wrong rejection. `generate_jwt` + backdates `iat` by `ttl` when given `expires_at`, for exactly this. +- **`enter_client` fails on an anonymous connection.** The server answers basic auth + with `clientId: "*"`, `Auth._configure_client_id` records that as validated while + leaving the client id `None` (`ably/rest/auth.py:335`), and `can_assume_client_id` + then refuses every id with 40012. Pass `client_id='*'` to `sandbox_realtime_client`. + Await CONNECTED before entering, too — an enter on a CONNECTING connection is queued. +- **Waits here are wall-clock, the inverse of the unit tier's rule.** `poll_until` + yields to the event loop, which against a real server spins a core on a network wait. + Use `wall_clock_poll_until`, and no `FakeClock`. `writing-derived-tests.md` has a + section on this ("Integration timeouts are wall-clock"). + ## Timers -The realtime client has a timer seam; the REST client does not. -`ably/http/http.py` calls `time.time()` directly. Where a REST spec calls -`enable_fake_timers()` / `ADVANCE_TIME(ms)`, prefer short real timeouts driven by client -options (`fallback_retry_timeout=100`), which is what the specs themselves do. The global -pytest timeout is 30 seconds, so keep waits well under it. +Three regimes; pick by tier. -On a realtime client, prefer a short real interval through a client option +**REST unit.** No clock seam reaches it: `TestOptions(timer=...)` is read by the +realtime connection alone, and `ably/http/http.py` calls `time.time()` directly. Where +a REST spec calls `enable_fake_timers()` / `ADVANCE_TIME(ms)`, drive it with short real +timeouts through client options (`fallback_retry_timeout=100`), which is what the specs +themselves do. + +**Realtime unit.** The `timer` seam exists, and `realtime_client(mock, clock=clock)` +installs a `FakeClock` on it. Still prefer a short real interval through a client option (`realtime_request_timeout`, `disconnected_retry_timeout`, `suspended_retry_timeout`, `channel_retry_timeout`) or through `connected_message(maxIdleInterval=...)`, and reach for `FakeClock` only for `connection_state_ttl`, which no option sets and whose default costs 120 real seconds. See the fake-time section of `test/uts/deviations.md`. +**REST integration.** Real time, deliberately. There is no seam to install in front of +a server the tests exist to talk to, and shortening a timeout would only make the tier +flaky. Poll with `wall_clock_poll_until` rather than sleeping a guess. + +The pytest timeout is 30 seconds for the suite and 120 for `rest/integration`, so keep +waits well under whichever applies. + ## Deviations Diagnose per the decision tree in `writing-derived-tests.md`, then apply one of: @@ -539,7 +668,8 @@ the reasoning. The next reader will otherwise reach the same first conclusion. ```bash uv run --frozen --extra crypto --extra dev ruff check ably/ test/ -uv run --frozen --extra crypto --extra dev pytest test/uts -q +uv run --frozen --extra crypto --extra dev pytest test/uts/rest/unit test/uts/realtime/unit test/uts/helpers -q +uv run --frozen --extra crypto --extra dev pytest test/uts/rest/integration -q RUN_DEVIATIONS=1 uv run --frozen --extra crypto --extra dev pytest test/uts -q ``` @@ -547,8 +677,13 @@ RUN_DEVIATIONS=1 uv run --frozen --extra crypto --extra dev pytest test/uts -q environment's cutoff — and `--extra dev` carries pytest. Line length is 115. If `uv.lock` changes, `git checkout -- uv.lock`. -The first two must pass. The third is the check that the deviations record is still -true: **every gated test must fail when enabled**, so gated + unimplementable under the -third run must equal the skip count under the second, and nothing may pass under both -behaviours. The expected counts are in the header of `test/uts/deviations.md`; update -them from a measured run rather than copying them forward. +The second command is the offline tiers, which need no network. The third provisions a +sandbox app and does; `pytest test/uts -q` runs both together, so run the tiers +separately when only one is in question. All three must pass. + +The fourth is the check that the deviations record is still true, across both tiers: +**every gated test must fail when enabled**, so gated + unimplementable under it must +equal the skip count under the other runs, and nothing may pass under both behaviours. +It reaches the network for the same reason the third does. The expected counts are in +the header of the deviations record; update them from a measured run rather than +copying them forward. From 1836c88502c3e6030eef7c2671a5d8c44a9a6e1f Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 15:51:56 +0100 Subject: [PATCH 07/10] docs: record the integration deviations and the specification faults found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The eleven REST integration specifications add eleven gated Test IDs to the record. Ten of them land on root causes the unit tiers already found — no Auth#revokeTokens, no Rest#batchPresence, no PushChannel surface, no clientId filter on RestPresence#get — so those entries grow a tier rather than gaining a twin. Both halves now gate on the same spelling, and go green together. Two SDK defects are new. Rest#request never renews an expired token: it is the only call site passing raise_on_error=False, so the HTTP layer returns the 401 instead of raising and the reauthorise-and-retry branch never runs, while the pre-emptive check is separately inert without a time offset. And a basic-auth connection records its clientId as validated and None, so enterClient can never match; every test that needs presence on an anonymous connection passes '*' around it. Four specification faults are recorded and not filed: a device identity token push_channels.md hard-codes that the server rejects, two tests that close the realtime connection the following read depends on, and a time-range test whose assertions hold with the range dropped. The header now separates Test IDs, derived tests and pytest cases, which the integration tier is the first to make diverge in both directions. Co-Authored-By: Claude Opus 5 (1M context) --- test/uts/deviations.md | 498 ++++++++++++++++-- .../rest/integration/batch_presence_test.py | 5 +- test/uts/rest/integration/presence_test.py | 5 +- test/uts/rest/integration/push_admin_test.py | 5 +- .../rest/integration/push_channels_test.py | 4 +- .../rest/integration/revoke_tokens_test.py | 8 +- 6 files changed, 481 insertions(+), 44 deletions(-) diff --git a/test/uts/deviations.md b/test/uts/deviations.md index 309ac227..5076aba5 100644 --- a/test/uts/deviations.md +++ b/test/uts/deviations.md @@ -15,19 +15,40 @@ issues, ranked, for a maintainer deciding what to file; and **How the specificat are adopted here**, which records the choices behind the harness rather than the behaviour. -Of 1017 derived tests, 811 pass, 191 are gated behind `RUN_DEVIATIONS` and 15 -cannot be run at all. Every gated test has been confirmed to fail when enabled, so -none of them passes under both behaviours. 536 of the derived tests come from -`uts/rest/unit` and 481 from `uts/realtime/unit`; of the gated tests 110 are REST -and 81 realtime. A further 122 tests under `helpers/` cover the mock infrastructure -itself and are not derived from a specification. - -The 181 gated tests that record SDK non-compliance reduce to **65 distinct root -causes** — 25 on the REST side and 40 on the realtime side. Two further realtime +Three counts differ here, and every figure below says which of them it is. A **Test +ID** is the specification's own identifier for a test, carried in a `# UTS:` comment. A +**derived test** is a test function written under one. A **pytest case** is one run of +one function. They diverge in both directions. + +One Test ID can become more than one derived test: five Test IDs in `rest/unit` — in +`error_types_test.py`, `fallback_test.py`, `rest_client_test.py` (two) and +`paginated_result_test.py` — assert several independent things under a single id, and +the derivation writes a function for each rather than one function with an unrelated +second half. That turns 1051 Test IDs into 1060 derived tests. Going the other way, one +derived test can become more than one case: five of the eleven `rest/integration` +specifications carry a `## Protocol Variants` section and run every one of their tests +twice, once per protocol, and nine `rest/unit` tests are parametrized over a table of +fixtures the specification gives inline. That turns 1060 derived tests into 1131 pytest +cases. + +Of **1051 Test IDs, derived as 1060 tests and run as 1131 pytest cases**: 834 Test IDs +(843 tests, 910 cases) pass, 202 (202 tests, 206 cases) are gated behind +`RUN_DEVIATIONS`, and 15 (15 tests, 15 cases) cannot be run at all. Every gated test has +been confirmed to fail when enabled, so none of them passes under both behaviours. 494 +of the Test IDs come from `uts/rest/unit` (503 tests, 536 cases), 481 from +`uts/realtime/unit` (481, 481) and 76 from `uts/rest/integration` (76, 114); of the +gated Test IDs 121 are REST and 81 realtime, which is 125 REST cases and 81 realtime. +A further 122 pytest cases under `helpers/` cover the mock infrastructure itself and are +not derived from a specification. + +The 192 gated Test IDs that record SDK non-compliance — 192 tests, 196 cases — reduce to +**66 distinct root causes**, 26 on the REST side and 40 on the realtime side. Three further defects are recorded below with no test of their own, because the specification's test -for each cannot discriminate (RTP18a) or has nothing to assert against (the timezone -split on synthesized LEAVE timestamps), so the file carries **67 SDK root causes** in -all. The remaining 10 gated tests are specification faults, and reduce to 7. +cannot discriminate (RTP18a), has nothing to assert against (the timezone split on +synthesized LEAVE timestamps), or is worked around in the setup of every test that +would otherwise trip over it (`enterClient` on an anonymous connection), so the file +carries **69 SDK root causes** in all. The remaining 10 gated Test IDs are +specification faults, and reduce to 7. Entries closed by a fix are removed rather than kept as history; `git log` holds that. @@ -102,8 +123,11 @@ same housekeeping categories in `realtime/unit`, and RSA4c3 contradiction, since that issue is what decides it. Not every entry has an issue of its own: the URL-safe base64 alphabet is recorded below -and not filed, because ably-python's own encoding settles the tests either way. Line -references in these entries are against `ably/specification@d9a04ca`. +and not filed, because ably-python's own encoding settles the tests either way. Neither +are the faults in `uts/rest/integration`, which have no issue numbers against them. Each +of those is of the second kind — a fixture, a setup step or a header label — so the +derived test keeps the corrected fixture and passes, and none of them is among the ten +gated above. Line references in these entries are against `ably/specification@d9a04ca`. ### `/time` is stubbed as an object rather than an array @@ -230,6 +254,18 @@ claim is the one to revisit. `revoke_tokens.md` has the same internal split: `Content-Type: application/json`; CSV2b templates the version, the sibling spec says ">= 3", and the binary protocol default makes the content type msgpack. +The revocation half of that split is settled by the server. `POST +/keys/{keyName}/revokeTokens` with `X-Ably-Version: 5` answers 201 with the +`{successCount, failureCount, results}` envelope `revoke_tokens.md`'s "Server Response +Format" section describes, for a mixed success/failure batch as well as an all-success +one, so it is the two mocks that stub a bare array that are wrong rather than the +assertions that read the envelope. Two further details of that response were measured +at the same time: an invalid target type comes back as code **40001** where the +specification's example writes 40000, though only the status code is asserted either +way; and `issuedBefore` is echoed unchanged while `allowReauthMargin: true` pushes +`appliesAt` about 30.1 seconds past server time (30089 ms in one run), which is what +RSA17e's two assertions need. + ### Two presence specifications contradict themselves over the wildcard clientId The same contradiction appears twice, mirrored, and the two should be settled together. @@ -468,6 +504,85 @@ the path, or drop the reference and keep the inline cases as the definition. | `connection_auth_test.md`, `auth_callback_errors_test.md` | RSA4c2 is the same test in both files: `callback-error-causes-disconnected-0` and `callback-error-connecting-disconnected-0` have the same authCallback, the same mock and the same four assertions, and the second adds only `useBinaryProtocol: false` and a `state_changes` listener. The closing note of `auth_callback_errors_test.md` acknowledges the overlap without removing it. Both are derived, since each has its own Test ID | | `channel_properties.md` | `RTL15b/serial-not-updated-irrelevant-3`'s closing comment reads "RTL15b2 clears it on DETACHED/FAILED, then ATTACHED sets it fresh". The DETACHED it injects arrives while the channel is ATTACHED, so RTL13a reattaches and the DETACHED *state* is never entered. Nothing clears the serial; it is simply never written from the DETACHED message. The assertion the comment sits above is still the right one | +### `push_channels.md` hard-codes a device identity token the server rejects + +**Spec points:** RSH7a, RSH7c, `rest/integration/RSH7a/subscribe-unsubscribe-device-0`. + +The setup's own comment says "The deviceIdentityToken is obtained from the registration +response", and the pseudocode immediately beneath it writes +`deviceIdentityToken: "test-device-identity-token"`. The comment is right and the code is +not. RSH7a2 and RSH7c2 authenticate as the device, and the server refuses a token it did +not issue — `POST /push/channelSubscriptions` with `X-Ably-DeviceToken: +test-device-identity-token` answers 400/40005, "Invalid accessToken in request". The real +one comes back from `PUT /push/deviceRegistrations/{id}`, under `deviceIdentityToken` as +an object of `{token, keyName, issued, expires, capability}`, and its `token` is accepted: +the same subscribe and unsubscribe answer 201 and 204. + +`test_rsh7a_subscribe_unsubscribe_device` takes the issued token, through +`issued_device_identity_token(registration)`, and would fail 40005 whatever the SDK did +if it took the literal. `test_rsh7b_subscribe_unsubscribe_client` keeps the placeholder, +because RSH7b2 and RSH7d2 subscribe by clientId and neither sends `X-Ably-DeviceToken` — +the unit tier pins that, and the server accepts the clientId subscription on the client's +ordinary credentials. + +### Closing the realtime client destroys the presence the following REST read is about + +**Spec points:** RSC24 and BGF2 (`batch_presence.md`, +`rest/integration/RSC24/restricted-key-channel-failure-1`); RSP4b2 (`presence.md`, +`rest/integration/RSP4b2/history-direction-forwards-0`). + +Two specifications put `AWAIT realtime.close()` between the presence operations that +create their fixture and the REST read that asserts on it, and in both the close is what +breaks the assertion. + +`batch_presence.md`'s restricted-key test enters `member-1` on the allowed channel and +`member-2` on the denied one, closes the realtime client, and then requires +`success.presence.length == 1` with `success.presence[0].clientId == "member-1"`. A +presence member belongs to the connection that entered it, so closing the connection takes +it away. Measured against the sandbox: after the close, `GET +/presence?channels=channel6,denied-…` answers `{"successCount": 1, "failureCount": 1, +"results": [{"channel": "channel6"}, {…"error": {"code": 40160, "statusCode": 401}}]}` +every time, the allowed channel carrying no `presence` at all. The other two tests in the +same file get it right and say so — "Keep realtime open during the REST query so the +presence member persists on the server." + +`presence.md`'s RSP4b2 closes the connection while the member is still present, which +produces a LEAVE carrying no data, and then reads +`history(direction: "backwards").items[0].data == "third"` — which now reads the +synthesized LEAVE rather than the last update. + +In both places the close belongs in cleanup, as the sibling tests put it. The derived +tests omit it and leave the suite's autouse teardown to close every client a test built, +which is what the specifications' own cleanup steps amount to; the five `presence.md` +tests that generate presence events do the same. Every assertion is the specifications', +unchanged. + +### RSL2b3's assertions cannot detect an ignored time range + +**Spec point:** RSL2b3, `history.md`, `rest/integration/RSL2b3/history-time-range-0`. + +The test publishes two "early" messages, waits 2 ms, publishes two "late" ones, computes a +boundary from the server-assigned timestamps, and queries twice — once from before the +early batch up to the boundary, once from just after the boundary to beyond the late +batch. Its four assertions are that each page is non-empty, that the early page contains a +name beginning `early`, and that the late page contains one beginning `late`. + +None of those discriminates. A client or a server that dropped `start` and `end` entirely +would answer both queries with all four messages, and every assertion would still hold. +The test exists to show that the range filters, and it passes when the range is ignored. +What discriminates is the converse — that each window *excludes* the other batch. The +sandbox does filter exclusively: the early window returns `early2, early1` and the late +window `late2, late1`, and widening the early query's `end` to `min_late_ts + 1000`, which +is the mutation an ignored `end` amounts to, is caught only by the exclusion check. + +`test_rsl2b3_history_time_range` carries the specification's four assertions verbatim and +then the two exclusion assertions it omits, under a `# UTS SPEC ERROR:` comment at the +site. It also asserts `min_late_ts > max_early_ts` first, which is the premise the +specification's own 2 ms wait exists to establish and which its boundary arithmetic +depends on: with both batches inside one millisecond there is no side of the boundary to +put them on, and the test should fail on the stated premise rather than on an exclusion +that cannot hold. It passes. + ### Smaller faults | Spec | Fault | @@ -481,6 +596,9 @@ the path, or drop the reference and keep the inline cases as the definition. | `fallback.md` | REC3a, REC3b and REC3 drive a Realtime client but sit in `rest/unit` | | `message_encoding.md`, `msgpack_interop.md`, `annotations.md` | Six sections carry no Test ID; ids were inferred by sibling convention | | `publish.md`, `rest_presence.md`, `message_encoding.md`, `history.md`, `idempotency.md` | All point at `/Users/paddy/data/worknew/dev/dart-experiments/...` for the mock contract | +| `publish.md` (integration) | The `Spec points:` header reads RSL1d, RSL1l1, RSL1m4, RSL1n, and the file carries a fifth section, `## RSL1k5 - Idempotent publish with client-supplied IDs`, with its own Test ID. The section is sound; only the header is short. Same housekeeping class as [#532](https://github.com/ably/specification/issues/532) | +| `auth.md` (integration) | RSC10's expired-JWT fixture is `generate_jwt(expires_at: now() - 5_seconds)`, naming `exp` and leaving `iat` open. Ably reads a JWT's lifetime as `exp - iat` and rejects a negative one with 400/40003 "Invalid value for ttl" before it considers expiry, so `iat` at now produces a token that fails the wrong way and never reaches the 40140–40149 renewal path the test is about. Backdating `iat` past `exp` gives the already-expired token the test wants, answered 401/40142. An SDK signing its own Ably JWT has to choose, so the fixture should say which | +| `batch_presence.md` | BGR2 says a channel with no members "returns a success result with an empty `presence` array", and the unit tier's mocks all send `'presence': []`. The server sends no `presence` key at all, so an implementation has to default the field for the assertion to hold. The derived test asserts the specification's `length == 0`, with the wire shape in a comment | ## Failing Tests @@ -489,21 +607,35 @@ the mark is the only change needed once the SDK behaviour lands. ### Unimplemented features -Nothing to fix here, only something to build. Each row is one feature, and the test -count is the number of gated tests that fall with it. - -| Spec points | Missing | Tests | +Nothing to fix here, only something to build. Each row is one feature, and the count is +the number of gated Test IDs that fall with it, with the pytest case count beside it +where the two differ. + +Four of these rows are gated at both tiers. `batchPresence`, `Auth#revokeTokens`, the +`PushChannel` surface and the `clientId` filter on `RestPresence#get` each carry +`uts/rest/integration` tests as well as unit ones, written against the spelling the unit +tier already gates on, so both tiers go green together when the API lands. Those +integration tests do all their real work first — the sandbox app, the channels, the +presence members entered over a realtime connection, the registered device and the +issued token are all real, and each test reaches the missing call before it fails, so the +assertions either side of it are known to hold against real server responses. The +`batch_presence` and `push_channels` files were additionally run against throwaway shims +— a `batch_presence` forwarding to `GET /presence`, and a `PushChannel` posting and +deleting `/push/channelSubscriptions` with `X-Ably-DeviceToken` — and pass in full +against them. + +| Spec points | Missing | Test IDs | |---|---|---| -| RSC22, RSC24, BSP2, BPR2, BPF2, BAR2, BGR2, BGF2 | `batchPublish` and `batchPresence`, and all six result types. `grep -rn batch ably/` finds nothing | 41 | -| RSA17, RSA17b–g, BAR2, TRS2, TRF2 | `Auth#revokeTokens`, `TokenRevocationTargetSpecifier`, `BatchResult` | 17 | -| RSH7, RSH7a–e, RSH6, RSH8 | `PushChannel`: `channel.push`, `client.device`, `LocalDevice`. The push *admin* surface (RSH1) does exist | 10 | +| RSC22, RSC24, BSP2, BPR2, BPF2, BAR2, BGR2, BGF2 | `batchPublish` and `batchPresence`, and all six result types. `grep -rn batch ably/` finds nothing | 44 (47 cases) | +| RSA17, RSA17b–g, BAR2, TRS2, TRF2 | `Auth#revokeTokens`, `TokenRevocationTargetSpecifier`, `BatchResult`. Gated against `auth.revoke_tokens(targets, issued_before=, allow_reauth_margin=)` returning `success_count` / `failure_count` / `results`, with `target` / `issued_before` / `applies_at` / `error` per result. RSA17d is the one case that needs no server at all — a token-authenticated client must refuse locally with 40162/401 — so it can be satisfied before any of the wire work | 21 | +| RSH7, RSH7a–e, RSH6, RSH8 | `PushChannel`: `channel.push`, `client.device`, `LocalDevice`. The push *admin* surface (RSH1) does exist | 12 | | RTN16, RTN16f–k, RTC1c (TO3i) | Connection recovery, entire. `recover` is in the `Options` signature, stored, and given a property and a setter (`options.py:30,111,193,196`), and read nowhere. No `Connection#createRecoveryKey`, no `recover` connect parameter, no recovery-key decoding | 6 | | RTL22, RTL22a–d, MFI1, MFI2a–e | `MessageFilter`. `RealtimeChannel.subscribe` (`channel.py:262-273`) accepts only a `str` or a callable, and there is no filter type of any shape to spell. Each test builds its filter through the module's `message_filter()` helper, which is the one place to repoint when the type lands | 5 | | RTS5, RTS5a, RTS5a1, RTS5a2, DO2a | Derived channels: `DeriveOptions` and `Channels.getDerived`. `grep -r derive ably/` is empty. Each test imports `DeriveOptions` inside its body so the module still loads | 5 | | RTB1, RTB1a, RTB1b | Retry backoff, jitter and `retryIn`. Retry timers schedule the flat configured timeout (`connectionmanager.py:753`, `channel.py:866-871`); `grep` for jitter/backoff/retry_in returns nothing, and neither `ConnectionStateChange` nor `ChannelStateChange` carries `retryIn` | 4 | | RTL25, RTL25a, RTL25b | `RealtimeChannel#whenState`. `Connection._when_state` exists (private, awaitable), so this is a gap on the channel rather than a house style; the tests are written against a `channel.when_state(state)` matching the shape the connection already has | 4 | | RSC2, RSC3, RSC4, TO3b, TO3c, TO3c2 | `log_handler` as a client option, and any use of `log_level` — it is stored on `Options` and read by nothing | 4 | -| RSP3a2, RSP3a3 | `clientId` and `connectionId` filters on `RestPresence#get`. `Presence.get` takes only `limit`, while `Presence.history` does take its documented params | 3 | +| RSP3a2, RSP3a3 | `clientId` and `connectionId` filters on `RestPresence#get`. `Presence.get` is `get(self, limit=None)` (`ably/types/presence.py:216`), while `Presence.history` does take its documented params. `presence.get(client_id=...)` raises `TypeError: get() got an unexpected keyword argument 'client_id'`. The absence also forces the RSP5 decoding adaptation below | 4 (5 cases) | | TP3a, TP3d, TP3g | Presence attributes defaulted from the encapsulating ProtocolMessage. There is no ProtocolMessage type; `ably/realtime/channel.py:751-761` passes the presence array through without context. Matters for synthesized-leave detection and `memberKey` | 3 | | TB4, RTL7h, RTP6e | `attachOnSubscribe`. `ChannelOptions.__init__` (`channeloptions.py:22-26`) takes only `cipher`, `params` and `modes`, and `subscribe()` on the channel, on presence and on annotations all end unconditionally with `await attach()`. This absence also forces the largest single adaptation in the suite, below | 3 | | RSL7 | `RestChannel#setOptions`. The realtime channel implements it; the REST `options` setter expects the kwargs dict `Channels.get` collected, so a `ChannelOptions` raises `TypeError` | 2 | @@ -1145,6 +1277,84 @@ tests pass. **Status:** open bug. +#### `Rest#request` never renews an expired token — 1 test + +**Spec points:** RSC10, and RSC19 for the path it is observed on. + +RSC10 requires a REST request that fails with a token error (40140–40149) to have its token +renewed and the request retried. That happens on every REST operation except the one +`rest/integration/auth.md` drives the test through: `Rest#request`. + +`Http.make_request` is wrapped by `reauth_if_expired` (`ably/http/http.py:19-42`), which +renews on two triggers. The pre-emptive one is inert for a token-authenticated client: +`Auth.token_details_has_expired()` returns `False` whenever `time_offset` is unset +(`ably/rest/auth.py:139-140`), and the offset is only ever set by `query_time`, so an +`authCallback` client never has one. That leaves the reactive trigger, which fires on a +raised `AblyException`. `AblyRest.request` asks for `raise_on_error=False` +(`ably/rest/rest.py:145`) so that its `HttpPaginatedResponse` can report an error status to +the caller, per RSC19e and RSC19d3; `make_request` therefore skips +`AblyException.raise_for_response` (`http.py:233-234`) and returns the 401 as a `Response`. +Nothing raises, so the reactive branch never runs — the `authCallback` is invoked once, the +expired token is sent, and the 401 reaches the caller. The two requirements are in direct +conflict in the code as it stands: the flag that suppresses the exception also suppresses +the renewal. + +`Rest#request` is the only call site in the library that passes `raise_on_error=False`, and +this is specific to it rather than a general failure of RSC10. Confirmed on one client in +one run: given the same expired JWT, `channel.publish()` renews correctly — the callback is +invoked twice and the client ends up holding the second JWT — while `client.request()` +invokes the callback once and returns 401/40142. + +**Tests affected:** `test_rsc10_token_renewal_expired_jwt`, which fails on the +specification's `result.statusCode >= 200 AND < 300` as `assert 401 < 300`; the two +assertions after it, `callback_count == 2` and the check that the client holds the renewed +JWT, fall with it. + +**Status:** open bug. A fix has to separate the two meanings `raise_on_error` carries — +renew and retry on a token error whatever the flag says, and only then decide whether the +final response is raised or returned. + +#### A basic-auth connection records its clientId as validated and `None`, so `enterClient` can never succeed — no test + +**Spec points:** RSA7b4, RTP14, RTP15. + +A client built from a full-access key alone and asked to `enter_client("user-1", …)` +raises `AblyException: 40012 400 Unable to enter presence channel with clientId user-1 as +it does not match the current clientId None`. + +The server tells such a connection `clientId: "*"` in CONNECTED, and +`Auth._configure_client_id` (`ably/rest/auth.py:335-353`) opens with a branch that exists +to stop a server wildcard overwriting a clientId the caller configured: + +```python +if original_client_id != '*' and new_client_id == '*': + self.__client_id_validated = True + self.__client_id = original_client_id + return +``` + +With no configured clientId, `original_client_id` is `None`, `None != '*'` holds, and the +client id is recorded as **validated and `None`**. `can_assume_client_id` +(`auth.py:356-363`) then takes the validated path and answers `None == '*' or None == +assumed`, which is `False` for every clientId, so `enter_client` cannot succeed for any of +them. The `original_client_id is None` escape in the unvalidated branch is unreachable once +CONNECTED has arrived. The branch should apply only where a clientId was configured, and +`__client_id` should become `'*'` otherwise, which is what RSA7b4 asks for. + +**Tests affected:** none, which is why this is recorded here rather than gated. The three +`batch_presence.md` tests build their realtime client with `client_id='*'` instead, in +`entering_client()`, with the diagnosis in its docstring — the repository's own presence +suite carries the same workaround for the same reason, commented "Use wildcard auth for +enterClient" (`test/ably/realtime/realtimepresence_test.py:394-396`). It is setup rather +than subject: those tests are about `batchPresence`, and nothing they assert depends on how +the members got there. A fix would let the setup halves be written exactly as the +specification writes them. + +This is distinct from the wildcard-clientId contradiction recorded under UTS Spec Errors, +which is about a client that *does* configure `clientId: "*"`. + +**Status:** open bug, one line from a fix. + #### Auth behaviour on the REST client | Spec points | Behaviour | @@ -1202,6 +1412,7 @@ through an internal object to get at a value the specification makes public. | RTP19, and the `Interface Under Test` block of `presence_sync.md` | `endSync() -> List`, the synthesized LEAVEs | `end_sync()` returns `(residual, absent)` of the *stored* members; the synthesis lives one level up in `RealtimePresence.set_presence` (`presence.py:575-587`). Tests reading only counts and clientIds concatenate the two lists exactly as `set_presence` does; tests reading the LEAVE itself drive a `RealtimePresence` and assert on what its subscribers receive | 4 in `presence_sync_test.py` | | TB2, RTS3b, RTS3c, RTS3c1, RTL16 | `channel.options` as a `ChannelOptions` | a dict keyed by wire names, because `RealtimeChannel` passes `ChannelOptions.to_dict()` to the REST `Channel` constructor (`channel.py:84`). Assertions read `channel.options['params']['rewind']`. On `ChannelOptions` itself the cipher attribute is spelled `cipher`, not `cipherParams`. `set_options_without_reattach` replaces the stored mapping wholesale rather than merging, which `test_rts3c_options_updated_existing` pins | 5 | | RTS2, RTS4a | `channels.exists(name)`, `channels.names`, and an awaitable `release()` | `name in client.channels` (`Channels.__contains__`); the collection iterates over its channels rather than their names; `release` is synchronous. Genuinely idiomatic spelling rather than an absence — recorded only because of the `__getattr__` hazard noted below | 4 | +| RSH1b1, RSH1b2, RSH1b3, RSH1b4, RSH1b5, RSH1c3 | `DevicePushDetails`. The specification builds every device as `DeviceDetails(…, push: DevicePushDetails(recipient: {…}))`; ably-python has no such type | `DeviceDetails.__init__` takes `push` as a plain dict and stores it unchanged (`ably/types/device.py:10-40`), and `DeviceDetails.push` hands that dict back, so the tests read `{'recipient': {…}}` directly. The recipient's `transportType` is still validated against `DevicePushTransportType` in the constructor, which is the only part of `DevicePushDetails` carrying behaviour | the 7 in `push_admin_test.py` that register a device, through its `apns_device()` helper | **Status:** open bugs of the missing-API kind, not of the wrong-behaviour kind. Adding the accessors would leave every assertion above unchanged; only the spelling would move. @@ -1404,7 +1615,30 @@ DISCONNECTED, citing RSA4a2, and ably-python matches that one — `test_rtn15h1_token_error_no_renew` asserts 40171/403 with the specification's expectation in a comment. +`revoke_tokens.md` reaches the same code from a third direction, and is recorded here +rather than as an entry of its own. Its "Verification Strategy" watches a realtime client +built as `Realtime(ClientOptions(token: token_details))` — a token and nothing else — and +asserts a DISCONNECTED whose `reason.code` is 40141. The server does exactly what the +specification describes: revoking the token pushes `{'action': 6, 'error': {'message': +'token revoked', 'code': 40141, 'statusCode': 401}}`. ably-python reads that as a token +error (RTN14b, `connectionmanager.py:474`), `on_token_error` (`:456`) tries +`_ensure_valid_auth_credentials(force=True)`, a client holding only a `TokenDetails` has no +way to obtain another, and the 40171 displaces the server's 40141 as above. What differs is +that no DISCONNECTED is emitted at all: the observed result is a single state change, +`connected -> failed`, carrying 40171/403. Where the client *can* renew, `on_token_error` +reaches `notify_state(DISCONNECTED, exception, retry_immediately=True)` (`:464`) and does +report DISCONNECTED with the server's 40141, so the divergence comes from the +specification's token-only setup meeting RSA4a rather than from anything about revocation. +`test_rsa17g_revoke_token_prevents_use` and `test_rsa17c_mixed_success_failure` assert the +FAILED state change and its 40171/403 with the specification's expectation in a comment; +both are also gated for the absent `revoke_tokens`, so those assertions cannot run until +the API lands. The behaviour was established outside the suite, with a throwaway script +that provisioned a sandbox app, issued a token, connected a realtime client with it and +POSTed to `/keys/{keyName}/revokeTokens` directly, reproduced three times. + **Status:** arguably correct as it stands; the specifications should be reconciled first. +RSA4a does not say which state a token error observed mid-connection should be reported in, +and other SDKs may report DISCONNECTED first and fail afterwards. ### `errorReason` is not cleared by a successful reconnect @@ -1482,6 +1716,27 @@ assertions out of the run for one missing option. **Status:** the adaptation stands until RTL7h is implemented. +### Adaptations forced by the absent `clientId` filter on `RestPresence#get` — 3 Test IDs, 6 cases + +**Spec point:** RSP5. + +Three of the four RSP5 decoding tests in `rest/integration/presence.md` open with +`presence.get(clientId: "client_")` and then assert `items.length == 1`. The filter +is incidental to what each test is about — it selects one of the six pre-populated fixture +members so the decoded `data` can be asserted — so rather than gate three decoding tests +behind the missing parameter recorded under Failing Tests above, each fetches the whole +member set and selects in Python through the file's `member_for(page, client_id)` helper. +The `items.length == 1` assertion becomes `member is not None`, with the specification's +expectation in a comment; every assertion about `data` is the specification's, unchanged. + +`RSP3a2/get-with-clientid-filter-0` is *about* the filter, so it is gated rather than +adapted. + +**Tests affected:** `test_rsp5_decode_string_data`, `test_rsp5_decode_json_data` and +`test_rsp5_decode_encrypted_data`, each run under both protocols. + +**Status:** adapted; the gap itself is the open issue recorded above. + ### Channel and presence behaviours asserted as they are | Spec points | Specification | ably-python | Tests | @@ -1510,6 +1765,9 @@ pseudocode is not mistaken for non-compliance. | `Connection#whenState`'s registration window | `_when_state`'s deferred branch is an `async def`, so its `once` registration happens when the coroutine *starts*, not when `_when_state` is called. A caller that needs the registration in place before the state can change must schedule it and yield first, which the derived tests do. A literal callback API would have no such window | | RTP17b's synthesized-LEAVE filter | RTP17b's own implementation note allows the check to live "either inside the presence map's `remove()` method, or at the calling level". ably-python uses the calling level (`presence.py:557-558`). Compliant | | RTP19a's route | the specification models an ATTACHED without HAS_PRESENCE as `startSync()` then `endSync()`. `on_attached(has_presence=False)` calls `_synthesize_leaves(...)` then `clear()` (`presence.py:611-618`), which is the requirement itself rather than the model of it | +| `hasNext` as a value | `push_admin.md` RSH1b2 writes `ASSERT result.hasNext == true`. In Python `has_next` is a bound method (`ably/http/paginatedresult.py:63`), truthy whatever the page holds, so the direct translation asserts nothing. `test_rsh1b2_list_devices_pagination` writes `result.has_next() is True`, confirmed to fail when inverted | +| Push `remove` return values | the specification's remove steps assert nothing about a return value ("should not throw"). `PushDeviceRegistrations.remove` / `remove_where` and `PushChannelSubscriptions.remove` / `remove_where` return the `ably.http.http.Response` from the DELETE rather than `None` (`ably/rest/push.py:112-127`, `:176-192`), so the six derived removal tests assert `response.status_code == 204`, which is stronger than the specification asks and matches what `test/ably/rest/restpush_test.py` already asserts | +| RSP4b1's time bounds | the specification records `time_before = now_millis()` before generating the presence events and `time_after = now_millis()` after, then asserts a `history(start=, end=)` over that window returns them. Read from the runner's clock the window is only as good as the skew against the sandbox, which decides the timestamps actually stored, so a runner running a little fast would exclude the very events the test generated. Both bounds come from `await client.time()` instead — the same instant on the clock that stamps the events. `Presence.history` passes an `int` straight through as milliseconds (`ably/types/presence.py:232-241`), which is what `client.time()` returns, so no conversion is involved | ### REST behaviours asserted as they are @@ -1523,7 +1781,7 @@ pseudocode is not mistaken for non-compliance. | RSC18 | The constructor rejects basic auth over HTTP | Construction succeeds; 40103 is raised from `make_request` when a request needing Basic Auth is attempted, and no request goes out. RSA1/RSC18 say only "any attempt to use" | Compliant; the UTS is stricter than its source | | REC1b1, REC1c1 | Code 40000, or a message containing "invalid" or "conflict" | 400/40106 with a specific message. The features spec mandates no code | Cosmetic | | RSAN1a3 | Code 40003 for a missing `Annotation.type` | 400/40000 | Cosmetic; worth aligning cross-SDK | -| RSH1a | Empty `recipient` or `data` rejected with code 40000 | `TypeError` / `ValueError`, not an `AblyException`. The "no HTTP request" half is satisfied | Open bug, minor | +| RSH1a | Empty `recipient` or `data` rejected with code 40000, the error reaching the caller from the server | `PushAdmin.publish` validates its arguments itself and raises before touching the HTTP layer (`ably/rest/push.py:49-59`): a non-dict `recipient` or `data` raises `TypeError`, an empty one `ValueError`. So there is no request, no server error and no `code` to read. The "no HTTP request" half is satisfied, and the repository's own sandbox suite already pins the exception types (`test/ably/rest/restpush_test.py::test_admin_publish`). `test_rsh1a_push_publish_invalid_recipient` in the integration tier asserts `pytest.raises(ValueError)` alongside the unit-tier test | Open bug, minor — a stricter precondition rather than wrong behaviour. The specification's test would need a recipient the SDK will send and the server will reject, an unknown `transportType` say, to exercise the server-side path it describes | | HP6 | `errorCode` is a number | The raw header string, `'40101'` | Open bug, trivial | | HP8 | `headers` is a map | A list of `(name, value)` pairs, so the lookup the spec describes is impossible without converting, and case-insensitivity is lost | Open bug; changing the return type is breaking | | RSC19e | An error indicated idiomatically | `httpx.ConnectError` / `ReadTimeout` reach the caller unwrapped, because `AblyRest.request` carries no `@catch_all` unlike `time()` and `stats()`. The messages do name the failure | Borderline; defensible under RSC19e | @@ -1675,6 +1933,83 @@ ably-python does it with `loop.call_soon` (`connectionmanager.py:668`), so DISCO left within the same turn of the event loop. The awaited state is unreachable because the SDK is compliant, not because it is not. Recorded under Adapted Tests as a translation note. +### Push admin filter keys have to be camelCase, and the snake_case form filters nothing + +**Spec points:** RSH1b2, RSH1b5, RSH1c1, RSH1c4, RSH1c5, RSH7a, RSH7b. + +`PushDeviceRegistrations.list` / `remove_where` and `PushChannelSubscriptions.list` / +`list_channels` build their query string with `format_params(params)` — the collected +`**params` dict passed *positionally*, so it bypasses the `snake_to_camel` conversion +`format_params` applies only to its own `**kw` (`ably/rest/push.py:94,127,147,158`, +`ably/http/paginatedresult.py:18-25`). `list(client_id=x)` therefore sends `?client_id=x`, +which the server does not recognise. + +Measured against the sandbox with three devices registered, two of them sharing a +clientId: `list(clientId=cid)` returned 2, `list(client_id=cid)` returned 3, and `list()` +returned 3. So the snake_case form does not match *nothing*, it matches *everything* — the +unknown query parameter is dropped and the response is the unfiltered page. That is the +more dangerous of the two failure modes, because a test asserting `items.length >= 1`, or +asserting that a subscription it just created is present, passes with the filter doing +nothing at all. `PushChannelSubscriptions.remove_where` is the one exception — it spreads +its params (`format_params(**params)`), so both spellings work there — which makes the +surface inconsistent with itself. + +Not a compliance failure, since the specifications name the filters in camelCase and the +derived tests write them that way. Every filtered list in `push_admin_test.py` and +`push_channels_test.py` carries a control assertion proving the filter narrowed: a decoy +registration the `deviceId` filter must exclude, a count before the `limit` is applied, a +second channel the `channel` filter must exclude, and a surviving `clientId` that +`removeWhere` must leave alone. An inconsistency worth tidying — the four positional calls +could spread their params like the fifth does — rather than a defect. + +### Device deletion is asynchronous on the server + +**Spec points:** RSH1b5, RSH1c4, RSH1c5. + +`removeWhere` answers 204 before the rows are gone, so the specification's immediate +`ASSERT result.items.length == 0` is racy. `test/ably/rest/restpush_test.py` already +carries the same observation — "Deletion is async: wait up to a few seconds before giving +up" — and the derived tests poll with `wall_clock_poll_until(..., timeout=20.0)` rather +than asserting straight away. Server behaviour, not the SDK. + +### RSL1m4's clientId mismatch is rejected by the server, not locally + +`Channel.__publish_request_body` does carry a local clientId check that raises +`IncompatibleClientIdException` 400/40012 (`ably/rest/channel.py:73-77`), so there was a +question of whether `rest/integration/publish.md`'s RSL1m4 test verifies a local check +rather than the server interop it describes. It does not. The specification builds its +client with `token: token_details.token` — the bare token string — so `auth.client_id` is +`None`, the library cannot tell which clientId the token carries, `can_assume_client_id` +allows the publish, and the message goes to the wire. The sandbox rejects it with +`AblyException 40012 400 "Malformed message; invalid clientId"`, which is the code and +status the specification asserts, raised from the response rather than locally. The test +is genuine server interop under both protocols. + +Passing `token_details=token_details` instead would hand the library the clientId and move +the rejection client-side, to the same 400/40012, before any request left. The test notes +this at the site so that nobody "simplifies" it into a local check. + +### The presence fixture channel returns all six members, and the cipher has to go in first + +`rest/integration/presence.md` allows for `>= 5` members on `persisted:presence_fixtures` +and warns that `client_encoded` may not decode. Against a real provision all six are +returned on both protocols, and with `fixture_cipher_params()` supplied at `channels.get` +time `client_encoded` decodes to `{'example': {'json': 'Object'}}` — the same payload +`client_decoded` carries in the clear. `test_rsp5_decode_encrypted_data` asserts that value +rather than the specification's `IS NOT null`, which would also hold for the raw ciphertext +bytes the same call returns without a cipher. + +The cipher has to go in on the **first** `channels.get` for a client: `Presence.__init__` +snapshots `channel.cipher` (`ably/types/presence.py:207-212`) and `Channels.get` caches the +channel, so setting the option afterwards leaves the presence object decrypting with +nothing and silently yielding the ciphertext. Each test builds its own client, so the order +is not shared between them. + +`RSP3/full-pagination-3` is sound on the same fixture: presence `get` paginates at +`limit=2`, walks three pages of two and recovers exactly the six fixture clientIds with no +duplicates, on both protocols. The test asserts the full set rather than only the +specification's `>= 5`, since the fixture is fixed. + ## Candidate issues `writing-derived-tests.md` asks for the deviations above to be classified into distinct @@ -2021,6 +2356,32 @@ none of these shows up as a failure — which is why they are easy to lose. | `ChannelStateChange#event`, and a `ChannelEvent` type | the key the listener was registered against | RTL2, RTL5, RTL12, TH5 | | A public `Connection#whenState` | the private `Connection._when_state`, which `test/ably/realtime/realtimepresence_test.py` already reaches for in two places | RTN26 | +### From the integration tier + +The five tiers above classify the realtime derivation; the REST unit derivation's +candidates went upstream as [#709](https://github.com/ably/ably-python/issues/709)–[#712](https://github.com/ably/ably-python/issues/712). +Two further defects came out of `rest/integration`, and neither is filed. Both are +tier 2 by the ranking above — an error where there should be none. + +**I.1 `Rest#request` never renews an expired token.** RSC10, RSC19. Every other REST +operation renews and retries on a 40140–40149; `Rest#request` returns the 401 to the +caller. It is the only call site passing `raise_on_error=False` (`ably/rest/rest.py:145`), +so `make_request` skips `raise_for_response`, nothing raises, and the reactive branch of +`reauth_if_expired` (`ably/http/http.py:19-42`) never runs; the pre-emptive branch is +separately inert, because `token_details_has_expired()` returns `False` with no time +offset. The same expired token renews correctly through `publish()`. A fix has to separate +the two meanings `raise_on_error` carries. +`test/uts/rest/integration/auth_test.py -k rsc10` + +**I.2 `enterClient` cannot succeed on a basic-auth connection.** RSA7b4, RTP14, RTP15. A +connection built from a key alone is told `clientId: "*"`, and the branch of +`Auth._configure_client_id` (`ably/rest/auth.py:335-353`) that guards a configured clientId +against a server wildcard fires when there is no configured clientId, recording it as +validated and `None`. `can_assume_client_id` then refuses every clientId. One line from a +fix, and `enter_client` is unusable without one. No test gates on it — the three tests that +would hit it pass `client_id='*'` in setup instead, as the repository's own presence suite +does — so it will not show up as a failure. + ## How the specifications are adopted here Choices about the approach, as against the behaviour recorded above. @@ -2275,18 +2636,73 @@ outright. Only wrong behaviour is gated. +### The integration tier runs against one provisioned sandbox app, once per protocol + +`uts/rest/integration` is the first tier with a server behind it, and three harness +choices follow from that. + +The app is provisioned once for the whole tier and deleted at the end, which is the +specifications' `BEFORE ALL TESTS` / `AFTER ALL TESTS` pair; a fresh app per test would +make the tier several times slower and invite the sandbox's rate limiting. +`sandbox.key(0)` is the full-access key the specifications call `api_key`, and the other +indices are the capabilities each specification's app-provisioning section names. + +A specification carrying a `## Protocol Variants` section runs each of its Test IDs twice, +through a `use_binary_protocol` fixture parametrized `[False, True]` with the ids `json` +and `msgpack`. Five of the eleven specifications carry that section, so 38 of the 76 +integration Test IDs are two pytest cases each. The six that do not are json only and +take the default `sandbox_rest_client` applies. This is why the counts in this file give +Test IDs and cases separately. + +An autouse fixture closes every client a test built, whether or not its assertions held +(`test/uts/conftest.py`, `close_open_clients`). The specifications write +`AWAIT realtime.close()` inline, which is redundant against that fixture and in two +places actively destroys what the following REST read is about — see the UTS Spec Error +above. Tests omit the inline close and leave it to teardown. + +### A hedged integration setup is provisioned so its guarded assertions bite + +`time_stats.md` hedges its setup in a way that lets both its tests pass without testing +anything, and the harness removes the hedge rather than the assertion. It allows for an empty result — "stats may be empty for a new sandbox app" — +and guards its assertions on an interval's shape behind `IF result.items.length > 0`. A +freshly provisioned app has no stats at all, verified, so against one that branch never +runs and both RSC6 tests assert only that the call returned something. The module-scoped +`app_with_stats` fixture records a minute of traffic against the app through the sandbox's +`POST /stats` injection endpoint — the mechanism the repository's own +`test/ably/rest/reststats_test.py` uses — and the tests then make the guarded assertions +unconditionally. Injection rather than real traffic is deliberate: real traffic is +aggregated on the server's own schedule, so there is no bounded wait after which a +published message is certainly counted, whereas an injected interval is queryable at once. +With traffic in place `test_rsc6_stats_with_parameters` also asserts that every returned +interval has `unit == 'hour'`; the specification asserts only `items.length <= 5`, which an +empty page satisfies whether or not the query reached the server. Both tests were confirmed +to fail with `assert 0 > 0` when the injection is removed, so neither passes vacuously. + +One further setup departs from the pseudocode, for latency rather than coverage. +`pagination.md`'s five setups are each `FOR i IN 1..N: AWAIT channel.publish(...)` — 15, +12, 10, 25 and 3 messages, a round trip apiece. The tests publish the same messages as one +list through `channel.publish([Message(...), ...])`, which `Channel._publish` accepts +(`ably/rest/channel.py:97,105`): the resulting message set, names and data are identical +and each message still gets a distinct id, so only the setup latency differs. Each test +then polls history until the expected count is visible before paginating, which is the +specifications' own `poll_until`, because history is not immediately consistent. + ### Deviation records are consolidated, not accumulated -Fourteen specification areas were derived in parallel, each writing its own +Each round of derivation runs a specification area per agent, and each writes its own `deviations-.md`. Those files are scaffolding and are not kept: `writing-derived-tests.md` requires one entry per **root cause**, and a per-area file -cannot see that two areas found the same defect. Three defects were in fact reported -by more than one area — the `on_error` bypass, the transposed `AblyException` -arguments, and the `EventEmitter` wrapper registry — and one was reported as a defect -and then refuted. - -So the per-area files were merged into this file and deleted, and the comments in -the tests that pointed at them now point here. +cannot see that two areas found the same defect. In the realtime round three defects +were in fact reported by more than one area — the `on_error` bypass, the transposed +`AblyException` arguments, and the `EventEmitter` wrapper registry — and one was +reported as a defect and then refuted. The integration round found four gaps the unit +tier had already recorded, which extend the rows they belong to rather than opening +new ones: `Auth#revokeTokens`, `Rest#batchPresence`, the `PushChannel` surface and the +`clientId` filter on `RestPresence#get`. The revoked-token 40171 it observed is the +same root cause as RTN15h1's, and sits in that entry. + +So the per-area files are merged into this file and deleted, and the comments in +the tests that pointed at them point here instead. A refuted claim is kept, under *Investigated and not defects*, because the reason a reader needs it is precisely that it looks like a defect. @@ -2294,7 +2710,19 @@ reader needs it is precisely that it looks like a defect. The header states how many derived tests there are, how many pass, how many are gated and how many cannot run. Those numbers are the check that the file is still -true: the gated count must equal the number of failures under `RUN_DEVIATIONS=1`, -and the sum must equal the number of skips without it. Anyone changing the suite -should re-run both and update the header, rather than copying the previous numbers -forward. +true: in pytest cases, the gated count must equal the number of failures under +`RUN_DEVIATIONS=1`, and gated plus unrunnable must equal the number of skips without +it. As of this writing that is 206 failures and 15 skips with the variable set, and +221 skips and 1032 passes without it, the 1032 being 910 derived cases and 122 +`helpers/` ones. + +The other two counts are measured from the source rather than from a run. The number of +**derived tests** is the number of `# UTS:` comments, 1060. The number of **Test IDs** is +the number of *distinct* ids in them, 1051 — not the same figure, because five ids in +`rest/unit` are carried by more than one test function. Counting the comments and calling +the result Test IDs is the easy mistake here, and it overstates the specification coverage +by nine. + +Anyone changing the suite should re-run both and update the header, rather than copying +the previous numbers forward. Keep the three units apart while doing it: one Test ID is +one or more derived tests, and one derived test is one or more pytest cases. diff --git a/test/uts/rest/integration/batch_presence_test.py b/test/uts/rest/integration/batch_presence_test.py index f41c58b7..2ccb356b 100644 --- a/test/uts/rest/integration/batch_presence_test.py +++ b/test/uts/rest/integration/batch_presence_test.py @@ -20,7 +20,10 @@ beside the assertions they bear on — the `presence` key the server omits for an empty channel, and the presence members a closed connection takes with it. -See [deviations-batch-push-channels-integration.md](../../deviations-batch-push-channels-integration.md). +See [deviations.md](../../deviations.md): the gating under *Failing Tests* -> +*Unimplemented features*, the omitted `presence` key and the closed connection under +*UTS Spec Errors*, and `enterClient` on an anonymous connection under *Failing Tests* -> +*Auth*. """ from ably.realtime.connection import ConnectionState diff --git a/test/uts/rest/integration/presence_test.py b/test/uts/rest/integration/presence_test.py index 8250a8a7..68a69cef 100644 --- a/test/uts/rest/integration/presence_test.py +++ b/test/uts/rest/integration/presence_test.py @@ -27,8 +27,9 @@ def member_for(page, client_id): """The one fixture member with `client_id`, or None. The specifications reach for this with `presence.get(clientId: ...)`, which - `Presence.get` does not offer; see - [deviations-presence-integration.md](../../deviations-presence-integration.md). + `Presence.get` does not offer; see *Adaptations forced by the absent `clientId` + filter on `RestPresence#get`* under *Adapted Tests* in + [deviations.md](../../deviations.md). """ return next((item for item in page.items if item.client_id == client_id), None) diff --git a/test/uts/rest/integration/push_admin_test.py b/test/uts/rest/integration/push_admin_test.py index cb4f768a..9c0ae7bf 100644 --- a/test/uts/rest/integration/push_admin_test.py +++ b/test/uts/rest/integration/push_admin_test.py @@ -71,8 +71,9 @@ async def test_rsh1a_push_publish_invalid_recipient(sandbox): # NOTE: the spec expects the server to reject the empty recipient and the # error to carry a `code`. `PushAdmin.publish` validates the recipient # itself (`ably/rest/push.py:57`) and raises `ValueError` before any request - # is made, so there is no server error and no code to read. See - # test/uts/deviations-push-admin-integration.md. + # is made, so there is no server error and no code to read. See the RSH1a row + # under Adapted Tests -> REST behaviours asserted as they are, in + # test/uts/deviations.md. with pytest.raises(ValueError): await client.push.admin.publish({}, {'notification': {'title': 'Test'}}) diff --git a/test/uts/rest/integration/push_channels_test.py b/test/uts/rest/integration/push_channels_test.py index d509e1f6..26741bbd 100644 --- a/test/uts/rest/integration/push_channels_test.py +++ b/test/uts/rest/integration/push_channels_test.py @@ -26,7 +26,9 @@ RSH7a setup below: the `deviceIdentityToken` it hard-codes is rejected, and the real one from the registration response is used instead. -See [deviations-batch-push-channels-integration.md](../../deviations-batch-push-channels-integration.md). +See [deviations.md](../../deviations.md): the gating under *Failing Tests* -> +*Unimplemented features*, and the hard-coded device identity token under *UTS Spec +Errors*. """ from ably import AblyException, DeviceDetails diff --git a/test/uts/rest/integration/revoke_tokens_test.py b/test/uts/rest/integration/revoke_tokens_test.py index 6fdf6309..7b2adacd 100644 --- a/test/uts/rest/integration/revoke_tokens_test.py +++ b/test/uts/rest/integration/revoke_tokens_test.py @@ -18,8 +18,9 @@ the issued token and the realtime connection the revocation drops. The two connection tests assert FAILED with 40171 where the specification asserts DISCONNECTED with 40141, which is an adaptation recorded in -[deviations-revoke-tokens-integration.md](../../deviations-revoke-tokens-integration.md) -and explained where it is made. +[deviations.md](../../deviations.md) under *Adapted Tests* -> *A token error with no +means to renew reports the renewal failure, not the server's error*, and explained +where it is made. """ import asyncio @@ -107,7 +108,8 @@ async def test_rsa17g_revoke_token_prevents_use(sandbox): # server does push exactly that — action 6 carrying `{"code": 40141, "message": # "token revoked"}` — but a connection holding only a `TokenDetails` has no way to # renew, so ably-python fails it under RSA4a with 40171 and the 40141 never reaches - # the connection's state. See deviations-revoke-tokens-integration.md. + # the connection's state. See "A token error with no means to renew reports + # the renewal failure, not the server's error" in test/uts/deviations.md. state_change = await dropped assert state_change.reason.code == 40171 assert state_change.reason.status_code == 403 From 24e559859a374b571530d493aaca171e8e7e949d Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 17:23:09 +0100 Subject: [PATCH 08/10] test: run the proxy integration specifications through uts-proxy uts/docs/proxy.md puts ably/uts-proxy between the client and the sandbox for the specifications that are about what the SDK does when a request goes wrong. The sandbox answers correctly, so the fault has to be injected in front of it: the proxy binds a port per session, takes plain HTTP on it and speaks TLS onwards, applies the session's rules, and records everything that crosses it. helpers/proxy.py supplies the proxy. The pinned release is downloaded on first use, checked against the sha256 the release publishes and extracted into ~/.cache/uts-proxy//, under a lock file so the several Python versions CI runs fetch it once between them; UTS_PROXY_LOCAL_PATH substitutes a locally built binary and UTS_PROXY_CONTROL_URL a control API already running. One control process serves a test run, on a free port rather than a fixed one so two suites on a machine do not collide, and it is reaped at the end of the run and again at interpreter exit. create_proxy_session and ProxySession are the specifications' own interface, with rules and log events left as the plain dictionaries their JSON describes. The package's proxy_session fixture opens sessions and closes every one of them, which is the specifications' AFTER EACH TEST. Its per-test timeout is 300 seconds, prepended so it is read in place of the tier's 120: a cold cache downloads the binary before the first test runs, and a specification that provokes a timeout sits through the delay it asked for. Co-Authored-By: Claude Opus 5 (1M context) --- test/uts/helpers/proxy.py | 585 ++++++++++++++++++++ test/uts/rest/integration/proxy/__init__.py | 0 test/uts/rest/integration/proxy/conftest.py | 80 +++ 3 files changed, 665 insertions(+) create mode 100644 test/uts/helpers/proxy.py create mode 100644 test/uts/rest/integration/proxy/__init__.py create mode 100644 test/uts/rest/integration/proxy/conftest.py diff --git a/test/uts/helpers/proxy.py b/test/uts/helpers/proxy.py new file mode 100644 index 00000000..d572a3a5 --- /dev/null +++ b/test/uts/helpers/proxy.py @@ -0,0 +1,585 @@ +"""The programmable proxy the proxy integration specifications route their traffic through. + +`uts/docs/proxy.md` puts [ably/uts-proxy](https://github.com/ably/uts-proxy) +between the SDK and the sandbox. The proxy runs a control API on one port and +binds a fresh port for each session opened through it; a client under test is +built with `endpoint='localhost'`, `port=session.proxy_port` and `tls=False`, +so its traffic arrives at the session port in the clear and the proxy speaks +TLS onwards to the sandbox. Rules attached to a session drop connections, +delay them, or answer them with a response the sandbox would never give, which +is how a specification exercises a fault path against the real server. + +Three things have to be true before a test can do any of that, and this module +is responsible for all three. + +The binary has to be on disk. The pinned release is downloaded from GitHub +into `~/.cache/uts-proxy//`, its archive checked against the sha256 +the release publishes, and the `uts-proxy` entry extracted out of it. The +download is serialised across processes by an `fcntl.flock` on a lock file +beside the binary: CI runs the suite once per supported Python version and a +developer may run pytest under `-n`, so several processes can arrive at an +empty cache together. Setting `UTS_PROXY_LOCAL_PATH` to a locally built binary +or to a `.tar.gz` holding one takes the place of the download, which is how a +change to the proxy itself is tried out ahead of a release. + +The control process has to be running. `ensure_proxy()` starts one for the +test session and `stop_proxy()` reaps it; between them the process is shared, +because a session port is per-session and one control process serves any +number of them. The port it listens on is chosen free rather than fixed, so +two suites on one machine do not collide, and `UTS_PROXY_CONTROL_URL` points +the harness at a proxy someone is already running instead of starting one. + +A session has to exist. `create_proxy_session()` is the specifications' +function of that name and returns the `ProxySession` they drive. + +Everything here talks to the control API over plain `httpx`, the way +`sandbox.py` talks to the sandbox's provisioning API. It is infrastructure, +and a proxy that could not be reached should look like broken infrastructure +rather than a failing assertion. +""" + +import asyncio +import atexit +import fcntl +import hashlib +import io +import logging +import os +import platform +import shutil +import socket +import subprocess +import sys +import tarfile +import tempfile +from urllib.parse import urlsplit + +import httpx + +from test.uts.helpers.sandbox import SANDBOX_ENDPOINT, SANDBOX_URL + +log = logging.getLogger(__name__) + +# The release the suite is pinned to. The cache directory is keyed on it, so +# moving the pin downloads afresh rather than reusing the old binary. +PROXY_VERSION = 'v0.3.0' + +# The sha256 of each release archive, copied from the release's own +# `checksums.txt`. The archive is what is verified, not the binary extracted +# from it, so a tampered download is rejected before anything is written to +# the cache and never becomes something a later run treats as a hit. +ARCHIVE_CHECKSUMS = { + 'uts-proxy_0.3.0_darwin_amd64.tar.gz': + '1355526543c3022f87efb7f564f55200b78edc68d84c7dba2e49f63429e3b788', + 'uts-proxy_0.3.0_darwin_arm64.tar.gz': + 'a948f99b7daf9b3bffff742f6405637d40a79947389309eed5f87e59026de9a5', + 'uts-proxy_0.3.0_linux_amd64.tar.gz': + 'de741ba21f3630fea4f59714d00585638d565005599ecd84179931eba248f280', + 'uts-proxy_0.3.0_linux_arm64.tar.gz': + '15b5ca87c40c2c4ff350c94af1911cea0ad6be5a2d890ba41029bc4b8bc52c61', +} + +RELEASE_URL = f'https://github.com/ably/uts-proxy/releases/download/{PROXY_VERSION}' + +BINARY_NAME = 'uts-proxy' +CACHE_ROOT = os.path.join(os.path.expanduser('~'), '.cache', 'uts-proxy') +CACHE_DIR = os.path.join(CACHE_ROOT, PROXY_VERSION) +LOCK_PATH = os.path.join(CACHE_DIR, 'uts-proxy.lock') + +# A locally built binary, or a `.tar.gz` distributive holding one, in place of +# the pinned release. The name is the one the Kotlin harness uses, so a +# developer with both checkouts sets it once. +LOCAL_PATH_VAR = 'UTS_PROXY_LOCAL_PATH' + +# The base URL of a control API someone else is running, as +# `http://localhost:9100`. When it is set nothing is downloaded and nothing is +# spawned, and the proxy outlives the test run because the test run did not +# start it. +CONTROL_URL_VAR = 'UTS_PROXY_CONTROL_URL' + +# Where a local distributive is unpacked. It is kept apart from the pinned +# version's directory so that unpacking a development build does not leave +# something in the cache that a later run, without the override set, would +# mistake for the release. +LOCAL_CACHE_DIR = os.path.join(CACHE_ROOT, 'local') + +# The upstream host a specification's `endpoint` names. The proxy is told a +# host rather than an endpoint, because it is not an Ably client and does not +# resolve endpoints; `nonprod:sandbox` serves both realtime and REST from the +# one host. +SANDBOX_HOST = urlsplit(SANDBOX_URL).hostname +TARGET_HOSTS = {SANDBOX_ENDPOINT: SANDBOX_HOST} + +# How long a release download may take. Generous: the archive is a few +# megabytes and a cold CI runner fetches it over whatever link it has. +DOWNLOAD_TIMEOUT = 60.0 + +# How long a control API call may take. A control call is local and answers +# immediately, so a wait this long means the proxy has stopped serving. +CONTROL_TIMEOUT = 15.0 + +# How long a freshly spawned proxy is given to answer `/health`, and how often +# it is asked. The single health check is bounded far shorter than a control +# call, so that the poll keeps its cadence while the port is still refusing. +STARTUP_TIMEOUT = 15.0 +STARTUP_INTERVAL = 0.2 +HEALTH_TIMEOUT = 2.0 + +# How long a proxy is given to exit on SIGTERM before it is killed outright. +SHUTDOWN_TIMEOUT = 5.0 + +# The session's idle auto-cleanup timer, in milliseconds, which is a +# specification's `timeoutMs`. The proxy's own default is 30000, measured from +# the last piece of traffic through the session; a test that delays a response +# by twenty seconds and then reads the event log spends longer than that idle, +# and would find its session already torn down. +SESSION_TIMEOUT_MS = 120000 + +__control_url = None +__process = None +__process_output = None + + +def control_url(): + """The base URL of the control API this test run is using. + + Raises if the proxy has not been started, since a control call made before + `ensure_proxy()` would otherwise fail as a connection error against + whatever happens to be listening. + """ + if __control_url is None: + raise AssertionError( + 'The uts-proxy control API is not running. A test reaching the proxy asks for the ' + 'fixture that calls ensure_proxy() first.') + return __control_url + + +async def ensure_proxy(timeout=STARTUP_TIMEOUT): + """Makes sure a control API is running, and returns its base URL. + + Called again once the proxy is up, this answers with the URL it already + has: the process is shared for the whole test run, so a second caller must + not start a second one. + + With `UTS_PROXY_CONTROL_URL` set, the proxy named there is used as it + stands. It is health-checked all the same, so that a stale value in a + developer's environment is reported as itself rather than as every proxy + test failing to create a session. + """ + global __control_url, __process, __process_output + + if __control_url is not None: + return __control_url + + external = os.environ.get(CONTROL_URL_VAR) + if external: + external = external.rstrip('/') + if not await _is_healthy(external): + raise AssertionError( + f'{CONTROL_URL_VAR} is set to {external}, where nothing answered GET /health. ' + 'Start a uts-proxy there, or unset the variable to have one started here.') + log.info(f'ensure_proxy(): using the uts-proxy already running at {external}') + __control_url = external + return __control_url + + loop = asyncio.get_running_loop() + binary = await loop.run_in_executor(None, _ensure_binary) + + # The port is bound, read back and released before the proxy is told to + # take it, so there is a moment in which something else could claim it. + # Nothing on the machine is hunting for a port to steal, and the + # alternative — a fixed port — collides with the suite running twice. + port = _free_port() + url = f'http://localhost:{port}' + + # The proxy's own output goes to a temporary file rather than to the + # terminal, where it would interleave with pytest's, and is read back if + # the process never becomes healthy. That is the one moment its output is + # worth having, and the one moment nothing else can explain the failure. + output = tempfile.TemporaryFile() + log.info(f'ensure_proxy(): starting {binary} on port {port}') + process = subprocess.Popen( + [binary, '--port', str(port)], + stdin=subprocess.DEVNULL, + stdout=output, + stderr=subprocess.STDOUT, + ) + + try: + await _wait_for_health(process, output, url, timeout) + except BaseException: + _reap(process) + output.close() + raise + + __process = process + __process_output = output + __control_url = url + return url + + +def stop_proxy(): + """Stops the proxy this test run started, if it started one. + + Registered to run at interpreter exit as well as being called from the + fixture that started it, because a `Popen` child does not die with its + parent: a suite interrupted between the two would otherwise leave a proxy + holding its control port for as long as the machine is up. + """ + global __control_url, __process, __process_output + + __control_url = None + process, __process = __process, None + output, __process_output = __process_output, None + + if process is not None: + _reap(process) + if output is not None: + output.close() + + +atexit.register(stop_proxy) + + +async def create_proxy_session(endpoint=SANDBOX_ENDPOINT, port=None, rules=None, + timeout_ms=SESSION_TIMEOUT_MS): + """Opens a proxy session against `endpoint`, as the specifications' function of that name. + + `rules` are the rule objects a specification writes out as JSON, passed + through to the control API unchanged — `{'match': {...}, 'action': {...}, + 'times': 1, 'comment': '...'}`, with the key spellings the proxy's API + reference gives. They are left as plain dictionaries rather than wrapped, + so that a derived test reads as the specification it came from and a rule + the proxy grows tomorrow needs nothing here. + + `port` asks for a particular session port, which the control API answers + with 409 if it is taken. Left out, the proxy binds a free one and reports + it back as `proxy_port`. + + `timeout_ms` is the session's idle timeout, not a deadline for the test. + """ + host = TARGET_HOSTS.get(endpoint) + if host is None: + raise AssertionError( + f'No upstream host is known for endpoint {endpoint!r}; ' + f'the endpoints this harness resolves are {sorted(TARGET_HOSTS)}.') + + body = { + 'target': {'realtimeHost': host, 'restHost': host}, + 'rules': list(rules) if rules else [], + } + if port is not None: + body['port'] = port + if timeout_ms is not None: + body['timeoutMs'] = timeout_ms + + base = control_url() + created = await _control_request('POST', f'{base}/sessions', body) + + session_id = created.get('sessionId') + assigned = (created.get('proxy') or {}).get('port', port) + if not session_id or not isinstance(assigned, int): + raise AssertionError( + f'The uts-proxy control API answered POST /sessions with {created!r}, which names ' + 'no session id or no port.') + + log.info(f'create_proxy_session(): session {session_id} proxying localhost:{assigned} to {host}') + return ProxySession(session_id, assigned, base) + + +class ProxySession: + """One session on the proxy, as a specification's `session`. + + A session owns a port of its own, and the rules and event log that go with + it. A test builds its client against `proxy_host` and `proxy_port`, runs + its scenario, reads `get_log()` for the traffic the proxy saw, and closes + the session — which is what frees the port again. + """ + + def __init__(self, session_id, proxy_port, control_url, proxy_host='localhost'): + self.session_id = session_id + self.proxy_host = proxy_host + self.proxy_port = proxy_port + self.control_url = control_url + self.closed = False + + @property + def _session_url(self): + return f'{self.control_url}/sessions/{self.session_id}' + + async def add_rules(self, rules, position='append'): + """Adds `rules` to the session while it is running. + + `position` is `'append'` or `'prepend'`. Rules are evaluated in order + and the first match wins, so a rule added to the front takes + precedence over everything the session was created with — which is how + a specification faults traffic only after the client has reached a + state it wants to fault it from. + """ + await _control_request('POST', f'{self._session_url}/rules', { + 'rules': list(rules), + 'position': position, + }) + + async def trigger_action(self, action): + """Performs `action` on the session's live WebSocket connection, now. + + This is the imperative half of the proxy's interface, for what a timed + rule says awkwardly: `{'type': 'disconnect'}`, + `{'type': 'close', 'closeCode': 1000}`, or an `inject_to_client` + carrying the protocol message to plant. The control API answers 409 + when no connection is open, which surfaces here as a failure naming + that, rather than as the action quietly doing nothing. + """ + await _control_request('POST', f'{self._session_url}/actions', dict(action)) + + async def get_log(self): + """Every event the proxy recorded for this session, in order. + + Each event is the dictionary the control API sent, with the field + names its reference gives: `type`, `direction`, `path`, `status`, + `queryParams`, `message`, `ruleMatched` and the rest, absent where + they do not apply. A specification's assertions are written against + those names, so they are carried through rather than renamed: + + requests = [event for event in await session.get_log() + if event['type'] == 'http_request' and '/time' in event['path']] + """ + body = await _control_request('GET', f'{self._session_url}/log') + return body.get('events') or [] + + async def close(self): + """Tears the session down, closing its connections and freeing its port. + + Teardown is best effort and never raises. A session the proxy has + already expired, or one whose proxy has gone, is nothing a passing + test should be failed over, and the idle timeout collects anything + left behind. Closing twice is harmless, since the fixture closes every + session it handed out whether or not the test closed it too. + """ + if self.closed: + return + self.closed = True + try: + await _control_request('DELETE', self._session_url) + except Exception as error: + log.warning(f'ProxySession.close(): session {self.session_id} was not closed: {error!r}') + + def __repr__(self): + return f'ProxySession({self.session_id!r}, {self.proxy_host}:{self.proxy_port})' + + +async def _control_request(method, url, body=None): + """Makes one call to the control API and returns its decoded body. + + A fresh `httpx.AsyncClient` per call, as in `sandbox.py`: these calls are + occasional, and a client held across calls would be bound to the event + loop that built it, which is not the loop every caller runs on. + """ + async with httpx.AsyncClient(timeout=CONTROL_TIMEOUT) as http: + response = await http.request(method, url, json=body) + if response.status_code < 200 or response.status_code >= 300: + raise AssertionError( + f'The uts-proxy control API answered {method} {url} with ' + f'{response.status_code} {response.text}') + if not response.content: + return {} + return response.json() + + +async def _is_healthy(url): + """Whether a control API is answering at `url`.""" + try: + async with httpx.AsyncClient(timeout=HEALTH_TIMEOUT) as http: + response = await http.get(f'{url}/health') + except Exception: + return False + return response.status_code == 200 + + +async def _wait_for_health(process, output, url, timeout): + """Waits for a freshly started proxy to answer `/health`. + + A proxy that exits — because its port was taken between being chosen and + being bound, or because the binary will not run on this machine — is + noticed as soon as it happens rather than at the end of the timeout, and + what it printed on its way out is quoted. + """ + loop = asyncio.get_running_loop() + deadline = loop.time() + timeout + while True: + if process.poll() is not None: + raise AssertionError( + f'uts-proxy exited with status {process.returncode} before answering GET ' + f'{url}/health:\n{_read_output(output)}') + if await _is_healthy(url): + return + if loop.time() >= deadline: + raise AssertionError( + f'uts-proxy did not answer GET {url}/health within {timeout}s:\n' + f'{_read_output(output)}') + await asyncio.sleep(STARTUP_INTERVAL) + + +def _read_output(output): + """Whatever the proxy has written to its output file so far.""" + try: + output.seek(0) + return output.read().decode('utf-8', 'replace').strip() or '(no output)' + except Exception as error: + return f'(the proxy output could not be read: {error!r})' + + +def _reap(process): + """Ends `process`, politely and then not.""" + if process.poll() is not None: + return + process.terminate() + try: + process.wait(SHUTDOWN_TIMEOUT) + except subprocess.TimeoutExpired: + log.warning(f'_reap(): uts-proxy {process.pid} ignored SIGTERM, killing it') + process.kill() + process.wait() + + +def _free_port(): + """A port nothing is listening on, from the ephemeral range.""" + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: + sock.bind(('127.0.0.1', 0)) + return sock.getsockname()[1] + + +def _ensure_binary(): + """The path to a `uts-proxy` binary, downloading the pinned release if need be. + + Blocking throughout — it holds a file lock across a network download — so + callers run it off the event loop. + """ + local = os.environ.get(LOCAL_PATH_VAR) + if local: + return _install_local(local) + + os.makedirs(CACHE_DIR, exist_ok=True) + target = os.path.join(CACHE_DIR, BINARY_NAME) + + # The lock is held across the check as well as the download, so that a + # process arriving while another is mid-download waits and then sees the + # finished binary rather than starting a download of its own. + with open(LOCK_PATH, 'w') as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + if os.path.exists(target) and os.access(target, os.X_OK): + log.debug(f'_ensure_binary(): {target} is already cached') + return target + archive_name = _archive_name() + archive = _download_archive(archive_name) + _verify_checksum(archive_name, archive) + _extract_binary(archive_name, archive, target) + log.info(f'_ensure_binary(): installed uts-proxy {PROXY_VERSION} at {target}') + return target + + +def _install_local(path): + """The binary `UTS_PROXY_LOCAL_PATH` names, unpacking it first if it is an archive. + + A path to a binary is used where it lies, so that rebuilding the proxy is + enough to pick the new build up. A `.tar.gz` is unpacked into a directory + of its own, away from the pinned release's, because a development build + sitting where the release belongs would be picked up by every later run + whether or not the override was still set. + """ + if not os.path.exists(path): + raise AssertionError(f'{LOCAL_PATH_VAR} is set to {path}, which does not exist.') + + if path.endswith('.tar.gz') or path.endswith('.tgz'): + os.makedirs(LOCAL_CACHE_DIR, exist_ok=True) + target = os.path.join(LOCAL_CACHE_DIR, BINARY_NAME) + with open(path, 'rb') as archive: + _extract_binary(os.path.basename(path), archive.read(), target) + log.info(f'_install_local(): unpacked {path} to {target}') + return target + + if not os.access(path, os.X_OK): + raise AssertionError(f'{LOCAL_PATH_VAR} is set to {path}, which is not executable.') + log.info(f'_install_local(): using the uts-proxy binary at {path}') + return path + + +def _archive_name(): + """The release asset for this machine. + + The platforms are the ones the release publishes, which are the ones the + checksum table covers; anything else is named in the failure rather than + being downloaded and found to be for the wrong architecture. + """ + if sys.platform.startswith('darwin'): + operating_system = 'darwin' + elif sys.platform.startswith('linux'): + operating_system = 'linux' + else: + raise AssertionError( + f'uts-proxy publishes no release for {sys.platform}. Build it and point ' + f'{LOCAL_PATH_VAR} at the binary.') + + machine = platform.machine().lower() + if machine in ('x86_64', 'amd64'): + architecture = 'amd64' + elif machine in ('arm64', 'aarch64'): + architecture = 'arm64' + else: + raise AssertionError( + f'uts-proxy publishes no release for {machine}. Build it and point ' + f'{LOCAL_PATH_VAR} at the binary.') + + name = f'{BINARY_NAME}_{PROXY_VERSION.lstrip("v")}_{operating_system}_{architecture}.tar.gz' + if name not in ARCHIVE_CHECKSUMS: + raise AssertionError(f'No checksum is recorded for {name}.') + return name + + +def _download_archive(archive_name): + """The bytes of the release archive, fetched anonymously. + + The release is public, and CI has no GitHub token to offer; the download + redirects to the asset CDN, which `follow_redirects` takes it to. + """ + url = f'{RELEASE_URL}/{archive_name}' + log.info(f'_download_archive(): downloading {url}') + with httpx.Client(timeout=DOWNLOAD_TIMEOUT, follow_redirects=True) as http: + response = http.get(url) + if response.status_code != 200: + raise AssertionError( + f'Downloading uts-proxy from {url} failed: {response.status_code} {response.text[:200]}') + return response.content + + +def _verify_checksum(archive_name, archive): + """Checks the downloaded archive against the sha256 the release publishes.""" + expected = ARCHIVE_CHECKSUMS[archive_name] + actual = hashlib.sha256(archive).hexdigest() + if actual != expected: + raise AssertionError( + f'Checksum mismatch for {archive_name}: expected {expected}, got {actual}.') + + +def _extract_binary(archive_name, archive, target): + """Writes the `uts-proxy` entry of `archive` to `target`, executable. + + The archive also carries a README, a changelog and a licence, so the one + entry wanted is pulled out by name rather than the whole thing being + unpacked. It is written beside its destination and moved into place, so + that an interrupted extraction cannot leave a truncated binary where a + later run would find it and take it for a complete one. + """ + partial = f'{target}.partial' + with tarfile.open(fileobj=io.BytesIO(archive), mode='r:gz') as tar: + for member in tar.getmembers(): + if not member.isfile() or os.path.basename(member.name) != BINARY_NAME: + continue + source = tar.extractfile(member) + with open(partial, 'wb') as binary: + shutil.copyfileobj(source, binary) + os.chmod(partial, 0o755) + os.replace(partial, target) + return + raise AssertionError(f'{archive_name} holds no {BINARY_NAME} entry.') diff --git a/test/uts/rest/integration/proxy/__init__.py b/test/uts/rest/integration/proxy/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/uts/rest/integration/proxy/conftest.py b/test/uts/rest/integration/proxy/conftest.py new file mode 100644 index 00000000..c12dd907 --- /dev/null +++ b/test/uts/rest/integration/proxy/conftest.py @@ -0,0 +1,80 @@ +"""Fixtures the proxy integration specifications share. + +The specifications in this package run their traffic through `uts-proxy`, so +each of them opens with a proxy session and closes it again afterwards. The +`sandbox` app they publish to comes from the parent package, which provisions +it once for the whole REST integration tier. + +`test/uts/helpers/proxy.py` describes the proxy itself and the two environment +variables that change how it is obtained and started. +""" + +import os + +import pytest +import pytest_asyncio + +from test.uts.helpers.proxy import create_proxy_session, ensure_proxy, stop_proxy + +# What a test in this package gets, in seconds, in place of the 120 the parent +# package gives the rest of the tier. A proxy test spends its budget on things +# the other integration tests do not: the first one to run waits for the +# binary to be downloaded on a cold cache and for the control process to come +# up, and a specification that provokes a timeout deliberately sits through +# a twenty-second delay before the request it is measuring even fails. +SUITE_TIMEOUT = 300 + +__package_dir = os.path.dirname(os.path.abspath(__file__)) + + +def pytest_collection_modifyitems(items): + # The parent package marks everything beneath it, this package included, + # with its own shorter timeout, and pytest-timeout reads the marker + # closest to the test — the first of the item's own markers. Putting this + # one at the front rather than on the end is what makes it the one read, + # whichever order the two hooks happen to run in. + for item in items: + if os.path.abspath(str(item.fspath)).startswith(__package_dir + os.sep): + item.add_marker(pytest.mark.timeout(SUITE_TIMEOUT), append=False) + + +@pytest_asyncio.fixture(scope='session') +async def proxy_control(): + """The running `uts-proxy` control API, shared by every test in the package. + + One control process serves any number of sessions, each on a port of its + own, so it is started once and reaped when the run ends. Asking for this + fixture is what guarantees a proxy is up; `proxy_session` asks for it, so + a test that opens sessions does not have to. + """ + await ensure_proxy() + yield + stop_proxy() + + +@pytest_asyncio.fixture +async def proxy_session(proxy_control): + """Opens proxy sessions, and closes every one of them when the test ends. + + This is the specifications' `create_proxy_session(...)` together with + their `AFTER EACH TEST: IF session IS NOT null: session.close()`. A test + calls it as it would the function: + + session = await proxy_session(rules=[...]) + + and leaves the closing alone. Each session holds a port and an event log + on the proxy until it is closed, so a test that fails part way through — + which, in a suite about faults, is the case worth planning for — still + gives them back. + """ + sessions = [] + + async def open_session(**options): + session = await create_proxy_session(**options) + sessions.append(session) + return session + + yield open_session + + for session in reversed(sessions): + await session.close() From a19333b2f0ce06d0cf9b48643ff775b35f355394 Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 17:23:20 +0100 Subject: [PATCH 09/10] test: derive the REST fallback proxy specification rest_fallback.md is the twelfth and last of the REST integration specifications, and the only one whose faults the sandbox will not produce on request: a request held past its timeout, a connection dropped mid-response, a CloudFront 403, a 5xx with and without a parseable body, a 4xx that must not be retried, and a publish the server persists while the client is told it failed. Each is a proxy rule firing once, so the retry that follows reaches the sandbox and the test is about what the SDK did in between. Every client authenticates through the specification's token_auth_callback: the session speaks plain HTTP, RSC18 refuses basic auth over it, and a token request routed through the session would be counted by the assertions that count requests, so the callback's own client goes straight to the sandbox. RSC15l2 is gated. The specification's httpRequestTimeout is milliseconds and ably-python's http_request_timeout is seconds, so its 3000 is three thousand seconds: against a session delaying /time by twenty, the request sat out the whole delay and succeeded on the primary host with no fallback attempted. Passing 3 makes the same test pass in 3.1 seconds, so the fallback path itself is compliant and the unit is the whole of the defect. Co-Authored-By: Claude Opus 5 (1M context) --- test/uts/rest/integration/conftest.py | 2 +- .../integration/proxy/rest_fallback_test.py | 359 ++++++++++++++++++ 2 files changed, 360 insertions(+), 1 deletion(-) create mode 100644 test/uts/rest/integration/proxy/rest_fallback_test.py diff --git a/test/uts/rest/integration/conftest.py b/test/uts/rest/integration/conftest.py index 911bd8dd..d6a5fbe1 100644 --- a/test/uts/rest/integration/conftest.py +++ b/test/uts/rest/integration/conftest.py @@ -52,7 +52,7 @@ def use_binary_protocol(request): client = sandbox_rest_client(api_key, use_binary_protocol=use_binary_protocol) - Only a test that asks for it is parametrised. The six specifications + Only a test that asks for it is parametrised. The seven specifications without that section are json only, and their clients take the JSON default `sandbox_rest_client` already applies. """ diff --git a/test/uts/rest/integration/proxy/rest_fallback_test.py b/test/uts/rest/integration/proxy/rest_fallback_test.py new file mode 100644 index 00000000..27eb1727 --- /dev/null +++ b/test/uts/rest/integration/proxy/rest_fallback_test.py @@ -0,0 +1,359 @@ +"""Derived from uts/rest/integration/proxy/rest_fallback.md in ably/specification. + +Spec points: RSC15l, RSC15l2, RSC15l4, RSL1k4 + +Every request a test here makes is routed through a `uts-proxy` session, which +answers the first `/time` or first publish with the fault the specification +names and passes everything afterwards through to the sandbox. What each test +is about is therefore what the SDK does next: retry on a fallback host, or +surface the error to the caller. + +`endpoint='localhost'` gives both the primary and the fallback host the same +name, so both attempts arrive at the same session port and both appear in the +same event log. A rule carrying `times: 1` fires on the first of them and the +second reaches the sandbox. + +There is no `## Protocol Variants` section, so these run against JSON only and +take no `use_binary_protocol`; the proxy reads text frames in any case. +""" + +import pytest + +from ably import AblyRest +from ably.util.exceptions import AblyException +from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until +from test.uts.helpers.deviations import deviation +from test.uts.helpers.sandbox import SANDBOX_ENDPOINT, random_id + +# The specification's `non_listening_port`: a port nothing is bound to, so that +# connecting to it is refused rather than answered. +NON_LISTENING_PORT = 19999 + +# The specification's `httpRequestTimeout: 3000`. +HTTP_REQUEST_TIMEOUT_MS = 3000 + + +def token_auth_callback(api_key): + """The specification's `token_auth_callback(api_key)`. + + The client under test points at the proxy, where a rule is waiting to fault + the first request that matches it. A token request made through that client + would be a request the rule could consume, and would show up in the event + log beside the requests a test counts. So the callback builds a Rest client + of its own aimed straight at the sandbox, asks it for a token, and closes + it: the token arrives over a connection the proxy never sees. + """ + async def auth_callback(params): + inner_rest = AblyRest(key=api_key, endpoint=SANDBOX_ENDPOINT) + try: + return await inner_rest.auth.request_token() + finally: + await inner_rest.close() + + return auth_callback + + +def http_requests(log, path, method=None): + """The `http_request` events the proxy recorded for `path`. + + `http_request` events carry the `method` and the `path` the client asked + for, which is what a specification filters them on. + """ + return [event for event in log + if event['type'] == 'http_request' + and path in event['path'] + and (method is None or event['method'] == method)] + + +def http_responses(log): + """The `http_response` events the proxy recorded, in the order it sent them. + + A response event carries `status` and `ruleMatched` and no path, so "the + first response was the injected one" is read off their order rather than by + filtering them down to a single endpoint. + """ + return [event for event in log if event['type'] == 'http_response'] + + +@deviation +# UTS: rest/proxy/RSC15l2/timeout-triggers-fallback-0 +async def test_rsc15l2_timeout_triggers_fallback(sandbox, proxy_session): + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'pathContains': '/time'}, + 'action': { + 'type': 'http_delay', + 'delayMs': 20000, + }, + 'times': 1, + 'comment': 'RSC15l2: Delay first /time request beyond httpRequestTimeout', + }]) + + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + fallback_hosts=['localhost'], + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + http_request_timeout=HTTP_REQUEST_TIMEOUT_MS, + ) + + result = await client.time() + + # The request should succeed (retried on fallback after timeout) + assert isinstance(result, (int, float)) + assert not isinstance(result, bool) + + # Proxy event log shows at least two HTTP requests to /time + log = await session.get_log() + assert len(http_requests(log, '/time')) >= 2 + + +# UTS: rest/proxy/RSC15l4/cloudfront-header-fallback-0 +async def test_rsc15l4_cloudfront_header_fallback(sandbox, proxy_session): + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'pathContains': '/time'}, + 'action': { + 'type': 'http_respond', + 'status': 403, + 'body': {'error': {'message': 'Forbidden', 'code': 40300, 'statusCode': 403}}, + 'headers': {'Server': 'CloudFront'}, + }, + 'times': 1, + 'comment': 'RSC15l4: CloudFront 403 on first /time request', + }]) + + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + fallback_hosts=['localhost'], + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + ) + + result = await client.time() + + # The request should succeed (retried on fallback after CloudFront error) + assert isinstance(result, (int, float)) + assert not isinstance(result, bool) + + # Proxy event log shows at least two HTTP requests to /time + log = await session.get_log() + assert len(http_requests(log, '/time')) >= 2 + + # First response was the injected 403 with CloudFront header + assert http_responses(log)[0]['status'] == 403 + + +# UTS: rest/proxy/RSC15l/unreachable-endpoint-error-0 +async def test_rsc15l_unreachable_endpoint_error(sandbox): + # No proxy session: the client is pointed at a port nothing is listening + # on, so the connection is refused before any request is written. The token + # still comes from the sandbox, so the refusal is the only thing the test + # provokes. + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + port=NON_LISTENING_PORT, + tls=False, + use_binary_protocol=False, + ) + + with pytest.raises(AblyException) as excinfo: + await client.time() + + # The error is an ErrorInfo-like object with a statusCode or code + # (the exact code/statusCode depends on the SDK's HTTP layer, but it must + # be present and non-null so callers can programmatically handle it). + # NOTE: the specification leaves the values open. This SDK reports 500 and + # 50000, the connection error having been wrapped by `catch_all`. + error = excinfo.value + assert error is not None + assert error.status_code is not None or error.code is not None + + +# UTS: rest/proxy/RSC15l/connection-drop-fallback-1 +async def test_rsc15l_connection_drop_fallback(sandbox, proxy_session): + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'pathContains': '/time'}, + 'action': { + 'type': 'http_drop', + }, + 'times': 1, + 'comment': 'Drop TCP connection on first /time request (ECONNRESET)', + }]) + + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + fallback_hosts=['localhost'], + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + ) + + result = await client.time() + + # The request should succeed (retried on fallback after connection drop) + assert isinstance(result, (int, float)) + assert not isinstance(result, bool) + + # Proxy event log shows at least two HTTP requests to /time + log = await session.get_log() + assert len(http_requests(log, '/time')) >= 2 + + +# UTS: rest/proxy/RSC15l/http-5xx-json-error-parsed-0 +async def test_rsc15l_http_5xx_json_error_parsed(sandbox, proxy_session): + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'pathContains': '/time'}, + 'action': { + 'type': 'http_respond', + 'status': 503, + 'body': {'error': {'code': 50300, 'statusCode': 503, + 'message': 'Service temporarily unavailable'}}, + }, + 'times': 1, + 'comment': 'Return 503 with JSON error body on first /time request', + }]) + + # No fallback_hosts -- endpoint='localhost' disables fallback (REC2c2) + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + ) + + with pytest.raises(AblyException) as excinfo: + await client.time() + + # The SDK parsed the error fields from the JSON response body + error = excinfo.value + assert error.code == 50300 + assert error.status_code == 503 + assert 'Service temporarily unavailable' in error.message + + +# UTS: rest/proxy/RSC15l/http-5xx-no-json-synthesized-1 +async def test_rsc15l_http_5xx_no_json_synthesized(sandbox, proxy_session): + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'pathContains': '/time'}, + 'action': { + 'type': 'http_respond', + 'status': 503, + 'body': {}, + }, + 'times': 1, + 'comment': 'Return 503 with empty JSON body (no error field) on first /time request', + }]) + + # No fallback_hosts -- endpoint='localhost' disables fallback (REC2c2) + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + ) + + with pytest.raises(AblyException) as excinfo: + await client.time() + + # The SDK synthesized an error from the HTTP status code + assert excinfo.value.status_code == 503 + + +# UTS: rest/proxy/RSC15l/http-4xx-not-retried-0 +async def test_rsc15l_http_4xx_not_retried(sandbox, proxy_session): + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'pathContains': '/time'}, + 'action': { + 'type': 'http_respond', + 'status': 403, + 'body': {'error': {'code': 40300, 'statusCode': 403, 'message': 'Forbidden'}}, + }, + 'times': 1, + 'comment': 'Return 403 with JSON error body on first /time request', + }]) + + # Fallback hosts ARE configured -- but 403 should NOT trigger fallback + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + fallback_hosts=['localhost'], + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + ) + + with pytest.raises(AblyException) as excinfo: + await client.time() + + # The SDK parsed the error fields from the JSON response body + error = excinfo.value + assert error.code == 40300 + assert error.status_code == 403 + + # Proxy event log shows exactly 1 HTTP request to /time (no fallback retry) + log = await session.get_log() + assert len(http_requests(log, '/time')) == 1 + + +# UTS: rest/proxy/RSL1k4/idempotent-retry-dedup-0 +async def test_rsl1k4_idempotent_retry_dedup(sandbox, proxy_session): + # `http_replace_response` forwards the publish to the sandbox and then + # discards the sandbox's answer in favour of a 503, so the message is + # persisted while the client is told it failed. What the test is about is + # what the server does with the retry that follows: the message carries the + # id the library generated for it, so the second copy is recognised as the + # first and dropped. + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'method': 'POST', 'pathContains': '/channels/'}, + 'action': { + 'type': 'http_replace_response', + 'status': 503, + 'body': {'error': {'code': 50300, 'statusCode': 503, + 'message': 'Service temporarily unavailable'}}, + }, + 'times': 1, + 'comment': 'RSL1k4: Forward first publish to server, then return fake 503 to client', + }]) + + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + fallback_hosts=['localhost'], + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + idempotent_rest_publishing=True, + ) + + channel_name = f'test-RSL1k4-idempotent-{random_id()}' + channel = client.channels.get(channel_name) + + # Publish a message -- first attempt succeeds server-side but client sees + # 503, SDK retries, server deduplicates the retry. + # The publish completed successfully: no error thrown. + await channel.publish(name='test', data='data') + + # Proxy event log shows at least two POST requests to /channels/. + # Read before history, which is a GET through the same session and so + # lands in the same log. + log = await session.get_log() + assert len(http_requests(log, '/channels/', method='POST')) >= 2 + + # Verify via history that only one copy of the message exists + # (server deduplicated the retry based on the library-generated message id) + async def published_message(): + page = await channel.history() + matching = [message for message in page.items + if message.name == 'test' and message.data == 'data'] + return matching or None + + matching = await wall_clock_poll_until( + published_message, description='the published message to reach history') + assert len(matching) == 1 From 35e37acbc2a22b6d5feb9a0ccbda15c33818bcf4 Mon Sep 17 00:00:00 2001 From: owenpearson Date: Thu, 24 Sep 2026 17:23:32 +0100 Subject: [PATCH 10/10] docs: record the proxy tier and the httpRequestTimeout unit defect MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The suite's README gains what a reader needs about the proxy package: where the binary comes from, the two environment variables that change how it is obtained, and the shape of a client built against a session. The translation skill gains what a writer needs — the harness table, a worked example and the traps the derivation hit, among them that a parent package's pytest timeout marker wins over a subpackage's unless the subpackage prepends its own, and that pytest-asyncio runs a session-scoped async fixture on a different event loop from the tests. deviations.md carries the one gated test and its measurement. The unit mismatch on http_request_timeout was already recorded twice, as something readable only off the options object; the proxy measures it reaching the wire, so both rows are corrected and the defect is counted once, as a root cause, where the gated test is. Its blast radius is bounded by the defaults coming out right by coincidence — 4 and 10 seconds being TO3l3's and TO3l4's 4000 and 10000 milliseconds — so only a client that configures the option is affected. Issue #709 is already filed against the same line for a different defect and the two are cross-referenced, since both change what one attempt may spend of the RSC15 retry budget. The counts are recomputed throughout: 1059 Test IDs, 1068 derived tests, 1139 pytest cases, and 67 gated root causes. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/uts-to-python/SKILL.md | 151 +++++++++++++++++++++++++- test/uts/README.md | 48 ++++++++ test/uts/deviations.md | 133 +++++++++++++++++++---- 3 files changed, 306 insertions(+), 26 deletions(-) diff --git a/.claude/skills/uts-to-python/SKILL.md b/.claude/skills/uts-to-python/SKILL.md index cba62713..1dbeb50e 100644 --- a/.claude/skills/uts-to-python/SKILL.md +++ b/.claude/skills/uts-to-python/SKILL.md @@ -16,10 +16,13 @@ gh api repos/ably/specification/contents/uts/docs/integration-testing.md --jq '. gh api repos/ably/specification/contents/uts/rest/unit/.md --jq '.content' | base64 -d gh api repos/ably/specification/contents/uts/realtime/unit/.md --jq '.content' | base64 -d gh api repos/ably/specification/contents/uts/rest/integration/.md --jq '.content' | base64 -d +gh api repos/ably/specification/contents/uts/docs/proxy.md --jq '.content' | base64 -d +gh api repos/ably/specification/contents/uts/rest/integration/proxy/.md --jq '.content' | base64 -d ``` `writing-derived-tests.md` governs, and `integration-testing.md` alongside it for -`uts/rest/integration`. This file covers only what is particular to ably-python. +`uts/rest/integration`, with `proxy.md` governing the proxy package within it. This +file covers only what is particular to ably-python. ## Layout @@ -285,9 +288,10 @@ All in `test.uts.helpers.clock`. ## The integration tier `uts/rest/integration/.md` becomes `test/uts/rest/integration/_test.py` and -runs against the real Ably sandbox — eleven specifications, 76 tests. There is no mock -and no `test_options`: the client reaches the network. `test/uts/README.md` covers the -same ground for someone reading the suite; this is what someone writing a test needs. +runs against the real Ably sandbox — twelve specifications, 84 tests, one of them the +proxy package the section below covers. There is no mock and no `test_options`: the +client reaches the network. `test/uts/README.md` covers the same ground for someone +reading the suite; this is what someone writing a test needs. What differs from the mock-backed tiers: @@ -333,6 +337,86 @@ async def test_rsl2a_history_returns_messages(sandbox, use_binary_protocol): history = await wall_clock_poll_until(one_message, description='the message to reach history') ``` +## The proxy tier + +`uts/rest/integration/proxy/.md` becomes +`test/uts/rest/integration/proxy/_test.py`, and routes its traffic through +[ably/uts-proxy](https://github.com/ably/uts-proxy) on the way to the sandbox. The +proxy binds a port per session, takes plain HTTP on it, speaks TLS onwards, applies the +session's rules and records what crosses it. `uts/docs/proxy.md` governs the tier and +is worth reading in full before deriving one of these; fetch it the same way as the +other governing docs. + +What differs from the rest of the integration tier: + +- **The client points at the session, not at the sandbox.** `endpoint='localhost'`, + `port=session.proxy_port`, `tls=False`, `use_binary_protocol=False`. `endpoint` and + `port` set the primary host, and `fallback_hosts=['localhost']` sets the fallback to + the same session, so both attempts land in one event log. Leave `fallback_hosts` out + where the spec does: `endpoint='localhost'` disables fallbacks by itself (REC2c2). +- **Authentication is a callback.** A plain connection cannot carry basic auth, so + every client takes `auth_callback=`; see the traps below. +- **A fault is a rule, and everything else passes through.** `times: 1` faults the + first request that matches and lets the retry reach the sandbox, which is the shape + every fallback scenario wants. +- **The proxy is the second witness.** The SDK's own result answers half the question + and `session.get_log()` the other half — how many requests were made, in what order, + and what the proxy answered them with. +- **The per-test timeout is 300 seconds**, set by the package's own `conftest.py`. + +| Name | Is | +|---|---| +| `proxy_session` fixture, in `rest/integration/proxy/conftest.py` | a specification's `create_proxy_session(...)`, together with its `AFTER EACH TEST: session.close()`. `session = await proxy_session(rules=[...])`, as many times as a test needs, and every session is closed afterwards | +| `proxy_control` fixture | the running control API, session-scoped. `proxy_session` asks for it, so a test does not have to | +| `create_proxy_session(endpoint=SANDBOX_ENDPOINT, port=None, rules=None, timeout_ms=SESSION_TIMEOUT_MS)` | the function itself, in `test.uts.helpers.proxy`, for the rare case that wants a session the fixture will not close | +| `session.session_id`, `session.proxy_host`, `session.proxy_port` | what a spec reads off its `session`. The host is always `localhost` | +| `session.add_rules(rules, position='append')` | rules added while the session runs. `position='prepend'` puts them ahead of the ones already there, which is how a spec faults traffic only once the client has reached some state | +| `session.trigger_action(action)` | the imperative half — `{'type': 'disconnect'}`, `{'type': 'close', 'closeCode': 1000}`, `inject_to_client`. 409 from the control API when no connection is open | +| `session.get_log()` | every event, in order, as the dictionaries the control API sends: `type`, `method`, `path`, `status`, `direction`, `queryParams`, `message`, `ruleMatched`. The field names are the ones a specification's assertions are written against, so they are not renamed | +| `session.close()` | best effort and never raises; the fixture calls it for you | +| `ensure_proxy()` / `stop_proxy()` | start the control process and reap it. Only the `proxy_control` fixture should need them | +| `PROXY_VERSION`, `ARCHIVE_CHECKSUMS`, `RELEASE_URL` | the pinned release and the sha256 of each platform's archive. Moving the pin means new checksums, copied from the release's own `checksums.txt` | +| `SESSION_TIMEOUT_MS` (120000) | the session's **idle** auto-cleanup timer, passed as `timeoutMs` | +| `SUITE_TIMEOUT` (300) in the package `conftest.py` | the per-test timeout for this package | +| `UTS_PROXY_LOCAL_PATH` | a locally built binary, or a `.tar.gz` holding one, in place of the pinned release | +| `UTS_PROXY_CONTROL_URL` | a control API already running, which is then neither started nor stopped by the suite | + +Everything but the two fixtures is in `test.uts.helpers.proxy`. `http_requests` and +`http_responses` in the example below are the derived file's own log filters, defined +at the top of it, because every test in the file reads the log the same two ways. + +```python +# UTS: rest/proxy/RSC15l4/cloudfront-header-fallback-0 +async def test_rsc15l4_cloudfront_header_fallback(sandbox, proxy_session): + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'pathContains': '/time'}, + 'action': { + 'type': 'http_respond', + 'status': 403, + 'body': {'error': {'message': 'Forbidden', 'code': 40300, 'statusCode': 403}}, + 'headers': {'Server': 'CloudFront'}, + }, + 'times': 1, + 'comment': 'RSC15l4: CloudFront 403 on first /time request', + }]) + + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', + fallback_hosts=['localhost'], + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + ) + + result = await client.time() + assert isinstance(result, (int, float)) + + log = await session.get_log() + assert len(http_requests(log, '/time')) >= 2 + assert http_responses(log)[0]['status'] == 403 +``` + ## Traps that cost the most time Ordered by how much they cost, not by subject. Every one was hit for real while @@ -596,6 +680,61 @@ that **passes while proving nothing**, rather than one that fails. Use `wall_clock_poll_until`, and no `FakeClock`. `writing-derived-tests.md` has a section on this ("Integration timeouts are wall-clock"). +## Traps found while deriving the proxy tier + +Established against the real proxy and the real sandbox while deriving +`rest/integration/proxy/rest_fallback.md`. + +- **Basic auth cannot be used through the proxy at all.** The session speaks plain + HTTP, and `tls=False` makes the SDK raise `40103 "Cannot use Basic Auth over non-TLS + connections"` (RSC18) before a request is written, so a client built with `key=` + never reaches a rule. `client.time()` is the one call that works, because it is + `skip_auth`. Every test in the specification therefore authenticates with + `authCallback`, including the `/time` ones, and a derived test should keep that even + where it looks unnecessary. +- **The token callback's own client must go straight to the sandbox.** A callback that + asks for a token through a client pointed at the session puts a request in front of + the waiting rule and an extra `http_request` in the log, which breaks every + assertion that counts requests exactly (`== 1` for RSC15l's 4xx test, `>= 2` for the + fallback ones). Build an inner `AblyRest(key=api_key, endpoint=SANDBOX_ENDPOINT)`, + request the token through that, and close it. +- **`http_response` events carry no `path`.** They have `status` and `ruleMatched` + only, so "the injected response fired" is read off the response events **in order** + — `http_responses(log)[0]['status'] == 403` — rather than by filtering to an + endpoint. `http_request` events do carry `method` and `path`. A rule with no + `comment` appears as `ruleMatched: "rule-0"`. +- **The parent package's timeout marker wins unless the subpackage prepends its + own.** `pytest-timeout` reads the first of an item's own markers, and + `rest/integration/conftest.py` marks everything beneath it with 120 seconds. The + proxy package's `pytest_collection_modifyitems` adds its 300 with + `append=False`; without that the marker order decides the timeout and a test that + waits out a twenty-second delay on a cold cache is cut off. Anyone adding a further + sub-tier under `test/uts/rest/integration/` hits this. +- **A session-scoped async fixture runs on a different event loop from the tests.** + pytest-asyncio 0.23 gives the session fixture its own loop, so an object bound to + the loop that created it — an `httpx.AsyncClient`, say — must not be held across the + yield: reusing it from a test raises `RuntimeError: Event loop is closed` or attaches + to the wrong loop. `helpers/proxy.py` opens a client per control call for exactly + this reason, the way `sandbox.py` does. +- **The session's `timeoutMs` is an idle timer, not a deadline.** The proxy's default + is 30000, measured from the last traffic through the session, and a test that has + the proxy delay a response by twenty seconds and then reads the log spends longer + than that idle, which is long enough for the session to be collected out from under + the test. The harness passes 120000. +- **Every request the test's client makes lands in the same log.** The log is + per-session, not per-endpoint, so a verification step that reads history through the + same client adds its own `http_request` events. In `RSL1k4` the log is read + **before** the history call, and the history read is a `wall_clock_poll_until` rather + than a single fetch, because a published message does not reach history at once. +- **`httpRequestTimeout` is milliseconds in the specification and seconds in + ably-python.** `ably/http/http.py:193` hands `(http_open_timeout, + http_request_timeout)` to `httpx`, which reads seconds, so the specification's + `http_request_timeout=3000` is a 3000-second deadline: measured, the request sat out + the proxy's whole 20-second delay and then succeeded on the primary host, and no + fallback was attempted. Passing `3` makes the same test pass in 3.1s, so only the + unit is wrong. The test is written as the specification has it and gated with + `@deviation`; `test/uts/deviations.md` carries the entry. + ## Timers Three regimes; pick by tier. @@ -617,8 +756,8 @@ costs 120 real seconds. See the fake-time section of `test/uts/deviations.md`. a server the tests exist to talk to, and shortening a timeout would only make the tier flaky. Poll with `wall_clock_poll_until` rather than sleeping a guess. -The pytest timeout is 30 seconds for the suite and 120 for `rest/integration`, so keep -waits well under whichever applies. +The pytest timeout is 30 seconds for the suite, 120 for `rest/integration` and 300 for +`rest/integration/proxy`, so keep waits well under whichever applies. ## Deviations diff --git a/test/uts/README.md b/test/uts/README.md index 724ab9ce..d3fea133 100644 --- a/test/uts/README.md +++ b/test/uts/README.md @@ -144,6 +144,54 @@ Tests here are given 120 seconds each, per `uts/docs/integration-testing.md`, ra than the 30 seconds `pyproject.toml` sets for the suite as a whole. The marker covers the integration package alone. +`rest/integration/proxy/` routes its traffic through +[ably/uts-proxy](https://github.com/ably/uts-proxy), a programmable proxy standing +between the client and the sandbox. Those specifications are about what the SDK does +when a request goes wrong — a connection dropped mid-request, a 503, a CloudFront 403, +a response held past the request timeout — and the sandbox answers correctly, so the +fault is injected in front of it. The proxy binds a port per session, takes plain HTTP +on it and speaks TLS onwards to the sandbox, applies the rules the session was opened +with, and records every request and response that crosses it. + +The binary is a pinned `uts-proxy` release. The first run that needs it fetches the +release archive for the machine, checks it against the sha256 the release publishes, +and extracts the binary into `~/.cache/uts-proxy//`, where every run +afterwards finds it; the download is serialised on a lock file, so several Python +versions starting at once on an empty cache fetch it once between them. +`UTS_PROXY_LOCAL_PATH` names a locally built binary, or a `.tar.gz` holding one, to be +used in place of the release, and `UTS_PROXY_CONTROL_URL` names a control API someone +is already running, which the suite uses as it stands and leaves running. Otherwise one +control process is started for the test session on a free port and reaped at the end of +it; it serves every session the run opens. + +`proxy_session` is a specification's `create_proxy_session(...)`, and closes every +session it hands out when the test ends. A client reaches its session by naming +`localhost` and the session's port with TLS off, which disables fallback hosts (REC2c2); +a scenario about a retry names the same session again as its fallback, so both attempts +arrive at the one port and appear in the one event log, which `session.get_log()` +returns: + +```python +async def test_rsc15l_connection_drop_fallback(sandbox, proxy_session): + session = await proxy_session(rules=[{ + 'match': {'type': 'http_request', 'pathContains': '/time'}, + 'action': {'type': 'http_drop'}, + 'times': 1, + }]) + client = sandbox_rest_client( + auth_callback=token_auth_callback(sandbox.key_str), + endpoint='localhost', fallback_hosts=['localhost'], + port=session.proxy_port, tls=False, use_binary_protocol=False) +``` + +A plain connection rules basic auth out, since RSC18 refuses it, so every client here +authenticates through a callback whose own request goes straight to the sandbox and +stays out of the event log. + +Tests in this package are given 300 seconds each: a cold cache downloads the binary +before the first of them runs, and a specification that provokes a timeout sits through +the delay it asked the proxy for. + ## Running ``` diff --git a/test/uts/deviations.md b/test/uts/deviations.md index 5076aba5..1637afed 100644 --- a/test/uts/deviations.md +++ b/test/uts/deviations.md @@ -24,30 +24,33 @@ One Test ID can become more than one derived test: five Test IDs in `rest/unit` `error_types_test.py`, `fallback_test.py`, `rest_client_test.py` (two) and `paginated_result_test.py` — assert several independent things under a single id, and the derivation writes a function for each rather than one function with an unrelated -second half. That turns 1051 Test IDs into 1060 derived tests. Going the other way, one -derived test can become more than one case: five of the eleven `rest/integration` +second half. That turns 1059 Test IDs into 1068 derived tests. Going the other way, one +derived test can become more than one case: five of the twelve `rest/integration` specifications carry a `## Protocol Variants` section and run every one of their tests twice, once per protocol, and nine `rest/unit` tests are parametrized over a table of -fixtures the specification gives inline. That turns 1060 derived tests into 1131 pytest +fixtures the specification gives inline. That turns 1068 derived tests into 1139 pytest cases. -Of **1051 Test IDs, derived as 1060 tests and run as 1131 pytest cases**: 834 Test IDs -(843 tests, 910 cases) pass, 202 (202 tests, 206 cases) are gated behind -`RUN_DEVIATIONS`, and 15 (15 tests, 15 cases) cannot be run at all. Every gated test has -been confirmed to fail when enabled, so none of them passes under both behaviours. 494 -of the Test IDs come from `uts/rest/unit` (503 tests, 536 cases), 481 from -`uts/realtime/unit` (481, 481) and 76 from `uts/rest/integration` (76, 114); of the -gated Test IDs 121 are REST and 81 realtime, which is 125 REST cases and 81 realtime. +Of **1059 Test IDs, derived as 1068 tests and run as 1139 pytest cases**: 841 Test IDs +(850 tests, 917 cases) pass, 203 (203 tests, 207 cases) are gated behind +`RUN_DEVIATIONS`, and 15 (15 tests, 15 cases) cannot be run at all. The three groups are +disjoint: two Test IDs, and one parametrized test, have a gated part and a passing part, +and are counted with the gated. Every gated test has been confirmed to fail when +enabled, so none of them passes under both behaviours. 494 of the Test IDs come from +`uts/rest/unit` (503 tests, 536 cases), 481 from `uts/realtime/unit` (481, 481) and 84 +from `uts/rest/integration` (84, 122), 8 of those (8, 8) from the `proxy` package within +it; of the gated Test IDs 122 are REST and 81 realtime, which is 126 REST cases and 81 +realtime. A further 122 pytest cases under `helpers/` cover the mock infrastructure itself and are not derived from a specification. -The 192 gated Test IDs that record SDK non-compliance — 192 tests, 196 cases — reduce to -**66 distinct root causes**, 26 on the REST side and 40 on the realtime side. Three further +The 193 gated Test IDs that record SDK non-compliance — 193 tests, 197 cases — reduce to +**67 distinct root causes**, 27 on the REST side and 40 on the realtime side. Three further defects are recorded below with no test of their own, because the specification's test cannot discriminate (RTP18a), has nothing to assert against (the timezone split on synthesized LEAVE timestamps), or is worked around in the setup of every test that would otherwise trip over it (`enterClient` on an anonymous connection), so the file -carries **69 SDK root causes** in all. The remaining 10 gated Test IDs are +carries **70 SDK root causes** in all. The remaining 10 gated Test IDs are specification faults, and reduce to 7. Entries closed by a fix are removed rather than kept as history; `git log` holds that. @@ -1377,6 +1380,51 @@ which is about a client that *does* configure `clientId: "*"`. |---|---| | TO3 | `endpoint` is left unset when none is given, per `TO/endpoint-affects-host-0`. `Options.__init__` resolves the default eagerly to the REC1a routing policy id `main`, so the attribute never reads back as null. Only the default case is gated; the two that name an endpoint are derived and pass | +#### `httpRequestTimeout` is seconds where the specification counts milliseconds — 1 test + +**Spec points:** TO3l4, RSC15l2, `rest/proxy/RSC15l2/timeout-triggers-fallback-0`. + +`ably/http/http.py:193` builds `timeout = (self.http_open_timeout, +self.http_request_timeout)` and hands it to `httpx`, which reads both as **seconds**. +TO3l4's `httpRequestTimeout` is milliseconds, default 10000, and +`CONNECTION_RETRY_DEFAULTS` holds `10`. So a caller passing the value the published +specification describes gets a deadline a thousand times longer than the one asked for: +`http_request_timeout=3000` is three thousand seconds. The **defaults** come out right +by coincidence — 10 seconds is TO3l4's 10000 ms, and `http_open_timeout`'s 4 is TO3l3's +4000 — which is why only a client that configures the option is affected, and why the +mismatch reads as cosmetic anywhere it is only compared against `options`. + +Measured through the proxy. Against a session delaying the first `/time` by 20 seconds, +a client built with the specification's `httpRequestTimeout: 3000` sat out the whole +delay, succeeded on the primary host, and tried no fallback — one `/time` request in the +event log where the test asserts two, which is the `assert 1 >= 2` the gated run shows. +Passing `3` in its place makes the same test pass in 3.1 seconds: the timeout fires, the +retry goes to the fallback host and succeeds. So RSC15l2's fallback path is compliant and +the unit is the whole of the defect. + +The same mismatch is recorded twice under *Adapted Tests*, at `TO3l1, TO3l5` and at +`RTC7 (TO3l3, TO3l4)` in *REST behaviours asserted as they are*, where it is what makes +the effective defaults unreadable from `options`. This is that defect seen from outside: +the same line of `http.py`, reached through a public client option rather than through an +attribute, so a caller is affected whether or not they ever read `options`. It is counted +as one root cause, here, because it is the only gated test on it; neither Adapted Tests +row is counted again. + +**Tests affected:** `test_rsc15l2_timeout_triggers_fallback`, the one gated test in +`rest/integration/proxy`. The test is written exactly as the specification has it, with +`http_request_timeout=3000`, so removing the gate is all that is needed once the unit is +fixed. + +**Status:** open bug. The fix converts at the boundary — `Http` dividing the option by +1000 before it reaches `httpx`, with `CONNECTION_RETRY_DEFAULTS` restated in +milliseconds — and has to move `http_open_timeout` (TO3l3) with it, since both halves of +the tuple are read the same way. That same line carries a second, filed defect: +[#709](https://github.com/ably/ably-python/issues/709) is that the value, in whatever +unit, bounds one socket read rather than the request, so a connection that keeps +producing frames is never timed out and `write` and `pool` are left unbounded. The two +want fixing together, since both change what one attempt may spend of the RSC15 retry +budget. + ### Requests | Spec points | Behaviour | @@ -1767,6 +1815,8 @@ pseudocode is not mistaken for non-compliance. | RTP19a's route | the specification models an ATTACHED without HAS_PRESENCE as `startSync()` then `endSync()`. `on_attached(has_presence=False)` calls `_synthesize_leaves(...)` then `clear()` (`presence.py:611-618`), which is the requirement itself rather than the model of it | | `hasNext` as a value | `push_admin.md` RSH1b2 writes `ASSERT result.hasNext == true`. In Python `has_next` is a bound method (`ably/http/paginatedresult.py:63`), truthy whatever the page holds, so the direct translation asserts nothing. `test_rsh1b2_list_devices_pagination` writes `result.has_next() is True`, confirmed to fail when inverted | | Push `remove` return values | the specification's remove steps assert nothing about a return value ("should not throw"). `PushDeviceRegistrations.remove` / `remove_where` and `PushChannelSubscriptions.remove` / `remove_where` return the `ably.http.http.Response` from the DELETE rather than `None` (`ably/rest/push.py:112-127`, `:176-192`), so the six derived removal tests assert `response.status_code == 204`, which is stronger than the specification asks and matches what `test/ably/rest/restpush_test.py` already asserts | +| `rest/proxy/RSC15l/unreachable-endpoint-error-0` | the specification asserts only that the error carries a non-null `status_code` or `code`, leaving the values open, so the derived test asserts exactly that and is the weakest of the eight proxy tests. What this SDK produces against a refused connection is 500 / 50000 with the message "All connection attempts failed", `catch_all` having wrapped the `httpx.ConnectError`, and that is recorded here rather than asserted, since pinning it would assert more than the specification does | +| `RSL1k4`'s history read | the event log is per session, so any request the client under test makes through it is recorded — including the `history()` the test verifies deduplication with. The log is therefore read **before** the history call, and the POST count assertion made against that snapshot. The history read itself is a `wall_clock_poll_until` rather than one fetch, for the same reason every other integration verification is: a published message does not reach history at once | | RSP4b1's time bounds | the specification records `time_before = now_millis()` before generating the presence events and `time_after = now_millis()` after, then asserts a `history(start=, end=)` over that window returns them. Read from the runner's clock the window is only as good as the skew against the sandbox, which decides the timestamps actually stored, so a runner running a little fast would exclude the very events the test generated. Both bounds come from `await client.time()` instead — the same instant on the clock that stamps the events. `Presence.history` passes an `int` straight through as milliseconds (`ably/types/presence.py:232-241`), which is what `client.time()` returns, so no conversion is involved | ### REST behaviours asserted as they are @@ -1794,8 +1844,8 @@ pseudocode is not mistaken for non-compliance. | CHM2g, CHM2h | `objectPublishers` and `objectSubscribers` on `ChannelMetrics` | Neither is modelled, so both are dropped on parsing. The test asserts their absence, and turns red once they are added | Open bug | | TO3l8 | `maxMessageSize` is a client option, default 65536 | Rejected by `Options.__init__`. `ably/realtime/channel.py:422` reads it with `getattr(..., 65536)`, so the default holds but cannot be configured, nor overridden by `connectionDetails` (CD2c) | Open bug | | RTN15, RTN23 | A DISCONNECTED `ErrorInfo` needs no `statusCode` | `ConnectionManager.on_disconnected` evaluates `exception.status_code >= 500` unguarded, so a DISCONNECTED whose error omits `statusCode` raises `TypeError` in a task whose exception is only logged, and the connection silently stays CONNECTED. `DISCONNECTED_MESSAGE` supplies 400 | Open bug | -| TO3l1, TO3l5 | `httpRequestTimeout` and `httpMaxRetryCount` carry their defaults on the options object | Left unset; the effective defaults are applied downstream by `Http` and by `Options.__get_hosts`. The spec's values are milliseconds, while ably-python's `http_request_timeout` is seconds | Intentional | -| RTC7 (TO3l3, TO3l4) | `client.options.httpOpenTimeout == 4000` and `httpRequestTimeout == 10000` | Both `None` on `Options`; `Http.http_open_timeout` / `http_request_timeout` fall back to `CONNECTION_RETRY_DEFAULTS`, which holds 4 and 10 — seconds, because that is what `httpx` takes. The three realtime timeouts the same test checks are defaulted on `Options` and match | Open bug for the default being unreadable from `options`; the unit difference alone is internal | +| TO3l1, TO3l5 | `httpRequestTimeout` and `httpMaxRetryCount` carry their defaults on the options object | Left unset; the effective defaults are applied downstream by `Http` and by `Options.__get_hosts`. The spec's `httpRequestTimeout` is milliseconds and ably-python's `http_request_timeout` is seconds, on the value a caller passes as much as on the default — gated, with the measurement, under *`httpRequestTimeout` is seconds where the specification counts milliseconds* in Failing Tests | Intentional for where the defaults are applied; the unit is an open bug, recorded there | +| RTC7 (TO3l3, TO3l4) | `client.options.httpOpenTimeout == 4000` and `httpRequestTimeout == 10000` | Both `None` on `Options`; `Http.http_open_timeout` / `http_request_timeout` fall back to `CONNECTION_RETRY_DEFAULTS`, which holds 4 and 10 — seconds, where TO3l3 and TO3l4 count milliseconds. The three realtime timeouts the same test checks are defaulted on `Options` and match | Two faults in one row: the defaults are unreadable from `options`, and the unit reaches the wire — `rest/proxy/RSC15l2/timeout-triggers-fallback-0` measures a request outliving its configured timeout by a factor of a thousand. Both open; the unit is gated under *`httpRequestTimeout` is seconds where the specification counts milliseconds* in Failing Tests | | RTC17 (RSA7b1) | `client.clientId == client.auth.clientId` | `AblyRealtime.client_id` reads `options.client_id` and returns the configured value, while `Auth.__init__` sets `self.__client_id = None` whenever `ably._is_realtime` (`rest/auth.py:34-41`), deferring it to whatever a CONNECTED confirms. The two disagree on a client that has not connected | Open bug. RSA12b only allows the realtime clientId to be unknown while it has not been *configured* | | RTC1f | a `transportParams` boolean appears as `"true"` / `"false"` | `True` / `False`, because `WebSocketTransport.connect` builds the query string with `urllib.parse.urlencode`, which renders each value through `str()` (`websockettransport.py:89`). Integers are unaffected | Open bug. A caller can pass the strings directly, but a bool is what the spec's Stringifiable type admits | @@ -2029,7 +2079,7 @@ is missing accessors over correct behaviour. Within a tier the order is blast ra | [#706](https://github.com/ably/ably-python/issues/706) | the fabricated `"None:0"` message id | 3.3 | | [#658](https://github.com/ably/ably-python/issues/658) | presence messages sent on reconnection before reattach | adjacent to 3.14, which is the other half of RTP17 automatic re-entry | | [#656](https://github.com/ably/ably-python/issues/656) | `utcfromtimestamp` deprecation | adjacent to 2.4 | -| [#709](https://github.com/ably/ably-python/issues/709)–[#712](https://github.com/ably/ably-python/issues/712) | REST request timeout, token-request nonce reuse, single-host retry, `dispose()` teardown | none — filed from the REST derivation, and distinct from everything here | +| [#709](https://github.com/ably/ably-python/issues/709)–[#712](https://github.com/ably/ably-python/issues/712) | REST request timeout, token-request nonce reuse, single-host retry, `dispose()` teardown | adjacent to I.3, which is a second defect on the line #709 is about and is not covered by it; otherwise none, these having been filed from the REST derivation | Every reproduction below is prefixed by: @@ -2360,8 +2410,10 @@ none of these shows up as a failure — which is why they are easy to lose. The five tiers above classify the realtime derivation; the REST unit derivation's candidates went upstream as [#709](https://github.com/ably/ably-python/issues/709)–[#712](https://github.com/ably/ably-python/issues/712). -Two further defects came out of `rest/integration`, and neither is filed. Both are -tier 2 by the ranking above — an error where there should be none. +Three further defects came out of `rest/integration`, and none of them is filed. The +first two are tier 2 by the ranking above — an error where there should be none. The +third is tier 1 for a client that configures the option it concerns, since the call does +not come back when the caller asked for it to. **I.1 `Rest#request` never renews an expired token.** RSC10, RSC19. Every other REST operation renews and retries on a 40140–40149; `Rest#request` returns the 401 to the @@ -2382,6 +2434,20 @@ fix, and `enter_client` is unusable without one. No test gates on it — the thr would hit it pass `client_id='*'` in setup instead, as the repository's own presence suite does — so it will not show up as a failure. +**I.3 `httpRequestTimeout` is applied as seconds where the specification counts +milliseconds.** TO3l4, TO3l3, RSC15l2. `ably/http/http.py:193` hands +`(http_open_timeout, http_request_timeout)` to `httpx`, which reads seconds, so a client +built with the specification's `httpRequestTimeout: 3000` waits three thousand seconds. +Measured through `uts-proxy` against a session delaying `/time` by 20 seconds: the +request sat out the whole delay, succeeded on the primary host and attempted no +fallback, where `http_request_timeout=3` timed out at 3.1 seconds and the fallback retry +succeeded. The defaults are unaffected, 4 and 10 seconds being TO3l3's and TO3l4's 4000 +and 10000 ms, so this reaches only a client that sets the option — but such a client gets +no timeout and no fallback at all. Distinct from +[#709](https://github.com/ably/ably-python/issues/709), which is about the same value +bounding a single socket read rather than the request; a fix wants to settle both. +`test/uts/rest/integration/proxy/rest_fallback_test.py -k rsc15l2` + ## How the specifications are adopted here Choices about the approach, as against the behaviour recorded above. @@ -2649,8 +2715,8 @@ indices are the capabilities each specification's app-provisioning section names A specification carrying a `## Protocol Variants` section runs each of its Test IDs twice, through a `use_binary_protocol` fixture parametrized `[False, True]` with the ids `json` -and `msgpack`. Five of the eleven specifications carry that section, so 38 of the 76 -integration Test IDs are two pytest cases each. The six that do not are json only and +and `msgpack`. Five of the twelve specifications carry that section, so 38 of the 84 +integration Test IDs are two pytest cases each. The seven that do not are json only and take the default `sandbox_rest_client` applies. This is why the counts in this file give Test IDs and cases separately. @@ -2660,6 +2726,33 @@ An autouse fixture closes every client a test built, whether or not its assertio places actively destroys what the following REST read is about — see the UTS Spec Error above. Tests omit the inline close and leave it to teardown. +### The proxy package runs against a pinned proxy, with a session per test + +`uts/docs/proxy.md` puts `ably/uts-proxy` between the client and the sandbox for the +specifications under `rest/integration/proxy`, and three harness choices follow from +having to supply the proxy itself. + +The release is pinned and verified rather than built or assumed present. The archive for +the machine is downloaded on first use, checked against the sha256 the release publishes, +and extracted into `~/.cache/uts-proxy//`, under a lock file so that the several +Python versions CI runs fetch it once between them. `UTS_PROXY_LOCAL_PATH` substitutes a +locally built binary or distributive, and `UTS_PROXY_CONTROL_URL` substitutes a control +API a developer is already running, which the suite then leaves alone. One control +process is started per test run on a free port, rather than a fixed one, so two suites on +a machine do not collide, and it is reaped at the end of the run and again at interpreter +exit. + +A session is per test and the `proxy_session` fixture closes every one it handed out, +which is the specifications' `AFTER EACH TEST: IF session IS NOT null: session.close()` +without each test having to carry it. The session's `timeoutMs` is set to 120000 against +the proxy's own 30000, because it is an idle timer and one of these tests sits through a +twenty-second delay before it reads anything; the package's per-test pytest timeout is +300 seconds against the tier's 120, for the delay and for the download the first test may +wait on. Every client in the package authenticates through an `authCallback` whose own +request goes straight to the sandbox: the session speaks plain HTTP, RSC18 refuses basic +auth over it, and a token request routed through the session would be counted by the +assertions that count requests. + ### A hedged integration setup is provisioned so its guarded assertions bite `time_stats.md` hedges its setup in a way that lets both its tests pass without testing