test: derive the REST integration specifications

test/uts/README.md

@@ -17,11 +17,12 @@ upstream, and the choices behind how the specifications are adopted here.
 
 ```
 helpers/     shared infrastructure the specifications assume, and its own tests
+assets/      fixtures the specifications name, vendored from elsewhere
 rest/        specifications under uts/rest
 realtime/    specifications under uts/realtime
 ```
 
-Every directory needs an `__init__.py`, because `test` is a package.
+Every directory holding tests needs an `__init__.py`, because `test` is a package.
 
 Unit tests serve every request from a mock and reach no network — neither the REST
 suite nor the realtime one. Integration tests run against a sandbox app.
@@ -80,18 +81,135 @@ that closes its own leaves nothing to clean up if it fails first.
 | [helpers/client.py](helpers/client.py) | client constructors, and the `AWAIT_STATE` / `AWAIT UNTIL` equivalents |
 | [helpers/clock.py](helpers/clock.py) | `FakeClock`, `settle()` and `advance_to_connection_state()` — `enable_fake_timers()` and `ADVANCE_TIME(ms)` |
 | [helpers/presence.py](helpers/presence.py) | the presence-map stubs and wire-message builders the presence specifications share |
+| [helpers/sandbox.py](helpers/sandbox.py) | the sandbox app the integration tier provisions, the presence-fixture cipher, `random_id()` and the JWT signing the auth specification asks a library for |
 | [helpers/deviations.py](helpers/deviations.py) | the `@deviation` and `@spec_error` gates |
 
 `SKILL.md` lists every name in each. The helpers have their own tests
 (`helpers/*_test.py`), which are not derived from a specification and are not counted
 in the derived-test totals.
 
+## The integration tier
+
+`rest/integration/` runs against the real Ably sandbox, so it needs network access;
+nothing there is served from a mock. Each specification's `BEFORE ALL TESTS` block
+provisions an app from the canonical `test-resources/test-app-setup.json` in
+[ably/ably-common](https://github.com/ably/ably-common), vendored at
+[assets/test-app-setup.json](assets/test-app-setup.json), and deletes it afterwards.
+One app serves the whole session, since an app per test would be slow and would invite
+the sandbox's rate limiting. Provisioning goes over plain `httpx` rather than through
+`AblyRest`: it is infrastructure, and a client that cannot form a request should fail a
+test rather than look like a broken fixture. Teardown is best effort — a sandbox app
+expires on its own, so a failed delete is logged and nothing more.
+
+The app arrives as the `sandbox` fixture, which is a specification's `app_config`:
+
+```python
+async def test_rsl1n_publish_returns_serials(sandbox):
+    channel = sandbox_rest_client(sandbox.key(0).key_str).channels.get('test-serials-' + random_id())
+```
+
+`sandbox.key(i)` is the specifications' `app_config.keys[i]`, carrying `key_str`,
+`key_name`, `key_secret` and `capability`. The index means what it means in a
+specification — `keys[0]` full access, `keys[1]` push admin, `keys[2]` the per-channel
+capabilities, `keys[3]` subscribe-only, `keys[4]` revocable tokens — which is why the
+asset is the `ably-common` file and not `test/assets/testAppSpec.json`, whose keys sit
+at other indices. `sandbox.app_id` and `sandbox.key_str`, the full-access key, are
+there too.
+
+`sandbox_rest_client(key, ...)` and `sandbox_realtime_client(key, ...)` in
+[helpers/client.py](helpers/client.py) build the clients. They set the endpoint to the
+sandbox, carry no `test_options`, and register the client for the same teardown the
+mock-backed constructors use. A specification that authenticates some other way leaves
+the key out and passes `token=`, `auth_callback=` or `auth_url=`. The protocol defaults
+to JSON; the realtime client keeps `auto_connect` and the fallback hosts at the library
+defaults, since connecting is the point.
+
+Waits are wall-clock here, which is the opposite of the unit tier: `poll_until` spins on
+the event loop, so the integration tier uses `wall_clock_poll_until(condition, timeout,
+description)`, which sleeps the specifications' interval between attempts and returns
+whatever the condition answered with. Its condition may be sync or async. Nothing waits
+on a fixed sleep for something that can be polled for.
+
+The five specifications carrying a `## Protocol Variants` section — `publish`,
+`history`, `presence`, `batch_presence` and `mutable_messages` — run once per protocol.
+A test asks for that by taking the `use_binary_protocol` fixture and passing it on; a
+test that does not take it runs json only.
+
+```python
+async def test_rsl1d_publish_failure(sandbox, use_binary_protocol):
+    client = sandbox_rest_client(sandbox.key(2).key_str, use_binary_protocol=use_binary_protocol)
+```
+
+Tests here are given 120 seconds each, per `uts/docs/integration-testing.md`, rather
+than the 30 seconds `pyproject.toml` sets for the suite as a whole. The marker covers
+the integration package alone.
+
+`rest/integration/proxy/` routes its traffic through
+[ably/uts-proxy](https://github.com/ably/uts-proxy), a programmable proxy standing
+between the client and the sandbox. Those specifications are about what the SDK does
+when a request goes wrong — a connection dropped mid-request, a 503, a CloudFront 403,
+a response held past the request timeout — and the sandbox answers correctly, so the
+fault is injected in front of it. The proxy binds a port per session, takes plain HTTP
+on it and speaks TLS onwards to the sandbox, applies the rules the session was opened
+with, and records every request and response that crosses it.
+
+The binary is a pinned `uts-proxy` release. The first run that needs it fetches the
+release archive for the machine, checks it against the sha256 the release publishes,
+and extracts the binary into `~/.cache/uts-proxy/<version>/`, where every run
+afterwards finds it; the download is serialised on a lock file, so several Python
+versions starting at once on an empty cache fetch it once between them.
+`UTS_PROXY_LOCAL_PATH` names a locally built binary, or a `.tar.gz` holding one, to be
+used in place of the release, and `UTS_PROXY_CONTROL_URL` names a control API someone
+is already running, which the suite uses as it stands and leaves running. Otherwise one
+control process is started for the test session on a free port and reaped at the end of
+it; it serves every session the run opens.
+
+`proxy_session` is a specification's `create_proxy_session(...)`, and closes every
+session it hands out when the test ends. A client reaches its session by naming
+`localhost` and the session's port with TLS off, which disables fallback hosts (REC2c2);
+a scenario about a retry names the same session again as its fallback, so both attempts
+arrive at the one port and appear in the one event log, which `session.get_log()`
+returns:
+
+```python
+async def test_rsc15l_connection_drop_fallback(sandbox, proxy_session):
+    session = await proxy_session(rules=[{
+        'match': {'type': 'http_request', 'pathContains': '/time'},
+        'action': {'type': 'http_drop'},
+        'times': 1,
+    }])
+    client = sandbox_rest_client(
+        auth_callback=token_auth_callback(sandbox.key_str),
+        endpoint='localhost', fallback_hosts=['localhost'],
+        port=session.proxy_port, tls=False, use_binary_protocol=False)
+```
+
+A plain connection rules basic auth out, since RSC18 refuses it, so every client here
+authenticates through a callback whose own request goes straight to the sandbox and
+stays out of the event log.
+
+Tests in this package are given 300 seconds each: a cold cache downloads the binary
+before the first of them runs, and a specification that provokes a timeout sits through
+the delay it asked the proxy for.
+
 ## Running
 
 ```
 uv run --frozen --extra crypto --extra dev pytest test/uts -q
 ```
 
+The offline tiers alone, which need no network:
+
+```
+uv run --frozen --extra crypto --extra dev pytest test/uts/rest/unit test/uts/realtime/unit test/uts/helpers -q
+```
+
+The integration tier alone, which provisions a sandbox app and needs network access:
+
+```
+uv run --frozen --extra crypto --extra dev pytest test/uts/rest/integration -q
+```
+
 `--frozen` is required: without it dependency resolution reaches past the
 environment's cutoff. `--extra dev` carries pytest.
 

test/uts/assets/test-app-setup.json

@@ -0,0 +1,81 @@
+{
+  "post_apps": {
+    "limits": {
+      "presence": {
+        "maxMembers": 250
+      }
+    },
+    "keys": [
+      {},
+      {
+        "capability": "{ \"cansubscribe:*\":[\"subscribe\"], \"canpublish:*\":[\"publish\"], \"canpublish:andpresence\":[\"presence\",\"publish\"], \"pushenabled:*\":[\"publish\",\"subscribe\",\"push-subscribe\"], \"pushenabled:admin:*\":[\"publish\",\"subscribe\",\"push-admin\"] }"
+      },
+      {
+        "capability": "{ \"channel0\":[\"publish\"], \"channel1\":[\"publish\"], \"channel2\":[\"publish\", \"subscribe\"], \"channel3\":[\"subscribe\"], \"channel4\":[\"presence\", \"publish\", \"subscribe\"], \"channel5\":[\"presence\"], \"channel6\":[\"*\"] }"
+      },
+      {
+        "capability": "{ \"*\":[\"subscribe\"] }"
+      },
+      {
+        "revocableTokens": true
+      },
+      {
+        "capability": "{ \"[*]*\":[\"*\"] }"
+      }
+    ],
+    "namespaces": [
+      {
+        "id": "persisted",
+        "persisted": true
+      },
+      {
+        "id": "pushenabled",
+        "pushEnabled": true
+      },
+      {
+        "id": "mutable",
+        "mutableMessages": true
+      }
+    ],
+    "channels": [
+      {
+        "name": "persisted:presence_fixtures",
+        "presence": [
+          {
+            "clientId": "client_bool",
+            "data": "true"
+          },
+          {
+            "clientId": "client_int",
+            "data": "24"
+          },
+          {
+            "clientId": "client_string",
+            "data": "This is a string clientData payload"
+          },
+          {
+            "clientId": "client_json",
+            "data": "{ \"test\": \"This is a JSONObject clientData payload\"}"
+          },
+          {
+            "clientId": "client_decoded",
+            "data": "{\"example\":{\"json\":\"Object\"}}",
+            "encoding": "json"
+          },
+          {
+            "clientId": "client_encoded",
+            "data": "HO4cYSP8LybPYBPZPHQOtuD53yrD3YV3NBoTEYBh4U0N1QXHbtkfsDfTspKeLQFt",
+            "encoding": "json/utf-8/cipher+aes-128-cbc/base64"
+          }
+        ]
+      }
+    ]
+  },
+  "cipher": {
+    "algorithm": "aes",
+    "mode": "cbc",
+    "keylength": 128,
+    "key": "WUP6u0K7MXI5Zeo0VppPwg==",
+    "iv": "HO4cYSP8LybPYBPZPHQOtg=="
+  }
+}