-
Notifications
You must be signed in to change notification settings - Fork 2
Tracker Protocol
Note
This applies to the PostgreSQL rewrite, which has not yet been fully released.
The tracker protocol is really simple. It uses WebSocket connections to reduce overhead. Connections have no state, except for HTTP Basic authentication to determine which pipeline the tracker is communicating with.
All commands must be atomic. Either they succeed, or they do not succeed, and there is no intermediate state. If something goes wrong, all persistent state must remain consistent. This allows for retries and makes the system more robust.
Requests currently have two fields — type and seq. type is the message type, and seq is a sequence number that the client generates to ensure messages don't get mixed up. Some requests will also require a third field, request, containing any additional parameters. The request field is a dictionary; keys must be strings.
Requests are sent from the client to the server.
Responses currently have three fields—status, payload, and seq. status is an HTTP status code corresponding to the result of the request. payload is the return value. seq is a copy of the seq value from the request.
In this article, when a response is described as having no payload, that should be interpreted as a payload of null. A server must provide a payload field even if it is null.
Responses are sent from the server to the client in response to requests.
Advisories are a special case. They are not directly triggered by a request; instead, the server may subscribe a client to certain advisories as a result of a request. Advisories have three fields — type, payload, and seq. seq is always None, to denote the message as an advisory.
Informs the server that the pipeline is still alive, and provides disk usage information.
Takes a disk parameter, which is an object containing free and total attributes (in bytes).
Elicits a response with a status code of 204 and no payload.
Request parameters:
-
slot: Pipeline slot that is dequeuing. Currently always 0.
Claims a page.
Elicits a response with a status code of 200. If an item was found, an object with further information is contained in the item field of the payload. Otherwise, the item field of the payload is null, and the message field is set to a human-readable reason why an item could not be provided.
Request parameters:
-
attempt_id: Identifier of the current attempt. -
message: Failure reason. -
fatal: Whether or not the error is fatal. (A fatal error won't requeue the page, even if there are still attempts remaining.)
Fails an attempt. It may or may not be cycled back into the queue.
Elicits a response with a status code of 204 and no payload.
Request parameters:
-
result_id: The result ID to use. (If the ID already exists, it will not be re-created.) -
attempt_id: The current attempt. -
type: The type of result. -
payload: The result data.
Saves a result associated with the item. In the case of an outlinks result, calculates the accept setting for each URL and adds all that apply.
Elicits a response with a status code of 201, and a new_id field containing the generated primary key of the result. In the case of an outlinks result, there is an additional urls_added field containing the number of accepted URLs (before deduplicating with the database).
Request parameters:
-
attempt_id: The current attempt.
Marks an attempt as finished.
Elicits a response with a status code of 204 and no payload.
There aren't currently any advisories, but this will likely be how task killing will be implemented.