diff --git a/docs/rfc/async-changed-targets.md b/docs/rfc/async-changed-targets.md new file mode 100644 index 00000000..b5e691ee --- /dev/null +++ b/docs/rfc/async-changed-targets.md @@ -0,0 +1,174 @@ +# Async Changed-Targets API + +## Summary + +`GetChangedTargets` holds a streaming RPC open while Bazel may run longer than the caller’s timeout. A disconnect aborts that work, and a retry repeats it because there is no durable handle to resume from. The result is already a content-addressed blob. + +This RFC adds an asynchronous API: register work, return a `request_id`, compute independently of the caller, then read status and fetch the completed result. The synchronous RPC stays. + +## Goals + +- Decouple computation lifetime from RPC and client lifetime. +- Preserve the existing changed-target response stream and output filtering. + + + +## Architecture + +Three client RPCs. Compute writes the request, publishes `request_id`, and returns success only after the publish succeeds. Status and fetch are later reads that never start work. + +The work consumer is blocking. It runs the same pipeline as today’s `GetChangedTargets` (two `GetTargetGraph` calls **in parallel**, then compare) and extends queue visibility until `Process` returns. There is no result-signal topic and no poll loop. A SubmitQueue build-signal split needs a pollable runner; native in-process Bazel is not that runner. + +```mermaid +sequenceDiagram + actor Client + participant RPC as Tango RPC + participant Store as Request store + participant Queue + participant Consumer + participant Orch as Orchestrator + participant Blob as Blob store + + Client->>RPC: ComputeChangeTargets + RPC->>Store: record request + RPC->>Queue: publish request_id + RPC-->>Client: request_id + + Note over Client,Blob: compute + Queue->>Consumer: deliver + par graphs + Consumer->>Orch: GetTargetGraph first revision + Orch-->>Consumer: graph + and + Consumer->>Orch: GetTargetGraph second revision + Orch-->>Consumer: graph + end + Consumer->>Consumer: compare + Consumer->>Blob: write compared-target blob + Consumer->>Store: CAS SUCCEEDED or FAILED + + Note over Client,Blob: read + Client->>RPC: GetChangeTargetsStatus + RPC->>Store: point read + RPC-->>Client: state, result_ref + Client->>RPC: GetComputedChangedTargets + RPC->>Blob: read result + RPC-->>Client: stream chunks +``` + + + +Large payloads stay in blob storage and queue messages stay small. Controller and consumer may co-locate or split. + +## API Contract + +Public surface is RPC, not the queue. Compute returns before Bazel. Status is unary metadata. Results stream from the blob. Cancel applies to that request. + +```proto +rpc ComputeChangeTargets(ComputeChangeTargetsRequest) returns (ChangeTargetsCompute); +rpc GetChangeTargetsStatus(GetChangeTargetsStatusRequest) returns (ChangeTargetsStatus); +rpc GetComputedChangedTargets(GetComputedChangedTargetsRequest) returns (stream GetChangedTargetsResponse); +rpc CancelChangeTargets(CancelChangeTargetsRequest) returns (ChangeTargetsStatus); + +message ComputeChangeTargetsRequest { + GetChangedTargetsRequest computation = 1; +} + +message ChangeTargetsCompute { + string request_id = 1; + ChangeTargetsStatus status = 2; +} + +message GetChangeTargetsStatusRequest { + string request_id = 1; +} + +message GetComputedChangedTargetsRequest { + string request_id = 1; + OutputConfig output_config = 2; +} + +message CancelChangeTargetsRequest { + string request_id = 1; +} + +message ChangeTargetsStatus { + string request_id = 1; + ComputeState state = 2; + TangoError error = 3; + string result_ref = 4; // set when SUCCEEDED + string download_url = 5; // optional, short-lived; omitted unless enabled +} + +enum ComputeState { + COMPUTE_STATE_INVALID = 0; + COMPUTE_STATE_PENDING = 1; + COMPUTE_STATE_RUNNING = 2; + COMPUTE_STATE_SUCCEEDED = 3; + COMPUTE_STATE_FAILED = 4; + COMPUTE_STATE_CANCELLED = 5; +} +``` + +- The RPC succeeds only after the request is stored **and** published. A failed publish is a failed RPC. A retry is a new store row, a new `request_id`, and a new publish. An orphan row from a failed publish is junk and is not repaired. +- Fetch before `SUCCEEDED` is a user error and does not start work. `OutputConfig` is send-time only. `download_url` is optional, short-lived, and omitted unless the deployment enables it. It is not required for v1 and must not be a long-lived URL on an enumerable id. +- Cancel CAS-marks the request `CANCELLED`. In-flight Bazel is not killed. The consumer must not overwrite `CANCELLED` with `SUCCEEDED` or `FAILED`. Status stays `CANCELLED`; fetch of that id is a user error. A cache blob may still appear from the race. + + + +## Identity + +One request is one unit of work. There is no shared in-flight computation. + +- `request_id` is minted by Tango (`platform/extension/counter`) when the store row is written. Clients do not supply it. There is no `idempotency_key`. A client retry always creates new work. Counter ids are enumerable. Status and fetch are unauthenticated in this RFC; treat the API as internal. Tightening auth is out of scope. + +Concurrent equivalent requests, and RPC retries, may run Bazel more than once. That matches today's sync path. The orchestrator still reads the result cache before computing. + +## Persistence + +Treat MySQL as a KV: point lookup, put-if-absent, compare-and-swap. No secondary indexes or queries over request values. Schema and key layout are implementation, not this RFC. + +The store is kinded and payload-opaque so later async APIs reuse it. Status is a point read by `request_id`. There is no unpublished-accept outbox. + +Write the compared-target blob, then CAS to `SUCCEEDED`. `Put` on the existing cache key is idempotent. Fetch of `SUCCEEDED` with a missing blob is infrastructure corruption, not a user miss. + +## Handoff, queue, and consumer + +Store, then publish, then return. The RPC succeeds only if publish succeeds. A failed publish fails the RPC; the store row is leftover junk and is not republished. A retry writes a new row and publishes again. If publish succeeds and the client never sees the response, the retry is a second request; the first may still run. + +The work queue carries `request_id`. The consumer loads the request and, unless it is already terminal, claims `RUNNING` and runs the compare pipeline: two `GetTargetGraph` calls **in parallel** (two workspace leases, two Bazel queries) then compare, same as `GetChangedTargets`. There is no handle to a live native query on another replica, so `RUNNING` from a dead owner is not a reason to skip work. + +While `Process` is alive, the consumer extends delivery visibility (cadence shorter than visibility) until it returns: both queries, compare, and blob write. Visibility is not a store claim other replicas honor. Budget is at least one query timeout plus compare, not two sequential timeouts. `RepoManager` still owns workspace capacity. Tenant is the configured repository. + +If the request is already `SUCCEEDED`, `FAILED`, or `CANCELLED`, ack and do not start work (queue redelivery of the same `request_id`). `PENDING` or `RUNNING` (including after a crash) claims and runs the pipeline again. Duplicate in-flight Bazel is accepted. Result reuse is the existing content-addressed cache: a prior attempt that already wrote graphs or the compared-target blob is a cache hit, not a join onto the old process. + +After a successful compare, write the blob then CAS status. First successful CAS wins; skip the CAS if the request is `CANCELLED`. `Put` on the cache key stays idempotent. + +Ack terminals; nack retryable infra; persist then ack user errors; DLQ must mark `FAILED`. + +## Request store + +`core/storage/requeststore` follows blob storage: interface, memory backend, MySQL backend, selected in `example/main.go`. It is not another `Storage` implementation. Do not wrap SubmitQueue's `Queue`. One store serves every async API; changed-targets is the first kind plus its topic. The store contract includes insert, status, claim, terminal CAS, and cancel CAS. + +## SubmitQueue platform + +Reuse queue, consumer, pipeline, counter, and delivery. Do not take SubmitQueue domain types, topics, or `service/messagequeue`. Tango keeps `core/errors`; map retryable infra onto nack. + +Two ways to get the packages: + +1. Extract `github.com/uber/submitqueue-commons` and depend on that. Clean module graph; requires extraction and a SubmitQueue migration first. +2. Import `github.com/uber/submitqueue` `platform/` (and `api/base/messagequeue`) directly. Faster; Tango tracks SubmitQueue's cadence and module. + +Tango owns one work stage and a DLQ reconciler. The counter mints `request_id` when the store row is created. + +## Alternatives + +**MySQL as the task queue.** Claiming `PENDING` rows needs a state query, which the KV constraint forbids, and reimplements leases, retries, and DLQ the platform already has. + +**Queue-only public API.** Exposes private transport and still cannot carry large results. + +**Stream results from status.** Status stays a small unary read. + +**Accept-before-publish plus idempotency key.** Returning success on the store write lets retries reuse a request that never reached the queue, which needs a repairer. This RFC returns success only after publish and drops the idempotency key: a retry is a new request; an unpublished row is junk. + +**Result-signal / build-signal poll.** Fits a remote `Trigger`/`Status` runner, not native in-process Bazel. Out of scope until such a runner exists. \ No newline at end of file