From 84da19e168b861a5f7e5205fba5d2d3840a529a1 Mon Sep 17 00:00:00 2001 From: Chenghan Ying <125171961+roychying@users.noreply.github.com> Date: Wed, 23 Sep 2026 09:38:34 -0700 Subject: [PATCH] docs: document service-scoped extension resolution and repoint the moved storage paths --- AGENTS.md | 14 ++++++--- doc/howto/TESTING.md | 2 +- doc/rfc/submitqueue/modular-queue-wiring.md | 4 +-- doc/rfc/submitqueue/workflow.md | 2 +- .../submitqueue/orchestrator/server/main.go | 2 +- .../extension/storage/mysql/schema/README.md | 31 +++++++++++++++++++ .../extension/storage/mysql/schema/README.md | 26 +--------------- 7 files changed, 47 insertions(+), 34 deletions(-) create mode 100644 submitqueue/gateway/extension/storage/mysql/schema/README.md diff --git a/AGENTS.md b/AGENTS.md index c39eca8ff..e87871100 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,10 +48,12 @@ submitqueue/ # repo root (Go module github.com/uber/submi │ └── extension/ # SHARED extension contracts + backends (counter/, messagequeue/, …) ├── submitqueue/ # SubmitQueue domain │ ├── gateway/ # Gateway service (port 8081) - entry point +│ │ └── extension/ # Aggregates/backends only the gateway resolves (storage/) │ ├── orchestrator/ # Orchestrator service (port 8082) - coordinates jobs +│ │ └── extension/ # Aggregates/backends only the orchestrator resolves (storage/) │ ├── entity/ # SubmitQueue-specific domain entities │ ├── extension/ # SubmitQueue-specific extension contracts and implementations -│ └── core/ # SubmitQueue-internal shared infra (batch, changeset, request, topickey) +│ └── core/ # SubmitQueue-internal shared infra (changeset, messagequeue, topickey) ├── stovepipe/ # Stovepipe domain (single service) │ ├── controller/ # RPC and queue-stage business logic │ ├── entity/ # Stovepipe domain entities @@ -131,7 +133,10 @@ Domain objects live under each domain's `entity/` tree, or under `platform/base/ Vendor-agnostic, pluggable interfaces with implementations in subdirectories: 1. **Shared across domains** — define interfaces at `platform/extension/{ext}/`, implementations at `platform/extension/{ext}/{impl}/`. 2. **Domain-specific** — define at `{domain}/extension/{ext}/`, implementations at `{domain}/extension/{ext}/{impl}/`. -3. Factory interface for dependency injection and lifecycle management (constructed in wiring, not inside `platform/extension` packages). +3. **Service-scoped resolution** — when only one service of a multi-service domain resolves an extension, its `Factory`, its aggregate, its implementations, its mocks and its schema live at `{domain}/{service}/extension/{ext}/`. The behavioural contracts stay at `{domain}/extension/{ext}/`. +4. Factory interface for dependency injection and lifecycle management (constructed in wiring, not inside `platform/extension` packages). + +**Split on reachability, not on declaration.** What a service may resolve is decided by the accessors on its aggregate, so that is the part worth scoping to a service — a gateway holding a four-accessor `Storage` cannot obtain an orchestrator store whatever else it can name. The contracts stay shared because moving them adds no enforcement and costs every domain-level caller a dependency on a service package. `submitqueue/extension/storage` is the worked example: thirteen store contracts, their mocks and the error vocabulary shared, while each service owns its own aggregate, MySQL implementation and schema. **Extensions hold contracts and implementations only — not factories or routing.** @@ -167,7 +172,8 @@ Paths follow the directory layout: shared packages live under `platform/` at the - Proto (generated): `github.com/uber/submitqueue/api/{domain}/{service}/protopb` (single-service: `.../api/{domain}/protopb`, e.g. `.../api/runway/protopb`) - Queue contracts: external `github.com/uber/submitqueue/api/{domain}/messagequeue`; internal `github.com/uber/submitqueue/{domain}/core/messagequeue` - Domain entities: `github.com/uber/submitqueue/{domain}/entity` (e.g. `.../submitqueue/entity`) -- Domain extensions: `github.com/uber/submitqueue/{domain}/extension/{ext}[/{impl}]` (e.g. `.../submitqueue/extension/storage/mysql`) +- Domain extensions: `github.com/uber/submitqueue/{domain}/extension/{ext}[/{impl}]` (e.g. `.../submitqueue/extension/storage`) +- Service-scoped extensions: `github.com/uber/submitqueue/{domain}/{service}/extension/{ext}[/{impl}]` (e.g. `.../submitqueue/orchestrator/extension/storage/mysql`) - Cross-domain consumer framework: `github.com/uber/submitqueue/platform/consumer`; internal topic keys live with the owning domain contract (for example `submitqueue/core/messagequeue` and `stovepipe/core/messagequeue`); external queue topic keys live with their published contract (for example `api/runway/messagequeue`) - Domain-internal infra: `github.com/uber/submitqueue/{domain}/core/{pkg}` (e.g. `.../submitqueue/core/request`) - Shared entities: `github.com/uber/submitqueue/platform/base/{pkg}` (e.g. `.../platform/base/messagequeue`) @@ -255,7 +261,7 @@ make clean # Clean Bazel cache 2. Add it to the owning service's pipeline topology when one exists (for example `submitqueue/orchestrator/pipeline.go`); otherwise wire it in `service/{domain}/{service}/server/main.go` **Add new extension:** -1. Define the interface, config, and `Factory` interface under `{domain}/extension/{ext}/` or `platform/extension/{ext}/`, and put implementation constructors under the `{impl}/` subdirectory. Keep concrete factory adapters and per-queue routing in service wiring. +1. Define the behavioural interface and config under `{domain}/extension/{ext}/` or `platform/extension/{ext}/`. Put the `Factory`, the aggregate and the implementation constructors alongside them, or under `{domain}/{service}/extension/{ext}/` when a single service resolves it. Keep concrete factory adapters and per-queue routing in service wiring. 2. Add `BUILD.bazel`, tests, and README.md **Add new entity:** diff --git a/doc/howto/TESTING.md b/doc/howto/TESTING.md index c85d941e5..f5401a798 100644 --- a/doc/howto/TESTING.md +++ b/doc/howto/TESTING.md @@ -23,7 +23,7 @@ SubmitQueue uses **two separate databases** to demonstrate proper architectural ### 1. Application Database - **Purpose**: Business data (requests, counters, batches) -- **Schema**: `submitqueue/extension/storage/mysql/schema`, `platform/extension/counter/mysql/schema` +- **Schema**: `submitqueue/gateway/extension/storage/mysql/schema`, `submitqueue/orchestrator/extension/storage/mysql/schema`, `platform/extension/counter/mysql/schema` - **Used by**: Gateway (receipts, request logs, and read models), Orchestrator (requests, batches, builds, and counters) - **Connection**: `MYSQL_DSN` diff --git a/doc/rfc/submitqueue/modular-queue-wiring.md b/doc/rfc/submitqueue/modular-queue-wiring.md index a3e716e10..6bcfa412a 100644 --- a/doc/rfc/submitqueue/modular-queue-wiring.md +++ b/doc/rfc/submitqueue/modular-queue-wiring.md @@ -338,7 +338,7 @@ import ( "github.com/uber/submitqueue/platform/lifecycle" "github.com/uber/submitqueue/platform/pipeline" "github.com/uber/submitqueue/submitqueue/orchestrator" - storagemysql "github.com/uber/submitqueue/submitqueue/extension/storage/mysql" + storagemysql "github.com/uber/submitqueue/submitqueue/orchestrator/extension/storage/mysql" mqmysql "github.com/uber/submitqueue/platform/extension/messagequeue/mysql" ) @@ -433,7 +433,7 @@ import ( "os/signal" "github.com/uber/submitqueue/submitqueue" - storagemysql "github.com/uber/submitqueue/submitqueue/extension/storage/mysql" + storagemysql "github.com/uber/submitqueue/submitqueue/orchestrator/extension/storage/mysql" mqmysql "github.com/uber/submitqueue/platform/extension/messagequeue/mysql" ) diff --git a/doc/rfc/submitqueue/workflow.md b/doc/rfc/submitqueue/workflow.md index d1d48d2bf..92e718f4c 100644 --- a/doc/rfc/submitqueue/workflow.md +++ b/doc/rfc/submitqueue/workflow.md @@ -89,7 +89,7 @@ click node_conclude "https://github.com/uber/submitqueue/blob/main/submitqueue/o click node_changeproviders "https://github.com/uber/submitqueue/blob/main/submitqueue/extension/changeprovider/change_provider.go" click node_buildrunner "https://github.com/uber/submitqueue/blob/main/submitqueue/extension/buildrunner/build_runner.go" click node_storagecontract "https://github.com/uber/submitqueue/blob/main/submitqueue/extension/storage/storage.go" -click node_queuestorage "https://github.com/uber/submitqueue/blob/main/submitqueue/extension/storage/mysql/storage.go" +click node_queuestorage "https://github.com/uber/submitqueue/blob/main/submitqueue/orchestrator/extension/storage/mysql/storage.go" click node_requestlog "https://github.com/uber/submitqueue/blob/main/submitqueue/core/request/log.go" click node_messagequeue "https://github.com/uber/submitqueue/blob/main/platform/extension/messagequeue/queue.go" diff --git a/service/submitqueue/orchestrator/server/main.go b/service/submitqueue/orchestrator/server/main.go index be46f7afb..4e941cb57 100644 --- a/service/submitqueue/orchestrator/server/main.go +++ b/service/submitqueue/orchestrator/server/main.go @@ -231,7 +231,7 @@ func run() error { pipeline.PublishOnly(orchestrator.PublishOnlyTopics...), pipeline.Classifiers( genericerrs.Classifier, - // Storage (submitqueue/extension/storage/mysql) and queue + // Storage (submitqueue/orchestrator/extension/storage/mysql) and queue // (platform/extension/messagequeue/mysql) both run on the same // MySQL driver, so a single classifier covers errors surfaced // from either backend. diff --git a/submitqueue/gateway/extension/storage/mysql/schema/README.md b/submitqueue/gateway/extension/storage/mysql/schema/README.md new file mode 100644 index 000000000..362e53235 --- /dev/null +++ b/submitqueue/gateway/extension/storage/mysql/schema/README.md @@ -0,0 +1,31 @@ +# MySQL Schema + +The gateway's read model: the append-only request log and the three materialized projections behind request-summary retrieval and `List`. The orchestrator's pipeline working state is a separate schema — see [../../../../../orchestrator/extension/storage/mysql/schema/README.md](../../../../../orchestrator/extension/storage/mysql/schema/README.md). + +## Queue-leading primary keys + +Every table leads its primary key with `queue`: `request_summary` on `(queue, request_id)`, `request_log` on `(queue, request_id, timestamp_ms, salt)`, `change_uri_request_mapping` on `(queue, change_uri, received_at_ms, request_id)`, and `request_summary_by_queue` on `(queue, received_at_ms, request_id)`. A queue-bound store instance prefixes every read and stamps every write with its bound queue, so one queue's rows are unreachable through another queue's binding and every table is shardable by queue. `//tool/linter/queueshard` enforces this, and also rejects any secondary index that does not itself lead with `queue`, since such an index would reintroduce a cross-queue access path. + +## Read model + +The gateway request read model uses three additive tables and requires no alteration of existing tables. Deployments create these tables empty and populate them only for requests received after rollout; historical request logs and orchestrator working tables are intentionally not backfilled. + +### `request_summary` + +`request_summary` is keyed by `(queue, request_id)` and serves direct Status lookup within one queue. It stores immutable receipt context plus the current materialized request-log winner and its optimistic-lock projection version. + +### `request_summary_by_queue` + +`request_summary_by_queue` is keyed by `(queue, received_at_ms, request_id)`. This key covers the List predicate, descending sort, and keyset continuation for one bounded receipt-time window without a secondary index. The row duplicates the complete List response so one page is served by one range scan rather than one follow-up read per request ID. + +### `change_uri_request_mapping` + +`change_uri_request_mapping` is keyed by `(queue, change_uri, received_at_ms, request_id)` and serves bounded newest-first Status lookup by change URI within one queue. The gateway reads at most 101 mappings to enforce the API maximum of 100 results without silently truncating. A change URI landed into several queues has independent mappings in each, so looking it up across queues is one call per queue. + +### `request_log` + +`request_log` is keyed by `(queue, request_id, timestamp_ms, salt)` and holds the append-only audit trail behind History. `salt` disambiguates entries sharing a request, queue and millisecond; it is part of the key but never exposed through the storage interface. + +### JSON collections + +`change_uris` and `metadata` are non-null application values. MySQL JSON columns can contain the JSON value `null` despite `NOT NULL`, so stores normalize nil slices and maps to empty values on both write and read. diff --git a/submitqueue/orchestrator/extension/storage/mysql/schema/README.md b/submitqueue/orchestrator/extension/storage/mysql/schema/README.md index 59fef6c4b..520ab2366 100644 --- a/submitqueue/orchestrator/extension/storage/mysql/schema/README.md +++ b/submitqueue/orchestrator/extension/storage/mysql/schema/README.md @@ -2,7 +2,7 @@ ## Queue-leading primary keys -Every table leads its primary key with `queue`: `request` and `batch` on `(queue, id)`, `build` on `(queue, id)`, `batch_dependent` on `(queue, batch_id)`, `request_batch` on `(queue, request_id, batch_id)`, `change` on `(queue, uri, request_id)`, `queue_batch_state` on `(queue, state, batch_id)`, `speculation_path_set` on `(queue, head)`, `request_summary` on `(queue, request_id)`, `request_log` on `(queue, request_id, timestamp_ms, salt)`, `change_uri_request_mapping` on `(queue, change_uri, received_at_ms, request_id)`, and `request_summary_by_queue` on `(queue, received_at_ms, request_id)`. A queue-bound store instance prefixes every read and stamps every write with its bound queue, so one queue's rows are unreachable through another queue's binding and every table is shardable by queue. `//tool/linter/queueshard` enforces this, and also rejects any secondary index that does not itself lead with `queue`, since such an index would reintroduce a cross-queue access path. +Every table leads its primary key with `queue`: `request` and `batch` on `(queue, id)`, `build` on `(queue, id)`, `batch_dependent` on `(queue, batch_id)`, `request_batch` on `(queue, request_id, batch_id)`, `change` on `(queue, uri, request_id)`, `queue_batch_state` on `(queue, state, batch_id)`, and `speculation_path_set` on `(queue, head)`. A queue-bound store instance prefixes every read and stamps every write with its bound queue, so one queue's rows are unreachable through another queue's binding and every table is shardable by queue. `//tool/linter/queueshard` enforces this, and also rejects any secondary index that does not itself lead with `queue`, since such an index would reintroduce a cross-queue access path. The `build` key also removes a cross-queue uniqueness assumption: build IDs are runner-minted, so two queues sharing one CI pipeline may legitimately mint the same identifier. `speculation_path_set` relies on the same property for its head: a batch ID is unique only within its queue. @@ -27,27 +27,3 @@ Terminal-state records (and their batches) accumulate as the queue processes wor ### Composite primary key: `(queue, uri, request_id)` The `change` table records per-URI claims by in-flight requests. `request_id` is part of the primary key so that concurrent claims on the same URI by different requests coexist as distinct rows — a same-request retry collides on the PK and is a no-op (`INSERT IGNORE`), while a different-request claim is a new row that `GetByURI` surfaces for overlap detection. `queue` leads the key so queue-scoped lookups are primary-key-prefix scans and the table is shardable by queue. - -## Gateway request read model - -The gateway request read model uses three additive tables and requires no alteration of existing tables. Deployments create these tables empty and populate them only for requests received after rollout; historical request logs and orchestrator working tables are intentionally not backfilled. - -### `request_summary` - -`request_summary` is keyed by `(queue, request_id)` and serves direct Status lookup within one queue. It stores immutable receipt context plus the current materialized request-log winner and its optimistic-lock projection version. - -### `request_summary_by_queue` - -`request_summary_by_queue` is keyed by `(queue, received_at_ms, request_id)`. This key covers the List predicate, descending sort, and keyset continuation for one bounded receipt-time window without a secondary index. The row duplicates the complete List response so one page is served by one range scan rather than one follow-up read per request ID. - -### `change_uri_request_mapping` - -`change_uri_request_mapping` is keyed by `(queue, change_uri, received_at_ms, request_id)` and serves bounded newest-first Status lookup by change URI within one queue. The gateway reads at most 101 mappings to enforce the API maximum of 100 results without silently truncating. A change URI landed into several queues has independent mappings in each, so looking it up across queues is one call per queue. - -### `request_log` - -`request_log` is keyed by `(queue, request_id, timestamp_ms, salt)` and holds the append-only audit trail behind History. `salt` disambiguates entries sharing a request, queue and millisecond; it is part of the key but never exposed through the storage interface. - -### JSON collections - -`change_uris` and `metadata` are non-null application values. MySQL JSON columns can contain the JSON value `null` despite `NOT NULL`, so stores normalize nil slices and maps to empty values on both write and read.