diff --git a/docs/architecture.md b/docs/architecture.md index 0218a40..fd8083e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -199,6 +199,20 @@ See [Architecture Decision Records](decisions/) for detailed trade-off analysis. > stage model, and exception sources are pinned in > [`docs/ops-surfaces-spec.md`](ops-surfaces-spec.md); implementation is staged > (D2–D4). +> +> **A three-node demo topology is designed ([ADR 0012](decisions/0012-three-node-demo-topology.md), 2026-07-04).** +> The public Hugging Face demo grows from one container to a center (`msk` HQ) +> plus two edge branches (`spb`/`ekb` — [`domain.md`](domain.md) §1) that push +> operational events over HTTPS, so the artifact shows the branch-distribution +> story live. This is a **distinct axis from horizontal scaling**: the three +> nodes are separate single-replica Spaces with embedded control planes — *not* +> the [ADR 0010](decisions/0010-control-plane-externalization-postgres.md) +> PostgreSQL scale profile — HTTPS substitutes for the Kafka→Flink transport, +> and the federated live layer is ephemeral on a deterministic re-seeded +> baseline. Node roles, the ingest contract, sleep choreography, and the N1–N12 +> test invariants are pinned in +> [`docs/three-node-demo-topology.md`](three-node-demo-topology.md); +> implementation is F2 and the Space deploy is an owner gate. ## v1-v6 Capability Map diff --git a/docs/decisions/0012-three-node-demo-topology.md b/docs/decisions/0012-three-node-demo-topology.md new file mode 100644 index 0000000..1137fee --- /dev/null +++ b/docs/decisions/0012-three-node-demo-topology.md @@ -0,0 +1,248 @@ +# ADR 0012: Three-node demo topology on Hugging Face Spaces + +## Status + +Accepted - 2026-07-04 (design; F2 implements per `plan_endgame_02_07_26.md`, +deploy of the Spaces is an owner gate). Concrete build contract: +[`docs/three-node-demo-topology.md`](../three-node-demo-topology.md). + +## Context + +The public demo today is a **single** Hugging Face Docker Space +(`liovina/agentflow-demo`, live at `https://liovina-agentflow-demo.hf.space` - +`deploy/hf-space/`): one read-only container that seeds DuckDB on boot, runs in +demo mode (admin routes `404`, mutating routes `403` except `POST /v1/query`), +and tracks `main`. It demonstrates the event-native metrics story **within one +node** - the in-process pipeline (`src/processing/local_pipeline.py`) generates +events, and serving reads reflect them. + +The production topology is multi-service and multi-branch, and none of it fits a +single Space: + +- ADR 0006 fixes serving on **ClickHouse**; ADR 0007 makes **Kubernetes/Helm** + the substrate; ADR 0009/0010 externalize the control plane to **PostgreSQL** + so the API can run `replicaCount > 1`. +- The legend (`docs/domain.md` §1) is a **multi-branch** importer: `msk` (HQ, + central warehouse, fulfils all three RU channels, main WMS), `spb`/`ekb` (RU + regional warehouses), `dxb` (JAFZA re-export hub, UAE), `ala` (EAEU hub, KZ). + Each branch is a separate legal entity, which is why per-jurisdiction PII + satellites and per-branch row policies exist in the DV2 vault. + +Phase F of the endgame plan calls for a **three-node** demo - a center plus two +edge branches, events flowing over HTTPS - so the public artifact *shows* the +distribution story (a branch produces operational events; the HQ's cross-branch +metrics reflect them live) rather than a single box plus a diagram. + +### Hugging Face Spaces constraints (verified 2026-07-04) + +- Free `cpu-basic` hardware: **16 GB RAM, 2 vCPU, 50 GB non-persistent disk**; + a Space **sleeps after 48 h of inactivity** and cold-starts on the next HTTP + request (the image is already built - a wake is seconds, not a rebuild). Free + hardware cannot configure a custom sleep timer. + ([Spaces Overview](https://huggingface.co/docs/hub/en/spaces-overview)) +- Non-persistent disk is the load-bearing constraint: **local state does not + survive a restart**, so every node must re-seed on boot. This is a property to + design *with*, not around. +- **Multiple public Spaces per free account** need no organization and no second + account - proven empirically: the `liovina` account already runs + `agentflow-demo`, `nl-sql`, and `vacancyradar` as concurrent public Spaces. + +## The honesty boundary (what this demo does and does NOT show) + +This section is the crux for the Phase G adversarial audit. There are **two +orthogonal axes**, and conflating them would be a claim the system does not run: + +1. **Branch distribution / event federation** - *this* is what the three-node + HF demo demonstrates: distinct geographic branch nodes emit operational + events to a hub that aggregates a cross-branch view. +2. **Horizontal scaling of one logical node** - ADR 0007/0009/0010: `replicaCount + > 1` pods of the *same* service sharing an externalized PostgreSQL control + plane. This is the Kubernetes scale profile and is **not** what the HF demo + shows. The demo's three nodes are three *different* single-replica services, + each with its own **embedded** control plane - which is exactly the correct, + documented single-replica profile per ADR 0009 §Decision(2). + +Three substitutions the demo makes, to be stated plainly in the node READMEs so +the artifact never over-claims: + +- **Transport:** direct HTTPS `POST` replaces Kafka -> Flink as the inter-node + event carrier. Event *semantics* are preserved (the same canonical events + drive the same `pipeline_events` journal and metric freshness); only the + transport differs, because Spaces cannot run Kafka. +- **State:** each node is single-replica with the embedded control plane, **not** + the PostgreSQL scale profile. No shared database spans the Spaces. +- **Liveness:** the federated cross-branch layer is **ephemeral** - it re-seeds + to a deterministic baseline on every restart; live events that arrived since + the hub last booted are lost on the hub's next restart. + +## Options considered + +### 1. Keep one node; describe multi-node only in docs + +Rejected. Phase F's entire value is a *running* distribution, not a diagram; a +live cross-branch reflection is what makes the portfolio point. + +### 2. One Space, three processes (supervisord / compose inside the image) + +Rejected. HF Docker Spaces run a single app on one `app_port`; three processes +inside one container share a kernel and filesystem, so they are not real nodes, +do not cross a network boundary, and never exercise the HTTPS event path the +design is meant to show. Less honest than three real Spaces, for more plumbing. + +### 3. Three Spaces, one per node, federated over HTTPS - chosen + +A real network boundary and real per-node isolation; each node is a real +deployment of the **same image** differentiated only by environment. Matches how +production separates branches, at demo scale. + +### 4. Second HF account / paid persistent Spaces for the branches + +Rejected as unnecessary. One free account hosts all three public Spaces (proven +above), and ephemerality is acceptable for an on-demand demo given the sleep +choreography below. No second account, no paid tier, no organization required. + +## Decision + +**1. Node roles and legend mapping.** + +- **Center = `msk`** (HQ). Runs the full serving API in demo mode **plus** an + authenticated node-ingest endpoint, and hosts the cross-branch aggregated + view. It is the hub analog of the production control plane - but embedded and + single-replica, **not** the PostgreSQL scale profile (see honesty boundary). +- **Edge 1 = `spb`, Edge 2 = `ekb`** - both RU regional warehouses. Each runs + the same image in edge role: seeds its branch slice, serves its own read + surface, and emits its operational events to the center over HTTPS. +- **Foreign branches `dxb`/`ala` stay narrative, not live nodes - deliberately.** + The legend's load-bearing governance claim is "RU PII never crosses the + border" (`domain.md` §1 - per-jurisdiction legal entities; §6 - the + PII-officer persona: "prove RU data never crosses the border"). Standing up a + live UAE/KZ Space + that receives faux-PII-bearing events over the public internet would visually + contradict that claim. Cross-jurisdiction is demonstrated where it actually + lives - DV2 vault governance (per-jurisdiction PII satellites, per-branch row + policies) - not the node graph. A future foreign node is a **non-goal** here; + if ever added it must carry zero PII in its event payloads. + +**2. One image, three roles, differentiated by environment.** The existing +`deploy/hf-space` image is extended (not forked). New env, consistent with the +`AGENTFLOW_*` convention: + +- `AGENTFLOW_NODE_ROLE = center | edge` (unset or `standalone` = today's + single-node demo, unchanged - a strict superset, so the current Space keeps + working with no new env). +- `AGENTFLOW_NODE_BRANCH = msk | spb | ekb` (scopes the boot seed to that + branch's slice). +- `AGENTFLOW_NODE_CENTER_URL` (edge only - where to emit, e.g. + `https://liovina-agentflow-center.hf.space`). +- `AGENTFLOW_NODE_TOKEN` (shared secret, stored as an HF Space **secret**, never + a public variable; authenticates edge->center ingest - distinct from the + public `demo-key`). + +**3. Event flow is push, edge -> center, over HTTPS.** + +- A new endpoint on the center - `POST /v1/node/events` (bearer = + `AGENTFLOW_NODE_TOKEN`) - accepts a batch of the **same canonical events** + `local_pipeline._process_event` already understands (order / payment / click / + product), tagged with the originating branch. The center applies them through + that existing path (`_process_event` -> `pipeline_events` journal + serving + tables), so cross-branch metrics, Order 360 timeline, and freshness update + with **no new serving logic** - it reuses the event->metric axis that already + exists (ADR 0006 Phase 1). +- The endpoint is **allow-listed in the demo-mode guard** exactly like + `/v1/query` (`src/serving/api/main.py:286-299`), so public callers with the + `demo-key` still get `403` on it; only the node token authorizes it. It is + mounted **only** in center role; in edge/standalone role it does not exist. +- **Push, not pull:** edges emit on their own activity (the existing background + generator tick, reused). This is what makes sleep choreography work. + +**4. Statelessness + re-seed.** + +- Every node seeds deterministically on boot (`AGENTFLOW_SEED_ON_BOOT=true`, + already the demo default). Edges seed their branch slice; the center seeds the + catalog plus an **aggregate baseline covering all three branches**, so a + center visitor sees a coherent cross-branch picture even before any live event + arrives. +- The federated live layer sits **on top of** the re-seeded baseline and is + **ephemeral**: a restart re-seeds the baseline and resets the live layer to + zero. Stated in the READMEs/UI. No shared database, no cross-Space persistence + - that is the point of stateless + re-seed. +- **Determinism + idempotency:** the seed is a fixed function of + `AGENTFLOW_NODE_BRANCH` plus the pinned demo dataset, so any node comes up with + the same baseline on any restart; event application is idempotent (order upsert + by id, journal keyed by `event_id`), so a retried push after a wake never + double-counts. + +**5. Sleep choreography (no keep-alive).** + +- Push + on-demand is self-healing without any cron. A visitor hits a sleeping + edge -> the edge wakes, seeds, and its generator emits to the center; if the + center is asleep, **the first emit wakes it** (it seeds its baseline, then + applies the event). +- The edge emitter treats a cold center **tolerantly**: short timeout + a couple + of backoff retries so the first (wake) request that times out is retried once + the center is up; on give-up it drops the event (best-effort for a demo, + logged) rather than failing the visitor's page. +- The center's cross-branch view **degrades gracefully**: a branch that has sent + nothing this lifecycle (asleep or just booted) shows its re-seeded baseline + plus a "branch waking / no live events yet" state and a per-branch last-seen + timestamp - the sleep behavior is **visible by design**, never an error. +- **No keep-alive scheduler** is part of the design. A GitHub Actions cron + pinging all three to fight sleep is possible but an explicit non-goal: it burns + minutes for no demo value and hides the honest sleep story. Always-warm, if + ever wanted, is an ops toggle, not this design. + +**6. Namespaces.** Three Spaces under the single existing `liovina` account: +`liovina/agentflow-center`, `liovina/agentflow-edge-spb`, +`liovina/agentflow-edge-ekb`. **No second account and no HF organization are +required** (proven above). An HF org (free) would only buy a prettier +`agentflow/center` namespace and shared secret management - recorded as an +**optional** nicety for F2, not a requirement. + +## Consequences + +### Positive + +- The public demo shows the distribution story **live**, on real network + boundaries, with per-node isolation. +- One image, three environments (DRY); no new infrastructure, account, tier, or + organization. +- New code is thin and low-risk: an ingest endpoint, an edge emitter hook, a + node-mode config, and a cross-branch view - all reusing the existing event + path (`_process_event`, the background generator, the `pipeline_events` + journal) rather than adding serving logic. +- Honest by construction: transport substitution, single-replica embedded state, + and ephemeral liveness are all stated where a viewer meets them. + +### Negative + +- Real F2 surface to build and test (ingest endpoint, emitter, node config, + aggregation, graceful degradation) - not a config-only change. +- The federated live layer is ephemeral (the baseline is not); a hub restart + resets it. +- Sleep adds first-request latency and a "waking" UX state; a center-first + visitor sees only the baseline until an edge is awake and emitting (mitigated + by UI copy directing "open a branch Space to see its events flow to the hub"; + an optional center-initiated wake-poke is noted in the spec, with its + sleep-timeout cost). +- Three Spaces to deploy and refresh instead of one. + +### Non-goals (recorded so they are not mistaken for gaps) + +- Not the multi-replica PostgreSQL scale profile (that is ADR 0010 + the K8s + cutover, verified separately). +- No live foreign-branch (`dxb`/`ala`) node - PII border. +- No keep-alive / always-warm; no cross-Space shared database. + +## Follow-up (F2 - `plan_endgame_02_07_26.md` Phase F) + +- Implement per the build contract in `docs/three-node-demo-topology.md` + (Space matrix, env matrix, endpoint contract, event payload, boot/emit/sleep + sequences, node invariants N1-Nx as the test spec, deploy runbook, + verify-live checklist). +- **Deploy of the three Spaces is an owner gate** (external publish under + `liovina`, `HF_TOKEN` from `D:/VacancyRadar/.env` - never printed). +- Verify-live: health of all three, plus an edge->center event visibly moving a + center cross-branch metric. +- ADR 0007/0009/0010 remain accepted and unchanged; this ADR neither relaxes nor + depends on the multi-replica gate - it is a different axis (distribution, not + scaling). diff --git a/docs/three-node-demo-topology.md b/docs/three-node-demo-topology.md new file mode 100644 index 0000000..7def3d5 --- /dev/null +++ b/docs/three-node-demo-topology.md @@ -0,0 +1,247 @@ +# Three-node demo topology - build contract (F2) + +Concrete spec implementing [ADR 0012](decisions/0012-three-node-demo-topology.md). +This is the F2 target: what to build, how the nodes talk, and the machine-checkable +invariants (N1-N12) that are the test spec. Deploy of the Spaces is an owner gate. + +Read ADR 0012 first for *why*; this doc is *how*. Terminology, roles, and the +honesty boundary are defined there and not repeated. + +## 1. Space matrix + +| Node | Role | Branch | Space | Live URL (planned) | +|--------|----------|--------|--------------------------------|---------------------------------------------| +| Center | `center` | `msk` | `liovina/agentflow-center` | `https://liovina-agentflow-center.hf.space` | +| Edge 1 | `edge` | `spb` | `liovina/agentflow-edge-spb` | `https://liovina-agentflow-edge-spb.hf.space` | +| Edge 2 | `edge` | `ekb` | `liovina/agentflow-edge-ekb` | `https://liovina-agentflow-edge-ekb.hf.space` | + +All three build the **same image** (extend `deploy/hf-space/Dockerfile`, tracking +`main`). The existing `liovina/agentflow-demo` Space stays as-is (standalone role); +the three-node set is additive. + +## 2. Environment matrix + +One image; role is pure env. Public **variables** unless marked secret. + +| Variable | Center | Edge (spb / ekb) | Standalone (today) | +|------------------------------|-----------------------------------------|------------------------------------------|--------------------| +| `AGENTFLOW_NODE_ROLE` | `center` | `edge` | unset / `standalone` | +| `AGENTFLOW_NODE_BRANCH` | `msk` | `spb` / `ekb` | unset | +| `AGENTFLOW_NODE_CENTER_URL` | - | `https://liovina-agentflow-center.hf.space` | - | +| `AGENTFLOW_NODE_TOKEN` | **secret** (accepts) | **secret** (sends; same value) | - | +| `AGENTFLOW_DEMO_MODE` | `true` | `true` | `true` | +| `AGENTFLOW_SEED_ON_BOOT` | `true` | `true` | `true` | +| `DEMO_API_KEY` | `demo-key` | `demo-key` | `demo-key` | + +Rules: + +- `AGENTFLOW_NODE_ROLE` unset or `standalone` => **no** node endpoint mounted, **no** + emitter started - byte-identical to today's demo (N1). +- `AGENTFLOW_NODE_TOKEN` is the **same** string on all three; store as an HF Space + **secret**, never a variable; never logged (N10). +- Center's `AGENTFLOW_NODE_CENTER_URL` is unused (center does not emit); edges + require it, and boot must fail fast if an edge has role=`edge` but no center URL + or no token. + +## 3. Role dispatch (where the code hangs) + +- Resolve role/branch once at startup in the `lifespan` (`src/serving/api/main.py`, + alongside `app.state.demo_mode` at line 86). Store `app.state.node_role`, + `app.state.node_branch`. +- **Center:** mount the ingest router (§4) only when role=`center`. Add its path to + the demo-guard allow-list set next to `/v1/query` (main.py:291-294). +- **Edge:** start the emitter task (§6) in the `lifespan` only when role=`edge`. +- **Seed scoping (§5):** the boot seed (`initialize_demo_data`, main.py:123) takes + the branch into account. +- Keep a single `src/serving/node/` module for role config + emitter + ingest + handler so the node concern is one seam, not sprinkled through serving. + +## 4. Ingest endpoint contract (center only) + +``` +POST /v1/node/events +Authorization: Bearer +Content-Type: application/json + +{ + "origin_branch": "spb", + "events": [ , ... ] // 1..N, bounded (reject > 500) +} + +200 OK -> { "accepted": N, "applied": N, "dead_lettered": M } +401 -> missing/malformed bearer +403 -> wrong token, OR demo-key/public caller (demo-guard), OR role != center +422 -> body shape invalid (not the per-event schema - that dead-letters, see below) +``` + +- Auth: constant-time compare of the bearer against `AGENTFLOW_NODE_TOKEN` + (reuse the pattern in `src/serving/api/auth/`); **not** the `demo-key` path. +- The demo-guard (`main.py:286-299`) blocks `POST` for the public key on every + path except the allow-list; adding `/v1/node/events` to that set lets the + **token-authenticated** node call through while the public `demo-key` still + gets `403` (N3). +- Apply each event via `local_pipeline._process_event(conn, event, clickhouse_sink=...)` + on the center's serving connection - **no new serving logic**. A per-event schema + failure dead-letters exactly as in-process events do (that path already writes + `events.deadletter` to `pipeline_events`); it does not fail the batch. +- Tag origin: set `event["source_metadata"]["branch"] = origin_branch` before + applying, so the journal/lineage carry the branch (N4). Reject a batch whose + `origin_branch` is not one of the known edge branches (N12). +- Idempotency: `_process_event` upserts orders by id and the journal is keyed by + `event_id`; re-POSTing the same batch must not double-count (N5). + +## 5. Canonical event payload + +The `events[]` items are exactly what the producers already emit - do not invent a +new schema. Produced by `generate_order|generate_payment|generate_click|generate_product` +(`src/ingestion/producers/event_producer.py`) via `json.loads(model.model_dump_json())`; see +`local_pipeline._generate_random_event`. Shape (illustrative order event): + +```json +{ + "event_id": "evt-...", + "event_type": "order.created", + "tenant": "default", + "order_id": "ORD-20260404-...", + "source_metadata": { "tenant": "default", "branch": "spb" }, + "...": "producer payload fields" +} +``` + +Keys the pipeline reads: `event_type` (prefix routes upsert: `order.`/`payment.`/ +`product.`/`session.`), `event_id` (journal key), `tenant` or +`source_metadata.tenant`, and the entity-id field per prefix (`order_id`/`user_id`/ +`product_id`/`session_id`, `_derive_entity_id`). `validate_event` gates admission. + +## 6. Edge emitter (edge only) + +Reuse, do not rebuild: + +- The edge already can run `local_pipeline.run()` (the background generator). In + edge role, run a **slow** generator loop (e.g. 1 event / few seconds - low, this + is a demo not a load test) that for each produced `(topic, event)`: + 1. applies it locally via `_process_event` (edge's own read surface stays live), then + 2. **forwards the same dict** to `POST {AGENTFLOW_NODE_CENTER_URL}/v1/node/events` + with the node token, batching a few events per request. +- Local apply and forwarded payload are the **same** canonical dict (N7). +- Cold-center tolerance (N9): short timeout (~3-5 s), 2-3 retries with backoff; on + give-up, drop + log at info, **never** raise into the loop (the edge must stay up + and its own page must stay live even if the center is down/sleeping). +- The emitter only runs while the edge Space is awake; that is the intended sleep + behavior, not a bug (§8). + +## 7. Seeding and branch scoping (§ ADR 0012 Decision 4) + +- Edge boot seeds **its branch slice**: `initialize_demo_data` scoped/filtered by + `AGENTFLOW_NODE_BRANCH` (store/branch column in the demo tables per `domain.md` + §5.2 `store_code` `msk-01`/`spb-shr-02`/...). If the current seed is not yet + branch-parameterizable, the minimal change is a post-seed filter by branch; keep + it a pure function of the branch so restarts are deterministic (N11). +- Center boot seeds the **catalog + an aggregate baseline across all three + branches**, so a center-first visitor sees a coherent cross-branch picture + before any live event (N8 baseline half). + +## 8. Sequences + +**Boot (any node):** container start -> `lifespan` resolves role/branch -> seed +(branch slice for edge, all-branch baseline for center) -> demo guards on -> +(center) ingest mounted / (edge) emitter task started. + +**Live emit (happy path):** visitor opens edge `spb` -> edge awake -> generator +produces `order.created` -> applied locally (spb read surface updates) -> forwarded +to center -> center `_process_event` -> center cross-branch metric for `spb` moves; +Order 360 timeline / lineage carry `branch=spb`. + +**Sleep + wake:** center asleep, edge awake and emits -> first POST wakes the center +(cold start seconds) -> that POST may time out -> emitter retries -> center up, seeds +baseline, applies the retried batch. Center-first visitor while both edges sleep -> +sees baseline + "no live events yet" per branch (N8); UI copy: "open a branch Space +to see its events flow to the hub." + +**Restart (ephemerality):** any node restart -> disk gone -> re-seed baseline -> +live layer reset to zero. Deterministic baseline (N11); README states the live layer +resets on hub restart. + +## 9. Cross-branch view + graceful degradation + +- Center exposes a cross-branch summary (extend the existing admin/UI surface or a + read endpoint composing per-branch counts from the journal - reuse + `QueryEngine`/serving reads, no new store). Per branch: baseline figure, live + delta this lifecycle, and **last-seen** timestamp (null => "waking / no live + events yet"). Never error on a silent branch (N8). + +## 10. Optional: center-initiated wake-poke (record, do not default) + +A center visitor could trigger a best-effort GET to each edge's `/v1/health` to wake +them, so cross-branch liveness appears without the visitor opening each edge. This +reintroduces a pull and its sleep-timeout handling; keep it **optional/off by +default** behind a flag - the core design is push-only (ADR 0012 Decision 5). + +## 11. Deploy runbook skeleton (owner gate) + +Mirrors `deploy/hf-space/DEPLOY.md` x3. `HF_TOKEN` from `D:/VacancyRadar/.env` - never +printed. For each of `agentflow-center`, `agentflow-edge-spb`, `agentflow-edge-ekb`: + +```bash +# 1. Create (one-time) +huggingface-cli repo create --repo-type space --space_sdk docker +# 2. Push Dockerfile + README.md (per-role frontmatter/title) to the Space repo root +# 3. Set env in the Space Settings: +# - variables: AGENTFLOW_NODE_ROLE / _BRANCH / _CENTER_URL (edges) / DEMO_MODE / SEED_ON_BOOT +# - secret: AGENTFLOW_NODE_TOKEN (same value on all three) +``` + +Bring-up order: **center first** (edges need its URL live), then the two edges. + +## 12. Verify-live checklist (F2 done-gate) + +- `curl {center}/v1/health`, `{edge-spb}/v1/health`, `{edge-ekb}/v1/health` -> all 200. +- Public demo-key `POST {center}/v1/node/events` -> `403` (demo-guard holds, N3). +- `POST {center}/v1/node/events` with the node token, one seeded `spb` order -> + `200 applied:1`; then a center cross-branch read shows `spb` last-seen set and the + metric moved (N4). +- Re-POST the same batch -> center count unchanged (N5). +- Open edge `spb`, wait, open center -> `spb` shows a non-zero live delta (end-to-end). +- Center-first while edges idle -> baseline + "no live events yet", no error (N8). + +## 13. Node invariants (test spec - N1-N12) + +Machine-checkable; each is a unit/integration test F2 must add (mirrors +`ops-surfaces-spec.md` I1-I12). + +- **N1** Standalone role (no `AGENTFLOW_NODE_ROLE`): `/v1/node/events` is not mounted + (`404`), no emitter task; behavior byte-identical to today's demo. +- **N2** `/v1/node/events` mounted iff role=`center`; edge/standalone => `404`. +- **N3** Ingest rejects the public `demo-key` (`403`, demo-guard) and missing/wrong + bearer (`401`/`403`); accepts the correct `AGENTFLOW_NODE_TOKEN`. +- **N4** A valid POSTed event is applied via `_process_event`, lands in + `pipeline_events` tagged `branch=`, and moves the matching center metric. +- **N5** Idempotency: the same `event_id` POSTed twice does not double-count. +- **N6** Edge boot seeds only its `AGENTFLOW_NODE_BRANCH` slice; center seeds the + all-branch baseline. +- **N7** Edge emitter applies locally and forwards the **same** canonical dict. +- **N8** Center cross-branch view: a branch with zero live events shows baseline + + null last-seen + "waking" state, never an error. +- **N9** Emitter tolerates a cold/unreachable center: bounded timeout+retries, drops + on give-up, never raises into the generator loop. +- **N10** Node token: constant-time compare, sourced from env/secret, never logged. +- **N11** Restart determinism: re-seed after restart yields the same baseline (pure + function of branch + pinned dataset). +- **N12** Role/branch guard: ingest refused in non-center role even with a valid + token; batch with an unknown/mismatched `origin_branch` rejected. + +## 14. F2 implementation order (small PRs, full suite green each) + +1. `src/serving/node/` config resolution (role/branch/center-url/token) + fail-fast + boot validation + N1/N2 tests. No behavior change for standalone. +2. Center ingest endpoint (`POST /v1/node/events`) + demo-guard allow-list + auth + + `_process_event` wiring + branch tag + N3/N4/N5/N12 tests. +3. Edge emitter task (reuse generator; local-apply + forward; cold-center tolerance) + + N7/N9 tests. +4. Branch-scoped seed (edge slice / center baseline) + N6/N11 tests. +5. Center cross-branch view + graceful degradation + last-seen + N8 test. +6. Per-role Dockerfile/README frontmatter + deploy runbook doc; **deploy = owner + gate**; then verify-live (§12). + +Steps 1-5 are autonomous (no Docker, unit/integration-testable on the existing +single-container path with a mocked peer). Step 6's deploy is the external gate.