Let a source push statements in the open contract

[WaylandYang]

Record 0054, cut 1. Surfaced by #875: a system that already holds {thing, relation, value, when} could only get it in by spelling it into prose, chunking it on a model-sized budget, and paying a model call to read the prose back into a statement.

What lands

• A statements source kind (next to api, same push token, same identity/version/tombstone semantics, same run history) and POST /sources/{id}/statements.
• The body is the contract. The request carries the open extraction shape verbatim (e / s / n, the same positions the model is asked to fill) inside the api push's envelope (external_id, doc_time, deleted). No second representation.
• The payload is the document, in one piece. Stored as the document's content and its single chunk; the chunker is not consulted (its budget is sized for a model's attention, and no model reads this).
• No model, same path. For a document under a statements source, extraction parses the chunk instead of prompting for it and then continues unchanged (identity, described things, name facts, evidence, time mentions, qualifiers, drop signals). The job runs with no chat model configured.
• The item is its own evidence. Statements carry no quote; the evidence row names the chunk and phrase with null offsets, which is what fact_evidence already means by "not located" (0061).
• No slot for a type. Keys outside the envelope and the contract are refused at the door (422), so a client cannot believe it wrote a typed fact. Pushed statements reach the typed graph only through alignment.
• Events, not state; an update marks, it does not close. Both stated in the record and in the in-app guide, because the first caller will try with a table and the second will expect the old observation to close.
• Library: the source appears like an api source (status, token dialog, guide link); Braces icon; both language packs; the ingest guide has a new section and its chooser table a new column.

No migration: sources.kind is unconstrained text.

Not in this cut

Batch pushes, a typed write, a table importer (0036), synthesised quotes, per-item confidence. Listed with reasons in the record.

Verified

• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings: clean.
• a_source_kind_is_listed_once (frontend list ↔ backend enum, order included): passes.
• New tests (sources_statements_tests.rs): the door refuses unknown keys / a non-null quote / a short item / a blank identity / an empty s (no DB); against pgvector/pgvector:pg16 on a fresh database: a pushed statement becomes one open fact with phrase is on, both entities, and an evidence row on the payload's own chunk with null quote and offsets, with no chat model configured; a second push under the same identity is unchanged then updated with a second version; the route answers 422 / 404 (an api source) / 401.
• cargo test -p utopia-server on that fresh database: 398 passed, 0 failed.
• pnpm guard, pnpm typecheck, pnpm test (132), pnpm build: pass.

One thing worth knowing if you run the DB tests locally: a dev server connected to the same database will have its job worker pick up the process_document jobs the route tests enqueue and fail them against its own data dir. Point tests at a database the server is not using.

Refs #875.

🤖 Generated with Claude Code

[WaylandYang]

Review (posted as a comment because GitHub refuses a formal review on a PR under the same account).

The design and the door are solid: constant-time token check, every write scoped by the source's own base, unknown keys refused at the door, deterministic re-serialisation, no migration, and ADR 0054 reads true for decisions 1, 2 and 4 to 7. One thing breaks decision 3 and the PR's own example, so please fix it before merging.

Blocking: a pushed statement's when / ended is never recorded. locate_time (crates/utopia-server/src/extraction_open.rs:76) starts with let (q, span) = quote?;, and the door requires s[i][0] to be null ("the item is its own evidence", sources_routes.rs:679), so for every pushed statement the lookup returns None before it searches the chunk and run_open takes the TIME_NOT_IN_QUOTE branch (extraction_open.rs:685): the fact lands with only the document's doc_time, a drop signal is written per statement and time_mentions::record is never called. Pushing the ADR's example ("when": "08:14:03") yields a fact with no time mention and one drop. The DB test passes because it asserts phrase, entities and evidence but never counts time_mentions. Suggest: in the pushed branch, search the chunk itself when the quote is null, and add an assertion that one time mention lands in a_pushed_statement_reaches_the_open_graph_without_a_model.

Two smaller asks

• A subject (or an n entity) that is not listed in e passes the door's shape check, answers 200 created, then is dropped silently as UNKNOWN_REF in extraction, which is exactly what decision 5 says the door should not do. Resolve s[i][1] and n[i][0] against e at the door (same folding as name_key) and answer 422.
• Tests: none for deleted: true (tombstone, missing_since, revival), the missing Authorization header, the 64 KiB / 200-item limits, or a non-null n[i][2] quote. The auth, unknown-key, api-source 404 and versioning paths are genuinely covered.

Pre-existing and not blocking: the body is buffered before the token check (axum's default 2 MiB), same as the existing push; a DefaultBodyLimit on the route would close it cheaply.

CI green, DCO fine, i18n in both packs. Happy to merge once the time path is fixed.

[WaylandYang]

Updated in a0b3875, all three points.

Time words (blocking). On the pushed path the words are now located in the chunk directly (the chunk is the payload), instead of going through locate_time, which returns at quote? before it searches. a_pushed_statement_reaches_the_open_graph_without_a_model now asserts that one time mention (08:14:03, role when) lands on the fact; I ran it with the fix stashed and it fails there, so it does guard the path.

Unlisted subject / n entity. The door resolves s[i][1] and n[i][0] against e with the same folding as name_key (whitespace and case only) and answers 422 naming the slot. An object not in e still lands as a literal value, as it does for a model's reply; the record and the guide say both.

Tests. Added: tombstone marks and a new push revives (missing_since set, then cleared, document never deleted); a tombstone for an identity never pushed is a no-op that still answers marked_missing; the missing Authorization header; the 64 KiB limit over HTTP (still 422 with the message, not a 413); the 200-item limit; an n entity not in e; a non-null n quote; and a subject that differs only by case and whitespace, which must pass.

The route also carries DefaultBodyLimit of four payloads now, so the 2 MiB default is no longer buffered ahead of the token check.

Local: clippy clean, the five statements tests and the extraction_open unit tests pass on a fresh database.

[WaylandYang]

Re-reviewed a0b3875. All three points are addressed: the pushed branch locates time words in the chunk itself (extraction_open.rs:680) instead of through locate_time, and the DB test now asserts the 08:14:03 mention lands; the door resolves s[i][1] and n[i][0] against e with the same name_key folding and answers 422 naming the slot; the new tests cover tombstone and revival, an unknown-identity tombstone, the missing Authorization header, the 64 KiB limit over HTTP, the 200-item limit, an n entity not in e, a non-null n quote and a folded subject that must pass, and the route carries a DefaultBodyLimit. Approved on my side once CI is green; merging waits for Wayland's explicit go, as asked.

[WaylandYang]

Ran the branch end to end against a real server (fresh database, port 8319, job worker on, no chat model): register → workspace → base → create a Statements source → read its token → push the record's example over HTTP → wait for the worker → read the graph back. All 26 checks pass at ecf32fb: door refusals (401 wrong or missing key, 422 unknown key, 422 unlisted subject), created/unchanged/updated/marked_missing, one JSON document with two versions, the open statement on the entity as an unbound relation, the when time mention, an evidence row with no quote, only known_as on the typed layer, the earlier statement listed under review's unconfirmed queue after the update rather than invalidated.

It found one gap, fixed in ecf32fb: GET /token and POST /rotate-token still gated on kind == "api", so a Statements source's token could be seen once at creation and never again. Both now accept either push kind, with a test that reads the token back, rotates it, and checks the old one is refused.

The resolve_time and align_types jobs log a warning without a chat model, as they do for any document; extraction itself reaches done.