Record a capture that can never become a Dataset as a Shortfall

[xmap]

Third and final slice of the capture-shortfall design (#796 widened the PII fitness test, #797 gave the refusal a discriminated reason). Both prerequisites are on main.

What this fixes

CORA could not distinguish "no scan ran" from "a scan ran and produced nothing". Both look like a terminated Run with no Dataset. Three real 2-BM Runs hold files with 1 projection of a commanded 1541 and no /exchange/theta; the sweep retried them 387 times in one log window at a measured 6.3s each and wrote nothing. The reader had already computed the frame counts and discarded them at the refusal, so the only non-substrate evidence was thrown away precisely when it disagreed with the substrate.

Shortfall is a terminal-at-genesis fact-chain beside Acquisition and Attestation, appended from ingest_scan as its second outcome. No new command, grant, route or MCP tool.

The refusal is unchanged

The 400 still raises. The caller asked for a Dataset and did not get one, so the refusal remains the truthful answer; recording what CORA learned while answering is a separate act (a failed login returns 401 and still writes an audit row). Nothing about the response shape, the route, the MCP tool or CaptureScanIngestor's code moves. The candidate simply stops being selected once the fact lands.

This settles the memo's open question about manual POST semantics. The three preconditions are evidentiary, not identity-based: a resolved capture_path_id, a terminal Run, and a file older than that terminal. A human POSTing an indirect locator is therefore treated exactly like the sweep; a hand-typed file:// path never qualifies, because it has no stream key.

Reviewer attention, in priority order

1. The memo's finality lock rested on a false claim. It said both sides of the comparison were "already in hand, so the check costs no additional I/O." They were not: Run state carries no terminal timestamp, load_run folds and discards the envelopes, and proj_run_summary.updated_at is projection-write time (mutable, re-derived on replay). All four terminal events do carry occurred_at, so load_run_ended_at reads it from the stream. Reading it lazily inside the refusal branch made the memo's handler-reordering lock unnecessary, so no existing error precedence changes and a successful ingest pays nothing.

2. The stream id is derived, against the sibling pattern. Acquisition and Attestation mint a fresh UUIDv7. This one derives from capture_path_id via uuid5, because the sweep re-selects a candidate until proj_data_shortfall_summary catches up: with a minted id, every tick of projection lag would add another duplicate Shortfall. Deriving makes the second append a version conflict, so uniqueness comes from expected_version=0 rather than from a read that can be stale. The ConcurrencyError is swallowed quietly, which is why the memo's ShortfallAlreadyRecordedError/409 is not implemented: a duplicate here is an expected race, not a conflict an operator can act on.

3. Only StructurallyIncomplete records a Shortfall. Unrecognized is permanent too, but yields no Description, so its Shortfall would carry a verdict and no counts. What a countless Shortfall means is a modelling question with zero observed instances. Keying on InvalidScanFileError itself would have permanently condemned every file caught mid-transfer, which is the one mistake in this design that loses data silently; test_ingest_transient_refusal_records_nothing pins that.

4. The event carries both sides of the judgement, beyond the memo's payload list. "Can never become a Dataset" is unfalsifiable otherwise, and this row can never be erased.

Verification

Four mutations, each caught by exactly one test and no others:

Mutation	Caught by
Finality comparison removed	..._still_being_written_records_nothing
Reference-required guard neutered	..._at_a_direct_path_records_nothing
ConcurrencyError no longer swallowed	..._repeated_incomplete_reads_record_one_shortfall
SQL exclusion re-keyed to run_id	..._one_location_still_leaves_the_other_a_candidate

That last one is why the _CANDIDATE_SQL change has a Postgres test rather than a fake: the obvious single-location test passes under the wrong key. Only the two-location case discriminates, and without it one bad copy on the acquisition tier would silently condemn every other copy of the same Run.

The fixture guards itself too. test_shortfall_fixture_instants_are_independent fails if the clock, the Run terminal and the file mtime are ever collapsed, so the finality check cannot drift into agreeing by construction. The test's nanosecond conversion walks a timedelta from the epoch while production uses divmod + fromtimestamp, so a rounding bug in one is not reproduced by the other.

Full local run: 51,726 unit + architecture, 3,457 contract, 110 deployments, 16 Postgres integration. ruff, format and pyright clean over src and tests.

Known, deliberate

• host and root generate as drop:text, so they are dropped from the published record. Bare str, and the generator cannot tell a storage tier from free text. AttestationRecorded.kind/.outcome sit the same way, and withholding a facility's internal hostname from an external publication is right by default. No override added; state.py says why.
• docs/architecture/model.md still says "forty-three" aggregates. Not an oversight: 20 architecture doc pages already differ from a fresh render (the Trust page says "Five aggregates" against six, with its slices table at the literal _Generated from the code at build time._ placeholder). The committed marker bodies are caches the mkdocs hook replaces at build time.
• Three architecture allowlist entries, each with reasoning in place: the command-name one is a discovery artifact (ingest_scan now constructs exactly one event class directly, so the scan reads it as single-event); the two fold-symmetry ones have different rationales, since file_modified_at has no CORA actor at all while run_ended_at's actor lives on the Run's own stream.

Still open

promote_dataset gates publication on producing_run_end_state == "Completed". All three affected 2-BM runs are Completed, so a dataset from a run that captured 1 frame of 1541 passes that gate today. Recording the Shortfall does not change that unless promote_dataset consults it. That is a behaviour change to a publication gate and wants its own slice.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
apps/api/src/cora/data
_projections.py
apps/api/src/cora/data/adapters
capture_path_locator.py
apps/api/src/cora/data/aggregates/shortfall
__init__.py
_stream_id.py
events.py
evolver.py
read.py
state.py
apps/api/src/cora/data/features/ingest_scan
handler.py
apps/api/src/cora/data/projections
__init__.py
shortfall_summary.py
apps/api/src/cora/infrastructure
schema_version.py
apps/api/src/cora/run/aggregates/run
__init__.py
read.py
Project Total

This report was generated by python-coverage-comment-action