Skip to content

Discriminate InvalidScanFileError's cause with a closed, non-PII enum - #797

Merged
xmap merged 2 commits into
mainfrom
worktree-agent-a3f6c59e3e6a34ac3
Sep 10, 2026
Merged

Discriminate InvalidScanFileError's cause with a closed, non-PII enum#797
xmap merged 2 commits into
mainfrom
worktree-agent-a3f6c59e3e6a34ac3

Conversation

@xmap

@xmap xmap commented Sep 10, 2026

Copy link
Copy Markdown
Owner

Summary

The capture-scan sweep retries a refused ingest every 30 seconds forever. A follow-on change needs to decide when to stop, and it cannot key that on InvalidScanFileError alone: nine raise sites share that one class, spanning causes as different as a half-transferred file (transient, retrying works once the writer finishes) and a scan with no rotation-angle dataset (permanent, no retry ever produces one).

Catching the class collapses that distinction. Misclassifying a transient cause as permanent means silently abandoning a file that would have ingested fine later, which is unrecoverable data loss.

A second constraint rules out the obvious alternative: the refusal messages embed the opened path verbatim, and 2-BM's directory layout carries {UserLastName}-{ProposalNumber}. So the discriminant cannot be the message either.

Changes

  • ScanFileInvalidReason, a closed StrEnum with one member per raise site. None merged — every pair considered keeps a real distinction (the two captured_at causes have opposite remedies; UNREADABLE and DIGEST_UNREACHABLE are different pipeline stages behind different ports).
  • is_transient as the machine-readable split, behind a single frozenset so the verdict has exactly one place to change. Each classification is cited in the docstring from the port docs that already document it, not from intuition.
  • Values are plain PascalCase labels that can never embed a path, so they stay safe to log and to persist forever.
  • InvalidScanFileError.reason carries it alongside the unchanged message. str(exc) is untouched, so this is purely additive at the HTTP boundary and every existing catcher that ignores reason behaves identically.
  • One behavioural use: the sweep's invalid_scan_file warning now names the reason. Diagnosing the current 2-BM stall required opening an HDF5 by hand precisely because that line could not say why.

Also corrects a comment claiming "the class alone already says structurally incomplete, unreadable, or no timestamp; that's enough for an operator" — it is not, which is the whole reason for this change.

Known soft spot, deliberately documented

STRUCTURALLY_INCOMPLETE bundles three sub-causes and only two are strictly permanent; the third is "post-processing not yet run". The member's docstring says so and points the finality work at it first. The follow-on requires a permanent reason and a finality invariant, so the enum was never meant to carry that weight alone.

Test plan

  • Mutation-verified: collapsing STRUCTURALLY_INCOMPLETE onto UNREADABLE fails test_ingest_transient_and_permanent_refusals_yield_different_reasons.
  • All nine raise sites carry distinct reasons; no raise sites exist outside ingest_scan/handler.py, so the required kwarg breaks nothing unseen.
  • Both modified tests were extended, not weakened; no existing assertion changed.
  • Tested against current main (merged in, not just mergeable): 14182 unit, 37443 architecture, all green.

🤖 Generated with Claude Code

xmap and others added 2 commits September 10, 2026 16:07
The capture-scan sweep retries a refused ingest every 30 seconds
forever, and a future change needs to decide when to stop. It cannot
key that decision on InvalidScanFileError alone: the nine raise sites
in ingest_scan.handler span causes as different as a half-transferred
file (transient, retrying helps once the writer finishes) and a
structurally incomplete scan with no rotation-angle dataset (permanent,
no retry will ever produce one). Catching the exception class collapses
that distinction, and misclassifying a transient cause as permanent
means silently abandoning a file that would have ingested fine later.

ScanFileInvalidReason gives each of the nine sites its own member, with
is_transient as the machine-readable split, cited per member from the
port docstrings that already document it (ScanReader's "may be
transient" language, ChecksumVerifier's Unreachable, and so on) rather
than from intuition. Values are plain PascalCase labels that can never
embed a path, so they stay safe to log and persist even though the
messages they're discriminated from carry personal 2-BM directory
paths. No sites were merged: every pair considered (the two
captured_at causes, the two locator causes, Unreadable vs
DigestUnreachable) has a distinct remedy or failure stage.

InvalidScanFileError.reason carries the new enum alongside the
unchanged message; str(exc) is untouched, so this is purely additive
at the HTTP boundary. The only behavioral use added is the capture
scan ingestor's warning log, which now names the reason an operator
can act on without opening the HDF5 file by hand.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  apps/api/src/cora/api
  _capture_scan_ingestor.py
  apps/api/src/cora/data
  errors.py
  apps/api/src/cora/data/features/ingest_scan
  handler.py 238, 479
Project Total  

This report was generated by python-coverage-comment-action

@xmap
xmap merged commit b7280f4 into main Sep 10, 2026
19 checks passed
@xmap
xmap deleted the worktree-agent-a3f6c59e3e6a34ac3 branch September 10, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant