Discriminate InvalidScanFileError's cause with a closed, non-PII enum - #797
Merged
Conversation
The capture-scan sweep retries a refused ingest every 30 seconds forever, and a future change needs to decide when to stop. It cannot key that decision on InvalidScanFileError alone: the nine raise sites in ingest_scan.handler span causes as different as a half-transferred file (transient, retrying helps once the writer finishes) and a structurally incomplete scan with no rotation-angle dataset (permanent, no retry will ever produce one). Catching the exception class collapses that distinction, and misclassifying a transient cause as permanent means silently abandoning a file that would have ingested fine later. ScanFileInvalidReason gives each of the nine sites its own member, with is_transient as the machine-readable split, cited per member from the port docstrings that already document it (ScanReader's "may be transient" language, ChecksumVerifier's Unreachable, and so on) rather than from intuition. Values are plain PascalCase labels that can never embed a path, so they stay safe to log and persist even though the messages they're discriminated from carry personal 2-BM directory paths. No sites were merged: every pair considered (the two captured_at causes, the two locator causes, Unreadable vs DigestUnreachable) has a distinct remedy or failure stage. InvalidScanFileError.reason carries the new enum alongside the unchanged message; str(exc) is untouched, so this is purely additive at the HTTP boundary. The only behavioral use added is the capture scan ingestor's warning log, which now names the reason an operator can act on without opening the HDF5 file by hand. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The capture-scan sweep retries a refused ingest every 30 seconds forever. A follow-on change needs to decide when to stop, and it cannot key that on
InvalidScanFileErroralone: nine raise sites share that one class, spanning causes as different as a half-transferred file (transient, retrying works once the writer finishes) and a scan with no rotation-angle dataset (permanent, no retry ever produces one).Catching the class collapses that distinction. Misclassifying a transient cause as permanent means silently abandoning a file that would have ingested fine later, which is unrecoverable data loss.
A second constraint rules out the obvious alternative: the refusal messages embed the opened path verbatim, and 2-BM's directory layout carries
{UserLastName}-{ProposalNumber}. So the discriminant cannot be the message either.Changes
ScanFileInvalidReason, a closedStrEnumwith one member per raise site. None merged — every pair considered keeps a real distinction (the twocaptured_atcauses have opposite remedies;UNREADABLEandDIGEST_UNREACHABLEare different pipeline stages behind different ports).is_transientas the machine-readable split, behind a single frozenset so the verdict has exactly one place to change. Each classification is cited in the docstring from the port docs that already document it, not from intuition.InvalidScanFileError.reasoncarries it alongside the unchanged message.str(exc)is untouched, so this is purely additive at the HTTP boundary and every existing catcher that ignoresreasonbehaves identically.invalid_scan_filewarning now names the reason. Diagnosing the current 2-BM stall required opening an HDF5 by hand precisely because that line could not say why.Also corrects a comment claiming "the class alone already says structurally incomplete, unreadable, or no timestamp; that's enough for an operator" — it is not, which is the whole reason for this change.
Known soft spot, deliberately documented
STRUCTURALLY_INCOMPLETEbundles three sub-causes and only two are strictly permanent; the third is "post-processing not yet run". The member's docstring says so and points the finality work at it first. The follow-on requires a permanent reason and a finality invariant, so the enum was never meant to carry that weight alone.Test plan
STRUCTURALLY_INCOMPLETEontoUNREADABLEfailstest_ingest_transient_and_permanent_refusals_yield_different_reasons.ingest_scan/handler.py, so the required kwarg breaks nothing unseen.main(merged in, not just mergeable): 14182 unit, 37443 architecture, all green.🤖 Generated with Claude Code