Skip to content

Harden 51Did parsing to answer with a reason instead of throwing - #118

Merged
jwrosewell merged 6 commits into
mainfrom
harden/51did-parse-without-throwing
Aug 31, 2026
Merged

Harden 51Did parsing to answer with a reason instead of throwing#118
jwrosewell merged 6 commits into
mainfrom
harden/51did-parse-without-throwing

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

What changed and why

The OWID library this package compiles in (SWAN-community/owid-java, branch harden/parse-without-throwing, PR #5) has been hardened so that an Owid only ever comes from a successful non-throwing read or from a Creator that signs it. The throwing factories Owid.fromBase64 and Owid.fromByteArray, the public constructors and the setters are gone. pipeline.did called the removed factories, so this PR adapts the package to the hardened surface and gives 51Did reading the same shape.

The 51Did reading contract is now:

  1. The OWID library reads the envelope through its non-throwing Owid.parse.
  2. An envelope fault is reported with the OWID status carried across unchanged, under the same name, never folded into a more general one.
  3. The payload must hold the five byte header (one byte of flags, four byte licence id) or the read reports PAYLOAD_TOO_SHORT. Random then needs the header plus 16 GUID bytes, and Probabilistic and HashedEmail need the header plus a 32 byte hash, or the read reports INVALID_TYPE_PAYLOAD_LENGTH. Reserved keeps its documented best-effort behaviour.
  4. Anything longer is accepted as it stands. The package has no upper bound on a payload, because the bytes past the value are a creator context section whose shape belongs to the cloud.
  5. A successful read is structurally valid and nothing more. The signature has not been checked, and the names and documentation say so.

Public API

Added, all in fiftyone.pipeline.did:

  • FodId.tryFromBase64(String) and FodId.tryFromByteArray(byte[]), returning FodIdParseResult.
  • FodIdParseResult with isSuccess(), getValue() (null on failure, never a half-read 51Did) and getStatus() (PARSED on success).
  • FodIdParseStatus, the OWID OwidParseStatus vocabulary mirrored member for member plus PAYLOAD_TOO_SHORT and INVALID_TYPE_PAYLOAD_LENGTH, with fromOwid(OwidParseStatus) mapping by name and refusing anything the mirror does not know. A test asserts every OWID member is mirrored.
  • FodId.verifyDetailed(String publicPem) returning the library's OwidVerificationResult, so "does not match" (SIGNATURE_INVALID) stays apart from "could not check" (KEY_UNAVAILABLE, INVALID_KEY).

Kept, with the same exception types as before: FodId.fromBase64, FodId.fromByteArray, FodId.fromOwid, FodId.verify(String). The throwing readers delegate to the same read as the non-throwing ones, so there is one walk of the payload. fromOwid still declares OwidException so existing callers compile, but no longer copies the envelope, because the library only hands out immutable ones.

Changed in DidClient: verify(String) and redeem(String, String, String) read the value after the existing length guard and before any key fetch or cloud call, and refuse a value that does not read with the IllegalArgumentException they already documented, now naming the status. verifySignatureDetailed uses the library's verification result instead of catching an exception per candidate key. The 4096 character guard is untouched and stays client policy, not a format limit.

Removed from callers in this repository: every use of Owid.fromBase64, Owid.fromByteArray, new Owid(...), Creator.sign(Owid), Owid.setDate and Owid.setPayload, which were in FodId, the test factory, two tests and the pipeline.developer-examples.fodid example. A repository-wide grep finds no other use.

Before and after for a caller

Before:

Owid owid = Owid.fromBase64(text);          // threw on bad input
FodId fodId = FodId.fromOwid(owid);

After:

FodIdParseResult read = FodId.tryFromBase64(text);   // never throws
if (read.isSuccess()) {
    FodId fodId = read.getValue();                    // not yet verified
} else {
    FodIdParseStatus why = read.getStatus();          // for example INVALID_BASE64
}
// or keep the exception style, which makes the same read:
FodId fodId = FodId.fromBase64(text);

Code that built an envelope with new Owid(domain, date, payload) and then creator.sign(owid) now asks the creator for one with creator.createBytes(payload). The README carries the same migration note.

The OWID pin

The owid-java submodule moves from 7a7f303 to 694f6333, the head of main on the 51Degrees/owid-java fork, which carries the squash merged hardening commit 802f7e4 and has the same tree as that commit. .gitmodules points at ../owid-java.git on branch main, as it did on this repository's main. pipeline.did/pom.xml compiles the submodule source straight into the module with build-helper-maven-plugin, so the build really compiles the hardened source and nothing comes from a cached artefact.

Tests

  • pipeline.did before: 99 tests (2 skipped, the live tests that need a resource key). After: 125 tests, 0 failures, 2 skipped. The new FodIdParseTests class holds the non-throwing contract and every existing test still passes.
  • pipeline.developer-examples.fodid: 9 tests before and after, 0 failures.
  • Every README code block was compiled and run against the built module with a recording transport standing in for the cloud, and every block ran to completion.
  • Full repository build as CI runs it (mvn install -DskipTests, then mvn surefire:test): all 25 modules built, and 587 tests ran across the repository with 0 failures and 7 skipped (the skips are pre-existing, being the live tests that need a resource key). Only pipeline.did and pipeline.developer-examples.fodid reference the 51Did package.

Neutralisation

Each new check was undone in turn, the pipeline.did suite run, and the check restored. Every neutralisation was caught, and the suite was green again after restoring.

Check undone Tests that failed
Header minimum (PAYLOAD_TOO_SHORT) 3: DidClientTests.verify_ShortPayloadStringIsRefusedBeforeTransport, FodIdParseTests.tryFromBase64_ShorterThanHeader_PayloadTooShort, FodIdTests.constructor_PayloadEmpty_Throws
Type minimum (INVALID_TYPE_PAYLOAD_LENGTH) 8: DidClientTests.redeem_MalformedStringIsRefusedBeforeTransport, FodIdParseTests.throwingReaders_ThrowTheDocumentedTypesForTheSameInputs, FodIdParseTests.tryFromBase64_ProbabilisticOneByteShort_InvalidTypePayloadLength, FodIdParseTests.tryFromBase64_RandomOneByteShort_InvalidTypePayloadLength, FodIdParseTests.tryFromByteArray_HashedEmailOneByteShort_InvalidTypePayloadLength, FodIdTests.constructor_HashedEmailPayloadOneByteShort_Throws, FodIdTests.constructor_PayloadOneByteShort_Throws, FodIdTests.constructor_RandomPayloadOneByteShort_Throws
OWID status carried unchanged (every envelope fault mapped to MALFORMED_ENVELOPE) 6: DidClientTests.verify_MalformedStringIsRefusedBeforeTransport, FodIdParseTests.throwingReaders_ThrowTheDocumentedTypesForTheSameInputs, FodIdParseTests.tryFromBase64_InvalidBase64_ReportsTheOwidStatus, FodIdParseTests.tryFromByteArray_DeclarationMismatch_PropagatedUnchanged, FodIdParseTests.tryFromByteArray_OtherEnvelopeFaults_PropagatedUnchanged, FodIdParseTests.tryFrom_NullOrEmpty_MissingInput
Client read before transport (both ensureReadsAs51Did calls removed) 3: DidClientTests.redeem_MalformedStringIsRefusedBeforeTransport, DidClientTests.verify_MalformedStringIsRefusedBeforeTransport, DidClientTests.verify_ShortPayloadStringIsRefusedBeforeTransport
Client result-based verify (any status treated as verified) 5: DidClientTests.verifySignature_EarlierNeighbourWithinToleranceAfterBoundary, DidClientTests.verifySignature_FalseWithAPublishedKeyFromAnotherPeriod, DidClientTests.verifySignature_FalseWithTheWrongKey, DidClientTests.verifySignature_LaterNeighbourWithinToleranceBeforeBoundary, DidClientTests.verifySignature_TamperedSignatureIsInvalidNotAnError

Checked with no issue

  • The 4096 character guard in DidClient is unchanged, still runs before reading, key retrieval and transport, and is still tested separately from the reader (verify_OverLongStringIsRefusedBeforeTransport, verify_OverLongObjectIsRefusedBeforeTransport, redeem_OverLongInputsAreRefusedBeforeTransport).
  • No constant, message, status or test names a removed envelope, payload, encoded-length or creator-context size. The only lengths in the package are the per type minimums that were already there.
  • Java 8 target holds. No records, sealed types or var, and the module's Animal Sniffer check against the Java 8 signature passed as part of the build.
  • Both base64 alphabets are still accepted, with the URL-safe form restored to the standard one in FodId before the OWID reader sees it, as before.
  • Comments and documentation no longer describe an OWID that can be built directly, changed after the fact, or exist unsigned.

What remains

  • DidClientLiveTests are skipped without a resource key, as before.

Produced with AI assistance under James Rosewell's direction and needs human review.

The OWID library removed its throwing parse factories, public
constructors and setters on branch harden/parse-without-throwing, so an
OWID now only comes from a successful read or from a creator that signs
it. The submodule points at that branch tip, 5d8facf, in the
SWAN-community repository for now, because the 51Degrees fork does not
yet carry the commit. Both the URL and the pin move back to the fork at
the merged commit once the branch is on main.
FodId.tryFromBase64 and FodId.tryFromByteArray answer with a
FodIdParseResult that reports whether the read worked, the 51Did only
when it did, and a FodIdParseStatus either way. The status vocabulary is
the OWID library's own, mirrored member for member under the same names
so an envelope fault is reported exactly as the envelope reader found
it, plus PAYLOAD_TOO_SHORT for a payload that cannot hold the five byte
header and INVALID_TYPE_PAYLOAD_LENGTH for one shorter than its type's
minimum. Nothing puts an upper bound on a payload, because the bytes
past the value are a creator context section whose shape belongs to the
cloud.

The throwing readers fromBase64, fromByteArray and fromOwid make the
same read and throw the exception types they always did, so there is
one walk of the payload and not two. fromOwid no longer copies the
envelope, because the library only hands out immutable ones. Reading
never checks the signature, and verifyDetailed says why a signature
does or does not verify, keeping a key that could not be used apart
from a signature that did not match.

The tests cover the three facts on every result, the type minimums,
longer domains and longer context sections, every OWID status carried
across unchanged, and the throwing readers agreeing with the
non-throwing ones. The test factory signs through the library's creator
and reads hand-written envelopes back through the library's parse.
DidClient.verify(String) and DidClient.redeem(String, ...) now read the
value with FodId.tryFromBase64 after the existing length guard and
before any key is fetched or the cloud is called, throwing the
IllegalArgumentException they already documented with the parse status
in the message. Malformed input therefore costs no network round trip
and no use, and the value is still sent to the cloud exactly as the
caller gave it.

verifySignatureDetailed uses the library's verification result rather
than catching an exception per candidate key, so a key that cannot be
read and a signature that does not match are both simply the next
candidate to try. The tests prove no transport call happens for
malformed input, that a longer identifier still reaches the cloud, that
a tampered signature reads and then verifies as invalid, and that a
key list that cannot be fetched is an error and never a false.
The hardened OWID library has no public constructor, so the example
asks the creator to stamp and sign the payload in one step instead of
building an envelope and signing it afterwards.
The README now explains the non-throwing readers and the three facts
every result reports, what each status means and which failures are
data results rather than exceptions, the per type lower bounds and the
absence of any upper bound, that the client's length guard is client
policy and not a format limit, and a before and after for a caller who
reached the removed OWID API through this package.
The owid-java submodule now records
694f63332734132f9e432b9bf1e9ecb734af1b1c, the head of main on the
51Degrees/owid-java fork, which carries the squash merged hardening
commit 802f7e4 and has the same tree. The .gitmodules entry returns to
../owid-java.git on branch main, as it was on this repository's main,
so the temporary SWAN-community URL, branch and pin at 5d8facf are gone.
@jwrosewell
jwrosewell merged commit ca6a296 into main Aug 31, 2026
1 check passed
@jwrosewell
jwrosewell deleted the harden/51did-parse-without-throwing branch August 31, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant