Harden parsing to answer with a reason instead of throwing, and close unsigned construction

[jwrosewell]

Hardening, and the Java half of the same change as owid-dotnet#9, owid-python#2 and owid-go#11.

Reading an OWID means reading whatever a caller was handed, which on a public end point is anything at all, so malformed data is an ordinary outcome rather than an exceptional one. Reporting it by throwing costs the construction and unwinding of an exception for every bad input, and whoever is sending the data chooses how often that happens.

What changes for a caller

Reading answers instead of throwing. Owid.parse, overloaded on the encoded string, the raw bytes and a ByteBuffer, reports three facts every time, being whether it worked, the OWID only when it did, and a named reason either way.

// before
Owid owid = Owid.fromBase64(value);        // throws on anything malformed

// after
OwidParseResult result = Owid.parse(value);
if (result.isSuccess()) {
    Owid owid = result.getValue();         // non-null, status is PARSED
} else {
    log(result.getStatus());               // which of the expected problems
}

OwidReader walks the buffer by index and checks every read against what is left, so a bad envelope is a comparison that fails. It deliberately does not call the old parser and catch, because the exception would still be built and unwound and only the surface would look different.

A framed read, so one OWID can be read out of something longer. Owid.parse(ByteBuffer) reads one envelope from where the buffer is positioned, moves the buffer to the first byte after it, and leaves what follows for the next read, which is how Java says read one item and move along.

ByteBuffer buffer = ByteBuffer.wrap(bytes);
while (buffer.hasRemaining()) {
    OwidParseResult result = Owid.parse(buffer);
    if (result.isSuccess() == false) {
        // buffer is still at the start of the frame that failed, and
        // result.getStatus() says why.
        break;
    }
    use(result.getValue());
}

The two contracts differ in one place. A whole buffer holds one envelope and nothing else, so the declared payload has to leave exactly the signature and a byte after it is BYTE_COUNT_MISMATCH. A frame only requires the declared payload and the signature to be present, because what follows is the next frame rather than rubbish. OwidParseResult.getByteCount() reports the same distance the buffer moved, for a caller that would rather do the arithmetic itself, and the whole buffer read reports it too, where it is the length of the buffer.

Two points in the contract worth naming. A frame that runs past the bytes supplied is UNEXPECTED_END rather than BYTE_COUNT_MISMATCH, because a caller reading from a source that is still arriving needs to know whether to wait for more bytes or to give up on these, and those are different answers. And nothing is consumed by a failed read, so the buffer is left at the start of the frame that failed and what to do with it is the caller's to decide.

Buffers with no array a caller may reach, being direct and read only ones, are read from a copy of what remains. Wrapped arrays, which is the ordinary case, are read in place.

A caller can no longer build an OWID. The constructor is package private and the fields are final with no setters, so an instance arrives from a successful read or from a creator that signs one into existence, never half made. An unsigned OWID is indistinguishable from a signed one to the code downstream of it, and the difference only surfaces later, somewhere that is not looking.

// before
Owid owid = new Owid();
owid.setPayload(bytes);
creator.sign(owid);

// after
Owid owid = creator.createBytes(bytes);    // creation names itself

createString and createBytes own the version, domain, date and signature, and both take an optional list of other OWIDs to cover with the same signature, which is what signWithOthers used to do. Creator.sign, signWithOthers, signString and signBytes are gone, along with Owid.fromBase64, Owid.fromByteArray, the setters and Version.fromByte. The README carries a before and after row for every one of them.

Naming. The new surfaces are spelled the way Java spells them rather than the way C# does. Reading is parse, overloaded on the encoded string and on the raw bytes, not a TryParse with a Boolean return and outputs. Verifying is verify, overloaded on the Crypto and on the public key PEM, named for what it answers rather than for how much detail it carries. Overloading means a caller passing a literal null has to say which one it means, as in Owid.parse((String) null), which the README records. createString and createBytes keep split names rather than being overloads, following the convention this codebase already set with Crypto.signByteArray and verifyByteArray.

State is read only, and getPayload and getSignature hand back copies, so writing into what a caller was given cannot alter an OWID whose signature was computed over the original bytes.

Status vocabulary

OwidParseStatus is the cross language set, so a failure means the same thing whichever language read the bytes.

Every member of both vocabularies either has a test or carries a comment on the member itself saying why it cannot be reached. Four are in the second group.

Member	Why there is no test
OwidParseStatus.INVALID_INPUT_TYPE	The compiler already refuses anything that is not a string or a byte array.
OwidParseStatus.IMPLEMENTATION_CAPACITY_EXCEEDED	A Java array cannot exceed Integer.MAX_VALUE bytes, so a larger declaration can never agree with the bytes present. The guard is kept so a future change to that arithmetic cannot silently truncate the cast below it.
OwidSignatureStatus.IMPLEMENTATION_CAPACITY_EXCEEDED	Needs an OWID and its chain to approach the two gigabyte limit of a Java array. The path is real, being the overflow guard on the serialized length, which raises a distinct exception so the status does not have to be told apart from VERIFICATION_ERROR by reading a message.
OwidParseStatus.MALFORMED_ENVELOPE	The declared payload count is already required to leave exactly the signature, so the check that an envelope ended where the input did cannot fire behind it. The check is kept as a backstop, because a later change to that arithmetic would otherwise start accepting bytes after the signature in silence, and loosening the count rule on purpose does make it fire.

OwidSignatureStatus.INVALID_SIGNATURE_LENGTH is tested from inside the package, because a consumer cannot produce it, as both routes an OWID arrives by settle the signature at 64 bytes. The reason is recorded on the member.

OwidSignatureStatus and the new verify keep "could not check" apart from "does not match". A key that cannot be obtained or cannot be decoded leaves the signature unjudged, and calling that invalid would report an outage as an attack. That is not hypothetical, since on 30 August 2026 the key end points served PEM that a strict parser rejects and every offline verification against them failed, with the keys and identifiers both fine. The existing boolean verifyWithCrypto and verifyWithPublicKey are unchanged.

Two behaviour changes worth review

A declared payload that cannot leave exactly the required signature is now BYTE_COUNT_MISMATCH whichever way the bytes fall short, rather than being reported as a truncation when the buffer also ended early. What a reader can say for certain is that the declaration and the data disagree.

Base 64 decoding is written out in the reader rather than handed to java.util.Base64. Neither JDK decoder answers the question this surface asks, since the strict one throws and the MIME one silently drops every character outside the alphabet, so a string of rubbish used to come back as an empty array and be reported as a missing OWID rather than as text that is not base 64 at all. The new decoder accepts the standard alphabet with or without padding and skips spaces and line breaks, which is what the old path accepted in practice, and refuses anything else as INVALID_BASE64. That is stricter than before for text containing junk, which is the point.

The empty marker says a node is absent

The single byte 0x00 marker for an absent optional OWID reports the new OwidParseStatus.ABSENT_NODE and hands back no value. It used to hand back an OWID, which was the one instance reaching a caller with nothing having signed it, and calling it UNSUPPORTED_VERSION instead was not accurate either, since version zero is supported and does mean something.

It still hands back no value, which is the part that matters, because the marker carries no domain, no date and no signature. Only what the caller is told has changed.

Read as a frame the marker is consumed, so a caller walking a run of frames steps over a node that is deliberately not there and reads the one after it.

Read as a whole buffer the marker is ABSENT_NODE as well, whatever follows it, because the version byte settles the question on its own and nothing after it can turn the value into an OWID. An earlier head of this branch reported MALFORMED_ENVELOPE for a marker with bytes behind it, which no other port does, so it was changed in 5d8facf to give the same answer as owid-rust#4, owid-php#2 and owid-js#9. MALFORMED_ENVELOPE therefore stays unreachable and stays in the untestable list above.

A buffer with no bytes in it is MISSING_INPUT rather than UNEXPECTED_END, because nothing supplied is not the same as data that stopped part way through a field.

Documentation

The README example had gone stale for the Go port without anyone noticing, so the Java example is now compiled and run as ReadmeExampleTest. A change to the library that would break it fails the build.

The payload limits section still said the domain had no encoded maximum, which stopped being true when the 253 character bound went in on the read, on the Creator and on the write helpers. It now records the bound, says that the library will not emit an OWID it would then refuse to read, and names both answers to a declaration that does not fit, being BYTE_COUNT_MISMATCH on the whole buffer read and UNEXPECTED_END on the framed one.

Verification

mvn test passes 99 tests, 0 failures, up from 52 on main. The complete CI matrix is green on this head, being Java 8, 11, 17 and 21 on both Ubuntu and Windows (run 33366341334). Locally only Java 21 was available, so the Java 8 runtime contract rests on that CI run rather than on anything checked by hand here. That matters more than usual for the framed read, since ByteBuffer.position(int) gained a covariant override in Java 9 and is called through Buffer so the class file stays readable on 8.

The new tests were then neutralised one at a time to prove they measure something.

Change undone	Tests that failed
Byte count compared as "not more than" rather than exactly	declaredLengthOffByOneRefused, trailingByteAfterSignatureRefused
Base 64 decoding handed back to the lenient JDK decoder	invalidBase64IsReported
Constructor made public and a setter restored	owidHasNoPublicConstructor, owidHasNoPublicMutation, reflectiveConstructionIsRefused
Byte arrays handed out without copying	writingIntoReturnedArraysDoesNotAlterTheOwid
Creator keeping the caller's payload array	payloadHandedToCreatorIsCopied
A key that cannot be used reported as an invalid signature	undecodableKeyIsInvalidKey
Empty marker allowed to parse again	emptyMarkerIsRefused
Empty buffer reported as a truncation	absentInputIsMissingInput
Framed read made to require the end of the input	8 tests, including twoEnvelopesReadOneAfterTheOther and the README example
Absent node made to consume nothing	3 tests, including anAbsentNodeIsSteppedOverAndTheNextFrameRead
Absent node made to hand back an OWID again	3 tests, through the assertion that no value comes back
Marker followed by bytes reported as a malformed envelope	emptyMarkerIsAnAbsentNodeAndNotAnOwid
A failed read made to report consuming a byte	15 tests, through the byte count assertion on every result

The construction boundary tests live in com.example.owidconsumer, outside the library's package, because a test beside the library shares its package and could reach what a consumer cannot, so asserting the boundary from inside would measure nothing.

Downstream

51Degrees/pipeline-java compiles this source into its 51Did package and calls Owid.fromBase64, Owid.fromByteArray, new Owid(), setPayload and creator.sign, all of which are removed here. That consumer needs updating in its own change when it advances the OWID source pointer.

Produced with AI assistance under James Rosewell's direction and needs human review.