hookproof

A conformance and claim-strategy harness for webhook delivery engines, in Java.

It answers two questions that documentation cannot:

1. What does a delivery engine actually do when a receiver rate limits it, resets the connection mid-response, or goes quiet? You point hookproof at a producer, it misbehaves at that producer in fifteen specific ways, and it prints what happened.
2. What does a durable queue cost per dialect? It measures compare-and-swap claiming against FOR UPDATE SKIP LOCKED and SQL Server's READPAST, on H2, PostgreSQL, MySQL and SQL Server, and checks first that each one claims every delivery exactly once.

It was built alongside an LFX Mentorship proposal for CloudEvents webhook notifications in Apicurio Registry, to replace quoting other projects' documentation with measuring them.

Why it exists

Webhook delivery looks simple. Anyone can POST some JSON. The difficulty is entirely in the failure modes, and the failure modes are where implementations quietly disagree with their own docs. Three examples the suite exists to catch:

• 429 is a 4xx, and retrying it is only half the job. Engines commonly split status codes into "5xx, retry" and "4xx, the request is wrong so retrying will not help". That reasoning is right about every code except one. 429 means the request was fine and the receiver wants it again later. An engine that gets that far and then retries on its own schedule, ignoring the Retry-After it was handed, looks correct in a log and still hammers a receiver that asked for room.
• A signature over the body alone is replayable forever. It authenticates the payload perfectly and stops nothing. Both widely deployed schemes sign a timestamp alongside the body for exactly this reason.
• A truncated 200 is not a delivery. An engine that resolves on the response head will record a success for a response that never arrived, and the event is gone with nothing in the log.

Quick start

Needs JDK 17 or later. Docker is only needed for the container-backed dialects.

mvn package

Run the suite against the built-in reference engine:

java -jar target/hookproof-0.1.0-SNAPSHOT.jar conform --target reference

Run it against both reference policies and diff them:

java -jar target/hookproof-0.1.0-SNAPSHOT.jar compare --out reports/comparison.md --json reports/comparison.json

Run a single check while iterating, instead of waiting out the whole suite:

java -jar target/hookproof-0.1.0-SNAPSHOT.jar conform --target reference --checks retries-on-429,replay-protection

Run it against a real Apicurio Registry with a webhook build deployed:

java -jar target/hookproof-0.1.0-SNAPSHOT.jar conform --target apicurio --registry-url http://localhost:8080 --out reports/apicurio.md

Benchmark the claim strategies:

java -jar target/hookproof-0.1.0-SNAPSHOT.jar claims --dialects h2,postgresql,mysql --rows 10000 --workers 8 --out reports/claims.md

Exit code is 1 if any check failed or any strategy proved incorrect, so it drops into CI as is.

The checks

Check	Question
signature-present	Is the delivery signed with a recognisable scheme?
signature-verifies	Does the signature verify with the subscription secret?
replay-protection	Can a captured delivery be replayed and still verify?
cloudevents-structure	Is the payload a valid CloudEvent, and in which HTTP mode?
idempotency-key	Do retries carry a stable, signed identifier for deduplication?
retries-on-5xx	Are server errors retried until they succeed?
retries-on-429	Is a rate limit retried rather than treated as permanent?
no-retry-on-permanent	Is a 410 Gone left alone?
retries-on-reset	Is a TCP connection reset retried?
no-false-success-on-partial	Is a 200 whose body is truncated treated as a failure?
backoff-shape	What retry schedule is observed on the wire?
honours-retry-after	Does Retry-After actually change the next attempt?
backoff-jitter	Is the schedule randomised, or identical every run?
gives-up-eventually	Does delivery stop against a dead receiver, and when?
retries-on-timeout	Is an unanswered request abandoned and retried?

Outcomes are pass, FAIL, warn, ? (inconclusive) and n/a. Inconclusive is deliberately distinct from a failure. A suite that reported "failed" when it simply could not observe enough attempts would manufacture findings, and manufactured findings are worth less than none.

Every command writes markdown with --out and JSON with --json, and both can be requested from the same run. The JSON carries the evidence lines alongside each verdict, so a pipeline can fail on a specific check id rather than on the overall exit code.

docs/design.md is the design this was built from, annotated with the places where the build disagreed with the plan.

How the measurements are made

The receiver speaks HTTP over raw sockets. Not because that is fun, but because several of the interesting faults cannot be expressed through a server library. Aborting a connection with a TCP reset after the status line has been written is the clearest case, and it is also the fault most likely to produce a false success. Keep-alive is supported, since producers built on Vert.x reuse connections and a receiver that closed every time would change the behaviour under test.

Knowing when a retry sequence has ended is the hard part. The first version waited a fixed period of silence, and it produced a fabricated finding within minutes of first running: a three and a half second wait against an engine whose fourth backoff step was four seconds reported "gave up after 3 attempts" about a producer that had done nothing wrong. A tool whose whole job is reporting defects cannot invent them.

So the wait adapts. A probe ends once the number of attempts the scenario implies has arrived and stayed quiet briefly. Failing that, it holds until the silence exceeds 2.5 times the largest gap already observed, on the reasoning that a schedule which has been doubling will not suddenly pause far longer than its own last step. FaultReceiverTest pins both halves of this.

Some properties are structural, not empirical. Replay protection is decided by what goes into the signature, so the suite reports it from the detected scheme and then demonstrates it by re-verifying a captured request out of band. Waiting to catch a replay in the act would prove nothing extra.

honours-retry-after is a controlled comparison. A producer whose ordinary first backoff already exceeds the requested delay would look like it honours the header while ignoring it. So the natural backoff is measured against a plain 500 first, and the two are compared.

Jitter needs two runs. One clean exponential sequence looks the same whether or not the delay was randomised, so the same scenario runs twice and the gap sequences are compared.

The reference engine

io.hookproof.reference.ReferenceProducer is a small but complete delivery engine: retry state on the queued delivery rather than on a thread, a scheduled dispatcher polling for work that has come due, and a compare-and-swap claim. Its queue is in memory rather than in a database, which is the one structural difference and does not affect anything the suite asks.

It exists to make the suite falsifiable. A conformance suite that has only ever run against one target cannot tell you whether a failure is a real defect or a bug in the check. Two presets ship:

• Policy.sliceMimic() models the design described in the Apicurio proposal: body-only HMAC, 1/2/4/8 second backoff with no jitter, 429 retried but Retry-After ignored.
• Policy.recommended() is the same engine with Standard Webhooks signing, 429 retried, Retry-After honoured and jitter on.

ConformanceDiscriminationTest runs the checks against both and asserts each one agrees with what was configured, including that the mimic passes the checks it should pass.

The mimic is a model built from a written design, not a measurement of the real code. Anything it reports is a prediction, to be confirmed by pointing the suite at a running registry with --target apicurio.

The claim benchmark

Four dialects, three strategies, and correctness checked before speed:

Dialect	Fast path available
H2	none
PostgreSQL	FOR UPDATE SKIP LOCKED
MySQL	FOR UPDATE SKIP LOCKED
SQL Server	WITH (UPDLOCK, READPAST)

The portability story is more nuanced than "PostgreSQL has SKIP LOCKED and nothing else does". PostgreSQL 9.5, MySQL 8.0 and Oracle all support it, and SQL Server reaches the same behaviour with READPAST. The one with neither is H2, which happens to be the dialect most test suites run on. So the real constraint is that a compare-and-swap fallback has to exist regardless, and the only open question is whether the faster path is worth maintaining alongside it.

Every id every worker claims is collected and compared, so a strategy that hands the same delivery to two workers is caught rather than rewarded for being quick about it.

What that caught. Against real containers, compare-and-swap was correct on both PostgreSQL and MySQL and FOR UPDATE SKIP LOCKED was correct on PostgreSQL, but skip locked deadlocked on MySQL with six workers draining three thousand rows. The select takes its locks through the (status, next_retry_at) index and the following update rewrites that same entry, since status is part of it, so two transactions can want the same locks in opposite orders. SkipLockedStrategy now sorts the locked ids so every worker rewrites those index entries in the same order, and retries on SQLState 40001 and 40P01 with jittered backoff. Skip locked is worth having, but it is not a drop-in optimisation: it buys throughput and costs you a deadlock path that compare-and-swap never has.

These are not publishable benchmark numbers. Everything runs on one machine, the databases are containers on that same machine, and no attempt is made to model latency between a dispatcher and its database. The ratio between two strategies measured in the same run is meaningful. The absolute figures are not, and the generated report repeats this.

Known issue: Docker Desktop on Windows

Testcontainers 1.20.4 (docker-java 3.4.0) could not reach Docker Desktop 29.6.1 on Windows during development. Every named pipe it tries answers /info with HTTP 400 and a near-empty body whose only content is the label com.docker.desktop.address=npipe://\\.\pipe\docker_cli. The docker CLI works perfectly throughout, which makes this look like a hookproof problem when it is not.

Tried and did not help: DOCKER_HOST pointed at dockerDesktopLinuxEngine (the active desktop-linux context), DOCKER_HOST pointed at docker_cli (the pipe the daemon itself advertises, which returns 404), and DOCKER_API_VERSION=1.44 against a daemon reporting a minimum of 1.40.

Until that is resolved, run the container-backed dialects on a Linux Docker host, where the socket is a plain unix:///var/run/docker.sock and none of this applies. H2 needs no container and runs anywhere.

Testing

mvn test

Container-backed dialect tests are excluded by default because they need Docker and the SQL Server image is large. To run PostgreSQL and MySQL:

mvn test -Dgroups=containers -DexcludedGroups=sqlserver

To include SQL Server as well, clear the exclusion entirely:

mvn test -Dgroups=containers -DexcludedGroups=

-DexcludedGroups is not optional here. The pom excludes the containers tag by default, exclusion beats inclusion in JUnit 5, and -Dgroups=containers on its own therefore selects those tests and then drops them again. That is not hypothetical: the first CI run did exactly this, reported Tests run: 0, and went green in sixteen seconds. The workflow now asserts a minimum test count so it cannot happen quietly again.

Pointing it at Apicurio Registry

ApicurioRegistryDriver encodes assumptions about a REST surface that is not upstream yet: the subscription endpoint path and its JSON field names, which follow the shape described in the proposal. If the merged API differs, two constants and one method need editing, and the checks themselves are unaffected.

Licence

Apache License 2.0.