Report what Hookdeck did when the retry test times out - #17
Open
garethx wants to merge 1 commit into
Open
Conversation
This test fails intermittently — roughly one full run in four, never when the file runs alone. `test:live` runs on every pull request, so it fails unrelated work, and "timed out waiting for a delivery" is all the evidence that survives. That message cannot distinguish an event that was never created, one still queued, one delivered against a connection the CLI was no longer attached to, or a retry rule that was replaced before it applied. Those have different fixes, and guessing between them has already cost several rounds. So the timeout now reports the event and its attempt triggers, the connection's paused and disabled state, and the rules actually in force. It does not fix the flake: it makes the next occurrence say what happened, in CI, without needing to be reproduced afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/live/delivery.test.mjs→ "a refused delivery is retried" failsintermittently: roughly one full
npm run test:livein four, and never when thefile runs on its own. It has failed on at least two unrelated pull requests.
Since
test:liveruns on every PR, that is a flake other people's work pays for.Why this instruments rather than fixes
All the failure leaves behind is:
That is consistent with several different causes, which need different fixes:
hookdeck listenwas no longer attachedto — the node's own setup text warns that events delivered while no CLI session
exists are not recorded against that connection at all
I have chased this three times on hypotheses and twice been wrong: first the CLI
health check reading a deliberate 503 as a dead origin (a real bug, fixed with
--no-healthcheck, but not this one), then an unchecked ingest response (also areal bug, also fixed, also not this one). Both were worth fixing. Neither was the
cause, because I could not reproduce the failure on demand and was reasoning from
absence.
So this PR makes the failure state its own case. On timeout it now reports:
What it does not do
It does not make the test pass more often, and deliberately does not add
retries or widen the timeout to hide it — masking a flake nobody understands is
how it becomes permanent. The next CI failure should be enough to fix it
properly.
Verified the instrumented test still passes:
test/live/delivery.test.mjs6/6.