Add event sequence defense: sequence unit tests + design review - #75
Merged
Conversation
The probe agent's integration tests can fail because of event interactions the agent didn't anticipate — e.g. config-changed sets a status message, then pebble-ready fires and clears it. The agent's unit tests pass in isolation (single event) but the integration test fails in CI (full sequence). This caused a false negative in PR #73. Two mechanisms, providing defense in depth: 1. Sequence unit tests: when an integration test observes a side effect of an event handler, write a companion unit test that fires events in order (chaining ctx.run() calls) and asserts on the observable after the full sequence. This runs in run_tox and catches interaction bugs before CI. 2. Test design review: after run_tox passes, do a structured self-review — what event triggers the behaviour? What observable? Which handlers modify it? Will any fire after the trigger? Will the test still pass? If not, fix before writing .PR.md. Updated unit test patterns to show sequence testing with a concrete example. Updated AGENT_DESIGN.md with a new 'Event sequences and observables' subsection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The probe agent's integration tests can fail because of event interactions the agent didn't anticipate — e.g. config-changed sets a status message, then pebble-ready fires and clears it. The agent's unit tests pass in isolation (single event) but the integration test fails in CI (full sequence). This caused a false negative in PR #73.
Two mechanisms, providing defense in depth
1. Sequence unit tests (the mechanism)
When an integration test observes a side effect of an event handler, write a companion unit test that fires events in order (chaining
ctx.run()calls in the ops testing harness) and asserts on the observable after the full sequence. This runs inrun_toxand catches event interaction bugs before CI — the only wayrun_toxcan catch them, since it doesn't run integration tests.2. Test design review (the reasoning step)
After
run_toxpasses and before writing.PR.md, do a structured self-review:If the answer to #5 is "no" or "not sure," fix the test before proceeding.
Why both?
Either alone is insufficient:
Together, they provide defense in depth: the review forces the agent to think about the sequence, and the sequence test catches the problem even if the reasoning is wrong.
Changes