Skip to content

Add event sequence defense: sequence unit tests + design review - #75

Merged
dwilding merged 1 commit into
mainfrom
event-sequence-defense
Sep 7, 2026
Merged

Add event sequence defense: sequence unit tests + design review#75
dwilding merged 1 commit into
mainfrom
event-sequence-defense

Conversation

@dwilding

@dwilding dwilding commented Sep 7, 2026

Copy link
Copy Markdown
Owner

The probe agent's integration tests can fail because of event interactions the agent didn't anticipate — e.g. config-changed sets a status message, then pebble-ready fires and clears it. The agent's unit tests pass in isolation (single event) but the integration test fails in CI (full sequence). This caused a false negative in PR #73.

Two mechanisms, providing defense in depth

1. Sequence unit tests (the mechanism)

When an integration test observes a side effect of an event handler, write a companion unit test that fires events in order (chaining ctx.run() calls in the ops testing harness) and asserts on the observable after the full sequence. This runs in run_tox and catches event interaction bugs before CI — the only way run_tox can catch them, since it doesn't run integration tests.

2. Test design review (the reasoning step)

After run_tox passes and before writing .PR.md, do a structured self-review:

  1. What event triggers the behaviour you are testing?
  2. What observable does your test assert on?
  3. List every handler in the charm that modifies that observable.
  4. After the trigger fires, will any of those handlers also fire?
  5. If a later handler modifies the observable, will your test still pass?

If the answer to #5 is "no" or "not sure," fix the test before proceeding.

Why both?

Either alone is insufficient:

Together, they provide defense in depth: the review forces the agent to think about the sequence, and the sequence test catches the problem even if the reasoning is wrong.

Changes

  • probe_issue.py: rewrote TASK_INSTRUCTIONS with a new "Event sequences and observables" section containing both mechanisms. Updated unit test patterns to show sequence testing with a concrete example. Added the test design review as a mandatory step (step 8) in the workflow.
  • AGENT_DESIGN.md: added a new "Event sequences and observables" subsection under the adversarial testing strategy. Updated the prompt composition section to mention sequence testing and the new task instructions content.

The probe agent's integration tests can fail because of event interactions
the agent didn't anticipate — e.g. config-changed sets a status message,
then pebble-ready fires and clears it. The agent's unit tests pass in
isolation (single event) but the integration test fails in CI (full
sequence). This caused a false negative in PR #73.

Two mechanisms, providing defense in depth:

1. Sequence unit tests: when an integration test observes a side effect of
   an event handler, write a companion unit test that fires events in order
   (chaining ctx.run() calls) and asserts on the observable after the full
   sequence. This runs in run_tox and catches interaction bugs before CI.

2. Test design review: after run_tox passes, do a structured self-review
   — what event triggers the behaviour? What observable? Which handlers
   modify it? Will any fire after the trigger? Will the test still pass?
   If not, fix before writing .PR.md.

Updated unit test patterns to show sequence testing with a concrete
example. Updated AGENT_DESIGN.md with a new 'Event sequences and
observables' subsection.
@dwilding
dwilding merged commit bb9de26 into main Sep 7, 2026
2 checks passed
@dwilding
dwilding deleted the event-sequence-defense branch September 7, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant