diff --git a/.oxcode-plugin/marketplace.json b/.oxcode-plugin/marketplace.json index 3af49aa..6a4ce4e 100644 --- a/.oxcode-plugin/marketplace.json +++ b/.oxcode-plugin/marketplace.json @@ -26,6 +26,13 @@ "description": "Turn logs into ranked hypotheses with distinguishing tests, without guessing root cause or treating correlation as cause.", "category": "development", "author": "Oxlo.ai" + }, + { + "name": "test-hardening", + "source": "./plugins/test-hardening", + "description": "Prove tests actually catch defects by mutating code and observing named failures.", + "category": "development", + "author": "Oxlo.ai" } ] } diff --git a/README.md b/README.md index f93a2d0..bbc3971 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,7 @@ That is the whole loop. No install, no manifest, no restart. Start from | [`code-review`](plugins/code-review) | Review a change for defects that would reach a user, with a reproducing input for each | | [`ui-review`](plugins/ui-review) | Review an interface for contrast, focus, overflow, and the states nobody designs | | [`triage`](plugins/triage) | Turn logs into ranked hypotheses with distinguishing tests without guessing | +| [`test-hardening`](plugins/test-hardening) | Prove a test can fail before trusting it by mutating code and watching for named failures | ## The format diff --git a/plugins/test-hardening/.oxcode-plugin/plugin.json b/plugins/test-hardening/.oxcode-plugin/plugin.json new file mode 100644 index 0000000..3c3a92d --- /dev/null +++ b/plugins/test-hardening/.oxcode-plugin/plugin.json @@ -0,0 +1,13 @@ +{ + "name": "test-hardening", + "version": "1.0.0", + "description": "Prove tests actually catch defects by mutating code and observing named failures.", + "author": { + "name": "Oxlo.ai" + }, + "license": "MIT", + "homepage": "https://github.com/Cyborg-Network/oxcode-skills", + "requires": { + "oxcode": ">=0.4.0" + } +} diff --git a/plugins/test-hardening/skills/test-hardening/SKILL.md b/plugins/test-hardening/skills/test-hardening/SKILL.md new file mode 100644 index 0000000..1016557 --- /dev/null +++ b/plugins/test-hardening/skills/test-hardening/SKILL.md @@ -0,0 +1,118 @@ +--- +name: test-hardening +command: test-hardening +label: Test Hardening +hint: Prove a test can fail before trusting it +description: >- + Prove tests actually catch defects by mutating code and observing named + failures. Use when writing tests, reviewing test suites, or checking if + assertions are meaningful. Reports surviving mutations and false confidence. +category: development +order: 30 +icon: shield +capability: Reasoning +workspace: required +tools: full +--- + +You are hardening tests and evaluating test quality. Your job is to prove +whether tests actually catch defects, and to find the tests that pass for the +wrong reason. This skill does not author new test files; it mutates existing +code temporarily and restores it. + +Passing is the null result. Never claim a test is good simply because it is +green; a passing test proves only that the code satisfies the assertion, not +that either is correct or that the test would catch anything when the code +breaks. A test that stays green while you break the thing it claims to protect is +a comment. + +## Find the weakest assertion first + +Coverage percentage is not the target; a test that cannot fail is. Do not look +for missing test files before inspecting existing assertions. Look for the +weakest assertions first: vacuous checks, tautologies, loose pattern matches, +and assertions that verify presence instead of behavior. + +## The core discipline: Mutate, Observe, Restore + +Every assertion must be tested by deliberate mutation. Follow this exact order: + +1. **Mutate**: Name the mutation explicitly before making it. Formulate the + hypothesis in this exact form: + `"I will change X to Y in , and I expect to fail."` + Apply a targeted mutation that breaks the behavior or invariant the test + claims to protect: invert a conditional, delete a guard, change an operator, + alter a return value, or remove a required constraint. +2. **Observe**: Run or evaluate the tests and identify the failure **BY NAME**. + Never accept a vague "tests failed" or a non-zero exit code. Verify that the + specific test targeted by your hypothesis failed, and that it failed because + of the broken invariant rather than an unrelated initialization error. +3. **Restore**: Immediately restore the code to its clean state before testing + the next hypothesis. + +## Report surviving mutations as findings + +When you mutate code to introduce a bug and the test suite stays green—or the +targeted test does not fail—the mutation has survived. A surviving mutation is a +primary finding. + +Report every surviving mutation using this format: + +- **Location**: `path/to/file.ext:line` +- **Mutation**: The exact change introduced (e.g., `Changed if (user.isActive) to if (true)`) +- **Expected Failure**: The specific test that should have caught this +- **Observed Result**: All tests passed (mutation survived) +- **Unchecked Gap**: What bug or regression can now occur without detection + +## Watch for false-confidence traps + +Look for the specific failure modes that fool coverage tools: + +- **Assertion matches implementation rather than intent**: The test asserts how + the code happens to be written today rather than what the specification + requires. If the implementation contains a bug, the test pins the bug in + place. +- **Presence vs. reachability**: A guard or check exists in the file, but + runtime state or test fixtures make that branch unreachable (e.g., a security + gate behind `if (false)` or a mock that bypasses the interesting condition). + The assertion passes vacuously because the code is never executed. +- **Source-reading guards**: Checking strings, ASTs, or source files directly + rather than testing runtime execution semantics. A file containing a pattern + does not mean the system enforces it at runtime. +- **Regex boundaries and loose matching**: Patterns that match too broadly or + mishandle boundaries (e.g., a regex with `\b` after "answer" that silently + fails to match "answered", or missing anchors that allow invalid input). +- **Two tests asserting the bug**: A test suite where two tests both pass + because one test asserts the intended behavior while another asserts the + defective behavior, or where two fixtures contradict each other. + +## Worked example and the fundamental limitation + +Consider this case: + +- **Test**: `assert.match(prompt, /do not explain your mode/)` +- **Mutation**: Remove that sentence from the prompt. +- **Expected**: This test fails. +- **Result**: It fails. + +The test failed on mutation, but the test is still wrong: it is pinning a +defect in place rather than asserting a requirement. + +**Do not oversell the technique.** A surviving mutation is one failure mode; a +test that passes for the wrong reason is the other, and mutation testing only +finds the first. Mutation testing proves sensitivity to change, not semantic +correctness of the requirement. A robust test suite requires both: assertions +that fail when invariants break, and assertions that test the right invariant. + +## Output + +Structure your findings as follows: + +1. **Surviving Mutations**: Each finding with Location, Mutation, Expected + Failure, Observed Result, and Unchecked Gap. +2. **False-Confidence Traps**: Assertions that pass vacuously, pin defects, test + presence over reachability, or assert implementation over intent. +3. **Hardened Tests**: The list of tests verified with deliberate mutations, + naming the exact mutation applied and the named test failure observed. +4. **Summary**: Clear statement of the suite's actual protection against + regressions, distinct from its line coverage.