Skip to content

Add snapshot/golden test for TerminalReporter output stability #63

Description

@Zious11

Background

Surfaced by PR review of #5 — pr-test-analyzer flagged that default_rendering_unchanged_when_mitre_flag_off (in tests/reporter_tests.rs) checks substrings (`MITRE: T1046`, absence of em-dash, absence of `## Uncategorized`) but does not assert byte-identical output against a known-good baseline.

What snapshot testing would buy

  • Catches incidental whitespace/format drift in the FINDINGS block, ANALYZER section, etc.
  • Forces explicit acknowledgment of any intentional render change (you regenerate the snapshot).
  • Covers behaviors the substring tests can't easily express (e.g., the section-header underline length, color-code byte sequences when `use_color = true`).

What snapshot testing costs

  • Friction on every legitimate format change (must regenerate the snapshot).
  • A new dev-dependency (most likely `insta` — de-facto Rust crate for snapshot testing).
  • Snapshots committed to git can grow large for analyzers with many summary lines.

Why deferred

Validated against external best-practice (Perplexity inconclusive on `insta` adoption rate); the trade-off is real — current substring tests cover the contract that matters today, and snapshot testing is the right tool for "this output should never change without me noticing," which is a different concern from "this output is correct."

Suggested approach if undertaken

  1. Add `insta = "1"` to `[dev-dependencies]`.
  2. Cover the default flat rendering against a fixture finding set first — this is the highest-stability path.
  3. Cover the grouped rendering second.
  4. Document in `tests/reporter_tests.rs` that intentional format changes require `cargo insta review` to accept the new snapshot.

Acceptance criteria

  • Snapshot tests for default + grouped FINDINGS blocks.
  • README or CONTRIBUTING note about regenerating snapshots.
  • No regression in existing substring tests (they remain as semantic assertions).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    reporterOutput/export formatstestTest coverage

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions