Skip to content

fix: #797 — restructure known_failures.json with audit metadata#819

Merged
proggeramlug merged 1 commit into
mainfrom
worktree-fix-797-known-failures-audit
May 16, 2026
Merged

fix: #797 — restructure known_failures.json with audit metadata#819
proggeramlug merged 1 commit into
mainfrom
worktree-fix-797-known-failures-audit

Conversation

@proggeramlug
Copy link
Copy Markdown
Contributor

Summary

  • test-parity/known_failures.json moves from a flat string-to-string map to per-entry structured records with issue, added (date skip-listed), category, and reason.
  • Adds an inline _schema block at the top of the file (we don't generally add free-standing READMEs without an explicit ask, so the schema lives in the file it documents).
  • Updates .github/workflows/test.yml to filter _schema out of the test-name set and to validate every other entry has non-empty category + reason — fails the build on malformed entries so the format can't silently drift back.

Audit findings (relayed in the entry reason strings)

Test plan

  • jq -r 'keys[] | select(. != \"_schema\")' test-parity/known_failures.json | wc -l → 58 (matches original 59 minus the new _schema key).
  • Schema-validator jq returns no malformed entries.
  • CI gate logic unchanged for the failure-diff computation.

Closes #797.

`test-parity/known_failures.json` was a flat string-to-string map. The
issue (#797) asked for per-entry provenance: an issue reference and the
date the test was skip-listed, so stale entries can be closed and
orphans can be filed.

Restructures each entry as an object:

  "test_name": {
    "issue": "793" | null,    # open tracker, or null for ci-env / pending triage
    "added": "YYYY-MM-DD",    # date skip-listed
    "category": "ci-env | module-inventory | bug-open | bug-stale | gap-categorical | gap-bisect",
    "reason": "..."
  }

Adds a top-level `_schema` key that documents the format inline (we'd
otherwise need a separate README; per project convention we don't add
those without an explicit ask). The CI gate in `.github/workflows/test.yml`
is updated to:

- Filter `_schema` out of the keys it diffs against test failures, so
  it isn't treated as a phantom known test.
- Validate that every other entry is an object with non-empty
  `category` and `reason` — fails the build on malformed entries so the
  format doesn't silently drift back to the legacy flat-string shape.

All historical entries inherit `added: 2026-05-15` (the audit date). All
cross-referenced issues were checked: closed ones (#233, #422, #462,
#510, #561, #616, #654, #685, #91) are flagged as `bug-stale` and rolled
under the parity roadmap meta (#793) until a Linux-specific tracking
issue is filed for the surviving sub-case. Net tests that need a
fixture (`test_net_min`, `test_net_socket`, `test_net_upgrade_tls`,
`test_issue_422_socket_connect`, `test_sock_write_map`) are tagged
`ci-env` with `issue: null` — those need a CI fixture, not a Perry fix.
Module-inventory trackers point at #789 (async_hooks) or #793 (rest).
@proggeramlug proggeramlug merged commit 13e8cb4 into main May 16, 2026
9 checks passed
@proggeramlug proggeramlug deleted the worktree-fix-797-known-failures-audit branch May 16, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test-parity: audit known_failures.json — every entry needs an issue # and date

1 participant